Kyle Robbertze

Result Summary

Original Goals

The original goal was to add audio support to the HDMI2USB FX2 USB firmware. This was intended to be integrated into the current firmware which provides HDMI video capture and serial control over the Opsis and Atlys boards

[](https://opsis.hdmi2usb.tv/) [](https://store.digilentinc.com/atlys-spartan-6-fpga-trainer-board-limited-time-see-nexys-video/)

The original proposal can be found here

Final Outcome

The current version of the firmware is being rewritten, so I did not merge the audio firmware into it, however I create standalone audio firmware which generates a tone that can be seen in Pulseaudio.

The other outcome I achieved was to update documentation and create examples to make starting out with the project easier for new contributors. This also included consolidating our build system to use a common base and integrating continuous integration testing so that the code is automatically built on every commit and pull request. This makes review easier and means issues with the firmware can be picked up quicker.

I started out working on a FX2 mini-board and did most of the development on that. I had the opportunity to test the firmware on an Atlys and an Opsis board at DebConf in Canada. Getting there and back was quite a trip, I collected my passport with visa inside 5 hours before I flew because the visa application centre took so long to process my application. On the way back I got detained by Belgian ImmigrationImmigrations because of European visa issues with one of my connecting flights. Thankfully they were able to reroute me via London instead so I could get home.

Community Outcome

The main outcome of this project is to add audio capture functionality to the HDMI2USB FX2 firmware. I achieved a large portion of this, as the groundwork is done. With the addition of the FPGA capture code, this project will allow cameras and laptops connected to the Opsis to send their audio over the HDMI cable and have it captured in the same way as the video. This simplifies the capture of cameras and speakers’ presentations.

Learning

I learned a lot about embedded programming throughout this project. The major things in my opinion were what I now come to think of as the basics: initial conditions matter, advanced Makefile and git usage and a better grasp of compiler generated assembly and other files. These skills were all crucial to making any progress whenever I got stuck and I am extremely grateful to mithro for teaching me much of them.

I plan on continuing my involvement with TimVideos and the HDMI2USB project with the intention of getting the audio firmware fully integrated and working. There are a couple of other things I would also like to tackle such as HDMI audio capture to output from the Milkymist IO Expansion board. I have caught the bug and don't want to stop!

Replicating My Results

I did most of my development on an EZ-USB FX2 Development Board, however I did test the firmware (audio tone generator) on both the Atlys and Opsis boards.

Required

  • Opsis, Atlys or FX2 board
  • Git

Repositories

Link: https://github.com/timvideos/HDMI2USB-FX2-Firmware

The audio firmware exists under the audio directory and the examples under examples.

Audio Tone Generator

  1. git clone https://github.com/timvideos/HDMI2USB-FX2-Firmware
  2. cd HDMI2USB-FX2-Firmware/audio
  3. make conda
  4. Plug in the FX2 miniboard to an USB port
  5. make BOARD=fx2miniboard load
  6. If that fails with an error message about permissions sudo make BOARD=fx2miniboard load

You should now find a new audio device appears in your audio settings, below is an example in PulseAudio

FX2 Miniboard Serial Example

  1. sudo apt install minicom
  2. git clone https://github.com/timvideosHDMI2USB-FX2-Firmware
  3. cd HDMI2USB-FX2-Firmware/examples/serial
  4. make conda
  5. Plug in the FX2 miniboard to an USB port
  6. Plug a Pmod USBUART board into another USB port
  7. Wire Pin 2 on the USBUSRT to pin PD3 on the FX2 miniboard
  8. Wire Pin 5 on the USBUSRT to GND on the FX2 miniboard
  9. make BOARD=fx2miniboard load
  10. If that fails with an error message about permissions sudo make BOARD=fx2miniboard load
  11. minicom -s
  12. Select Serial Port Setup
  13. Enter a
  14. Replace /dev/modem with /dev/ttyUSB0 and press <return>
  15. Select Exit
  16. You will see the serial output from the board appear on the screen

Lights Example

  1. git clone https://github.com/timvideos/HDMI2USB-FX2-Firmware
  2. cd HDMI2USB-FX2-Firmware/examples/lights
  3. make conda
  4. Plug in the FX2 miniboard to an USB port
  5. make BOARD=fx2miniboard load
  6. If that fails with an error message about permissions sudo make BOARD=fx2miniboard load
  7. You will see D1 and D2 on the FX2 miniboard alternate

To Do

There is still several things that need completing before the audio firmware can be considered complete:

  • The video firmware rewrite needs to be completed so that it makes use of C descriptors instead of the current assembly version. Once this is done the audio firmware can be merged in.
  • The audio firmware needs integrating with the FPGA chip on the board. Currently the audio firmware generates a tone from the FX2 chip and doesn't get sent anything from the FPGA.
  • The FPGA gateware still needs to be written to complete the audio capture from HDMI.
  • Add support for capture from the Milkymist IO Expansion board. This expansion board allows audio to be captured from a 3.5mm jack instead of being extracted from the HDMI input.

Documentation

Kyle Robbertze

Mithro and I managed to fix the SET_INTERFACE issue with the Opsis. Turns out the HDMI2USB video firmware was setting something to do with endpoint 2 that I was not resetting. Switching to endpoint 8 and fixing a minor error fixed the issues.

audio-data:

  • 23a84f - audio: Use TRM recommended endpoint config
  • f469d0d - audio: Use endpoint 8 as it is free

travis-boards:

  • d5a650b - Build for all target boards
Kyle Robbertze

After a fairly long hiatus thanks to DebConf17, today I got back to working on the HDMI2USB audio firmware. I opened a PR against the HDMI2USB-mode-switch for be able to load test firmware onto the Opsis and Atlys boards. I also spent time debugging the SET_INTERFACE error I am getting on the Opsis when loading the audio firmware.

device-id:

  • 09e075f - Add test device IDs to board list

audio-data: The issue is with the streaming code in the firmware.c file:

 while(TRUE) {
        if (got_sud) {
            printf("Handle setup data\n");
            handle_setupdata();
            got_sud = FALSE;
        }
        /* ISO endpoint config type is 01 in the enpoint configuration buffer */
        if ((EP2CFG & bmTYPE) == bmTYPE0) {
           // This is the issue
            while (!(EP2468STAT & bmEP2FULL)) {
                d1on();
                /* Send max data. Larger than 0x30 causes an EOVERFLOW */
                EP2BCH = 0x00;
                SYNCDELAY;
                EP2BCL = 0x30;
            }
            // To here
            d1off();
        }
    }

If this is included, dmesg shows an error:

usb 1-3: 1:1: usb_set_interface failed (-110)

Which fails repeatedly.

Kyle Robbertze

The last 2 weeks have been DebConf 17. This gave me the opportunity to test firmware on an actual Atlys and Opsis board. Getting there also allowed me to work on new firmware while in the plane.

During testing, I found the blinking lights example worked well on both, however the audio firmware, which is supposed to generate a tone that can be picked up by PulseAudio, only worked on the Atlys board. The Opsis throws an error trying to send a SET_INTERFACE request. I was unable to debug it while at DebConf, but plan to do so when my luggage, with an Opsis board inside, arrives home.

While I was travelling, I started working on the FIFO version on the audio firmware. This version creates a loopback audio device, using two FX2 boards. One acts as the PulseAudio source and sends this data over the FX2 built-in FIFO interface to the second dev board which reads the audio from the FIFO and presents this to the host PC. This allows us to fully test the FIFO and descriptor setup of the audio firmware.

While at DebConf, I also got hold of a Milky Mist expansion I/O board. With this I can work on the next set of the audio firmware, which is to capture sound from the I/O board and present that over USB.

Next week I need to try debug the audio firmware on the Opsis and get the FIFO version working on the FX2 dev boards.