Kyle Robbertze

Result Summary

Original Goals

The original goal was to add audio support to the HDMI2USB FX2 USB firmware. This was intended to be integrated into the current firmware which provides HDMI video capture and serial control over the Opsis and Atlys boards

[](https://opsis.hdmi2usb.tv/) [](https://store.digilentinc.com/atlys-spartan-6-fpga-trainer-board-limited-time-see-nexys-video/)

The original proposal can be found here

Final Outcome

The current version of the firmware is being rewritten, so I did not merge the audio firmware into it, however I create standalone audio firmware which generates a tone that can be seen in Pulseaudio.

The other outcome I achieved was to update documentation and create examples to make starting out with the project easier for new contributors. This also included consolidating our build system to use a common base and integrating continuous integration testing so that the code is automatically built on every commit and pull request. This makes review easier and means issues with the firmware can be picked up quicker.

I started out working on a FX2 mini-board and did most of the development on that. I had the opportunity to test the firmware on an Atlys and an Opsis board at DebConf in Canada. Getting there and back was quite a trip, I collected my passport with visa inside 5 hours before I flew because the visa application centre took so long to process my application. On the way back I got detained by Belgian ImmigrationImmigrations because of European visa issues with one of my connecting flights. Thankfully they were able to reroute me via London instead so I could get home.

Community Outcome

The main outcome of this project is to add audio capture functionality to the HDMI2USB FX2 firmware. I achieved a large portion of this, as the groundwork is done. With the addition of the FPGA capture code, this project will allow cameras and laptops connected to the Opsis to send their audio over the HDMI cable and have it captured in the same way as the video. This simplifies the capture of cameras and speakers’ presentations.

Learning

I learned a lot about embedded programming throughout this project. The major things in my opinion were what I now come to think of as the basics: initial conditions matter, advanced Makefile and git usage and a better grasp of compiler generated assembly and other files. These skills were all crucial to making any progress whenever I got stuck and I am extremely grateful to mithro for teaching me much of them.

I plan on continuing my involvement with TimVideos and the HDMI2USB project with the intention of getting the audio firmware fully integrated and working. There are a couple of other things I would also like to tackle such as HDMI audio capture to output from the Milkymist IO Expansion board. I have caught the bug and don't want to stop!

Replicating My Results

I did most of my development on an EZ-USB FX2 Development Board, however I did test the firmware (audio tone generator) on both the Atlys and Opsis boards.

Required

  • Opsis, Atlys or FX2 board
  • Git

Repositories

Link: https://github.com/timvideos/HDMI2USB-FX2-Firmware

The audio firmware exists under the audio directory and the examples under examples.

Audio Tone Generator

  1. git clone https://github.com/timvideos/HDMI2USB-FX2-Firmware
  2. cd HDMI2USB-FX2-Firmware/audio
  3. make conda
  4. Plug in the FX2 miniboard to an USB port
  5. make BOARD=fx2miniboard load
  6. If that fails with an error message about permissions sudo make BOARD=fx2miniboard load

You should now find a new audio device appears in your audio settings, below is an example in PulseAudio

FX2 Miniboard Serial Example

  1. sudo apt install minicom
  2. git clone https://github.com/timvideosHDMI2USB-FX2-Firmware
  3. cd HDMI2USB-FX2-Firmware/examples/serial
  4. make conda
  5. Plug in the FX2 miniboard to an USB port
  6. Plug a Pmod USBUART board into another USB port
  7. Wire Pin 2 on the USBUSRT to pin PD3 on the FX2 miniboard
  8. Wire Pin 5 on the USBUSRT to GND on the FX2 miniboard
  9. make BOARD=fx2miniboard load
  10. If that fails with an error message about permissions sudo make BOARD=fx2miniboard load
  11. minicom -s
  12. Select Serial Port Setup
  13. Enter a
  14. Replace /dev/modem with /dev/ttyUSB0 and press <return>
  15. Select Exit
  16. You will see the serial output from the board appear on the screen

Lights Example

  1. git clone https://github.com/timvideos/HDMI2USB-FX2-Firmware
  2. cd HDMI2USB-FX2-Firmware/examples/lights
  3. make conda
  4. Plug in the FX2 miniboard to an USB port
  5. make BOARD=fx2miniboard load
  6. If that fails with an error message about permissions sudo make BOARD=fx2miniboard load
  7. You will see D1 and D2 on the FX2 miniboard alternate

To Do

There is still several things that need completing before the audio firmware can be considered complete:

  • The video firmware rewrite needs to be completed so that it makes use of C descriptors instead of the current assembly version. Once this is done the audio firmware can be merged in.
  • The audio firmware needs integrating with the FPGA chip on the board. Currently the audio firmware generates a tone from the FX2 chip and doesn't get sent anything from the FPGA.
  • The FPGA gateware still needs to be written to complete the audio capture from HDMI.
  • Add support for capture from the Milkymist IO Expansion board. This expansion board allows audio to be captured from a 3.5mm jack instead of being extracted from the HDMI input.

Documentation

Kyle Robbertze

The last 2 weeks have been DebConf 17. This gave me the opportunity to test firmware on an actual Atlys and Opsis board. Getting there also allowed me to work on new firmware while in the plane.

During testing, I found the blinking lights example worked well on both, however the audio firmware, which is supposed to generate a tone that can be picked up by PulseAudio, only worked on the Atlys board. The Opsis throws an error trying to send a SET_INTERFACE request. I was unable to debug it while at DebConf, but plan to do so when my luggage, with an Opsis board inside, arrives home.

While I was travelling, I started working on the FIFO version on the audio firmware. This version creates a loopback audio device, using two FX2 boards. One acts as the PulseAudio source and sends this data over the FX2 built-in FIFO interface to the second dev board which reads the audio from the FIFO and presents this to the host PC. This allows us to fully test the FIFO and descriptor setup of the audio firmware.

While at DebConf, I also got hold of a Milky Mist expansion I/O board. With this I can work on the next set of the audio firmware, which is to capture sound from the I/O board and present that over USB.

Next week I need to try debug the audio firmware on the Opsis and get the FIFO version working on the FX2 dev boards.

Kyle Robbertze

I finished exams last week Thursday (6 July), so I have had time to get things done. This this week has seen a fair amount of work. I have started working on USB audio traffic and opened four pull requests and had two merged for other aspects of the project, mostly involving documentation.

My merged pull requests were for documentation generation and a common Makefile for building examples. The document generation uses doxygen to generate documentation from code comments. This is useful for developers, as they can easily access and search for explanations around what they are working on. The common Makefile reduces code duplication and allows more examples to be easily integrated into the build system.

I ported the lights example to the Opsis and Atlys boards. The lights example is intended as a way for developers and users to test loading firmware onto the boards and so it is important that it works on both production boards. The Opsis support was fairly easy to implement as I made use of an LED intended for JTAG status indication. The atlys support was a little more difficult to implement as, initially, I could not find a LED connected to the FX2 USB chip. Eventually mithro found LD14, which indicates USB DONE and I used that. CarlFK, one of the other developers, gave me access to his Atlys board and this allowed me to debug the port remotely.

I also ported the lights header in the fx2lib repo to the atlys and opsis boards and this is still awaiting review. The lights header is useful, as it allows for a visual indication of progress and problems while the HDMI2USB firmware is running. This pull request also benefited from my remote debugging on CarlFK’s board.

Finally I started working on the USB traffic handlers for the audio firmware. These handlers are how the firmware communicates with the host the device is connected to. This is slow going as there are several different types of handlers, namely standard and audio, and only some need implementing. I am using uvc.h and uvc.c as references.

Next week I plan on continuing with the audio traffic, ideally completing it. If I get time I would also like to rewrite how the version is embedded into the firmware and get Travis Continuous Integration working for the HDMI2USB repo. These will help with reproducibility of builds and code quality.

Kyle Robbertze

Thanks to my exams, this week has been a mixed bag. My two pull requests for serial and lights examples were merged and I opened three new ones, one of which has been merged. I also started analysing the USB traffic that occurs between the host PC and an audio device.

The serial and lights examples are useful for helping developers test that their development setup is correct, by giving them specific firmware that tests individual aspects of the tool chain, namely loading firmware and reading serial debugging information. With these merged, I have been able to work on adding Opsis support to the lights example and moving a large part of the Makefiles for building either example into a common file that both depend on. This improves maintainability as the base build is defined in a single place. This common Makefile exposed a bug in the lights example, where a bitmask was named incorrectly, which I corrected and this fix was merged.

I opened a pull request for Audio Descriptors, which is currently being reviewed. It gave rise to a discussion about the licenses used by the HDMI2USB, which caused an issue to be raised against files that are currently incompatible with the Linux USB Headers library that we are using.

The final pull request to be merged was moving the string table generator to the linux USB headers repo. This move was also for ensuring maintainability, as this generator is used by all firmware images that use the linux USB headers to generate the string table for things like their vendor and product names. I improved the initial script by adding more details to the comments, fixing the code style, making the argument parsing a little more robust (using the argparse module) and updating the string formatting.

The USB traffic analysis proved interesting. It highlighted the traffic that the host expects for setting up an audio device and the packet structure of the actual audio data packets that are requested and received. It will be useful when I start working on actual traffic transfer between the host PC and the FX2 dev board.

My exams finished this week, so I expect to be much busier on this project next week. I hope to implement USB packet transfer and port the serial example to the Atlys board. If I get time, there are a few things extra that I can do, such as adding Travis CI support to the repo. This will help code review and code consistency.

Kyle Robbertze

This week has seen a fair amount of work. Exams are taking up more time than I would like, but not unbearably so. I have spent it fixing things in my two outstanding pull requests, submitting another pull request and getting audio working.

The two pull outstanding requests are for a Serial Example and a Lights Example. I have largely added documentation. These changes have been reviewed and there are still some more changes I need to make before they can be merged.

I submitted a pull request to the fx2lib linux headers repo which moves the strings table generation for the USB audio descriptors into the library itself. This means that the generator will not need to be included into every firmware image that uses C-style linux headers. I still need to make the changes requested on the PR.

The final thing I did this week was to get audio USB descriptors working. Both lsusb -v and wireshark report that the FX2 appears as an audio device when flashed with the audio firmware. This is the first required step towards getting USB audio working!

After initially trying to copy the descriptors from a Logitech webcam, I ended up using using mithro’s bulkloop example as the base, changing the Makefile to be independent of the FX2 library and then adjusting the descriptors to be the ones required for audio.

This week I plan to add a second USB interface for audio control so that I can do things like adjust the input volume and start transferring data between the host and the FX2.

I am not sure how much I will be able to get done as I have two exams during that week.

Kyle Robbertze

Thanks to lectures winding down, this week has been productive. It has produced two pull requests and a git branch. I largely focussed on documentation through examples and written docs, with the occasional stab at development environment work and USB audio descriptors.

The two pull requests are for a Serial Example and a Lights Example. Currently they are being reviewed and I’ve had to make a few adjustments to documentation and style.

The serial example simply writes “This is the serial example for the HDMI2USB firmware” out over serial once every two seconds. It is intended as a method of testing that developers’ and users’ serial setup is correct and working. It also includes documentation about how the serial connection is wired in the example and in the HDMI2USB firmware itself. The wiring diagram is as follows:

  • RXD -> PD3
  • TXD -> RDY0
  • GND -> GND

The lights example is a way for developers to test that their development environment is setup correctly as all it does is flash the lights on the FX2 development board. It looks like this when it is running:

I also flashed one of my FX2 dev boards as a Saleae Logic Analyser to use with Sigrok. This is useful as it allows me to view the output of the port B pins when working with the HDMI2USB firmware as a debugging tool. I found this instructables as well as the Sigrok wiki useful when working out how to flash it, and what ports were available. I used the hdmi2usb-mode-switch to load the .ihx firmware file that is created after building. A demo capture looks like this:

The final thing I did this week was start working on the USB audio descriptors. These are modelled on the Logitech C170 webcam. The descriptors are now compiling, but when flashed to the FX2, it does not disconnect and reconnect to the host machine. I have spent several days trying to find the cause of this issue, but to no avail. This is largely what I plan on doing in the next couple of days, hopefully finding and fixing the issue.