I’ve long been fascinated by the idea of automated testing of production software. The problem I faced in implementing anything like “automated testing” was that I couldn’t figure out how to do it with embedded firmware.
Writing firmware can be tricky. There are often dependencies that you can’t control. Typical problems include: How do you test complicated protocol stacks? How can you test the control of analog hardware whose behavior changes with temperature and/or humidity? How can you automate observations of whether the LED is blinking?
In my many years of working as a “seasoned” embedded software engineer, I have been responsible for the creation of multiple large gobs of code that have seemed difficult or even impossible to test.
My VHDL-slinging coworkers would often share their experiences of creating “test benches” to verify their complicated FPGA designs. It sounded good but these same coworkers would end up walking away in frustration when I started asking questions like: “Yea, but how can I write a test case for this piece of hardware that has dramatically different analog responses over time?”
Some of you might be more evolved than I was and may have just rattled off a bunch of answers to the questions above. My problem was I just didn’t understand how to do it. I knew in my heart that there had to be a way but I didn’t understand what it was.
I often looked in envy at my peers that were writing server-based and PC-based code. I’d mockingly say to myself: “It must be nice to work with ‘testable software’.” – I would talk myself into believing it was impossible to develop meaningful tests for embedded firmware.
At one really low point of my career, I even went as far as to learn VHDL in the hopes that working with test benches would help show me a path forward. – I was beginning to understand what my coworkers had been talking about but I still failed to see how it could be applied in a meaningful way to the art of embedded software.
Then I discovered James Grenning‘s book, Test Driven Development for Embedded C. It took me a while to work through his examples but it felt as though this book was specifically written to address my concerns, systematically. – I’ve since discovered others writing about the topic, which is awesome, but this was the first light of hope that I had ever seen.
To help myself work thorough my frustrations/confusion, I took a software module I had written ages ago and used it to create some test cases that could be called any time it was compiled with GCC on Linux (long live make!).
It was all interesting but I needed a project to apply the knowledge to.
It took a while to identify a good pilot project but it eventually came in the form of a need to develop a firmware driver that could dynamically discover the midpoint of a noisy sinusoidal curve. (Sounds like a crazy piece of hardware, right?) I had finally found an excuse to develop a piece of production embedded code using TDD!
My tools were a little clunky since I was choosing to use a Linux Virtual Machine to run my test suite on but it worked! I was able to develop a working solution relatively quickly, with a significantly shorter code-debug-test-repeat development cycle.
There were a few challenges along the way:
- It was tough to effectively sync code changes between the development machine and the test-target, a Linux Virtual Machine. Distributed Version Control (long live Mercurial!) quickly solved that problem.
- Collecting reasonable data to model tests with. This was solved thanks to a very patient test engineer and a bit of Python code to interact with “dumb” firmware that could perform analog reads-and-writes.
- But! The. Biggest. Problem. I fought was during the integration of the firmware with the real target hardware. There was a behavior in the real system that was never effectively captured by my model-generating firmware and Python code. The target hardware had some hysteresis effects that I failed to capture in my models due to the slower read-write cycle of my PC-based controller. This made for a very frustrating integration
sessionmonth but in the end, we were successful. I had been working with an inaccurate model and it took a frustrating amount of time to recognize the hysteresis.
The takeaway is that TDD for embedded projects is far more feasible than I once thought. In talking with some of my colleagues that also write embedded software, it seems a bit atypical in practice but I believe it should be a best-practice for our field and should probably be embraced. I certainly need to refine my processes but there are a lot of benefits to this style of development.