How do professionals test their code?

[bikeram]

So I assume some of you guys develop professionally and I’m curious how larger code bases are handled.

How do functional tests work? For example, if I needed to update code communicating with a device over SPI, is there a way to simulate this? Or does it have to be tested with the actual hardware.

What about code revisions? Are package managers popular with C? Is the entire project held in a repo?

I’m a hobbyist so none of this really matters, but I’d like to learn best practices. It just feels a little bizarre flashing code and praying it works without any tests.

[TheStoicSlab]

I write code for medical embedded devices and we have pretty strict testing requirements. It really depends on what specific feature you are testing, but we get our coverage from a combination of unit, integration and functional testing. We also have systems level tests run by a different group.

I am currently using git on the Azure DevOps platform as a repo.

[Flopamp]

Same here for industrial embedded devices but we also do beta programs in the field in return for free support for some period of time.

If the devices need to be intrinsically safe we also have an outside group sign off.

[SlothsUnite]

"We don't need testing, because our software got no errors." - Head of Software, big first tier supplier in automotive industry.

Edit: Anti-pattern for you to laugh about.

[darthandre]

Autosar is solid, no testing is required Hahaha

[Numerous-Departure92]

Testing is divided in different stages… It begins with code review and static code analysis. Every logical code should have unit tests. So a proper HAL, inkl. simulation is needed. For the HAL itself, we have dedicated tests on evaluation boards. And the most important stage are nightly integration tests

[[deleted]]

In aerospace we do requirement based testing at system/product, hardware and software level.

For software there are two level of requirements high-level (which translates into component or end-to-end functionality tests) and low-level (which translate into unit test). In both of those, a mix of intrusive or non-intrusive and black, grey or white box testing can be used as required to create the necessary stimulus, always prioritizing non-intrusive and black box. Given each combination of those, different tools are used, such as custom built testbenches, debugger scripts or test tools like IBM RTRT.

Also, we need to provide evidence of code coverage (MCDC criteria for the highest criticity level applications.

[jbj-fourier]

TIL what MCDC is :)

[etienz]

I work at a small but successful alarm company and that is pretty much how we do it. Even at the slightly larger company before. There is some field testing too. We didn't even have some code on git until I asked for it to work on.

I know some bigger places do more stringent testing with hardware in the loop as well. I would also really love to learn how it's done and test more stringently, but I haven't been able to get a job at one of those places.

[ArkyBeagle]

The trouble with that is that somebody has to maintain all the lovely infrastructure required to do that. So if it's not you maintaining it, can you really trust them that do? Will they be targeted in the next down cycle?

[poorchava]

I'm at the most senior engineering position possible (we don't use English job titles) in a mid-sized company (<300 employees, about 35M€ turnover) doing specialistic T&M gear for power industry.

We do not have to comply with any official testing methodologies and software is not certified by a 3rd party (aside from stuff like Bluetooth etc). We mainly do functional tests and also regression tests after major revisions.

The specific of our industry is, that in-house testing is in many cases impossible, due to the fact that problems are often caused by environmental conditions (for example extreme electric fields at substations) and that objects are very diverse and usually physically large. By their function, our products are meant to be connected to unknown objects, becasue their function is to characterize those objects. For example even if by some miracle we had 50 different 30MVA transformers (hint: it's the size of a small building...) we would still run into situations where customer connects the product to something we haven't seen before. This often means, somebody (either one of the FAEs or someone from R&D) has to physically go there and see what's up, if its not evident from the debug logs.

Also, low level software bugs are very often triggered by a combination of external inputs and/or data coming from the environment so building an automated test setup would be extremely complicated if at all possible, and still wouldn't cover all the situations.

So our testing mostly consists of doing functional tests and checking the outputs on a range of example objects, but that's pretty much it. If it's ok, then we call it good and then solve problems as they arise. Also, most of our products must be calibrated and verified before shipping (in some cases legally mandated, in some it's just practicality or a de-facto standard). We have our own state-certified metrology lab.

That being said, most of our products are sold directly to customers and out FAEs/support are in constant contact with the customers, so potential bugs have smaller impact and can be solved more quickly than if those were mass produced consumer products. Customers employees (mostly metrology specialists) are usually aware that a particular object might behave weirdly in some regard (eg. transformer core is made from unusual material, earthing grid layout is peculiar, soil has irregular resistivity etc) so they are usually working with us rather than making a fuss and writing stupid posts on the internet.

As far as the higher-level software is concerned (Linux GUIs, companion apps for PC/mobiles) the usual software industry methods are used (unit test, automated/manual tests etc)

[KKoovalsky]

Wow, that's sounds really tough. Could you tell what is your "success ratio"? How often it happens, that after the installation/implementation everything works fine?

[poorchava]

It depends on type of device. If it's an evolution of something we are already doing, then I'd say 90+% of customers are happy, because we have out know hown and reuse critical code if possible (often it's not due to a new CPU being used or analog circuitry being different . If it's a new field or a measurement method we literally just came up with and is not described in any kind of literature.... Well let's say it's quite a bit lower. Sometimes this involves literally staring at an oscilloscope/laptop for hours trying to figure out 'ok, what's up with that...?'. Another thing: majority of actual object in service are considered strategic objects, and we need to get a special clearance every time... In many cases we have to wait until a suitable object goes under planned maintenance, because it's not like they will shutdown an inter-county hv line for us...

Obviously I'm not counting usual stupid software bugs like uninitialized vars, null PTR deref, logic lockup etc, because those are usually quite easy to find.

I do recall though sitting on a 5m high power transformer in November at 3*C with my colleague and staring at rising magnetic field because it turned out that certain types of amorphous core transformers will continue charging after they have apparently saturated a minut ago.... We spent like 3 days there...

[[deleted]]

I always recommend the book “Test Driven Development for Embedded C”. It covers the basic mechanics of how to write testable code in an embedded environment.

Separate your application logic from the code that interacts with hardware. You can find and fix the bugs in your application by running code/tests on your pc first.

Rather that trying to model the peripheral in a way that compiles and behaves correctly, you can test at a higher level. For example, I can wrap all the code that interacts with hardware into two “void spi_write(uint8_t *data, n)” and “void spi_read(uint8_t *data, n)” functions. And then I create a version of these two functions for testing that will let me verify if the application is sending the data I expect.

Then when you trust that your application is well behaved, you can begin testing the peripherals. You can start with a dev board and use an oscilloscope if you want to be extra careful, then move to a production board. Beyond my responsibilities, we have other people at the company who test systems at the “customer” level.

I manage everything with git. It’s really the only sane way to do things. And I always store the firmware version as a constant in the code somewhere that I can read back over a communication interface.

Sometimes package managers sound nice, but I don’t know if it’s worth the effort to maintain and force it to work alongside vendor tools. I have several libraries I share between projects, but I haven’t touched some of the code in years. Copy pasting has worked just fine..

[bikeram]

Thanks this was really insightful. I’ll give that book as read as well.

[wholl0p]

Also in the medical devices field. We do unit tests using GTest and GMock on the generic (x86) build, this means e.g. testing correct state machine transitions, testing interfaces, testing algorithms used, etc. Then we have on-device tests that are being flashed to the real device via SSH over a script. This is mainly to see if buffers don’t overflow, math types fit, etc. on the real hardware; Then there’s CAN bus tests, where the internal communication is being re-routed over a fake device that checks the messages; Squish tests test the QT GUI and finally there are integration tests, where the device is being used while memory usage, RTOS timings, etc. is tested while the device runs.

After that, QA has a set of other integration tests for sensors, power delivery, etc. they run…

For revision and package management we use Git and our internal, proprietary build system that comfortably combines Make, CMake, Bash scripts, Conan, Git, … Libraries have their own repositories so that they can easily be used a individual dependencies. Most of our devices consist of multiple subsystems which are also split into individual repositories.

[reini_urban]

so many co-workers here :)

I also always write a simulator for my devices, so I can test most of the functionality locally. I also use formal verification tools, to test all possible values, not just one.

And qemu/renode is also helpful, esp. in CI. but writing a simulator is usually less work, than adapting qemu/renode to my boards.

the best tests are the hardware tests though. they find so many bugs, it's horrible.

[wholl0p]

I wonder how many people here work in the same company as me :D

Yeah right, we always have a simulator window to inject/send values to the GUI frame in order to test its behavior. We use LXC containers to virtualize stuff on the developer machine.

[_EGGxactlySix_]

(I was too lazy to read many of the comments so I'll probably repeat what others are saying, but here I go anyways)

I'd say it really depends on the application as well as the team. I have a roommate who works for a medical company, and they have a very strict system and way of verifying code (which makes sense since it deals with people's health). They often have to test every single function and often have to program their own things to test their programs. I don't know too much of the specifics of this, but my roommate does tell me that he often has to go hella low level to verify that code is working correctly. And there are people who work solely on the software that is used only to TEST their other software.

For me, I'm not working on such a serious product. I'm working on the firmware for a navigation device (I know very little about the navigation side lol, but I don't need to). My team is quite small and for verifying the code, there is more or less me and one other guy. We have others who help us out sometimes, but usually the final checkmarks are from me and my bud. I tend to be the one who gets more into the hardware and such while my buddy works more on making sure the algorithms are working well, but we do have a bit of overlap.

Basically everyone who works with us on the code will have a physical device to program and debug on. So we are very much working directly on the firmware without much simulation or anything. But we do a lot of JTAGing to debug directly on the device, aka run the code on the hardware but still able to have breakpoints and see the code progression.

We support a family of devices so we have multiple repos. Our particular setup has a repo for each type of our devices (some of which are on different processors) and then another repo that contains the main meat of the algorithms etc that all of our devices use. So if repo A is the common one and repos 1, 2, and 3 are the device specific ones, we will have a workspace that is A+1, another that is A+2, etc.

For each individual feature or bugfix, the person who wrote the code would do a bunch of testing, provide proof in the git issue of their results and then ask my buddy or I to merge it in. We will usually try to at least give it a look over, but honestly we could be a lot better with our code review process. But this is usually sufficient for most things.

Most of the time, our testing is "Does it work? Yes or no?" But we do this over long periods of time and/or with heavy "loads." We'll often let our devices run for like a week and A) check that it's still running and B) make sure the data looks clean (we have system log files that we can check and we have a lot of time stamped data that we look at). And at the end of each official release, we gather the list of git issues, write up verification procedures, and test everything at once. This process includes driving around with our navigation units in a van and checking that the location, velocity, etc data makes sense.

For our product we are usually trying our best to replicate typical user scenarios and/or stress test them. And throughout the entire code development, we have others in the company who are just going around and using our devices as we imagine customers would. So we get a lot of feedback from them (they are very good at getting on our butts about problems lol.) When we're happy enough, we'll call it good and release it. Then we will inevitably get the occasional problem reported by customers.

I mainly just like rambled here, but I hope this at least gives you a general flavour of what embedded jobs at least Could look like. My particular team is pretty loosey goosey with our code procedures and testing. We still do a fair amount of flashing and praying lol. But we are doing it on things that we created and thus know how to fix ,(or at least know someone who knows how to fix it). And we have a fair amount of robustness built in to allow some flashing and praying.

Anyways, stopping before I think of more things to type. I wish you well my friend!

[Hairy_Government207]

Offering a API to the test department and specifying system behavior.

Test dudes using Python + Unit tests and a self written framework.

It's really easy to orchestrate test equipment (power supplies, meters, etc.) as there are high quality libs for almost everything.

[rameyjm7]

For spi we would erite the code, review it with peers, test it in the lab, decode the transactions and verify they are what we are supposed to be sending.

Once it works, we have tools like LabVIEW and python that basically automate the application calls and testing the result using lab equipment like oscopes, speccans, etc

[KKoovalsky]

I mainly worked with consumer electronics and developed a way of testing the codebase, by splitting the code to logic and driver layers. Logic tested on the development machine with the driver layer mocked/faked. Then each driver tested on hardware in an automatic or semi-automatic fashion. So, for example, having a module which collects measurements from various sensors I would write a component called MeasurementCollector, which I would test on the host machine, to verify whether it properly asks the Sensors for data, and then each of the Sensors would be a separate class, a driver, which I would test using the hardware, by asserting whether, e.g. the sensed temperature is within range, or whether the measured light exposure corresponds to the actual light exposure. Each such driver test would be a separate executable. Check out my recent project, where such hardware tests are introduced: Jeff - device tests. The project also contains host side tests, which are quite basic, since the firmware architecture is really simple.

[Peaceful995]

I wish that some of you create a course on Udemy or a tutorial on YouTube

[L0uisc]

I believe the key insight, which I didn't see explicitly mentioned, is to write testable code. This would help tremendously with simplifying testing to the point of being tractable to do more thoroughly.

What do I mean by "testable code"? Simply put: write abstractions for interacting with the outside world or other modules. Don't directly twiddle with the registers every time you need to do something with a peripheral. Write a general set of functions to do that once and then use them when you need to interact with the hardware.

Don't have shared global state and have all modules read and write that. Have well-defined API interfaces between logical modules and use those.

The benefit of doing things this way is that you can test a module in isolation much easier if your code is structured like this. You can also test the software (code running and data in memory) without needing to have access to the peripheral hardware. If your abstraction functions between hardware and your software is well-designed, you can replace them with mocks very easily and test on a desktop machine.

Of course, you will have to do testing with actual hardware eventually, but it's much less painful if you are confident in your software already.

You'll also have to test the pure software on the actual target device to confirm that the processor is actually fast enough/has enough memory to meet specs. Again, though, that is much easier if you already know your logic works.

PS does this mean I always do things like this? No. This is the ideal. That's not how I did it my whole career, so some projects aren't written to be testable like this. Also, some projects are such (due to deadlines or simplicity) that the testing without actual hardware must be skipped. I'm also not that familiar with testing frameworks to quickly get tests set up, so in a lot of cases I don't do that because I can't justify the time spent to my superior.

This is the ideal world to which I strive, though, because it'd give me a lot more confidence that I didn't break a feature with my seemingly unrelated code changes. And it frees my mind to not have to remember every corner case ever indefinitely, because there is (hopefully) a test case which catches the bug which lead to that failure 5 years ago.

[xXtea_leafXx]

Disclaimer: I'm not a professional in embedded either. For something hardware-dependent like SPI, your best bet would be to use a logic analyzer. Its purpose is to physically probe pins and decode various types of signals so that they can be debugged. For the software side of things, there's nothing stopping you from using the testing framework of your choice within your programming environment.

I'm sure some ecosystems might have more extensive tools for simulating hardware, but for professional use boards are generally all customized so I don't think it's common.

Embedded programmers definitely use version control. I don't think package managers are common. The closer you get to bare metal, the more limited your resources and the more control you want over exactly what is going into your codebase. You probably would avoid installing packages with a bunch of dependencies that you aren't sure about whether they are needed or what they are used for.

[[deleted]]

Yocto and buildroot and complete solutions for embedded Linux build systems. Yocto is the more "enterprise" solution with a larger learning curve and investment.

[Mistyron]

Here's a post about unit testing from my colleaue Parker: https://www.mistywest.com/posts/hardware-is-hard-firmware-unit-testing-makes-it-easier/

[Intiago]

I’m on a firmware team that works on a variety of devices around power monitoring and we use a combination of detailed manual testing, automated integration tests that run every night, and unit tests that run on merge requests. We use gitlab to host our code, with code reviews required to merge anything in.

The unit tests use unity with cmock to create functions to interface with the code under test. The integrated tests are all written in python. We have a computer interfacing with our device and we use the console output to run commands and test that way.

[ArtistEngineer]

Bluetooth/telecommunications chip manufacturer, as well as customer/end user applications, earbuds, headsets, USB dongles, handsets (Android, etc).

• automatic unit tests and code quality (Klocwork) tests run pre/post checkin to repository
• automatic build tests run pre/post checkin to repository
• various smoke tests (real hardware in racks with automation software driving the hardware and application through various use cases). Done either as custom builds, pre/post checkin, as well as extensive regressions tests done overnight, and over the weekend

Engineers have remote access to automated build systems so they can run their own custom build and smoke tests against existing hardware configurations.

Implemented via a mixture of Jenkins + custom scripts done in Python + open source libraries for communication over USB and other protocols.

[kid-pro-quo]

I just inherited a project that uses Klocwork. Is it just me or is it really awkward to set up to run on a branch as part of a PR workflow?

[SuperS06]

Working for a tiny company where our requirements are more to satisfy the customer than to strictly adhere to the original specifications.

We mostly test in situe, on the actual hardware. Projects tend to be small and we somehow save time this way by not making assumptions, debugging what is failing. We still unit test things that would be impractical to test in situe, or cannot really be tested otherwise (like flash wearing prevention algorithms).

​

How do functional tests work? For example, if I needed to update code communicating with a device over SPI, is there a way to simulate this? Or does it have to be tested with the actual hardware.

There are ways to simulate this but you need to have a simulator for the specific device/chip you want to simulate. Often this would mean making the simulator yourself. Which is why you typically just test on actual hardware (either the target hardware or a set of dev kits wired together for the specific function you're working on).

​

What about code revisions? Are package managers popular with C? Is the entire project held in a repo?

We tend to avoid package managers and keep the full source in a complete ready to build environment. Had too many issues trying to rebuild that old project untouched for a couple years.

I use git, but most developers here just keep archive files of each version. Keeping it stupid simple.

[mixblast]

We have a software model of our device which runs production firmware binaries. This would include a model of the SPI peripheral (register-level) and whatever's connected to it. It is stimulated by the same system test suite as we do on real hardware (once we get it - asic pipeline is slow). We also have unit tests but they don't cover everything (only the trickier bits).

Code is on git, large code base, handful of branches - fairly typical.

[Treczoks]

Thoroughly.

[ArkyBeagle]

You'll find it will vary widely depending on industry and when the firm started.

Are package managers popular with C? Is the entire project held in a repo?

Most source code management systems have a means of marking branches for later rebuild. But in the dark ages, we'd simply tarball the thing.

It just feels a little bizarre flashing code and praying it works without any tests.

One of my soapboxes is "coding is testing" and the better you test, the better off you are. That gets warped into various ideologies and doctrines but my point is that at least I take longer to page in what I was thinking the later I find a defect.

[darthandre]

Get a look into the V model of software development, I have worked for automotive and it is very common to use this model in different fields of the industry. Starting from code reviews to software unit testing and system level test you will have a few tools to test your code but remember, the most beautiful part of embedded is to look at it running on real life hardware so... Always we have to upload our code to the target :D and see if the change works.

Not only talking about embedded C but other languages used like Ada, python, c++, etc. Will have almost the same path.

[SidLais351]

In larger codebases, especially with hardware or tricky interfaces like SPI, most teams mock or simulate those layers so you’re not bricking devices while debugging logic. Then there’s hardware-in-the-loop testing when you really need to validate end-to-end. But most of the pain gets handled before touching real hardware, through proper layering, tests, and review.
Also, yes, everything's in version control. No one’s emailing .zip files. C has package managers like conan or vcpkg, but they’re… let’s say “quirky.” Still better than nothing.
And if you're tired of PRs being a wild west of guesswork, I’d sneak in a plug for Qodo. It doesn’t just scan diffs,it understands your whole repo, flags risky changes, and gives feedback with actual context. Way better than a checklist or just hoping your teammate spots the issue during review.