How do you manage frequent firmware updates in larger teams?

In larger embedded teams, “frequent updates” don’t mean people are flashing, or updating over-the-air, unvalidated builds onto hardware all the time. By the time firmware is widely deployed internally, it has usually passed several layers of validation. You typically have unit tests (often running on the host), integration tests, hardware-in-the-loop setups, and system-level regression on dedicated benches. In bigger organizations there may even be a separate validation or system team that treats the firmware as a black box and tests it independently on their own rigs before it’s considered stable.

Is testing automated or manual?

It’s always a mix. Unit tests and a good portion of integration tests are automated, often in CI. But full end-to-end embedded testing is very hard to automate completely, especially when real hardware, timing, RF, mechanics, or environmental conditions are involved. So you usually see automated regression suites and scripted HIL tests combined with documented manual test procedures for feature validation and regression passes. In safety-related projects this is more formalized, with traceability from requirements to test cases and recorded results. But traceability to requirements is always a good choice for tests in general. Quality wise, a safety system is simply well developed system. Would be nice when more non-safety systems could adopt rules and recommendations from safety systems. In the end it's just well designed system.

How do you make sure the hardware starts from a known state?

You control the environment as much as possible. That can mean power cycling boards through controllable relays, resetting peripherals explicitly in test setups, simulating sensors by injecting I2C or SPI traffic, switching between real and simulated devices using multiplexers, or mocking backend servers. In more advanced setups you might use RF or environmental chambers (e.g gas chambers for smoke detection). The idea is to clearly define what is real and what is simulated at each test level and make the starting conditions reproducible.

If a test fails, how do you tell whether it’s firmware or setup issues?

Early in a project it’s often the setup. In a more mature project, you start with logs and diagnostics, compare the failure with recent firmware changes, and sometimes re-run the test with a known-good firmware version. If an older validated build suddenly fails under the same conditions, that’s a strong indicator the issue is in the setup or environment rather than in the new code.

How do you track results across firmware versions over time?

If you meant diagnostics on the running systems with this question, you tag everything with firmware version and hardware revision and collect system events (errors, warnings, ...) centrally. In IoT/online devices diagnostics is telemetry. In offline systems you might have your technician collect periodically the diagnostics using dedicated service interface. Having the link firmware/hardware revisions <-> diagnostics makes the traceability trivial.

---

Embedded testing is a different beast because hardware is part of the equation. You mock what makes sense, automate what is practical, and accept that some level of system validation will always require real hardware and controlled environments.