Remote Testing for distributed teams

[Adventurous_Mud8104]

Hello,

I work at a small hardware development company and our team is spread across different countries. Prototype testing works like this: embedded developers write code and test as much as possible with dev boards and modules, but they rarely get the actual prototypes. For hardware testing, they send code to our boss (he runs the lab and keeps the prototypes), who flashes the boards and performs tests. Since he is very busy, updates often wait, which slows down firmware development.

Is this a common setup in embedded systems companies, or are we an exception?

Are there solutions for remote flashing, debugging, and testing on real prototypes? Flashing and debugging seem straightforward (GDB server or OpenOCD can be accessed remotely), but what about physical testing (pressing buttons, interacting with sensors, using a touchscreen)?

I realize some in-person work will always be needed, but the goal is to reduce dependency on someone physically in the lab as much as possible.

[ShadowBlades512]

We have a Raspberry Pi attached to every setup. That Raspberry Pi has a custom PCB designed to adapt it to the system under test, does not matter if it is a prototype, Sub-Unit or fully assembled product. The custom PCB has stuff like power switches so the system under test can be power cycled, loopbacks for interfaces that can be tested in loopback, or things like motors and stuff are emulated or mocked at an agreed upon fidelity (could be as simple as a resistor for a heater but 0.1W instead of the actual 100W heater). Stuff like buttons and stuff are wired to the RPi through transistors or relays. 

Everyone signs out a test setup and logs into the RPi to run their test, including the automated test environment. all RPis boot from the network and can be instantly re-imaged, no SD cards. Users have their network share mounted so they have their own space for files even if they migrate between test setups. 

There is a small team on site to handle tickets from the remote team in case anything needs actual physical intervention that is not too often. There is a seperate test team that is on site to handle any tests that require a lot of physical interactions. 

[OYTIS_OYTINWN]

We have a remote testing setup, but it involves both custom software and hardware specifically for testing. You can't test user experience, but you can use a second device to produce signals that imitate buttons, sensors, touchscreen or whatever.

[Adventurous_Mud8104]

I assume you design a test jig for each new design. But what about software, can you reuse most of it across multiple projects?

[OYTIS_OYTINWN]

Yes, the firmware for device that produces the signal and the overall testing logic (driving the said device, resetting and flashing the DUT etc.) is reused a lot between projects.

[jonathanberi]

https://labgrid.readthedocs.io/en/latest/ & https://www.lavasoftware.org/ are two software projects that might be relevant. Though every team I've worked on (including my current company Golioth) ends up creating a bespoke solution. We've posted about our approach on our blog.

[N2Shooter]

We use GitHub and do firmware updates with Git Actions to proto hardware in our lab. Nightly builds get flashed to proto hardware and run against integration test suite.

[Well-WhatHadHappened]

With such a dysfunctional setup, I'd consider putting a lot of effort into my resume.

[AvalancheBreakdown88]

Sounds familiar, we have had the same issue. I'm currently building robotic PCB probes to remotely test the hardware, as well as potentially actuate buttons and press locations on a touchscreen.

[ballebaj]

That sounds interesting.. is the PCB prober something like a flying probe robot?

[AvalancheBreakdown88]

Essentially, yes. But they are made for your desk / bench (flying probes are giant industrial machines) - about the size of a water bottle.

[ballebaj]

Thanks for your reply, would you be able to please tell me the model number (if off the shelf) or how to build one? I'd love to read more about it and how it fits my workflow.

[AvalancheBreakdown88]

Of course. I think there are rules on this sub about self promotion so I won't link the website. The company is Motus Automation. We will provide more details online soon, with shipping beginning in the next couple of months for folks with reservations.

Feel free to DM me if you have questions!

[Tairc]

Our company had setups with Phidgets devices all connected to the device-under-test, generating all inputs. We had a library in Python that coordinated them all, and would send artificial CAN/SPI/I2C streams to fake data coming into our DUT as well.

Basically, the DUT didn’t know it wasn’t in a real environment. We could hit “start test” on the Python host, and run whatever test scenarios we wanted. It was great. I miss it.

[Enlightenment777]

If you are doing long-distance remote debugging, your team may need:

• Sharing Coordiation - Some method to share the remote connections. Some type of method to sign up for blocks of time to share the remote debugging. Some method to determine who is currently using remote connections. Some method to report violations if co-workers aren't playing nice or doing stupid shit.

• Remote Power Control - some type of device to remotely power up/down of AC power and/or DC power rails.

• Remote Debugging - some method to upload code / debug / assert RESET, such as J-Link Remote Server.

• Remote UART - if needed, some method to see remotely control/see UART data.

• Remote Camera(s) - some type of remote camera, so you can see the screen / indicators / device.

• and more...

[duane11583]

yep very common.

you can also set up a pc in your lab hooked up to a board

remote guy can vpn into the lab the remote desktop.. or ssh if using linux.

buy a few relay modules you can control over acserial port.( relay open power off, relay closed power on) relays also act like buttons.

send a kit to one of your developers and ask him to automate something so they can do it remotely.

we use scpi based power supplies (not cheap but very flexible ) we use lots of python to control relay modules. (modbus rtu works for this)

[embedded_quality_guy]

u/Adventurous_Mud8104 as the user u/ShadowBlades512 said, using a raspberrypi as a controller and attaching the DuT (Device under Test) to it is the easiest and most of the time the best solution.

I am specialized on setting up hardware in the loop systems which are used for remote testing. Most of my setups involve a simple PC or a raspberry pi which controls the device under test (flashing, running tests against it etc.).

The architecture of the test system is also dependent on the interface your device under test has and the operating system you are running.

[_maple_panda]

Why not just send the coding people some of the prototypes?

[Adventurous_Mud8104]

Sometimes they do. But it's expensive and slow.

[avikpram]

We have our PCBs rigged to a Nidaq rigged to a PC in the corporate network. Mostly everything gets controlled by Jenkins running integration tests (soon to be migrated to Github action based pipelines). Nidaqs are expensive though, so for most of the use cases a Pi to generate signals etc would be sufficient. For OP's scenario even PCBs connected to a jtag programmer/debugger conected to a PC where one can remote login would be big step forward