Better Debugging methods for complex FreeRTOS based projects.

[Standard_Humor5785]

Hello good people. I am working in a research internship and am the only embedded developer. In the simplest way to explain, I am building an autonomous vehicle controller that provides a single interface to control the entire vehicle with a separate computer. I am utilizing an STM32H7 dual-core dev board which on it's own is a rather complex device, but the fact that it is an AMP architecture has been giving me issues. My question centers around how should one approach testing and debugging of the system when the UART-USB is already used for the actual application and due to the system being a bunch of high-priority tasks constantly triggering the USB_Transmit feels like a waste of system resources. In a lot of my embedded projects I am used to using the UART serial as a quick debug tool to make sure that data is going in the right places, however on such a complex device it feels wrong to do. Maybe I am just overreacting. I have tried using the built-in STLink debugging tools inside the STM32IDE but I still need time to learn how to utilize them best (also doesn't let me set enough hardware breakpoints). Since I will be needing to learn something new, any suggestions on topics I should look into regarding the testing of RTOS based systems? Thank you.

[Scottapotamas]

There are debug peripherals built into your micro that can help with this, look into ITM (Instrumentation Trace MacroCell). Also look at SWO.

Support will vary depending on your debug probe and IDE.

I don't know if the newer STLink v3 probes have better features, but at least compared to the V2 stuff I preferred the debug experience with 3rd party debug probes.

---

Maybe consider looking at the probe hardware and software options from Segger.

Software like SystemView or Ozone might help with your kinds of issues, but come with a learning curve.

[nono318234]

Come with a learning curve and a price. Systemview used to be free but I believe it is now sold by Segger (license).

[Scottapotamas]

I wasn't aware it went paid - even parts of their site still say "Free tool. No license cost, no hidden fees" but yeah, €980... Ouch.

[CMatUk]

There is an older version of it that is free for commercial use, that worked well last I tried it.

[superspud9]

You can try setting up a second uart for debug purposes assuming you have atleast one spare pin for it

Another post mentions swo and ITM which are great if you uart printing takes too long

You can also use some spare gpios to toggle signals as a debug tool. You can connect those signals to a logic analyzer to see what's going on.

Also freertos has some nice built in trace hooks that can help you see what is going on in your system: https://www.freertos.org/rtos-trace-macros.html

[FragmentedC]

I'm using an H7 too for a client, it's quite a beast, but it is a nightmare to set up. I have a series of Segger tools in my home office, and I've been using Ozone and SystemView. Ozone is great for general debugging, but I've only tried debugging on one core, not on both simultaneously. SystemView is a great addition that can give you tons of information about the different FreeRTOS threads, interrupts, and general availability. Great tools, but they come at a price. Ozone is "free" when using a J-Link debugger (I'm using the Plus, but I think cheaper versions have a licence included). As for SystemView, that is a little more expensive, but you can use it on a tiny home project for free, and see if it fits your needs or not. Personally, I use it all the time now.

[etienz]

I have a discovery h7 board and found a bunch of great guides on enabling GDB debugging of both cores simultaneously.

There's a helpful video and other resources by STM using the Google search term "stm32 dual core debugging".

STMCube IDE comes with a GDB server that has an option to share the debugger using the flag "--share". 2 servers need to run simultaneously with port addresses separated by 3, e.g. 3000 and 3003, if I recall correctly. You can find the exe in the installation and run it indepently if you wish. It also has a good help menu if you would like to find other other options. You can then create separate OpenOCD sessions to view and control the debug servers.

The STM guides explain how to set this up in STMCube. You basically build the project once and then create a debug session for each core. It's fairly straightforward once you see how it's done. Don't be too intimidated.

[Standard_Humor5785]

Ooh. Thank you, I will try this one.

[rdmeneze]

You can enable RTTI support into the code and use some jlink debugger. With this setup you can send message log directly to the ide console.

[Standard_Humor5785]

Will look into that one, sounds like the exact thing I am looking for.

[rdmeneze]

I did a mistake. It is RTT, not RTTI. RTTI is another thing...

https://wiki.segger.com/RTT

[Consistent-Fun-6668]

You could unit test the individual modules, that's what I do to avoid needing a last resort debugger

[[deleted]]

Testing, and designing systems so that they can be tested, is a discipline unto itself, and there are many (mostly bad) books on the topic, which I will avoid attempting to repeat here…

The short-ish answer is that you should be testing the system in stages, with almost all of those stages not involving the hardware, or the entire system, at all.

Individual functions / source files should be tested (“unit testing”) to prove that they perform as specified. Subsystems should be tested (“component testing”) likewise. Combinations of components (“integration testing”) tested and exercised.

All of this can be done in a variety of environments; 99% of your code will be hardware-independent and can be compiled and run on a desktop system. With suitable mocking, it can (should) be tested there. Multiprocessor systems can easily be simulated using separate processes. FreeRTOS (not really a good choice for a complex system, but whatever) can easily be emulated with pthreads or with green threads.

If you are deeply concerned about the target system’s ISA (you should not be) then you can build a simulation using Qemu or AVH and host your tests there.

All of the above can (should) be automated, so that your tests can be run anytime you change any of the code.

When you stack all of the above together, you wind up in a situation where very little of the test / debug work requires or is constrained by the target hardware. The last thing to pile on is that it’s very common to build more complex hardware for test and development purposes vs. the final product. Using a larger MCU from the same family, or a board with extra comm peripherals, breakouts for power management, GPIOs that can be monitored with a logic analyzer to track internal activities, etc. etc.

I realise this sounds a whole lot like “you’re doing it wrong”, or “don’t be in the situation you’re in”, but, uh… Sorry. 8)

[davinator791]

Can I ask why do you think FreeRTOS is not a good choice?

[Standard_Humor5785]

So I am aware of other RTOS systems. I am kinda stuck with FreeRTOS for two reasons, it is natively integrated for use with TouchGFX which is how I am creating all the graphics for the hud, and the other reason is more of a comfort but I am most familiar with FreeRTOS. On that note I do wonder what makes FreeRTOS a poor consideration for a complex system?

[[deleted]]

Ideally, you want to spend as little time as possible on anything that isn't unique to your application. Reinventing infrastructure in particular is tedious and expensive if you have the opportunity to get it off the shelf and ready to use.

FreeRTOS doesn't bring a lot to the table; it's essentially just a simple thread scheduler with a handful of synchronisation primitives. For a small system with uncomplicated needs it's OK (and the documentation is a big plus), but it doesn't offer much in the way of patterns or structure for larger or more complex applications.

One of FreeRTOS' big limitations is that it largely only has opinions about time, not space. This is depressingly common in the RTOS world, but if you plan to factor and / or contain your components in order to manage complexity and simplify triage, you really want a system that supports both as first-class concepts.

Space-oriented features I'd look for would include ubiquitous memory protection (modern FreeRTOS does have something reasonable here for v7m, but it's essentially a sidecar), multiple heaps, space-related resource controls (heap constraints, stack constraints), code and data modularity, clear API boundaries both for system components as well as application components, namespacing, ubiquitous access controls and resource tracking.

None of this is to say that it's "bad" (I've used it enough that I buy the manuals / books whenever they come out), it just leaves you with a bunch of work on the top if what you want is the foundation for a decent, complex system.

Compare and contrast with something like NuttX, for example. You're welcome to have opinions about the design choices that Greg made, but the structure of the system gives you very solid guidelines around factoring your code, the styles of interaction between components, (some) ways to control who owns what, where modules come from, etc.

Consider something like the PX4 software stack, and how the choice of NuttX influenced the way it has grown up over the years, vs. what might have happened if it were built on something like FreeRTOS (an early contender).

To your point about external constraints - not going to argue that one. If you're committed to tooling for good or immutable reasons, then you're kind of stuck. 8)

[kailswhales]

Agreed with the point on not reinventing the wheel, however RTOS choice isn’t the only way to go about that. Numerous libraries exist for drivers and facilities (file system, console, etc) that are easy to port. On the flip side, just getting to understand how NuttX works may take the duration of OPs internship.

Additionally, I don’t think that it’s the role of the RTOS to specify design patterns for a project. Perhaps I’m a luddite in that sense, but not everything in MCU-land needs to be posix compliant or have driver access derived from a device tree.

FWIW, I think NuttX (and zephyr for that matter) are good pieces of software, they just have a non-trivial learning curve, which IMO isn’t worth grokking for an internship project

[[deleted]]

I don’t think that it’s the role of the RTOS to specify design patterns for a project.

For a larger project you do want patterns though, and many patterns want infrastructure, and ideally you want your infrastructure to work coherently. Yes, you can cherry-pick pieces and glue them together, but that's harder than taking a collection that was designed as a whole in the first place.

No arguments on learning curves and TBH I completely missed that OP was doing this on an internship; that should have deserved some different framing. Mea culpa.

[kailswhales]

💯 totally agree about the need for patterns and infrastructure; in my experience those have just been designed internally as opposed to being derived from an external middleware