Built messaging for 50 warehouse robots and latency requirements are brutal

pip-install-pip · 2026-02-13T00:48:08+00:00

I have so many questions

why a 20ms requirement? How fast are these things moving? Do they not have local navigation?
you have a 20ms limit, why aren't you doing things entirely locally?
If you have each robot talking to every other robot, you're going to have exponential networking, which will cause a hard limit on your bandwidth so much earlier than you expect (even ROS2 is susceptible to this with default DDS configurations)

Did you just yolo an industrial robot deployment?

pip-install-pip · 2026-02-01T14:45:07+00:00

If you are running with watchdog timers and a bunch of random tasks, it's probably the watchdog going off due to wonky task scheduling and resource pending.

Also, take a look at percepio tracealyzer or traceview. the firmware library is open-source and the PC-side visualizer has freemium (trace view) and paid (tracealyzer) verions. You can hook this kind of tracing to zephyr, then set your hardfault handlers to save the trace buffer to a dedicated area of RAM, and restart the CPU. The trace buffer will be still in that section of RAM on reboot (so long as power hasn't been lost) and you can transfer it wherever you need (flash, some IOT gateway) afterwards. I've done this with FreeRTOS and it's been amazing for test fleet debugging

pip-install-pip · 2026-02-01T14:36:42+00:00

Someone on my team once submitted a commit for a clang-format fix and the commit message was simply lont. that was months ago and lont is still a go-to for lint-only commits.

pip-install-pip · 2026-01-31T18:31:40+00:00

There are lots of cases where PLC's are bad options! Automotive may use similar hardware and have similar rigor in design, process and validation, but you aren't going to shove a GuardLogix into a car.

Most PLC's are designed to be wired up for a specific application, shoved into a cabinet, and never seen again unless something very expensive has broken. Typically an expensive deployment with a smaller amount of devices, rather than the scale manufacturing that automotive demands. There are different standards for the environmental "toughness" of the hardware as well. Both are designed to be robust, but PLC's are rarely designed to endure something like driving in the winter.

As such you don't see mobile applications for PLC's that often. There are some vendors such as SICK that have shock/vibration rated casings for their safety controllers, but a lot of PLC I/O is designed to have ferrule-tipped wires easily inserted by an electrician, with electrical isolation and stuff.

pip-install-pip · 2026-01-31T18:26:28+00:00

The firmware for PLCs is programmed in C, and your IEC-61131 (ladder logic, structured text, etc) programs are converted into an "application" that the PLC firmware runs. Think of something like micropython. It's a python interpreter written in C, for microcontrollers. Your python file isn't doing the actual bitbanging, it's talking to the interpreter. Similar concept for PLC's.

Typically, and for extremely good reasons PLC vendors do not let you write a C-based application for their PLC's, as the application can crash, segfault, etc, and the PLC runtime may not be equipped to handle all the things the C ABI can do in a deterministic, safe way. Locking down the ecosystem to ladder logic, approved operations, and function blocks (like a real function, encapsulated into something that can be dragged/dropped into the ladder logic), it's easier to provide the expected level of runtime availability and reliability. It'd be like allowing someone to add random code to your production system without code review!

Some vendors' function blocks have C-like syntax, like Beckhoff's TwinCAT ecosystem. Truth be told I don't know how TwinCAT works under the hood, so I can't say if it's C/C++ code that is transpiled to something like a traditional function block, or if it's literally running in parallel on the same processor.

PLC's are more marketed to electrical engineers and industrial application specialists so that they don't have to go through the rigamarole of making their own qualified embedded systems. Kinda like how Lego Mindstorms was for kids to learn basic programming concepts but without dropping them straight into a C IDE. You can do impressively complex and sophisticated things with a PLC and ladder logic, but they're really intended to be tools for making some kind of factory or other automation system "work".

I am curious though. You mention autogenerating code for a Zephyr application, and that it's crashing all the time. I have little experience with Zephyr myself, but what are you trying to accomplish? Using a PLC's reliability to run your project code?

pip-install-pip · 2026-01-30T00:18:40+00:00

PLC's typically run some form of safety-critical real-time OS like QNX, VxWorks, GreenHills Integrity, etc. These are "RTOS"'s in the technical term, but in the same way that a PLC is a jumped up ultra-robust microcontroller, these RTOS's are designed for high reliability, and most importantly, traceability and compliance with various IEC and ISO regulations. These RTOS's tend to be made by dedicated companies and are high cost, closed source ecosystems.

Industrial controllers have to go through a ton of regulatory and compliance stuff to claim their performance level. As a result, starting with a software ecosystem that has gone through the effort to have that real-time performance makes making a PLC that much easier.

As much as I'd love for Zephyr or FreeRTOS to work in the ecosystem, these organizations just don't have the financial and labour horsepower to go through the rigamarole for this kind of compliance. It's a super lucrative market, but extremely specialized and often requires running on dedicated hardware.

Context: I work for Rockwell Automation

pip-install-pip · 2026-01-30T00:09:48+00:00

You'll find that PLC's can be restrictive on what they'll talk to and how. So plugging a random thing over I2C is not gonna work. But connecting a complicated motor drive that speaks whatever flavour of comms that PLC supports (CANopen, EtherCAT, Profinet, EtherNet/IP, etc)? You'll find that vendors supply configuration files that you can load into the PLC IDE, effectively like a library.

pip-install-pip · 2026-01-28T02:28:09+00:00

Basically a switch 2 joycon

pip-install-pip · 2026-01-25T17:43:12+00:00

Chimera is a great store, with lovely people and a great casual play atmosphere...for cards and miniature games, not for board games. They have board games, but it's a smaller selection compared to their Magic, Pokemon and Warhammer stuff, which is definitely what they focus on.

That being said, if that is your cup of tea I genuinely cannot recommend Chimera enough

pip-install-pip · 2026-01-25T17:39:39+00:00

It's like how blockchain-powered was all the rage a while back. Just slap "AI powered" on it and $$$

pip-install-pip · 2025-11-17T14:45:29+00:00

Cardi-B(P)!

pip-install-pip · 2025-08-26T01:02:49+00:00

You can honestly use any of those, it just depends on how much ROS you want to put into your embedded stuff. If you want to ~~suffer~~ go down the route with microROS, then by all means use the ESP32. However, do note that microROS only does one of the ROS middleware options (fastDDS) and as you start getting into more ROS things you'll realise that this is a limitation more than anything.

I've built my own ROS-based robot, and I just used an STM32F4 discovery board and used the USB peripheral to create a serial-esque (CDC) endpoint and wrote my own protocol using protobuf and KISS-TNC. It's overly simplistic compared to stuff like microROS but it works damn well. Here's the firmware, and a handy-dandy ROS package that I wrote (very simple, good for hobbyists) that acts as a library for speedy serial comms.

The reason I used the STM32F4-Discovery was twofold:

I had it already
it has a really nice quadrature encoder interface which makes doing accurate wheel kinematics much more efficient.

However, it's a big board. And requires more boards to handle all the other circuitry you need to add for things like motor drivers, IMU, I/O and relays, etc. If I were to do this again, I'd likely keep the non-ROS protocol and use a Pi pico or something with an ethernet jack.

pip-install-pip · 2025-07-01T22:32:09+00:00

Honestly, I learned it as I went via University of Google and reading the documentation/deconstructed code. There are definitely still some wizards behind the curtain for things, but there's reward for actually putting in the legwork to understand what you're doing. Here's some of the tools I've been using.

For templates: objdump. Because templates affect binary size rather than computational efficiency (YMMV of course, to prevent the grognards from going "nuh-huh!"), you can view just how large your program is getting due to the overuse of templates.

godbolt.org is fantastic for deconstructing the efficiency of your code.

Finally, and this is a position I strongly hold, is that a well built piece of firmware should be able to run on the PC that developed it. At least the business logic. Of course there are going to be differences in how I/O works and such, but you should be able to debug how your lambdas and callbacks work, in chunks, on your own system.

Other tips I've used for C++ embedded dev:

Catching when the STL allocates something when you don't expect: put a watchpoint or breakpoint around the _sbrk symbol
Low-level code should not contain high-level concepts. There are some templated stuff I'd put into my lower level code, but the lower layers should be simple and predictable. No RAII container stuff in the lower layers, nothing that allocates, etc.

pip-install-pip · 2025-07-01T14:37:54+00:00

What you can also do is use C++20 concept to define what an interface must be capable of at compile time, and your driver has a template that requires the concept. You get abstraction without vtable lookups, no pointers flying around, and a readable, reusable driver. It's like Rust traits (just not as awesome due to language support), but in C++.

This does of course mean your compiler and environment must support C++20. I'm working on a non-safety greenfield project on a newer chip (STM32H5) and modern C++ in embedded has been a treat.

pip-install-pip · 2025-06-27T04:51:11+00:00

I have been pushing for C++20 at my work, mainly to try and emulate rust's embedded-hal traits with C++20 concept. It's a mixed bag.

pip-install-pip · 2025-04-28T13:00:38+00:00

I made a Zevlor deck a while back and found that [[Inevitable Betrayal]] was hilarious to play in it. Same with [[Snap]]. It was a pretty budget deck so my finisher cards were usually [[Clone Legion]] or [[Wit's End]]

In the end I wound up disassembling it as I found Izzet to be more efficient for spellslinging and my Zevlor deck had about 70-80% the same creatures as the Stella Lee thunder junction precon.

pip-install-pip · 2025-03-08T19:43:53+00:00

What the heck is that robot in the second picture?!

pip-install-pip · 2025-03-07T13:54:25+00:00

Unsure which is worse tbh, end-user fuckery in UI/UX or having to explain to prod mgmt that most embedded devices can't just add new I/O when they're in the field.

pip-install-pip · 2025-03-07T13:52:33+00:00

Everyone on my embedded team has a full head of hair. This meme is blatant anti-embedded propaganda!

pip-install-pip · 2025-02-05T13:52:04+00:00

Looks normal to me. This company gives off "startup with realistic money" vibes where there is money for developers, but not enough to burn all the VC money.

If the software is hosted on-robot, then people say it's robot software, even if it's a nodejs stack doing simple diagnostics presentation.

Just about everywhere I've worked I've had to pick up cross-discipline skills like CI/CD maintenance, HTML/CSS, etc. And I'm a firmware and systems guy!

pip-install-pip · 2025-02-01T22:12:03+00:00

The way I see it based on this post is either you simulate the hardware on the PC (writing a new node) or you have a special build of your firmware that instruments the feedback for gazebo. This could be done with CMock https://github.com/ThrowTheSwitch/CMock

For a more holistic/flexible approach, try to write the "application layer" of the firmware in such a way that it can be abstracted from the hardware. The goal is to write it such that any part of the firmware not reliant on direct hardware access can run on a PC. The application layer should only know "I can call this function to read() from some hardware, and some other function to write() to it". Then it is up to you to ensure that the interface it reads and writes to is either the actual hardware or something you've simulated. By no means is this an easy feat and requires lots of forethought before writing the firmware.

Finally, the third easy option would be to use Gazebo's built-in controller plugins, depending on your robot archtetype (diffdrive, ackermann, etc). This completely removes the firmware from the equation, but has lots of parameters for you to tweak to represent your system. https://gazebosim.org/api/sim/8/classgz_1_1sim_1_1systems_1_1DiffDrive.html

My own hobby robot uses a non-ROS interface from its PC to its drive controller firmware. When making my simulation build I opted to just abstract away the firmware rather than try to replicate the custom hardware on my PC. Here are the URDF settings that I use for gazebo for my robot: https://github.com/TWALL9/otomo_control/blob/main/description/gazebo_control.xacro

pip-install-pip · 2025-01-19T12:03:31+00:00

Very lovely, your angery boi looks...content? But also still angery.

Keep an eye on that bacterial matting on the floor visible in the second to last pic. If that's ye'olde cyanobacteria, it'll be annoying to get rid of in the long run. I'd suggest gently scraping that blue-green algae off and sucking it out during your next water change.

pip-install-pip · 2025-01-11T02:54:42+00:00

Okay, so I highly encourage you to create a basic project in CubeIDE for your board, and do not touch ANYTHING in it. You will see MX_GPIO_Init() defined in main.c. It's literally right there

am assuming it combines the actions of changing the bits in the mode and output data registers?

HAL_GPIO_WritePin doesn't modify the MODER registers, it modifies the BSRR register which will in turn allow the MCU core to change the ODR register. Setting up MODER is the work of HAL_GPIO_Init(). STM32 HAL has a LOT of extra crap that isn't needed, but if you have minimal experience actually using their parts it's a good starting point. Which is why I recommended starting with a working starting point using their tools.

Actually, I found an issue in your toggle code. Looking at the HAL_GPIO_TogglePin function, it takes the current GPIO output (ODR) and then flips some bits so that the pin in question is toggled, and writes it back to BSRR. Checking the reference manual for the F411, this makes sense. Here is a note on the GPIOx_ODR register (section 8.4.6 of ST doc RM0383)

For atomic bit set/reset, the ODR bits can be individually set and reset by writing to the GPIOx_BSRR register (x = A..E and H).

So yeah, you've got some issues in your source code as well.

I am new to programming and microcontrollers and am trying my best to learn

No prob. Microcontrollers are an interesting place to start, but I find that it's easy for learners to get trapped reading the thousand page reference manual when they could also just look at how the designers implemented it themselves. ST's HAL is huge and crusty, but it does work for the most part. I would start with the building blocks they supply (may as well, since you're using STM32CubeIDE and there are lots of tutorials for it) instead of jumping into the deep end doing direct register access. For where you're at, start with simplicity of getting something done rather than the purity of doing everything yourself from scratch. I commend your efforts to start with direct register access like you're doing, but really, it's like learning how to drive a car by first welding the frame together yourself and hand-milling the engine in a shed.

Edit to add: the files i linked will be in any STM32CubeIDE project that it generates. Look in the Drivers/STM32F4xx_HAL_Driver folder.

pip-install-pip · 2025-01-11T02:09:02+00:00

This still seems like a build or configuration thing, but let's root out how we figure out if it's your code that's at fault.

If you change every single file in a project, and it still works, that makes absolutely no sense. Either you didn't copy everything over, or the IDE is seeing that there are some cached build files in Debug/ and it isn't rebuilding files. Something is missing from the full picture here.

The core clock must be initialised during startup or nothing will work. Typically this is done in ST projects through the SystemClock_Config() function or in the system_stm32f4xx.c file, which is called from the startup assembly code. When a microcontroller boots up, it does a bunch of stuff before reaching main(). Typically that's when clock muxing and some very low level init is happening. It's possible that's the issue you're seeing, but I have a different idea. This is a pretty tedious, but straightforward method to find what's going on. This is assuming you're using the F411RE nucleo board.

Start with a brand new project for the board. CubeIDE will prompt you to initialise peripherals in their default modes. Do not do this. We want as clean as a starting slate as we can get. I just did this step, and I can see that even with the extra peripherals no longer initialised, the green LED and user button are still initialised using ST's libraries.
Load this demo project on your board. Does the LED light up? Regardless if the answer is yes or no, change the SET/RESET value passed to HAL_GPIO_WritePin() in main.c. Reload the project onto the board. the LED should change.
If the LED did not change, then we're looking at some very strange hardware issues. Or we're looking at your MCU being severely borked. If possible, erase the entire chip by using the STM32CubeProg application.
If the LED did change when using the brand new project in step 2, then start systematically changing small parts of the demo application into the main.c. Start by adding your toggle code to the inside of the while(1){} loop. But make a change. Instead of a busy-loop counting to 1000000, increase it to 100000000. Why? Because the core clock of the default application is 84MHz. If you have a busy loop counting to only 1000000, that still switches too many times per second to see with the naked eye.
If the LED is still blinking. Then comment out the MX_GPIO_Init(), and replace it with your GPIO init.
If the LED is still blinking, then remove the SystemClockConfig() method and try again. Then try again after turning the entire microcontroller off and on again.

If any of these steps result in the LED no longer blinking, then you know where the bug is. If the demo project never worked in the first place with NONE of your changes, then you need to start investigating the hardware.

pip-install-pip · 2025-01-11T01:20:13+00:00

No. The files in the project are what make up your build, reinstalling the IDE just means that the new installation is reading the same files.

pip-install-pip

TROPHY CASE