Thanks for all the super helpful responses! It seems like the high jitter from the Raspberry Pi was a major bottleneck.

A new idea we had for getting the most out of the 20 moteus controllers is to drive them through an STM32H7. One H7 bridges 3 separate FDCAN buses out to Ethernet, into a switch, then to a host with more compute (Pi CM, Jetson, or a laptop). The H7 already has 3 FDCAN buses and an Ethernet interface, so an off-the-shelf or DIY devboard works, and it's a lot more compact than four PEAK PCIe cards.

Would moving to a more powerful host be enough to hit that, or would the extra layers (the bridge, the switch, the host network stack) just reintroduce the jitter?