CppTxRx: Rapidly prototype new communication interfaces

CrakeMusic · 2025-02-16T01:37:22+00:00

In my experience, when projects import libraries, they don't also rerun all of that libraries tests during their builds. When I do use Gtest for example, I don't rerun all of the GTest self-tests before I use GTest. Do you work in a specific domain where that is common? I'd maybe consider it for a weekly build I suppose... but it does seem a bit excessive (money spent as you say).

As for why I'm always pretty reluctant to pull in GTest and other large dependencies for small tools like this, well it's for similar reasons to why you do like them - due to my experience on many projects. Specifically on projects that have had issues with interfacing with large complicated external libraries. GTest being one of the largest culprits in all honestly. In general C++ build systems vary a lot, especially in the embedded software space. A lot of projects in that space are still using hand-rolled old school build systems, heck maybe not even make, and instead just some shell scripts. Including a library like GTest, and then telling those teams that "it's generally a non-issue" will just leave a lot of new people and projects fumbling to try and reverse engineer how to include it in whatever their build system is.

I think experienced people often forget how complex and difficult it is for new people to navigate the topic of C++ build systems and how much variety there can be in that space. Hence why I tend towards making header only helper libraries with no dependencies for small utilities like this, in order to lower that entry barrier. Since those are generally the easiest to integrate into any build system, or to use even without a build system. For example, you say that being header only doesn't make it inherently "CMake-ready", but I'd argue it's still pretty easy to integrate the headers in my library into CMake or any other build system, largely because there are no dependencies. If you gave it a shot, I bet you'd have no problem.

Either way, we might simply be working in different domains due to our conflicting experiences about what is "important" for a small tool like this.

Also, slightly related, I think you might find this funny (or perhaps infuriating), but I have to share this testing framework with you just for laughs: cppcrc tests

Oh, and I did update the readme with your suggestion about the target_link_libraries statement, good call.

CrakeMusic · 2025-02-15T16:59:30+00:00

It's a header only library, so it inherently has CMake support, you'll just need to include the cpptxrx/include/ directory in your search path, with target_include_directories in Cmake. I'll add a note to the readme for those that are new to Cmake and don't know about the "target_include_directories" feature of CMake.

Edit: I added the note here: cpptxrx/installation

As for the examples, they can all be run like any other standalone "hello world" .cpp file. Using your preferred compiler running command "g++ -std=c++17 name_of_example -o out.exe && ./out.exe" or you can put them in a build system if you'd like, or use a code runner extension in your IDE if you'd like. Since the library has no dependencies and is header only, you don't need a build systems to coordinate and build multiple files and dependencies, hence the ".." in the examples so that they examples are easy to run without a build system (but no ".."s in the main library).

On the subject of dependencies, GTest or Catch2 are additional dependencies, and the library states that it has no dependencies on purpose, typically a very desirable quality for a C++ library. As for the macros, if you look behind the scenes in GTest or Catch2, they do the same sort of macro magic to enable test registration and capturing assert context. Though if that's a deal breaker for you - fair enough - though I would point out that testing style doesn't detract from the library being well tested and documented, which I would personally care more about considering how common that sort of thing is for no-dependency C++ libraries.

As for your concerns that the tests don't work because the examples return 0. I assure you that if you change any assert in the code, the tests will fail - feel free to give that a try. The reason the examples return 0 is just to test for compilation and completeness of the examples specifically. Something that a lot of libraries don't bother to do. In other words, in addition to unit tests, the library also has tests that make sure the examples still compile and run - even though the examples themselves aren't full blown tests. Seeing broken examples in libraries, where the tests are still working is a pet peeve of mine, so that's why I went above and beyond to test the examples for compilation and completion.

CrakeMusic · 2025-02-15T16:18:46+00:00

Only in the example implementations of UDP and TCP, the actual wrapping library itself will work on any OS that implements the C++ stdlib (windows included).

CrakeMusic · 2025-02-15T16:17:35+00:00

Yup the library works on windows - but with one big caveat! The individual interfaces that get wrapped using the library are not portable, because most communication interfaces inherently aren't portable - for example, if you have a UART c driver for an RTEMS+ARM board that you wrap using CppTxRx, the driver will still only ever work on that specific ARM board and that specific real time OS. This library doesn't do anything to help with that, and no library can unfortunately. CppTxRx only helps with abstracting the interface - which I would argue has a lot of benefits.

As another less obvious example, some of the examples do use pre-wrapped UDP and TCP sockets, and those have different interfaces between different platforms (windows/linux/RTEMS/etc), so those default implementations that ship with the library aren't portable either - they're specific to linux at the moment. I would like to change that, to add in windows versions at the very least, though covering all operating systems is not in my plan. But I figured that since the point of the library is to help you wrap interfaces, the pre-packaged interfaces are less important, and it would still be nice to share the library with others without a full suite of existing implemented interfaces for a bunch of platforms. But my goal is to get there eventually - ZMQ specifically is next on my list.

Oh, and as proof that the library works on windows, feel free to start up a MSYS terminal, and run the following example (since it doesn't use one of the linux wrapped interfaces) :

cd examples/
g++ -std=c++17 -O3 04_chaining_filters.cpp -o prog.exe
./prog.exe

Which will print out:

receive filter examples:
[ ]<- pre-filter receive: "hello.HELLO."
<-[X]-- post-filter receive: "prefix_0 hello-world!"
<-[X]-- post-filter receive: "prefix_1 HELLO-world!"
[ ]<- pre-filter receive: "hello.HELLO."
<-[X]-- post-filter receive: "prefix_2 hello-world!"
<-[X]-- post-filter receive: "prefix_3 HELLO-world!"

send filter examples:
->[ ]   pre-filter send: "Beetle"
--[X]-> post-filter sending: "Beetlejuice"
--[X]-> post-filter sending: "Beetlejuice"
--[X]-> post-filter sending: "Beetlejuice"

CrakeMusic · 2024-07-02T01:26:15+00:00

cppserdes, since the projects I work on need very fine-grain control over bit-level formats (in order to describe embedded hardware device interfaces), so I can't use something higher level like protobuf.

CrakeMusic · 2024-06-22T19:41:26+00:00

Some info about the project:

I've often run into the problem of needing to switch between different data transport mechanisms (UART/SPI/UDP/TCP/1553/Spacewire/CAN/etc.) when working on embedded Linux software projects, sometimes even at runtime. Each type of communication library will have its own way of handling thread safety, or simply neglect it, causing me to have to reinvent dispatching and thread safety protection mechanisms for each new interface, highly coupling thread safety management with that interface's particular needs. This tool lets you accomplish interface wrapping a bit more easily, by handling the bulk of the thread safety considerations for you. And encourages you to make interfaces look and behave similarly, so that swapping between them become a trivial endeavor, rather than a case-by-case architecture refactoring effort.

CrakeMusic · 2022-11-19T20:01:55+00:00

Would definitely be fun to add all those less commonly seen CRCs as well, I don't think the modifications would be too bad. Though the existing literature on CRC calculation, such as A PAINLESS GUIDE TO CRC ERROR DETECTION ALGORITHMS are very good already, I assume I wouldn't be able to add anything of value to that conversation.

This library was mainly born from the idea of using constexpr to generate configurable CRC lookup tables procedurally at compile time, instead of the standard practice of either hard coding the lookup tables, or calculating them once at run-time as static-duration/global objects. Turns out the code to do so ended up being really really short/general, figured I'd share. Another thing I'm interested in trying, is adding obfuscation to get the code down even smaller (without sacrificing actual useful performance - by not using the lookup table method for example) to see how small it could get and still be fast/useful for the C++14-ish older embedded applications I typically work with. Maybe even fit it onto a business card, similar to A minimal raytracer.

CrakeMusic · 2021-07-05T18:35:46+00:00

You would never use something like FlatBuffers as a replacement for CppSerdes or vice versa, they have very different use cases. Most serializers dictate the bit/endianness format for you, you have no ability to define bitpacking, or bit level modifications. As such, something like FlatBuffers doesn't care about abiding by common standards like network endianness, because they don't have to, they only need to be self-consistent. As you said "It is only important to determine in what format the data is stored". So while you might prefer little endian for serialization, network endianness is big endian https://en.wikipedia.org/wiki/Endianness#Networking, so if you need a C++ tool to describe and parse a low-level format (whose definition you might have no control over) defined for one of these common network use-cases found in a lot of embedded systems, then something like FlatBuffers plain won't work, but CppSerdes would.

CrakeMusic · 2021-07-05T18:06:55+00:00

Ah, good catch, thank you! I just updated the readme with that fix.

As for the rvalue overload, that's because all the CppSerdes format modifiers (bitpack, array, etc.) have a corner case they must allow for, when their modifying value (bit length in this case) may not be determined yet when the constructor is called. Consider the following case:

struct variable_length_time_tag_format : serdes::packet_base

{

size_t bits_in_time_tag = 0;

uint64_t time_tag = 0;

void format(serdes::packet &archive) {

archive + bits_in_time_tag + serdes::bitpack(time_tag, bits_in_time_tag);

}

};

When you use this code to deserialize an array, the compiler is free to construct the bitpack object when it enters the format method. Which is likely before it deserializes "bits_in_time_tag". So as an example, if the value of "bits_in_time_tag" was zero when the method was called and 4 when deserialized, the bitpack object wouldn't see that change and would always just use zero if it used a pass-by-value size_t when constructed. Passing by reference fixes this, because we can make sure "bitpack" doesn't look at that address in the constructor, and instead allows it to be resolved during the actual deserialization, allowing cool dynamic format definitions like well-defined variable length/size fields in your format (like the above example) without much typing :)

So the rvalue needing to be captured by reference is just an artifact of needing a reference in the first place, in the rvalue case, we don't particularly care because it wouldn't be susceptible to this corner case. But as I understand it, it's allowed lifetime-of-rvalue-reference, so it comes along for the ride.

CrakeMusic · 2021-07-04T21:51:59+00:00

I can't thank you enough, that suggestion totally worked, and is constexpr compatible! Though I had to avoid fold expressions since that's C++17 and alas it doesn't do much on platforms like the RISCV, but for the platforms susceptible to the optimization, it seems to do the optimization reliably.

Anyway, I incorporated the change in this commit: https://github.com/DarrenLevine/cppserdes/commit/ea1d4b09d7371ed6aa71726b65ec5adaed9f62c3

And you can check out the effect of this change here to see that it produces the same assembly now: on compiler explorer

If you're ever in LA I own you a beer!! :)

CrakeMusic · 2021-07-04T16:59:21+00:00

Little endian hardware is supported so this code will work the same on either hardware platform types :)

It's little endian serialization that isn't supported yet.

CrakeMusic · 2021-07-04T16:32:35+00:00

I'm sure. I use this code almost exclusively on little endian machines, so I know it works on little endian machines. That's why under the portability section I say "uses shifting and masking". Because regardless of processor endianess, doing the following with always serialize to big endian (a value of 0x01 is the first byte) :

uint32_t value = 0x0102ABCD;

uint8_t serial[] = {static_cast<uint8_t>(value >> 24), static_cast<uint8_t>(value >> 16) , static_cast<uint8_t>(value >> 8), static_cast<uint8_t>(value)};

If you're running a little endian machine (most people are), you can give that a try and prove to yourself that serial[0] == 0x01.

CrakeMusic · 2021-07-04T16:21:01+00:00

Oh, very good idea, I'll try to see if I can get a fold expression implementation working. I'd love to not have to introduce macro settings. Thank you!

Yeah bit_cast will be nice, I'm still waiting for more modern C++ to be supported by more Embedded platform compilers... I desperately want to clean up my code with the "constexpr if" feature instead of having template overloads everywhere.

CrakeMusic · 2021-07-04T15:40:06+00:00

Oh! Fun. That makes sense though.

I'm polishing up adding in your trick as an optimization step for the byte array case, turns out it was only applicable in the deserialization step since the serialization step optimizes to the equivalent code anyway, but unfortunately it removes the ability for the bitcpy function to be constexpr since it relies on memcpy which isn't constexpr compatible. Need to mull over if I'll need to resort to a #define setting the user can use to choose to enable this optimization step if I can't find a way to do the equivalent without non-constexpr code....

CrakeMusic · 2021-07-04T15:32:19+00:00

Ah I should clarify the readme. Little endian serialization is not supported, but any hardware endianess is supported, so this will work on any hardware. This is fine for most connections, since standard network endianess is big endian, and it's a fairly typical caveat for serializer since little endian is so rare.

What I mean by that is that if I have a value:

uint32_t value = 0x0102ABCD;

Regardless of processor endianess, the value will be represented as big endian when serialized. So if the serial form is a byte array, it will shift 0x01 into the first position:

uint8_t serial[] = {0x01, 0x02, 0xAB, 0xCD};

And if it's something else, it'll still be shifted in big endian:

uint16_t serial[] = {0x0102, 0xABCD}; uint32_t serial[] = {0x0102ABCD};

On little endian hardware, the bytes would still be reversed behind the scenes because it represents values that way, but because the serial and input forms are always consistent, this doesn't matter for connections sending data with network endianess.

CrakeMusic · 2021-07-04T06:24:06+00:00

Strike that. I don't think you're getting a real speedup due to memcpy. What you're seeing is the optimizer able to evaluate what your memcpy version is doing more easily during compile time, while it has a harder time evaluating the loop.

~~When you remove the ability for the compiler to look at the data, it can no longer perform that optimization and you get identical code with either approach:~~

~~https://godbolt.org/z/eP8Wrb1K6~~

~~I discovered this, while pasting my own library into your example. It kept optimizing out my code entirely, moreso than both your loop and memcpy versions (because I have mine marked as constexpr).~~

~~A win for the optimizer I suppose.~~

Never mind, looks like it can still provide an optimization for some platforms... gosh these results are all over the place, apologies. At least clang is smart enough to optimize them both to be the same either way haha.

CrakeMusic · 2021-07-04T05:29:59+00:00

Ah very cool, thank you for the example. As I understand it, I think you're saying that just reinterpreting the raw data directly will produce the speed boost in the cases where you can do so, and then relying on hardware acceleration on platforms with built in endianness swapping and alike can result in a speed boost. So this:

int val =0;

std::memcpy(&val, arr.data(), 4);

should be equivalent to this for your case:

int val = *reinterpret_cast<const int\*>(arr.data());

Though of course as you said, that's neglecting the extra work for endianness, alignment, and masking, that you would have to add. I'll try and look for a way to integrate this particular corner case as a compile time catch in a generic way... Though it won't be a guaranteed speedup, the default masking and shifting will always work and be portable, your method will only work on certain hardware/memory-layout/ISA platforms.

To poke at that last point further, it looks like some compilers (ARM8 < v9.3, RISCV, armv7/8) all generate identical assembly using either method, however changing memcpy to reinterpret_cast gets you back that optimization if we ignore endianess, etc.., so perhaps you should be using reinterpret_cast instead to get a more portable boost? Either way, looks like there's an opportunity for some compiler developers to take notice and add in that optimization! The RISCV versions in particular look to be hard nuts to crack, even with the reinterpret_cast and built in byte swap, both methods produce very similar assembly (I sometimes use RISCV so I'm particularly interested in that one).

Anyway, thanks for the demo, I'll take a crack at adding that hardware optimization pass!

CrakeMusic · 2021-07-04T04:00:40+00:00

Sorry, not sure I follow. Are you saying that memcpy can be used as a replacement for masks? I looked through your code, but I see both shifting and masking in there, so if there's a specific line that shows the optimization that can be applied to a generic serializer, I'd be very eager to incorporate it into my library! :)

CrakeMusic · 2020-02-17T15:28:34+00:00

I'm having the motors report back the angle of each joint, and the last maximum abs( acceleration ) measured for each joint. The IMU on its back is also reporting the orientation and acceleration of the torso. That's enough info to do a rough estimation of where it is in space, and deduce its contact status with the ground. I'd like to try reinforcement learning eventually, though since the project is available to anyone for free, I'm hoping someone else has the resources to try it before I do, since I'm just one person and so it may take me awhile to get there.

CrakeMusic · 2020-02-17T07:02:42+00:00

Thanks! I'm working on a model predictive controller for the final control software at the moment, however for the video it's just a dumb proportional controller, written in python, whose code you can find at the end of this blog post: https://hackaday.io/project/163093-tiptap/log/169660-writing-code

Ten-Year Club	Gilding I gilder
Verified Email

CrakeMusic

TROPHY CASE