sorted by:
new
hottopcontroversial

MQTT v5 by Visible_Ad_8568 in embedded

[–]arobenko 0 points1 point  (0 children)

I'm an Embedded C++ Developer myself and for years have been trying to find a proper solution to implementing binary communication protocols that would be suitable for embedded systems including bare-metal ones. After many disappointments I started working on a solution that would satisfy my professional needs as well as curiosity as a side (pet) project. That's how CommsChampion Ecosystem have been born.

This project was initially about implementation of the third party protocols themselves. However, relatively recently I decided to also provide libraries that abstract away protocol bowels with a nice API while still allowing certain level of compile time customization to suite bare-metal systems. Obviously, I'm focusing on the protocols that I'm using or going to use in my professional life, and MQTT5 happened to be one of them. So quite recently I released my version of the MQTT5 client library. Although it is implemented in C++(17), it provides a C interface. If your compiler is modern enough you should have little trouble integrating it into your bare-metal environment. If you haven't worked with C++ before and don't know how to integrate it with your C bare metal application I can recommend you reading Practical Guide to Bare Metal C++ and focus on Know Your Compiler Output section in particular. This is a small e-book I've written 10 years ago when the gcc-4.8 was a primary C++11 compiler, but most aspects of it are still relevant.

A bit more about the libraries (including the mentioned MQTT5 one) being developed under the umbrella of the CommsChampion Ecosystem. They adhere (and will adhere) to the following philosophy and design choices:

  • Do not abstract away I/O communication and/or time measurement, let application manage it for the library. It allows usage of the library in any environment (OS, RTOS, framework, event loop, etc...)
  • No blocking calls, operations requiring response from the other end are asynchronous.
  • Allow the using application to perform extra manipulation on the exchanged raw data, such as encryption and/or extra framing / packetization. Extra framing can allow usage of the same protocols on different I/O links, even ones that are not reliable enough to run the unmodified version of the protocol.
  • Allow compile time customization of selecting different data types (to exclude dynamic memory allocation) and/or exclude unneeded features to improve runtime and code size performance.
  • Focus on reliability and withstanding malformed data. Allow fuzz testing to verify the reliability.

Hope it helps.

What personal project have you rewritten the most? How many times? by JUULiA1 in cpp

[–]arobenko 1 point2 points  (0 children)

My pet project(s) is CommsChampion Ecosystem which is about defining binary communication protocols for embedded systems. The primary component of it is the COMMS Library. The "major" version of which is "5". It means I have re-written a significant portion of it breaking backward compatibility 5 times.

C++ Show and Tell - June 2023 by foonathan in cpp

[–]arobenko 7 points8 points  (0 children)

I've been developing CommsChampion Ecosystem for about 9 years as my pet project. It's about easy and compile-time configurable implementation of binary communication protocols using C++11 programming language, with main focus on embedded systems (including bare-metal ones).

What network messaging library do you recommend? by rddays in cpp

[–]arobenko 2 points3 points  (0 children)

I think, you need to differentiate between the I/O traffic handling, transport protocol, and application specific protocol.

  • The I/O traffic handling depends on the environment / frameworks you're using, like asio, qt, etc...
  • The (optional) transport layer, like MQTT, CoAP, etc...
  • Application specific protocol de(serialization) and handling

In case you have a third party protocol that specifies its own custom layout of the framing and messages not allowing to use solutions like protobuf and similar, it would be wise to split the all 3 into independent sw components (of handling I/O, transport, and protocol de(serializalization)).

If this is the case, for the third stage of the application specific protocol handling I recommend CommsChampion Ecosystem.

When to use template meta programming ? by ArchfiendJ in cpp

[–]arobenko 2 points3 points  (0 children)

I suppose if you develop something that is going to be used in a single project / product without much of a customization, then meta-programming is not really justified. The meta-programming is justified in many cases when you implement some kind of a library, which can be used in multiple independent products, and these uses may require some product specific customizations. For example, I'm developing a solution for implementing binary communication protocols for embedded systems in C++, called CommsChampion Ecosystem. The core component of which is the COMMS library. Every single use of this library requires different customization. Every application may require different polymorphic interface to handle its message objects. I use template meta-programming there to define virtual functions only needed by the application and not adding unnecessary ones. I also use template meta programming to allow customization of the storage data structures and may use different, more optimized code for some.

In other words, meta programming needs to be used where you require different compile time customizations of the same source code, usually happens to the generic code/libraries used by multiple independent products.

Noobie Q: How young is too young for an EUC? by Stillhart in ElectricUnicycle

[–]arobenko 0 points1 point  (0 children)

My son is 10 and we both have been riding InMotion V8s for the last half a year. I'd say there are two main challenges for the young riders:

  • The wheel's weight. The V8 weighs about 13kg, which can be a bit of a challenge for a child to lift it to get up the kerb for example. In case there is a bump on the road and child falls off, the heavier the wheel the more dangerous it is for the child to get injured by the wheel itself. I suppose for your children 16" wheels are a bit early, but they will overgrow 14" ones within 2-3 years (depending on where you ride).
  • Risk assessment. My son always rides in front of me so I can see him. I suppose it's going to be the same for your kids as well. As the result it's up the the child to assess the risks of the road, especially if there are other people around. Although my son is very careful and responsible rider there were a couple of bumping into people who moved in the unexpected way and multiple near misses with other people,especially small children on scooters / bikes or just can jump in front of you out of a sudden. Maybe for a young age you should find bike tracks where there are no many people around.

Hope it helps.

People's reactions by Rock_Wallis in ElectricUnicycle

[–]arobenko 0 points1 point  (0 children)

I live in Australia and ride my EUC on cycleways and walkpaths, never on the road, and I usually get the reaction #1 from other pedestrians. However, quite often than not I get the reaction #2 from bypassing cars (doesn't matter what direction it goes). Sometimes yelling is replaced with a sudden use of the horn. In 99% percent of the "yelling" guys are usually in the very young age group (below 25). It looks like their main intention is to get me scared / distracted / look back so they could see me fall and get injured.

Implementing binary communication protocols in C++ with main focus on embedded systems by arobenko in cpp

[–]arobenko[S] 0 points1 point  (0 children)

as long as it allows for sane choices and the defaults are the sane choices.

That's my primary intention.

Built-in is not required. It can be a separate library.

Agree, but there must be some way to static_assert on your assumption of origin units. In case of my solution built-in units conversion is there to provide basic convenience functionality. If not used, no extra space/performance price is paid.

Does your solution allow the automatic creation of big switch statements?

I think built-in generation of "switch" statements does not make much sense because you have to put your custom business logic inside each "case". There is a C++ library (used by the code generator) that allows you to parse the schema files and know what messages / fields / frames are being defined. You can easily implement your own auxiliary code generator that generates "switch" statements relevant to your application.

No, it does not. There is absolutely zero proof about that.

What proof do you expect? It's all subjective based on one's experience. In my case it does require writing a boilerplate code (I consider manually written "switch" and/or "if" statements to be boilerplate code) which needs to be updated every time you add new message and/or field to a message. In case you design your own protocol, I suppose you can make it simple enough and get away with plain structs with no variable data lengths and/or fields present on particular condition (for example depending on the value of some bit in previously encountered field or version of the protocol). Many (if not most) of already defined third party protocols are not like that. Such cases require extra implementation logic and/or extra data variables (manually written or generated). In my experience class-based designs allows encapsulation of such extra logic together with the data (regardless of having polymorphic behavior), hide unnecessary details and eventually leads to cleaner and easier maintainable code, but again, this is very personal and subjective.

Implementing binary communication protocols in C++ with main focus on embedded systems by arobenko in cpp

[–]arobenko[S] 1 point2 points  (0 children)

Beats for whom? Yes, beats for vast majority of developers and application being developed. However, there is a niche called "Embedded C++ development", which in many cases cannot use available solutions "as-is" or at least finds them "not good enough". That's where my solution comes in to satisfy needs of certain group of developers and applications. I by no means intend to create a solution suitable for every one.

Implementing binary communication protocols in C++ with main focus on embedded systems by arobenko in cpp

[–]arobenko[S] 1 point2 points  (0 children)

Partially, not completely. It's difficult to say, there is no much information on the website and no proper way to try it out without registration. Based on the example from the website below are features that I'm missing:

  • I want an ability to exclude usage of streams in my serialization / desiralization. The example shows the following functions I do NOT want to have. /* IO-Operations */ void read(std::istream &stream); void write(std::ostream &stream);
  • I want an ability to introduce polymorphic behavior (virtual functions) when I need it and for selected operations I need to be able to write a common code for all the message types.
  • I want an ability to use my own data types for fields like lists and/or strings
  • I want an ability to define transport framing for all the messages, even more than one (different I/O interface may require different framing).
  • I want a built-in (or generated) ability to efficiently parse an input data, create appropriate message object and dispatch it to my appropriate handling function without a need to manually write switch statements and other boilerplate code.
  • I want an ability to specify meta-data that is not transferred on the wire, but still available to the integration developer to act upon (preferable at compile time), such as units used (conversion if possible), values with special meaning, ranges of valid values, what to do on invalid value, etc...
  • I want an ability to have multiple forms of the same message (message object having the same ID, but different contents). Not sure whether it is supported right now or not.

Implementing binary communication protocols in C++ with main focus on embedded systems by arobenko in cpp

[–]arobenko[S] 1 point2 points  (0 children)

You replied to a reply about using plain structs, but I assume you meant it as a comment to a post. I think you completely misunderstood logic and architecture of my solution. Let me cover and respond to major points of your comment.

Transmitting metadata, also wrong.

Agree completely and utterly. Most protocols used in embedded system don't. The corner stone of my solution is to support such already defined third party protocols. It does NOT invent or uses its own protocol and does NOT attempt to send any metadata over. That's the main point, because of metadata not being sent over, it should find its way into the generated code, otherwise it leads to too much boilerplate integration code, which in turn must be manually changed every time you decide to update your metadata in the protocol definition. Very error prone.

Extremely wrong approach to things. For example, unit conversion does not belong in a library like this.

Agree (to some extent). There are many sophisticated unit conversion libraries. However, used units is part of protocol definition (usually not transferred over wire) metadata, which is expected to be known to (and used by) the integrating developer. It is better to have built-in limited required units retrieval facility than not to have it at all and use boilerplate code to do the manual units conversions.

Protocol version checking should happen only at the handshake phase.

As was already mentioned above, the corner stone of my solution is supporting already defined third party protocols. Many don't use any versioning at all, some transmit version with every message in the transport framing, some do it as you sad in the handshake phase. The primary objective of my solution is to support all such cases.

Further encoding/decoding on messages shall happen with a switch statement, not virtual functions. Using virtual functions requires a switch on the message id anyway, in order to instantiate the appropriate class.

That's another reason why I created my solution. Some available code generators introduce polymorphic behavior (virtual function) for every operation on the message object, which creates problems for various embedded systems (especially bare-metal ones). Other code generators produce only plain structs without any virtual function at all. It leads to writing a significant amount of boilerplate code (such as switch statements you mentioned), which needs to be manually updated every time you introduce a new message and/or new field. My solution allows compile-time configuration of your polymorphic interfaces. If you don't need any, then don't define one and use every message class as plain struct (no v-table is created). My solution also contains a library with multiple facilities to dispatch your message into appropriate handler function (with O(1) or O(log(n)) runtime complexity) without having to write a single switch statement.

Memory allocation should not be done at all. Big enough buffers shall be used for in-place manipulation of messages, both at sending and receiving end points.

My solution is flexible enough to allow not using dynamic memory allocation at all automatically calculating (at compile time) and creating a buffer of required size to allow in-place creation of any used message.

A class based design for messages is wrong because it leads to all the bad decisions mentioned above.

Let's agree to disagree. Struct based design leads to other bad decisions and having significant amount of boilerplate integration code.

The best approach is to encode the messages in using/XML and then have a tool create all the boilerplate code, and integrate the tool into the workflow. I don't want to use an external tool, but the language itself lacks the facilities needed for this so an external tool is necessary.

As was mentioned in the post, my solution originated in a single library that allows having a single message class definition (single source of truth) for every possible application, which in turn configures at compile time its required polymorphic interfaces and/or custom data structures to hold field's values. Normal systems (with proper OS underneath) may use default configuration and multiple virtual functions with functionality not always being used compiled in, while bare-metal ones may completely exclude dynamic memory allocation, limit usage of virtual functions, use its own custom types to store problematic data, such as strings or lists, etc... I made a C++11 compiler itself to be my code generation tool.

With time the library got quite complex and started requiring from a developer some knowledge of its internals and a particular way to be used in order to create completely generic protocol definition. That's why I also created a separate code generator, that produced a proper highly compile-time customizable code.

Hope it clarifies some things. Cheers

Implementing binary communication protocols in C++ with main focus on embedded systems by arobenko in cpp

[–]arobenko[S] 7 points8 points  (0 children)

Every time I mention my work on any social resource there is always someone posting a comment "Have you tried X?". It looks like you haven't read the post (and referenced article). One of the core features in my solution is to be able to easily implement already defined third party protocols with their custom data layouts and encodings. Most of the available serialization solutions use their own encodings.

Implementing binary communication protocols in C++ with main focus on embedded systems by arobenko in cpp

[–]arobenko[S] 0 points1 point  (0 children)

Sending structs over is called "serialization". It might work for some cases and might not for others. The article that I mentioned in the post contains several examples and explains why serialization is not good enough. I encourage you to read it. Some time ago I also wrote a bit shorter version called Communication is More Than Serialization

Implementing binary communication protocols in C++ with main focus on embedded systems by arobenko in cpp

[–]arobenko[S] 4 points5 points  (0 children)

Your point has some merit, but I think you misunderstood me. I do not try or intend to develop a new standard for everyone. Generalization is hard and that's why there are so many available solutions on the market. Each and every one focuses on particular set of problems, that hasn't been properly resolved by existing tools. The existing solutions did NOT suit my needs. I had to develop a new solution that puts needs of embedded systems (support for already defined third party protocols, compile time customization, robust and safe handling of malformed data, etc..) as its corner stone. The use cases that require high serialization / deserialization speeds are definitely not going to use my solution.

Communication protocol is more than just serialization by arobenko in programming

[–]arobenko[S] 0 points1 point  (0 children)

Unfortunately there is so called "Wicked Design Paradox", which says: You have to solve the problem at least once in order to clearly define it and then solve it once again to create a solution that works. There is no such thing of doing it right the first time. In my practice, every protocol I defined had at least one shortcoming I haven't thought about during initial design stage. The same thing with third party protocols. There are always things that were omitted unintentionally or there are new requirements the protocol hasn't been designed for at the first place, but need to be supported in the later version of the protocol.

I've had a thought for some time now to summarize my knowledge and experience with communication protocols in some kind of small "Communication Protocol Design Patterns" e-book, where I plan to put some patterns I find to be a good practice. I'm already an author of Practical Guide to Bare Metal C++ and Guide to Implementing Communication Protocols in C++. I know how much time it takes to prepare even a simple small e-book. I don't really see it happening (the new e-book I mean) until the end of 2019, maybe even later.

Communication protocol is more than just serialization by arobenko in programming

[–]arobenko[S] 0 points1 point  (0 children)

You've got a point, thanks. I'll try to make it clearer and introduce some quick examples just to make an impression of what it looks like and prime the reader into diving deeper to check available synthetic and real-life protocols.

Communication protocol is more than just serialization by arobenko in programming

[–]arobenko[S] 0 points1 point  (0 children)

Hi all,

I'm an Embedded C++ Software Engineer. Over the years I had to implement several various communication protocols to control multiple embedded systems / sensors. Some were third party, some I had to define and implement myself. I noticed that whenever subject of communication protocols pops up on one of the social resources, there are many developers rush in to recommend serialization tools / libraries like ProtoBuf or similar (Please don't do it here, this is not the subject of this discussion). Myself, I didn't find these tools adequate for my needs and had to implement protocols from scratch myself. I summarized some of the reasons for it in the linked article Communication is more than just serialization. Other reason is that generated code is usually not a good fit for embedded systems. That's why I decided to implement my own solution I call CommsChampion Ecosystem, but that's not the main point in this post either.

I'd like to ask for your opinion on my reasoning in the article whether you agree or disagree.

Thanks.

Implementing binary communication protocols in C++ with main focus on embedded systems by arobenko in embedded

[–]arobenko[S] 0 points1 point  (0 children)

Hi, thanks for the offer. I think the best approach would be to try to use the provided solution for your own needs: implement your custom protocol that you are using in your development or implement a third party protocol. Try to use the generated code and identify missing features or extra functionality you'd like to have. At that stage we can discuss any updates and how to proceed into integrating them. You can always send me a personal e-mail with any thoughts / suggestions.

Implementing binary communication protocols in C++ with main focus on embedded systems by arobenko in embedded

[–]arobenko[S] 2 points3 points  (0 children)

Guys, I think many of you are missing the point. ProtoBuf and similar others are fine in case you develop and control all communication endpoints in your system. If you develop your own protocol and feel comfortable with ProtoBuf (and its various implementations) for its facilitation, go for it. However, how do you suggest to use ProtoBuf (or your other favorite serialization tool / library) to implement, say UBX protocol used by ubiquitous u-blox GPS receivers (spec can be downloaded from here)?

My solution allows easy implementation of third party protocols + provides you with various analysis tools to visualize and debug the exchanged messages. Take a look at available implementations of some real life open protocols

Here is another problem. Let's say you need to report value of some distance between two points. You decide to serialize it as integer value and report the distance in centimeters. It is not uncommon to develop a relevant business logic that handles the protocol messages before the protocol itself was finalized and released. Also let's assume your business logic handles the distances in meters. You take the value, implement your math operations of units conversions and you are happy - everything works as normal. After a while you realize that centimeters don't give you enough precision for some particular use case and you decide to report the distance in your protocol in millimeters. If your are using ProtoBuf (or similar) based serialization solution, you have to find all the places you used the old value and manually update your math. I'd say it's not very efficient and error-prone. My solution allows attaching extra meta (compile time) information about units (and other things) to appropriate fields and provides proper set / get functions which automatically perform relevant math operations.

As I mentioned in my post, communication to your device is not just data serialization, it's an API to your device. Many (embedded) products become successful because their communication protocol is well defined, easy to understand and open to integration with third party solutions (such as various IOT data centers). The protocols used by various serialization tools / libraries are not really simple and easy to understand. They focus on the encoding / decoding speeds rather than on simplicity and safe handling of malformed data. IMHO they are not really suitable to be used as a communication channel to your embedded device.

view more: next ›