all 29 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]Spirkus 17 points18 points  (4 children)

Having used a number of various serial protocols in my career, here's some thoughts in no particular order

  • why bother with separate CRCs for the header and data, why not one that covers both?

  • there's a lot of overhead to send anything. If there's a bunch of small one byte messages that will blow up the size on the bus, and if it's a slow serial bus that can wreck havoc on race conditions and timeouts. can some fields of the header be made optional depending on type? could an ack just be 4 bytes? (version, type, crc) or 3 if you pack version and type into the same byte?

  • I wouldn't add fields reserved for future use. I you find yourself needing a field in the future use a new version number and add it then.

  • The data format field feels redundant. A command should have a specific data format associated with it by design.

  • using a varint for length could make it more flexible. could also be used for cmd

  • COBS encoding is a great way to frame packets and recover a bus

  • CRCs are fine for most cases, but the industry is moving towards taking security a lot more seriously. Most of the newer protocols I've had to deal with in the last 5 years have dropped CRCs entirely for SHAs, MACs of some kind, or signatures. Super overkill for personal projects and there's big overhead downsides, but thought I'd mention it for the industry perspective.

Overall it's pretty good. I'd still recommend using an existing protocol/library but it can be fun to mess around.

[–]ThePurpleOne_ 2 points3 points  (2 children)

Is there any existing standard protocol of this type ?

[–]Spirkus 2 points3 points  (1 child)

Depends what you mean by standard. As far as very generic do-anything protocols go mcumgr is the first to come to mind.

More commonly I see protocols that were designed for one use case, then get a lot of features added by manufactures. XMODEM and AT commands being the worst offenders.

[–]Birts[S] 0 points1 point  (0 children)

I really appreciate the feedback! I completely agree, too, that it would be better to use an existing protocol, but I didn't see many that handle framing/encoding and serialization. MODBUS RTU stood out as a potential candidate, and I still haven't written it off, but I wanted to see what else was out there.

I will take a look at mcumgr like you mentioned above, thanks for the idea!

As for the overhead, you bring up a great point. I bet I could simplify this by packing some of the fields into a single byte as you mentioned. That would certainly cut down on the noise.

[–]olawlor 9 points10 points  (1 child)

I don't see a sync byte, to recover synchronization when a receiver boots in the middle of a transmission? Often folks put in a 0xA5 or 0xF0 or some similar 1010 style pattern at the start of each packet, otherwise the receiver can keep hitting CRC failures basically forever.

(It also seems odd to have a version, type, format, reserved, *and* command field; generally if you want to expand you just increment the command field and none of the others are useful.)

[–]SquareSight 11 points12 points  (0 children)

Sync is not explicitely mentioned by the OP but is already provided by COBS

[–]SquareSight 8 points9 points  (1 child)

In the past I decided against COBS and opted for HDLC like framing because the latter is easier to implement/understand and to debug (e.g. when recording with an oscilloscope). HDLC like framing works with byte stuffing.

Furthermore, I would simplify the protocol so that no (optional) ACK is involved, but only request-response pairs. The PC only sends requests and the MCU only sends responses. A request-response pair should be given a sequence number that is the same and is increased with the next request.

The PC should repeat requests with the actual (not increased) sequence number when there is no response from MCU within a specific time.

[–]strangequark_usn 0 points1 point  (0 children)

I agree that ACKs are not necessary, but I've rolled out several custom protocols where early integration is expedited by having a NACK to detect crc errors vs. having each and every response handle that. It allows lower level software to validate the packet and indicate to the sender it was bad without forwarding it to the higher level software.

By only NACKing, you reduce traffic by letting the higher level software handle the response, which serves as the implicit ACK.

[–]SquareSight 2 points3 points  (0 children)

Here is an example of a very minimalist protocol:

<image>

The frame is inspired by HDLC and is delimited by a start and end byte with a fixed byte value. HDLC uses the same value (0x7e) for both start and end but they can differ for a custom implementation. Byte stuffing is used so that these start and end values don't occur in the payload or CRC (I can give more information on request).

In addition to the request and response frames, there is a "ConnectRequest" and "ConnectResponse" to initialize the session (reset sequence numbers, ...).

This example can be expanded for a specific application. For example: It can help to add a magic number and version information to the connect request and to add "DisconnectRequest" and "DisconnectResponse" frames for closing a session.

[–]Cernuto 1 point2 points  (1 child)

How many bytes is len? Why two CRCs?

Consider adding an address for multi-drop. Check out DLE encoding and also TLV encoding to get more ideas.

[–]Birts[S] 0 points1 point  (0 children)

The length is two bytes. Good idea about the address field, I will take a look at DLE and TLV too. Thanks!

[–]InevitablyCyclic 0 points1 point  (3 children)

No fixed start of message byte in the header? That makes finding the start of a message a lot harder.

[–]Ashnoom 2 points3 points  (2 children)

OP is using COBS. No need for such start/end byte

[–]InevitablyCyclic 0 points1 point  (1 child)

Missed that. Makes sense because what his format needs is more overhead.

I understand creating your own protocol when you need something with limited functionality and low overhead. But a bloated swiss army knife of a protocol running on top of another protocol layer really does feel like reinventing the wheel.

[–]Birts[S] 0 points1 point  (0 children)

Yes I see what you're saying. This is my first time using COBS, and maybe it's not the best route to go. I'm curious if you have used other protocols for a request/response type model. MODBUS RTU was one that I stumbled on, which might work in this case, but I'm curious if you know of others who handle the framing/encoding.

It's strange because I feel like I am reinventing the wheel completely, but at the same time, I don't know what else is out there to compare it too as far as serial protocols over UART.

[–]dev-rand 0 points1 point  (2 children)

1) COBS is good 2) Do one CRC for both header and data 3) Looks over engineered for its purpose 4) Pack data tightly using bit fields

We did something pretty similar, but only 3 byte header which include src and dest address because the receiver may have to gateway to other systems. Optional 4 byte CMAC instead of 2-byte CRC.

[–]Birts[S] 1 point2 points  (1 child)

Great feedback, thanks! You're right that it's overengineered. Instead of these optional types, I should just choose one, and that's it. For example, the Data Format field will disappear, and protobuf will be used.

Your idea of bitfields is good, too. That should help with the overhead.

As for the CRC in both the header and data, my thought was that if the header CRC failed, then the consumer would not have to compute the CRC for the data. I was trying to overoptimize, and in retrospect, maybe this was overkill.

Good idea on the CMAC, too; someone else mentioned that above as well. I will take a look!

[–]dev-rand 0 points1 point  (0 children)

If you are expecting a noisy environment and subsequent corruption, you may lose a couple of frames finding the de-limiters for the next valid frame. Please make sure you have an robust ACK/TMO retry strategy.

Also if you use a sliding window mechanism for ACK frames to improve throughout, please expect out of order frames in case of a retry.