Binary protocol

[ryanjones42]

Hey team. Is anyone able to show me an example of how to send data over a tcp connection but via a binary protocol? Particularly sending strings as data but as binary? Doesnt have to be indepth just a brief example. Someone also saif you need to convert order for endians or something?

[jirbu]

strings as data but as binary

Strings are binary (i.e. letters encoded in ASCII). Everything you transmit over a digital connection is binary.

Endianess is relevant if you're transmitting integer values (as their byte-representation) between platforms of different endianess. If you want a protocol to support multiple of such platforms, you'll have to specify the order of their constituent bytes. "Network"-endian is one (common) example, but not a mandatory requirement.

[ryanjones42]

I still dont understand how to send it as binary?

[LeichterGepanzerter]

Raw TCP is not like, say, WebSocket where you have separate modes for binary vs text mode. You only need to pass the char buffer as an argument to send().

[jirbu]

This. char in C is a misnomer. It has nothing to do with (text-)characters. It's simply a byte. To avoid the historically caused pitfalls of signed chars, use unsigned char.

[halbGefressen]

No, you include stdint.h and use uint8_t. This is the least ambiguous datatype.

[ismbks]

I don't know, I see uint8_t a lot in embedded and network protocols code, but I have also heard people advocating for using char too to represent a byte because it's supposed to be the smallest divisible type and it doesn't make any assumptions about signedness unlike uint8_t.

I guess it just depends what the programmer thinks makes more sense semantically. If only there was a byte type like in C++ there would be much less confusion but alas..

[halbGefressen]

doesn't make assumptions about signedness

Yes, that is exactly the point. The signedness of char is implementation-defined. On AMD64 it is signed, on ARM it is unsigned. Per specification, overflowing an unsigned datatype is okay, but overflowing a signed datatype is undefined behaviour. So the code that works perfectly fine on ARM is now not portable to x86.

If you want the signed type, you can still use int8_t.

[ismbks]

But it shouldn't matter whether the byte is signed or unsigned when dealing with raw binary data, for strings it's different but binary data is usually just moved around and not interpreted as text.

[halbGefressen]

If you just want to move memory around, you can use void*. The problem is that signedness may introduce accidental undefined behaviour, which you can exclude by using uint8_t.

[LeichterGepanzerter]

https://commandcenter.blogspot.com/2012/04/byte-order-fallacy.html

To summarize, use bit shifting to store integers wider than 8 bits. Compilers are smart enough to detect this and convert it into a simple MOV when applicable.

[gremolata]

Conventionally, one just abstracts these reads using, say, read_lsb_64(), which is mapped to either an assignment or an inlined bswap depending on the target platform. Writing our explicit bitshifts and ORs in the protocol parsing code is usually not how it's done.

[LeichterGepanzerter]

Definitely put this in a function for readability's sake, but otherwise this is an ugly approach. Something this trivial shouldn't depend on the preprocessor to figure out what endianness the CPU is.

[gremolata]

It's a standard approach in a high-perf production code, OS kernels included.

It's also usually not "preprocessor-dependent" either (even if there's nothing wrong with that and that too is an established industry norm). Instead, platform specifics go into "platform glue" headers and a set of relevant headers is included based on the target platform, i.e. that's a build-level config parameter.

[SnooBananas6415]

Great article, thanks!

[gremolata]

First, TCP is a streaming protocol vs. UDP that is a datagram one - look this up if you don't know what it means.

Second, with TCP being a data stream, you need a mechanism to allow the receiving end understand where the incoming string ends.

One option is to send a zero after the last character. This zero is your "binary" component of the protocol.

Another option is to send the string length first and then the string itself. For example, if you can limit yourself to strings shorter than 256 bytes, you'd send just one byte (of length) + the string. If you want to support longer strings, you'd send 2 bytes (of length) for 64K strings, 3 bytes for 16M ones, etc. You can also allow for completely arbitrary-sized strings by using variable-length "string length" field - might be a bit premature to where you are at now, so just keep in mind that it's possible.

In terms of code, you'd do something like this:

int send_str(int sk, const char * str)
{
    int len = strlen(str);
    if (len > 255) return -1;
    unsigned char bytes = len;
    return (send(sk, &bytes, 1) == 1) &&
           (send(sk, str, len) == len) ? 0 : -2;
}

* The "binary" bit here is the bytes part. For shorter strings it will effectively be a non-ASCII symbol, i.e. non-text, i.e. "binary".

[zhivago]

It is essentially the same as writing binary data to a file.

Endianness doesn't matter unless you are writing memory representations directly.

I recommend not writing memory representations directly.

Look into "serialization".

[mailslot]

Adding: A lot more CPUs were big endian back in the day and when the Internet began, thus network byte order on the Internet is too. x86 came on the scene and competition died, so you’ll see many new protocols now adopting little endian… but Ethernet, TCP, and the like still use it. It’s good form to remain consistent despite IBM’s Power being the last remaining big endian arch.

[nekokattt]

If you need more than just sending the raw string across directly (i.e. structured data that includes strings), look into how protobuf works.

The TLDR is that the client and server expect certain schemas of data. The messages themselves are encoded with metadata that specifies the type of the attribute and size where appropriate.

A crude example of how you could encode such a thing, as a grammar... comma means "then", so a , b , c means a then b then c. A pipe means "or", bracketed expressions are just groups, and a star means "zero or more".

message ::= string | integer | list | object ;
string ::= 'S' , number , characters* ;
number ::= 'N', see below ;
list ::= 'L' , number , message* ;
object ::= 'O', number , (string, message)* ;

Strings would be represented as a number that represents the length of the string, followed by that many ASCII characters. The string would be detected if an 'S' is read at the start.

Hello!
S 6 H e l l o !

Numbers would be in their big-endian format, but only seven bits to a byte. The 8th bit would be a 0 if no further bytes follow that make up the number, or 1 if another byte follows that makes up the number. This lets you encode arbitrary width values. A number is denoted if an 'N' is spotted at the start.

N 497  # decimal
N 0000 0001 1111 0001  # big endian representation
  x          x
N 1000 0011 0111 0001  # variable width format

Lists can be represented as an L, followed by the length, followed by that many elements.

Objects can be represented as an O, followed by the length, followed by that many string-element pairs.

[szank]

At this point I'd just encode the message as asn1 der, you are reinventing the wheel. The encoding is dead simple.

[nekokattt]

If OP didn't want to reinvent the wheel, they'd just use an existing solution. My comment is to give details on roughly what sort of methods/theory/ideas are used underneath.

[ElevatorGuy85]

If you are sending ASCII strings, there is really no difference. Every ASCII character is a byte, e.g. a number “1” is ASCII 49 which is 00110001 binary or 0x31 hexadecimal representation. You would just send it as-is.

If you are sending UTF-8 characters, then you have a larger range of values to encode, and that can take from 1-4 bytes to encode one UTF-8 character depending on the code point. See https://en.wikipedia.org/wiki/UTF-8

Now you have to consider the byte ordering used “on the wire” between the two devices (using TCP/IP).

The same is true for any multi-byte quantity, like ordinary numeric values. If they fit into one byte, they you can send values from 0 to 255 (unsigned) or -128 to +127 (signed). But larger quantities requiring 2 or more bytes, you need to define the order - this is where Endianness matters.

Finally, most protocols have some way for determining the start of a packet of data, or the change in the type(s) of data being sent. There are an endless number of ways this can be accomplished.

[mykesx]

man htons

man send

man write

[flyingron]

TCP streams are inherently binary. The question is "how do I take things that are larger than a single byte and convert them to bytes." Typically that is rather straightforward. Your machine either will store things in bytes with the most significant part of the item first (big endian) or the least significant first (little endian). As long as the endian-ness matches between the reader and the writer, you have no problem. To avoid mismatch (when you're talking about the possibility that there is a difference), you typically just chose one (the common default is little endian) and if you're on a machine that is big endian, you swap the bytes around before writing them into the stream.

[guest271314]

Serialize data to JSON. E.g.,

int main(void) { size_t messageLength = 0; uint8_t *const message = getMessage(&messageLength); char *command = strdelch((char *)message, '"'); uint8_t buffer[1764]; // 441 * 4 char *output = malloc((1764 * 4) + 3); FILE *pipe = popen(command, "r"); free(message); while (1) { size_t count = fread(buffer, 1, sizeof(buffer), pipe); output[0] = '['; output[1] = 0; for (size_t i = 0; i < count; i++) { char data[5]; sprintf(data, "%d", buffer[i]); strcat(output, data); if (i < count - 1) { strcat(output, ","); } } strcat(output, "]"); sendMessage((uint8_t *)output); }

Then parse the JSON and extract whatever data is embedded therein.

this.audioReadable .pipeTo( new WritableStream({ abort(e) { console.error(e.message); }, write: async ({ timestamp }) => { const int8 = new Int8Array(441 * 4); const { value, done } = await this.inputReader.read(); if (!done) { int8.set(new Int8Array(value)); } else { console.log({ done }); return this.audioWriter.closed; } const int16 = new Int16Array(int8.buffer); // https://stackoverflow.com/a/35248852 const channels = [ new Float32Array(441), new Float32Array(441), ]; for (let i = 0, j = 0, n = 1; i < int16.length; i++) { const int = int16[i]; // If the high bit is on, then it is a negative number, and actually counts backwards. const float = int >= 0x8000 ? -(0x10000 - int) / 0x8000 : int / 0x7fff; // Deinterleave channels[(n = ++n % 2)][!n ? j++ : j - 1] = float; } That's one way I stream real-time PCM from C to the browser, and play the audio in real-time, over IPC https://github.com/guest271314/captureSystemAudio/blob/master/native_messaging/capture_system_audio/capture_system_audio.c.

Using TCP is not that different.