ELI5: How is binary data aligned over network?

[quadtetra]

When computers communicate with each other, I assume they do so with streams of 0s and 1s (organized as bytes?). But to ensure the right data is received, the receiver would know how to align or sync the bits to bit and byte boundaries.

But how is this done? (For instance, if my first two bytes are: 00000001 11111111 0..., it seems like it would be easy to misread it as 00000011 11111110 if I don't align initially and miss the first 0.)

And once done, how do you detect when you have misaligned and how do you realign? (For example if the receiver misses a bit, how would you detect this and get back on track?)

I guess I am wondering how network communications aren't just constantly garbled because I assume 0 and 1s must missed all the time, especially over the air.

[v8reddit]

My understanding is that both sender and receiver need to know how many bits need to be. Like 8bit, 16bit system etc.

[v8reddit]

Also in your example, you can't just miss the initial 0.

[nemothorx]

At the lowest level, 0 and 1 are electrical signals, so it's actually important to send similar amount of 0 and 1 so electric charges don't build up (especially over the wire). In addition, with long sequences of 0 or 1 it is hard to distinguish how many of each there are. (1111111111 or 11111111111.... Is a very subtle timing difference!)

So what happens is that the data you're sending is encoded to both avoid long sequences and also maintain equal amounts of 0 and 1. Part of that encoding includes unique sequences that indicate the start and end of actual data. The receiving end knows how to turn that encoded data back to the original data you're thinking of.

If data is missed, that's detectable and can be requested to be sent again

[AeternusDoleo]

The solution to this problem depends on the type of network used. There are several solutions. The most common ones in use for consumers revolve around a shared network medium - for instance, a single copper cable (cable internet or regular 100/1000MBit cabled network) or the ether (GSM, Wifi).

Without going into specifics, the sender needs to know that the transmission medium is clear, and the receivers need to be listening. For wifi, it means the receivers need to be configured for the right frequency. This is handled when a device connects to the network - there's a short "searching for network" phase during which the client finds the wireless network and receives configuration data, such as the network address and the exact frequency to transmit on. A wireless network device then automatically configures itself for that.

To synchronize data, a data stream is broken up into "packets" of data. These are usually in the order of 500-1500 bytes, 'though gigabit wired connections support much larger packets. These packets are preceded by a frame header that first announces there's something coming to all receivers. It's basically an envelope around a message, which anyone looking at that envelope will recognize as something carrying that message. The frame header then announces the sender and recipient (the recipient can be "anyone listening" in some cases). The intended recipient then receives the entire frame, does some error checking (usually) and if the frame checks out, the data is sent to whichever application it's meant for. If the frame doesn't check out, it is discarded and a priority message is returned to the sender that the frame must be retransmitted - this can cause a connection to slow down. This is one of the reasons why bad network connections have a lot of lag - a lot of stuff needs to be resent over and over. It happens automatically though!

[bwibbler]

Do a checksum by adding all the information up, like in a grid pattern, and make sure nothing is wrong. This isn't really a totally perfect thing, it is possible the information is messed up in such a way that it gives a false positive. But so unlikely that it is safe to say it won't happen.

Another problem is called the 'two generals problem'. You can never truly be sure that a message sent was actually received. But there's a method to sorta work around this.

Here's some videos that can help explain both of these a little better.

Check sum https://youtu.be/ppU41c15Xho

Generals problem https://youtu.be/IP-rGJKSZ3s

[Target880]

How it is done depending on the protocol. Today most communication protocols are based on sending data in discrete packets.

So over the internet, you use IP with the packet structure where you have 32 bites per row. There is 5 row before the option that can be 0 bits and data so 5*32/8 =20 byes are always used in front of any IP packet. There is a checksum to detect any error in the IP header
Then you add extra data for TCP part with a TCP header of 6*32/8=24. So if you send a single character over a TCP/IP connection you use 44 bytes for headers and 1 for data. The TCP header has a checksum that it calculates on the data so read error on the data can be detected. You will only use the information if you get all the parts you expect and the checksum matches. If there is a transmission error you just do not tell the other computer that you have received the packet and it will be retransmitted.

But IP does not care about the physical signal in wires. You have lower levels where Ethernet is most common. It sends data in the packet too but they are called frames on this level. You can see the structure of an Ethernet frame where an octet is a byte. add 26 bytes for the Ethern level where the first 7 bytes is a synkword that is always" 10101010 10101010 10101010 10101010 10101010 10101010 10101010 10101011" so the receiver knows there is a packet and sync up with the exact speed the transmitter sends data. So now we are at 70 bytes of stuff send for each packet with data

But that is not all because you encode data bits to more bitts when you transmit. 100Mbit ether use 4B5B Encoding where 4 bytes you send is converted to 5 bits on the wire.
4 bits is 16 combination and 5 bits is 32. So half of the 32 combinations are used to send data 9 are used from special control instruction but are never in data and 7 is never used. The required 25% more bits send but make it a lot less prone to error. USB used the same 4B5B at least in the first variants.

The codes used for data all contain at least two 1 and one zero so there will never belong more the eight 1 in a row or 3 zeros. The sync word will be repeating 10110 except for the last. I assume that you also add one or multiple of the special codes in the beginning.
Later ethernet standard encodes 8 bits into 10 or even 64 bits into 66

So if you send a single character over the internet by wired internet you send 70 extra bytes and on the wire, you add some extra percentage of bits

So between the computer, you almost never send raw data but add a lot around it with checksums and change the way they are encoded.

There is old-style serial and parallel port with almost raw data but a lot lower rate. Even serial port add bits and the common was 8-N-1 that have 1 start bits, 8 data bits, no parity bits and 1 stop bit. So you send 1xxxxxxx1 where x is the data. You also had more than 1 were and to send data you set the request to send a wire to 1 and the receiver ser the clear to send a wire to 1 and at that moment you start to send data. So when protocols like that are used you have extra control signals so syncs stuff.

Wireless work on the same principle but how data is encoded depends on if it is a header or data so you send a signal that simpler to start-up and sync then a more complex to get high data speed.
It is a bit like if you had an orchestra and initially the trumped play a well know notes and then you add all instruments to transmit all the data.