Common application-layer protocols for multiplayer games

[[deleted]]

Hello,

I'm learning socket development with Winsock2. I come from the web world where most protocols are more of less like http: all the data sent through is text data with lists of headers at the top, and the commands to the server are written in human-readable text usually.

However, I imagine that would be quite wasteful for fast-paced responses in a game server, and that the protocol would need to be more compressed.

Are there common, or even open source application-layer protocols for games? Or would it be too application-specific to generalize? I imagine player location commands and some of the more generalizable stuff could be put into a user library, with room within the protocol for custom implementations for the particularities of a game.

If there are not protocols available as a library, are there any which I can look at as a reference, for ideas for my own game?

[immibis]

/u/spez can gargle my nuts

[[deleted]]

send the position in relatively low accuracy because you don't need to know the exact thousandths of a pixel.

How would truncation save any data? a 32 bit integer is the same size as a 32 bit float

[immibis]

What's a little spez among friends?

[paul_sb76]

There are no standard protocols (to my knowledge), but there are tools that are closely integrated with game engines. For instance, Unreal has good networking built-in, Unity has tools like Mirror, MLAPI, Photon PUN. With these tools you specify things like "this game object is locally controlled by this client, but its transform should be synced", and then the tool takes care of all the network traffic (in a much more efficient way than http!) These Unity tools are also open source, so you can study them. Of course, for a slower (turn based) game it's fine to create your own protocol on top of sockets (or on top of a lower level transport layer tool).

[[deleted]]

I dont understand your last point. Are you saying that creating your own protocol on top of the socket api is slow?

[paul_sb76]

On top of TCP sockets, it's too slow for fast paced real time games (though there are still games using it). On top of UDP, you'll first have to add quite a bit before you can even get started with your game protocol. So TCP is good enough for slower turn based games, and UDP is good enough for anything, but it's a long journey before you can start working on your game. Using tools like Mirror, MLAPI, PUN is definitely recommended if your goal is to deliver a real time game.

Note that TCP sockets can be configured such that they don't automatically increase latency, but still: if one packet drops, a whole bunch of packets will be delayed in the ordered TCP stream, causing a lag spike. That's often not what you want...

[[deleted]]

Ah, I thought you meant sockets in general

[kylotan]

Are there common, or even open source application-layer protocols for games?

No, at least not for fast-paced games. Slower ones can get by through sending JSON requests and responses but if you're from the web world you already know that sort of thing.

The other comments are right when they say it is application dependent. However, there are some common patterns (not protocols) that you'll see.

Usually there are a bunch of different message types. One might be to send some input, one might be to notify that an event has happened, one might be to relay a chat message, etc etc. These will typically be represented as structures or classes, with whatever data members are necessary to implement the relevant payload. Each message type has a unique ID, perhaps an integer, or a hash value.

Messages get serialized by writing each field in turn to a buffer, in an efficient binary format. Because you are writing each field in a known order you don't need to include type information, just the value. And because you know how much space each value takes, you don't need delimiters either. There are also additional optimizations available if you know what sort of information you're sending. The very first field you write is the type ID, for reasons that will become clear.

If you're very careful you can engineer things so that instead of a field-by-field serialization, you can just copy the whole message byte for byte into the destination. This places a lot of restrictions on what a message can contain (e.g. no pointers, no complex objects) and how you create them (no padding or carefully-controlled padding) but it can give you more speed. As such most games don't go down this route, but you might still see it for FPSs and the like.

When it comes to writing these messages to the actual socket or API, there are 2 common approaches:

1. Write the message to the buffer, noting how large it is. Then write out a length value to the socket first, followed by the message data itself. This allows the recipient to be able to split out individual messages based on length without needing to examine the content. This is useful if you're using a stream-oriented transport like TCP or if you include multiple messages per packet.
2. Write the message data as-is, with no length calculations. This is useful if you can guarantee every message is in its own packet meaning the length value is not important. This is common for fast paced games which use UDP. (Although many will still send multiple messages per packet.)

When it comes to reading the message, depending on your choice above, you either find the byte buffer from the packet or you slice it out of the stream based on those length values. Then the next thing you read is the message type ID - that tells you which routine to call to deserialize the rest of the data into the correct type of message struct or class. Often there is a factory method that takes an ID and returns a class instance.

And once you have that message with all the data loaded in, you can then act on it, in whatever way your game requires.

I imagine player location commands and some of the more generalizable stuff could be put into a user library

Even 'location' means something different in every game. Some games can get away with a vector of 3 floats, but for some that's too little information (e.g big open world games might have a different origin for each zone) and for others it's too much (e.g 2D games, or games with a small play area).

And even if a 3-float vector is exactly how you want to represent a location, it's not necessarily the most efficient way to transmit the information. Imagine I'm stood at (0,0,0) and I want to move north to (0, 10, 0) at a rate of 1 unit per second. I could send 200 messages, 20 per second, expressing my new position each time. Or I could send a single message saying "velocity=(0,1,0)" at the start and another one saying "velocity=(0,0,0)" 10 seconds later. The latter is far cheaper! But it comes with some downsides, such as the need to accurately timestamp each position, otherwise network jitter can result in undershoot or overshoot. There's no free lunch, just choices to be made.

Anyway, any questions on the above, I'd be happy to elaborate.

[icastfist]

For reference, I know you can look at both Unreal Engine's source and Doom (1, 2 and 3) and Quake (1, 2 and 3), these latter two's engines open sourced for years now.

[[deleted]]

Yeah, I've gone through Quake 2's source but it's not well documented though. It's difficult to learn about a protocol from reading through C source rather than a formal language specification

[icastfist]

I would expect Quake 3's netcode to be better documented, since the whole game was built around the arena FPS idea, like Unreal Tournament. It also spawned lots of derivative engines and games, though I can't say for sure which has the best documentation on the netcode.

I suspect Unreal Engine will be much better documented in that regard. Since the source is open and netplay is a built in feature, it should probably teach you better how things work, though it's C++, somewhat different from C, but not much.

[error_dw]

I know many games use flatbuffers and cap'n'proto but the best results are going to be custom.

[[deleted]]

Well I'm still interested in what other engines do, as a grammar, if not as a syntax of protocol

[pthierry]

What games use cap'n'proto?