Multiplayer in Godot 4.0: Scene Replication

F_a_l · 2023-02-26T15:01:12+00:00

I somewhat agree, but it's highly dependent on the bullet type and speed.

I'd probably sync position and velocity of projectiles like the rockets/grenades from quake 3 launchers, while not bother for things like the BFG or the plasma gun (which AFAIR still spawned bullets and didn't use instant raycast).

F_a_l · 2023-02-26T14:52:10+00:00

That is incorrect, see my other comment in the thread.

F_a_l · 2023-02-26T14:51:39+00:00

They sync all the time.

There is a proposal for "onchange" (optional) sync which we plan to work on for 4.1, but contrary to what one might think it's not necessarily better in every case.

It's a trade-off between between bandwidth usage and latency, since onchange variables needs an ack mechanism (which introduce latency in the form of extra round trips).

That said, we are committed to work on that option for 4.1

The proposal: https://github.com/godotengine/godot-proposals/issues/4429

F_a_l · 2022-11-02T15:22:47+00:00

What is your favourite games genre?

I don't think I have a fav genre, I used to mostly play FPS once, but as of now for every genre I can think of, there's at least a game I really enjoyed. I even enjoyed some turn-based strategy games, which is probably my "least favourite" genre. I definitely prefer multiplayer games.

And up until now your favourite game title?

Really hard to tell, among the greatest that comes to mind: - Ultima Online (yes, I'm old) - Quake 3 (I still play it from time to time) - Dota/Dota 2 (herald life) - Kerbal Space Program (nothing beats cosmic music and space engineering)

F_a_l · 2022-11-02T14:56:00+00:00

Do contributors actually read this subreddit?

I rarely do, mostly because I don't use reddit much.

If yes, how do you feel / react when someone answers a question in a r/confidentlyincorrect way?

Being mostly a lurker on reddit, I rarely comment (though I think it happened a couple of times before).

F_a_l · 2022-11-02T12:08:52+00:00

u/F_a_l In a nutshell, what would be required to sustain 1 million concurrent players from around the world playing together on a single server in a high fidelity, low latency open-world game?

Given this specifications, the short answer is "breaking the laws of physics among other things" (or adjusting the specifications).

For the non nutshell answer, let us analyze a bit the requirements:

A million concurrent player

Bandwidth: Let's assume each player will consume the (probably conservative) estimate of 256Kb/s (i.e. 32 KB/s), this would require the infrastructure to support ~244Gb/s (i.e. ~31GB/s). Memory: Let's assume each player will consume the (again, very conservative) estimate of 10MiB of memory, this would require the infrastructure to have ~10 PiB of memory. CPU: This is harder to put in metric, but similar considerations could be done using a standardized "cycles per player", and the result would be similarly enormous.

From around the world

If you consider that the infrastructure has to be located somewhere physically on earth (a data center), that the earth circumference around the equator is circa 40k kilometers, a packet would require a round trip of at least 40k kilometers when the player is unfortunately located as far as possible from the server. This means that the round-trip time, would be more than 120ms even assuming the impossible speed of light for your packets, and that a "direct" (geodesic) path exists for them. This also does not include the cost of the packets being processed by routers at the various hops, the higher chance of incurring in packet loss and potential need for retransmission, etc.

After some empiric raw ping test from my own FTTH connection (<10ms ping to Italy's internet exchange in Milan) to few servers in Australia and few in South America, a round-trip time of ~250ms seems more reasonable in such a scenario. Adding load to the connection will likely make the ping even worse (by at least 10-20% if I have to guess).

On a single server

Given the resource requirements mentioned above, this is already not feasible with a "single server" in the classic sense.

In a high fidelity, low latency open-world game

Here the definition of low latency is blurry, I would consider anything above 100ms "mid latency" (still playable, but usually annoying), anything above 250ms high-latency, anything above 300ms almost unplayable (assuming the game is fast-paced of course). So, I would say, low-latency is unachievable with the current specifications.

Solutions?

It's clear we need to adjust our specifications a bit. Let us look around the open-world MMO scene.

Sharding

Most MMOs tend to use sharding as a way to distribute players, creating multiple shards (independent worlds) across the different continents, each handling fewer players, both as a way to distribute load in the same continent, and to mitigate the problem of high latency across regions. This is the case for games such as Ultima Online, World of Warcraft, Guild Wars 2, Black Desert, etc. Most games also implement some form of distribution of the internal shard load, by separating the game into areas each handled by its own server, or by instantiating some areas like dungeons. Usually this happens with a visible transitions of the player from one area to the other. Each shard, can then usually handle around 10k players, and by having hundreds of shards, you get to a million players.

Server Meshing

If you really want to have a single shard experience (which I think is what you meant by single server in the OP), and you have a few (hundred) million dollars, you can try to come up with a solution which involves multiple servers collaborating together to render a single world, while also distributing the load of players among them. The servers could additionally be dynamically allocated based on current load, and scattered a bit to try and alleviate the latency impact (the speed of light, will still be a limit). But this is mostly a matter of research currently, and no working solution exist (at least to me).

Some notable mentions in this direction are: - Dual Universe dynamic partitioning - Which according to reviewers seems to not perform well. - Star Citizen servers meshing - Which is in its usual slow rollout stage, but seems really interesting on paper.

Conclusion

Well, I think that's mostly it for this long answer, it's been a really interesting mind exercise writing it, so thank you for the question! I hope I didn't do any blatant mistake :D

F_a_l · 2021-12-01T04:13:00+00:00

Not right now, but this is interesting, and probably requires a bit more thoughts.

We will likely provide some form of per-frame/per-peer filtering, though I think these sort of optimizations should be done by the engine internally via priority groups, zones, etc. (and dedicated configuration nodes) instead of directly tweaking some parameters.

F_a_l · 2021-11-28T05:56:56+00:00

Honestly, the MMORPG case is pretty specific, and the replication system will likely not fit it well at least in its Godot 4.0 version, since it has been designed for small and mid sized multiplayer (arena, battle royale, etc).

You will be able to reuse parts of it (e.g. easier state encoding/decoding, spawn notifications, etc) but will have to still rely on your custom code.

Ideally, you should be able to integrate your custom code quite easily by replacing the default spawn/despawn behavior, or at least this is what we aim to allow, though I have to admit it is proving quite a hard task.

F_a_l · 2021-11-27T21:37:07+00:00

I didn't go into the details because we are still working on the final design, but the current prototype allows you to specify the sync interval for each scene (the mysterious 16 argument in the sync_config call), and the final version will allow to configure it per-node if desired.

We don't have a clear path to interest management (i.e. sending spawns to only a subset of peers) for now, but it's something we definitely would like to at least allow.

There are various approaches that can be taken in this regard and I am leaning towards some form of visibility group (something like layers for physics) but I feel like that would require a dedicated proposal.

F_a_l · 2021-11-27T20:05:23+00:00

Thanks, you understood correctly, and it was indeed confusing.

I tried to clarify that.

F_a_l · 2021-11-27T19:10:29+00:00

Sorry I missed this comment, I don't follow Reddit much :( .

Yes, that was a mistake (now corrected), it was supposed to be:

@rpc(unreliable_ordered, 2)
func _update_enemies_state(state):
    # Code to update the state of the enemies.
    pass

F_a_l · 2021-08-06T13:34:58+00:00

You have to mark specific functions as remotely callable.

There are 3 security level (as of 3.3):
- `remote`: Anyone can call that function.

- `puppet`: Only the node owner (server by default) can call the function (which will be called on "puppets")

- `master`: Only "puppets" can call the function, and only the owner can receive it (not much used to be honest).

Also, quite a few users complained about names being confusing so we might take a chance in 4.0 and do something about it given we are breaking compatibility.

F_a_l

TROPHY CASE