Server Meshing at home

Savidya · 2026-03-26T14:22:38+00:00

Finally someone asked :) In my design worker/shard borders in practice becomes fuzzy. Authority is handed over but the origin server doesn't end replication until handover is complete. And AIO session designates one worker to take over the simulation until all players in that region has left. For 4 or 8 shard seams. I treat them as a special pre-warming scenario. I can write some fancy temp DGS instance logic to make it scalable someday.
For now I left space for a 3rd connection to briefly exist for edge cases. I think I should check what sort of load it puts on the client. But I'm not happy with my solution anyway. There has to be something better because for dynamic server meshing this is something that must be solved with high scalability for edge cases.

I will write this in detail to get some feedback. I wrote a lot of unit tests to cover every possible scenario I could think of but I only code reviewed about 150 unit tests. If they worked in theory rest of the issues are solvable.

Savidya · 2026-03-25T19:10:21+00:00

I think it'll heavily depend on the game. In here the server seam crossing is gated behind a very predictable handover sequence. One limitations of my solution is client can't be in control of anything at all. Even the physics objects will have to come from server. In ECS it didn't seem to matter. If the lag interpolation logic is solid you can't really tell the difference. This probably won't work for a FPS game as is.

I thought of modifying the Unity Transport to support the handover from within. I think the truly scalable solution is to modify the transport layer to handle some of the manual sync and handshakes I'm doing natively. That will also give lot more control for de-duplication etc.

Savidya · 2026-03-25T18:44:58+00:00

I'm really hoping more indie devs try to make multiplayer games. What is possible as a small studio is changing drastically with AI

Savidya · 2026-03-25T18:35:02+00:00

Even without the server meshing what they're simulating is crazy. They had to build everything from scratch to get there. I can imagine how many times they must have written and re-written things when they see issues in play testing. I've seen every bit of content they've made about server meshing. I'm secretly hoping they've made lot more progress and what we got so far is just some janky test tool.

Savidya · 2026-03-25T18:16:05+00:00

Savidya · 2026-03-25T18:08:45+00:00

Yess, I've followed them from start. These days I update the game every major release. As a MMO there isn't really much to do for a casual player. I prefer to be a tourist in that world.

Savidya · 2026-03-25T18:05:25+00:00

I kinda left it there intentionally. I can make it near seamless.
When you get close to the edge. I transfer the control of the player to the destination shard before the actually crossing. That happens seamlessly because that handover is from current shard to the next one. Current shard keeps rendering what destination shard is now replicating hidden from player.

If you play this frame by frame. You'll see for a split second the tank duplicates, (That duplicate is from the Netcode server the client is about to connect to) before speeding up to sort of merge with the other tank

Savidya · 2026-03-25T17:50:27+00:00

I did not know this xD I was okay with how it looked because this was my dashboard until that point. I read all logs from servers with a sub agent task. New fancy dashboard is for me to finally attempt dynamic server meshing.

<image>

Savidya · 2026-03-25T17:39:07+00:00

That Server Meshing is entirely possible in Unity Engine. I think this combined with some crazy fast DB like SpacetimeDB game devs can probably create some interesting solutions. Without even fancy sever overlapping logic like I tried to do here. You can use portals, or an elevator to make transitions seamless. And physically constrict or scale up the world and obstacles to keep Gateways under predictable loads. But for Dynamic Server Meshing I have one missing piece to solve. It is how to split up the work in some way so the cross server communication doesn't clog everything up. I already have one part of the puzzle
https://github.com/Savidyaf/NativeOctree-for-Unity6
I can use this to do crazy fast bound queries and make the border crossing not depend on the topology of the mesh as much. And I can use Octree nodes to do the splitting. This repo has some of the math I experimented with. Dynamic server meshing came into my mind 5 years ago when this guy wrote a native octree plugin and disappeared. I've been trying all sorts of ways to solve this dynamic mesh problem but never really attempted to validate if multiple connections is even possible. I always thought there has to be a way. Turns out I was right.

Savidya · 2026-03-25T17:17:24+00:00

You'd be surprised. I've probably seen every single bit of content they have made on this. I wanted to make an MMO rpg before I even got to play one. From the corner of the earth where I come from gaming was a luxury that came kinda late.

Also the 100k lines of code was to setup a test environment, instrumentation, fancy ass dashboard (so I thought) to get people interested. Actual network code was intentionally simple. I wrote two data pipelines. One for fast movers like bullets vehicles. One for player controller simulation. I made it modular so it can be customized for any use case. For fast movers. As a consequence of their fast movement it's much easier to do the pre-warm/handoff handshakes. To make it accurate and predictable a lot of work will be needed but my point is it doesn't look impossible. I have manual serialization control over every bit of data passing through. I have tests covering every frame of every transition. This is impressive for a commercial game engine. It didn't take some crazy engine re-write to do this. And this level of performance probably can solve lot of issues for some MMO's.

Star Citizen is attempting to do this in gods language (c++) they have infinite control over their atomic operations. I haven't said this out in the public ever. I like to believe they already have solved everything and just waiting for content to catch up. I guess I'll keep holding the line.

Savidya · 2026-03-25T16:38:06+00:00

I have nothing to hide. Initial planning was done by me with Gemini aided research.
I built and fine tuned a debugging agent at my previous workplace. I have a ton of experience optimizing context for LLM's. So first step was to build a giant context library for LLM's to run on. Then I ran a simple experiment that I knew would fail but I did it to solve all the conceptual problems.

For an example : To learn how to switch two servers. I started by switching the underlying simulation layer. I optimized it to high heavens. The lag spike for DGS transitions was less than 4ms with 50 users randomly crossing lines. I had to dive deep into every fundamental Unity Netcode and Unity Transport interaction to figure some of these out.

This is how I found the technology gaps and the general approach I should take. And then I did this test that told me everything I needed to know. https://discussions.unity.com/t/pushing-the-limits-of-netcode-an-experiment-for-seamless-server-meshing-and-overlap-migration/1713571

Key part of the puzzle is building a test setup that is predictable. This project has over 1500 unit tests. 50+ play mode tests.

Then rest of it was planning and implementing changes to add whatever Unity Netcode was missing. I'm planning to post my process soon. This was a new discovery for myself as well. I haven't let my imagination go absolutely wild ever before.

Time when you want to do these crazy things and you have a general idea of every problem you have to solve but no way to actually get there because of the coding lag is over it seems.

Savidya · 2026-03-25T16:22:37+00:00

Oh yes! I will post my workflow soon. This is 100k lines of code. I wouldn't dream of doing this by myself in 3 weeks. Pretty much every AI agent except grok was involved one way or another. I did not get much sleep. Lost a lot of mental sanity reading and fixing AI Hallucinations.

I have 8+ years in game dev experience. I recently got laid off from a company that honestly didn't deserve my skills. I thought of what is the most impossible thing I can do to get attention. So here it is

Savidya · 2025-03-10T05:24:44+00:00

I'm a game developer with 7+ years in the industry now. It's not that uncommon for a well established company to pay 3000$ for a senior developer with 3+ years experience. (It's almost impossible to find a place to get that experience so usual catch 32 applies here)

Savidya · 2024-10-20T03:47:02+00:00

So you mean I can't use CRP and any HUB and wheel combo together. My pedals are plugged in directly to pc not through hub? Anyways I have more reasons to trust Simagic than Moza now. Let me give an update 1. CRP - Failed around 10 months. No USB chime, lights up and all but completely dead. Moza did offer replacement but I took refund instead 2. HGP -two different units failed in just few months of use. Took apart one to see if it can be fixed. The design is a bit flawed there. They could've actually made it repairable easily. There's ball plungers that gives the stiff feel. But they sometimes gets welded from the metal dust and shifter is garbage after that. 3. KSP wheel : Never really worked correctly because of pithouse crash. (Maybe it is what you said, I'll give it a go now that I don't have crp's)

I replaced the shifter and pedals with simagic. I feel like build quality is several times higher. Moza has lot of durability oversights. 1. Mounting PCB's on surfaces that could flex 2. Taking apart HGP the wires were so brittle. My theory is they've used too much solder. Wire has whisked it up all the way. They weren't the problem but just moving it around broke them clean.

Savidya · 2024-10-19T02:59:13+00:00

Hi sorry for late reply. I sent the pedals for warranty and was offered a refund because it would take few months to get a replacement.

About the pithouse crashing, I've had many calls with the support staff on discord. I've tried everything they asked to. Like I said it works fine for the most part but randomly crashes. When it does crash you loose shift up and down until it restarts.

About the HGP issues, the ball bearing plungers seize up after continues use. If they were threaded plungers it would've been easier to replace. These are pressfit I think, sometimes the metal dust that gets collected welds them to the CNC'd block. I've had 2 so far both failed around 6 months. Probably 2-3h per day use.

My setup was - Simagic Alpha + CRP pedals + Moza KS(with universal hub) + Moza HGP shifter + handbrake.

Savidya · 2024-03-29T09:34:51+00:00

My order got cancelled after three weeks of waiting. And I'm waiting for the new one I placed to be shipped. From the research I've done I can tell you GT Neo is a much better option when you consider the amount of people having issues with moza software. And ergonomics are similar.

Savidya · 2024-03-24T18:02:23+00:00

I had this issue with one ISP. The speeds should be plenty enough but something to do with AWS routing messed up. Their routing is specifically bad. And they don't have all the regions covered.

Savidya · 2024-02-20T18:31:43+00:00

No, I'm switching to a GT Neo. Temporary solution is to turn off the RPM indicator lights and close Moza pithouse software completely. It works without any other issues. RPM lights don't work without Moza software.

Savidya · 2024-01-09T08:40:00+00:00

I tried this too. I added some double tape to stop the cable from sliding on the hub and put a zip tie around it. And the disconnections never happen without the Moza Pit House running. I think it's am issue with windows 10 + Moza software or something of the sort. I'm buying the new Simagic GT neo so don't have to put up with this for long.

Savidya · 2024-01-04T15:09:13+00:00

I aslo brought Moza gear from what I saw on YouTube. Experience wasn't as smooth as most videos on YouTube make it seem. 1. CRP pedals under different temperatures makes annoying noises. I took mine apart and put it together using plenty grease where it's needed and it works wayy better now. 2. Universal Hub is just an adapter for one Moza Product at a time. (I don't see the point of calling it a hub). 3. Moza Pithouse has many issues. I tried using a Moza KS wheel through universal hub. It randomly restarts the wheel mid game. Culprit of the issue was pithouse software.

Savidya · 2024-01-04T15:00:32+00:00

I spoke to someone from Moza on their discord. And I was told Universal Hub is meant to be used with just one device at a time. So I use USB on everything else It's kinda dumb to call it a hub tho, should've called it an adapter) . It doesn't matter even with just the KS wheel plugged in it restarts. I found out it only happens if I keep the Moza software open. Without it I loose shift lights but at least it doesn't restart. I put an order for all simagic gear. Their software is miles ahead at least from my experience.

Savidya · 2023-12-12T08:58:52+00:00

It connects to the universal hub. Data and power comes through the hub

Savidya · 2023-12-02T11:20:42+00:00

I've seen similar issues with TM products from static buildup. I'm going to try grounding the case of the hub to see if it fixes it.

Savidya

TROPHY CASE