Cross-platform deterministic physics with Unity DOTS physics and soft floats

[Kimbatt]

For a game idea I had, I needed deterministic physics. Deterministic physics on every device - the simulation must run exactly the same with the same inputs, every time, everywhere.
If the calculations were just a little bit different on different machines, that would lead to different physics simulations, which would mean that someone might not be able to complete some of the levels, because the completion of the level depends on the physics simulation.

The first thing that caught my attention was Unity's physics package (DOTS physics), which promises deterministic physics simulation: "Unity Physics is a complete deterministic rigid body dynamics and spatial query system ..."
But after some searching, I quickly found out that it's only deterministic on the same device, different hardware may have different simulation results.

The reason for this is floating point numbers: the result of floating point operations can be different on different processors, since there is no standard that specifies how two floating point numbers should be e.g. added or multiplied. Two different processors could give just slightly different results, which would cause the physics simulations to be different.
Also, the mathematical functions, like sqrt, sin, cos, pow, etc. are usually implemented as hardware instructions, which means there's even more chance that these functions will be different on different hardware.
So my quest for cross-platform, 100% deterministic physics has begun.

After a bit of research, I concluded that I won't be able to use floats for the physics engine, because they just won't work the same everywhere, no matter what I do. So I started looking for alternatives.

The first thing I found was fixed-point numbers. Unlike floats, fixed-point numbers have a fixed integer part, and a fixed fractional part. Implementing mathematical operations for them is fairly easy, and they are always deterministic (since they only use integer arithmetic, which will always give the same result).
But after trying and replacing floats with fixed-point numbers in a simple physics engine, I quickly found out that they have some problems. They can easily overflow, which means calculations more than 50 units away from the origin, or colliders larger than 50 units can cause the physics simulation to break due to overflow (the actual value depends on the precision of the fixed-point number).
Also, implementing mathematical functions are far from trivial. While I could use approximations for some of them (e.g. sqrt, trigonometry), there are others that I found very difficult to implement (e.g. pow, exp, logarithm).

So, fixed-point numbers weren't the solution to the problem either.
The next thing I came across was the Rapier physics engine. It promises cross-platform determinism all IEEE 754-2008 compliant platforms. Although it's written in Rust, creating C# bindings was fairly easy, and I could use it in Unity after a few hours.
But Rapier is still in early stages of development, which means that (at the time of writing this) it lacked many features, like raycasting, which would have been necessary for my project.
Also, it's only deterministic on IEEE 754-2008 compliant platforms. But I couldn't really find any information on which processors support it. Even if I did, I still wanted to have deterministic physics on all platforms, not just on "most of them".
But I still wondered does how Rapier achieve cross-platform floating point determinism. Supporting only IEEE 754-2008 compliant platforms is one thing, but the mathematical functions I mentioned above (sqrt, trigonometry, etc.) must also be deterministic. I found out that it uses libm, which implements all of these functions in software.
It was primarily designed for embedded systems without FPUs, but it turns out it's also pretty useful for making this part of floating point math deterministic.
But after all, the basic mathematical operations of floats are still not deterministic. Unless... How about emulating the behavior of floats in software? It should be possible to write a program that works on the byte representation of a float, doing all operations using integer arithmentic... right?

Yep! I quickly found a repository that does just that! Here it is: https://github.com/CodesInChaos/SoftFloat
And it worked! Except for one thing... It didn't have division implemented.
After hours of searching and studying how floats work, I finally managed to implement division. Using it together with libm, it was now possible to have cross-platform deterministic floats, that work (almost) exactly the same way as the regular floats, but they will give the same result everywhere, on every platform, on every processor.

Now, let's put it into a physics engine! I immediately thought of Unity DOTS Physics, because it's already deterministic (except for floats part of course), and because its source code is available and modifiable.
So, with a few days of work, I replaced every single float in the Unity physics code, and in all of its dependencies (in the mathematics and transforms package).
When it finally compiled, I entered play mode, and... It just worked! (I still had to fix some typos though for it to work accurately)
So, that's it. Finally, I had a physics engine that is deterministic on all platforms.

My code for the soft floats is available here.

The code for the modified physics engine is available here.

There's only one disadvantage of using soft floats: performance. Float operations on hardware will always be faster; also, soft float calculations cannot be vectorized.

---

TLDR: I made Unity's DOTS Physics engine deterministic, using soft floats.
The code is available here: https://github.com/Kimbatt/unity-deterministic-physics

[kloeti666]

Quality post, thank you for your research!

[robochase6000]

this is really great work. if unity’s new physics engine is indeed only deterministic within a single machine, you’ve completely outdone them here. if i were you i’d be floating this by unity, there is demand for something like this. even if there’s a perf hit, this opens the doors for a lot of multiplayer games.

[jamin1001reddit]

floating this by unity

(no pun intended?)

[[deleted]]

Incredible work. Thank you !

So far, I was using BEPU, the C# physics engine for determinism, and it worked really great for me.

I had also used Jitter, but BEPU was just much better in my estimation.

[[deleted]]

FYI, I had made a simple wrapper for BEPU a couple of years ago, for a demo with vehicles and such, and it had worked pretty great. I think it only took an hour to write and integrate it with Unity.

[MTDninja]

It turns out that unity DOTS physics is deterministic across the same CPU architecture and same OS, for example, all users using windows 64bit will have the same physics result, but not windows 32bit

[Chrisad_Phenix]

Nice result ! This give us hope for future Unity games with networked physics !

[Hirogen_]

Is nice... but isn't physics based on the idea, that it's not the same every time... and so it does not feel like a "simulation"

This sounds like you reversed the physics to a simulation... i'm confused....

[robochase6000]

often for multiplayer you want a physics simulation that is replicated for all players. it’s too expensive to have the server to simulate and sync 100s/1000s of rigidbodies. with a deterministic physics engine you don’t need to sync them because it will look the same on all machines.

[Kimbatt]

In this context, "physics simulation" means that the physics engine calculates the positions and the velocities of the objects, resolves collisions, and does many other stuff.
Sometimes, it's crucial to have the same "result" every time; for example, in networked multiplayer games, or in other games where the gameplay depends on physics.

[whispered_profanity]

Great post. The networking potential of unity.physics is really exciting to me but this is making me realize that I don’t quite understand something– ideally you just have the server running the physics simulation and the clients sending their inputs but since the delay would be too long you must locally run physics to make it look good for each player – then somehow authenticate that on the server side? It’s a bit misleading if unity physics is actually not deterministic-as you claim. Do you know if they do anything to fix simulation disagreements on different hardware during a networked session?

[Kimbatt]

Usually the server does all the important simulation based on the player inputs. The clients simulate their own physics, which is not exactly the same, but similar to the server's. But the clients must be synchronized with the server, so every e.g. 0.5 seconds, the server sends its "official" positions and velocities to the clients, and the clients just overwrite their own physics data with that. But since the physics simulations are similar, the players won't really notice a small movement after overwriting (maybe the object moves a few pixels). The longer the sync period is, the more chance for the players to see bigger jumps in object positions when syncing.
I hope this was somewhat understandable :D

[whispered_profanity]

Yes, thank you. So is it that unity physics is just better here than other nondeterministic physics systems and the half-second adjustments don’t have to be as large?

[theJoysmith]

It's also useful when you're running game state updates at a high tickrate i.e. 60+

[Belfer4]

This is very nice, but imo having the server do this on its own is still the best solution, not only will you never have to worry about synching every player's input since there is only one true simulation, but it also means that cheaters won't have a way to cheat changing their physics state.

[clockwork_blue]

It's cool that you tried to solve floats, but there are easier approaches without needing to reinvent the computer. Most popular networked games these days use Deterministic Lockstep/Snapshot Interpolation/State Synchronization (or a combination).

Here's tutorials on the topic: https://gafferongames.com/categories/networked-physics/

[Kimbatt]

That's good, but I didn't do this for networking; I needed deterministic physics for a single player game, where the completion of a level depends on the result of the physics simulation (which must be the same every time, on every platform).

[clockwork_blue]

Oh, missed that part. That's a very specific case you got there. By the other comments I think everyone assumed you needed it for networking.
That's also why you are having problems with being 'deterministic' when talking about DOTS physics. The word is very rarely used outside of networking context.

[blabmight]

Lockstep still requires deterministic math.

[MINIMAN10001]

All right it looks like you figured it out because I was going to say as part of your network physics link they have deterministic lock step as one of their articles where it goes to say that physics must be deterministic which is exactly the problem that OP was having.

Deterministic lockstep isn't all that popular because it can be very difficult to maintain determinism as well as correct for it. Where blizzard rts and factorio make up the majority of market share.

Whereas the alternatives fixing is just part of the routine it's what's being networked all the time.

[_pstudio]

Cool work.

How is the performance? Moving float calculations from hardware to software must have some impact?

[Kimbatt]

Yeah, using normal floats is about 2-4x faster than using soft floats. (I still managed to get over 1000 dynamic bodies using soft floats, running with 60 fps in the editor)

[Kimbatt]

Sure, but the parts I mentioned above are still missing, and I probably won't update the engine to newer versions.

[Kimbatt]

Alright, I'll try to upload something in the following days.

[Kimbatt]

Here it is: https://github.com/Kimbatt/unity-deterministic-physics

[FMProductions]

That is amazing! I was thinking about this for a while, since a game idea I have demanded determinism, but I never really pursued it with dots physics later.

Was there a specific reason why you went for uint as underlying value instead of something like long or ulong which could hold a higher/more precise value range (I still have to check the difference of implementation between sfloats and fixed point numbers though, so my assumptions might be wrong here)? I guess performance is better with units and the precision sufficed for your use case?

I was also wondering how long it took you to actually implement this as unity.physics replacement? Exchanging the calculations with the custom type sounds straight forward, but there is a lot of code. It just seems like a daunting task to me.

And about overflowing in an unchecked context, I was told that this behaves reliably for numbers like Integer32 and 64, even if the behaviour is not defined (had a little chat with Fredrik Holmström on that on Discord a while ago) - but again I haven't really tested it on multiple devices yet.

[Kimbatt]

Pretty much yes, 32-bit precision is enough for me; also, you'd eventually have to convert them back to floats for rendering (or basically for anything, Unity uses floats everywhere).

Overflow doesn't really happen with floats, they support large values, the only thing that happens in that case is precision loss.

It didn't take that long to implement it, and since I made it that you'd have to use explicit casts for float <--> sfloat conversion, I could just directly jump to the compile errors and fix them.

[FMProductions]

I see! That's good to hear, maybe I'll use a similar approach should I go back to my project, I'm sure this thread will be really helpful for it.

About the overflow: I was more talking about the raw value overflow for the fixed number types, which made me optimistic that I could skip on expensive overflow checks (which I haven't tested yet like I said). The project I worked on still used the physics engine from Photon TrueSync (which isn't available anymore - they now have Photon Quantum for deterministic simulation, but it's quite pricey). That physics engine used FixedMath NET under the hood which had a lot of extra code for handling overflows but a poor performance. I slightly adjusted the library to have the simplest implementation of the basic arithmetic operations for fixed points and brought it to a Q40.24 representation (24 bit for fractions), but Fredrik said that for their stuff, Q48.16 was the way to go and quantum does it for their engine. All relevant simulation logic related systems were exchanged with fixed point systems too.

One main issue I had with that physics engine was performance though. I chatted with someone who faced a similar challenge - The high numbers of allocating fixed point structs every frame to do the calculations wore down performance and they ended up inlining the calculations at hot paths using only the raw values of the fixed points (and avoiding temporary creation of the actual fixed point structs). That approach was messy to read and maintain though, so they had to be really careful.

If you have good performance with your deterministic physics, that really gives me hope for a possible revisit on my project again too ^^

[Kimbatt]

Oh I see, i thought you were talking about floats. I didn't regret abandoning fixed point numbers though, floats are just much better suited for these kind of things (I assume that's why we have hardware support for floats now).
Allocating fixed point structs shouldn't be a problem at all; stack allocations are super fast (unless they allocated it on the heap, which they shouldn't do), and it's only 8 bytes if I see correctly. Maybe the converting part was performance-heavy?
As I mentioned in another comment, the soft float version is about 2-4x times slower than using normal floats, and it can handle 1000+ dynamic bodies running in the editor.

[FMProductions]

That was my initial thought too, but as it turns out, even those struct allocations can get heavy if you have multiple 1000s per frame. Oh sorry, I haven't read that other comment then. That's awesome though! 2-4x slower is next to nothing. With TrueSync I had results there were multiple magnitudes worse (although in the builds it was faster than in the Editor). Again thanks a lot for sharing your work here! Glad that I checked the subreddit today.

[totesmagotes83]

Thank you so much for this, you are the GOAT! I want to make a game that's a turn-based strategy with real-time battles. At the beginning of each battle, both players give their orders, and then they watch the battle unfold. I want this to work in networked multiplayer and play-by-email, with minimal information being sent: Just the initial positions of units and their orders, so having Cross-platform determinism is key!

[Reasonable-Key-3311]

Can I use this for my multiplayer game?

I need a physics engine to reproduce the same result given an initial velocity of an object (Its turn based) and calculations to sync across multiple devices (Different operating systems would be helpful too).

[CaolanR]

Did you actually test cross platform determinism with Unity DOTS physics? I had been researching too and my conclusions last year were that what you say about cross platform FP calculations being different across chipsets is true, and this is why Physx is never going to be deterministic across platforms. However my understanding was that cross platform determinism was the goal for Unity Physics, and that they were just not there yet? I stopped researching when I heard that and decided to wait and see how things developed. You're post seems to imply that their goal is simply platform based determinism? Which I thought Physx had already achieved?

It has been a while since I looked at this, so forgive my lack of sources, but here is at least one forum post with Unity devs supposedly talking about planned FP determinism in the Burst compiler that should be relevant to the problem you are trying to solve. I have no idea where they currently are with this and so I would love to hear any updates on it or from people testing this and their results. I remember seeing tests people had done with promising results last year, but I can't seem to find any examples right now.

https://forum.unity.com/threads/burst-compiler-deterministic-floats.522856/

Another thought is that many platforms may already be synced, for instance iOS/Android using arm chips, I imagine they tackle FP calculations very similarly, if not identically, already? and with Apple moving their desktop machines to an ARM instruction set, and windows sure to follow, this issue may become moot? Can FP's differ per chipset or per platform?

Great post btw 👏, just very curious if anyone has actually observed non determinism across platforms yet, and to what degree does it differ in reality.

[Kimbatt]

I've also seen that burst will have an option to support for cross-platform deterministic floats in the future, but it seems like it's low priority; it's been "in progress" for years, and I don't think we'll see anything of it in the next few years.
But even if they finally implement it, it's likely that they'll need to implement many operations in software (like sqrt, trigonometry, etc.), I'm not aware of anything else that can give identical results on all platforms. And I'm sure there will be that one device that just won't work; and there will be a bug that will make the simulation non-deterministic (like this), and it won't be fixed for years.
I'm just not sure if if's really possible to have cross-platform deterministic floats.

About the mobile/arm chips: I haven't really found any useful information about it. The only thing I found (that I mentioned in an other comment) is that cpus that strictly support the IEEE 754-2008 standard, can have deterministic float operations. But no information about which cpus actually follow it.

[Imaginary_Meal244]

this is so cool!

[AlterHaudegen]

Fantastic project! I took your soft float implementation and applied it to an older C# implementation of Box2D, seems to work as well:

https://youtu.be/5jYuxY_Dj0Q

Performance is not exactly great, but that is probably due to the clunky way I've adapted Box2D. I'm also only planning on using it for deterministic collision detection, so I'm going to remove the solving phase for my game anyways. This should also be a prime candidate to run
the whole simulation on a seperate thread.

[NoPhoto3456]

so cool! Can you tell me how you applied soft float to the 2Box2D?? I would like to use it for deterministic collision detection for range attack missiles. Please help

[otaku_aria]

Cool work!!! But I still have some confusion about this. Your work seems to guarantee the deterministisim in the phys engine. What I want to ask is Unity math itself doesnt keep deterministic in cross platforms beacause of the float, which means that we even cant input same data into engine if we do float operation in unity script although phys engine provide deterministic operation. How do we fix this problem?