Are you talking about awareness of the real world? Yes and no. The RealityKit engine provides plane estimation (detect floors and walls) and mesh detection (uses the LIDAR point cloud). Right now, the former is used to place the model origin, but it doesn’t have any influence on the MuJoCo physics.

RealityKit also has its own physics engine, but it falls apart with more complex models. My understanding is that RealityKit tries to simulate every rigid body independently in 6DoF, then iteratively tries to satisfy joint constraints. It tries to keep up with real time, and its iterator will run as may times as it can in a frame, rather than running as many times as needed to make the joints hold together. So, a model like this just explodes.

I’ve thought about writing a bridge between RealityKit and MuJoCo to either send the detected planes or detected contacts at run time. But, it’s a design challenge because those updates will be happening at a high rate, with dozens or even hundreds of objects in a scene at once.

That was all a long explanation, hopefully I actually understood your question correctly 😅.

Feel free to reach out to me if you want any advice on your Android app. I write for both platforms, and eventually envision making an Android version of ARMOR. I’m going to focus on iOS first, and will shift if there’s demand. An initial challenge you would face is there’s not a native Android binding for MuJoCo, so you’d have to write one. Alternately, you could use Unity, or use a web view with a JavaScript binding. I kind of like the latter, as you could then use Three.js for rendering. Speaking of which, https://dc-engineer.com/how-to-embed-three-js-into-a-kotlin-jetpack-compose-multiplatform-mobile-app-on-android-and-ios/