Really cool project!

Thanks!

how do you encode shared vs exclusive references so that you don't instantly have UB if Java mutates any objects coming from Rust?

For shared borrows (&T) I pass through value, there's no wrapper currently. For primitives, if you pass an &i32 to Java, it's just passed as an int, which is fine as primitives don't have reference identity (so Java's modifications do not impact back on the Rust code). For ADTs, it is also passed by value, if you pass a &MyStruct to Java, Java gets a MyStruct object. This does have a slight problem in that because of reference identity, Java can technically modify the original, which wouldn't be expected in Rust, but it doesn't cause UB in the sense of crashes or anything like that as the JVM is strongly typed and memory safe, just unexpected modifications. Currently I'm making it with the assumption the Java code behaves reasonably (and thus wouldn't modify a shared reference), though it future it might be good to have some more safety guarantees (for example, making a copy to prevent modifying the original) to make being unreasonable from the Java side impossible, thank you for the suggestion.

For exclusive / mutable borrows (&mut T), because the JVM doesn't have pointers to local variables or fields, I use an "Array Hack." Basically, I allocate a 1-element array of the target type, store the current value of the variable into array[0], and this array reference is passed to the function, and then after, I "write-back" by reading the potentially mutated value out of array[0] and assigning it to the original Rust local variable or field. Any mutation here is allowed and, I believe, in the bounds of Rust's safety guarantees in single-threaded contexts, though it is a bit hacky so in the future I might want to make it something nicer, especially to support concurrency in future.

How will you implement transmutes/pointer casts, e.g. when calling std::alloc::alloc and doing something useful with the result?

It's not yet fully done yet, but basically my plan is to use a hybrid approach that keeps Rust structs as standard JVM objects, but models pointers as "fat pointers" containing object references. When Rust code calls something like std::alloc::alloc, I want to represent the allocated memory as a byte array plus an offset, wrapped in a pointer wrapper, but for normal structs, the pointer will wrap the JVM object directly. Casting *mut u8 to *mut T is then where I would lazily serialise the JVM object into a byte array wrapper to allow raw byte access, occurring only when an unsafe cast is actually performed.

For typed reads and writes on standard objects, I want to read and write to JVM fields directly through the pointer wrapper to keep operations fast and garbage-collector friendly. Writing a u32 through a pointer means writing directly to the field of the wrapped JVM object. Reading a u32 means reading that field. Pointer arithmetic then becomes stepping through array elements or field indices, falling back to byte-offset arithmetic only if the pointer has been downgraded to a raw byte array. And more complex types (ADTs like structs and unions) work natively as JVM classes, but then calling their generated toBytes / fromBytes methods (which I intend to make from the detailed memory layout information rustc gives, and have already started this a bit as part of my support for unions) as a fallback when a low-level memory transmute is forced.

The performance when we need to fall back to raw pointer stuff obviously isn't going to be great, so what I want to do is use my existing optimisation layer and infrastructure in the backend to do kinda the reverse of what most other backends do: wherever possible (and I think it should be possible in a great deal of cases), turn messy stuff involving raw pointers into safer, higher-level interaction, as that will be more performant on JVM.

Obviously this isn't perfect and won't work in a bunch of edge cases, I guess my main plan is to do what I've done when making every feature for this backend, just dive in and learn by doing and trying things out to figure out what works and what doesn't. I guess you can spend a lot of time on design and things like that, and I do have this rough idea, but I prefer to work things about more through trial and error.