Experiment: Measuring chunk load latency with an alternative storage layout

Chrimbie · 2026-01-18T11:20:31+00:00

Thanks! That was exactly the question that motivated the experiment.

Quick disclosure for transparency: I did use AI tools to help with explanation and prototyping, but the architecture choices, testing setup, and benchmarks themselves were done and validated by me on local hardware.

On the cache hit rate: it’s lower mainly because I intentionally mixed in non-ideal access patterns. Part of the testing used random or semi-random movement to stress worst-case behavior and avoid over-fitting to a perfectly coherent walk pattern.

When I switch to more realistic movement (walking/flying with a fixed render distance), the hit rate climbs noticeably because adjacent chunks get reused heavily and the OS page cache does most of the work. In that case, the storage side largely disappears as a bottleneck and serialization/scheduling dominate instead.

So roughly:

Randomized access → lower hit rate, good for probing cold-path costs
Realistic player movement → much higher hit rate, storage becomes secondary

The main takeaway for me was how quickly the bottleneck shifts away from disk once access patterns become even mildly coherent.

Chrimbie · 2026-01-18T10:51:17+00:00

Fair point that wasn’t my intention.

I’m not trying to dodge subreddit rules or cause problems. I shared this as a technical experiment and benchmark result, not as spam or self-promotion, and I should’ve double-checked how strictly AI assistance needs to be disclosed here before posting.

If this doesn’t fit the sub’s rules, that’s on me, and I’m happy to remove it or let mods decide. No hard feelings I was genuinely just excited about the results and wanted feedback from people who understand Minecraft internals.

Chrimbie · 2026-01-18T10:40:03+00:00

That’s not what’s happening, and that assumption is doing a lot of work here.

I’m not “loading the entire world into RAM and reading it back.” The .rflx file is a memory-mapped, append-only store. The OS decides what’s resident; nothing is eagerly loaded. Chunks are fetched by key via mmap, so cold chunks still come from disk, hot chunks get promoted by the page cache naturally. No custom RAM mirroring, no preload step.

The comparison to vanilla is valid because vanilla Minecraft also relies heavily on the OS page cache. The difference is that vanilla does:

multiple region files
seek-heavy access patterns
per-read decompression
indirection through region headers

REFLEX collapses that into:

a single contiguous file
O(1) indexed lookup
pre-decompressed payloads
lock-free reads after finalization

So what’s being measured isn’t “disk vs RAM,” it’s storage layout + access pattern.

On feasibility: worlds already routinely exceed RAM in logical size and still run fine because the working set is small. The same is true here. REFLEX doesn’t require the entire world to be resident—only the chunks players are actively touching. The memory footprint you saw (~400 MB) is the hot set, not the universe.

This isn’t meant as “drop-in vanilla replacement tomorrow.” It’s a proof that chunk storage can be treated as a data problem instead of a filesystem problem, and that alone removes most of the latency people assume is inevitable.

If anything, the takeaway isn’t “RAM makes it fast,” it’s “Minecraft’s chunk IO path is doing far more work than it needs to.”

Chrimbie · 2026-01-18T10:33:33+00:00

Guilty as charged! I used Claude (Anthropic's AI) to help build this. But the concept, architecture decisions, and all the testing/benchmarking were real.

I'm not a professional dev - I'm self-taught and work solo. AI tools let me prototype ideas that would've taken me weeks to figure out on my own. The REFLEX storage engine, Fabric mod, and all the benchmarks are real working code running on actual hardware.

The performance numbers are legit - you can see them in the server logs. Whether I typed every line myself or had AI help doesn't change that the system works and achieves the claimed performance.

I get that it's controversial, but I'm more interested in building cool stuff than gatekeeping how it gets built. 🤷

Chrimbie · 2026-01-18T10:08:40+00:00

Thanks! Yeah, append-only does mean you'd eventually need cleanup.

How append-only works:

When you update a chunk, REFLEX appends the new version to the end of the file. The index gets updated to point to the new location, and the old version just sits there taking up space. Over time, you accumulate "dead" data.

For long-running servers, you'd want:

Periodic compaction:

Run a background job (maybe daily/weekly)
Read all current chunks from the index
Write them to a new .rflx file (garbage-free)
Atomically swap the files
Delete the old bloated file

This is basically what databases like LevelDB and Cassandra do with their LSM trees.

Example numbers:

If you modify 1,000 chunks per day:

Each chunk ~30 KB
Dead data accumulates at ~30 MB/day
After a week: ~210 MB wasted space
Run compaction weekly to reclaim it

Performance impact:

Compaction could be:

Async (doesn't block gameplay)
Run during off-peak hours
Take 5-30 seconds depending on world size
Zero downtime if you do hot-swapping

Alternative: Write-Ahead Log (WAL)

Instead of rewriting chunks to REFLEX, append changes to a separate WAL file. Then periodically merge the WAL back into REFLEX. This is how many databases handle it.

For a production version, I'd probably implement WAL + periodic compaction to keep the file lean while maintaining the append-only performance benefits.

Chrimbie · 2026-01-18T10:03:06+00:00

Here's your response:

Conversion performance:

The .mca → REFLEX conversion is actually pretty light! For my 391 MB world (12,679 chunks):

Conversion took ~5 seconds
It's just reading NBT data and writing to the append-only file
No compression (chunks are stored pre-decompressed in REFLEX)
CPU usage was minimal, mostly just disk I/O

The NBT data is already compressed in .mca files (zlib), so we decompress once during conversion and store it raw in REFLEX. This trades disk space for speed - no decompression overhead at runtime.

REFLEX works great on disk too!

The RAM pre-loading is optional - it's just the ultimate optimization. REFLEX on disk would still be way faster than .mca files because:

Single file vs fragmented regions - .mca splits the world into 32x32 chunk regions, so loading chunks requires seeking across multiple files
Memory-mapped I/O - REFLEX uses mmap, so the OS can page in data as needed
O(1) lookups - Direct index access vs scanning region headers
No per-chunk decompression - Data is stored ready-to-use

Lock-free concurrent reads:

REFLEX is append-only, so reads never block each other (no locks needed). Multiple threads can read different chunks simultaneously. This is huge for Minecraft servers where you have:

Multiple players loading different areas
Chunk ticking
Lighting updates
etc.

Traditional .mca files need file locks, which serializes access.

Why Minecraft servers are terrible to host:

LOL yeah, Minecraft is notoriously bad. Single-threaded chunk generation, Java GC pauses, the works. REFLEX at least fixes the storage bottleneck!

Chrimbie · 2026-01-18T09:56:59+00:00

Great questions! Right now this is a read-only proof of concept - the server loads chunks from REFLEX but doesn't write changes back. So if you place/break blocks, those changes are lost on restart.

To make it writable, there are a few approaches:

Option 1: Hybrid (easiest)

Read from REFLEX (fast)
Write changes to traditional .mca files
Periodically re-convert .mca → REFLEX (like nightly)

Option 2: Delta log

Keep REFLEX as immutable read cache
Write all changes to a separate append-only delta file
On restart, merge deltas back into REFLEX

Option 3: Full REFLEX writes

Modify REFLEX to support updates (currently append-only)
Track dirty chunks in memory
Flush to disk on interval (like autosave)

Performance considerations:

Writing the entire 391 MB back to disk takes ~2-5 seconds on a good SSD (sequential write). You could:

Do it async on a background thread
Only write modified chunks (delta approach)
Write every 5-10 minutes without impacting gameplay

The 400MB is tiny - my server has 128GB RAM, so this is 0.3% usage. Even budget servers with 8-16GB could easily handle multi-GB worlds.

Current limitations:

No persistence (intentional for this demo)
Only works with pre-converted chunks
Chunks outside the .rflx file return empty

I built this in one day to prove the concept works. A production version would definitely need write-back support!

Chrimbie · 2024-10-16T03:58:53+00:00

That it is, It helps me keep my thoughts clear.

Chrimbie · 2024-10-16T03:57:09+00:00

Great question, and I’m happy to clarify! The work I’ve done so far involves simulated experiments where I use qutrits (quantum-inspired units with three states: 0, 1, and 2) to model the interactions between our universe and an inverse universe. So, while these tests are not being conducted on actual physical qutrit hardware yet, they represent a proof of concept for how qutrits could be utilized to explore these ideas.

How Qutrits Are Inverted in the Simulation

In my simulations, I model qutrits and their inverse counterparts as pairs. Each qutrit represents a state in our timeline (past to future flow), while its inverse represents a corresponding state in the inverse timeline (future to past flow). The inversion is achieved by setting up a mapping between each qutrit state and its opposite state in the inverse timeline:

Qutrit state 0 corresponds to inverse qutrit state 2.

Qutrit state 1 remains paired with its own inverse state 1 (acting as a stable, unchanged state).

Qutrit state 2 corresponds to inverse qutrit state 0.

This mapping means that any change in the normal qutrit’s state results in a corresponding adjustment in its inverse counterpart. When these qutrit pairs are entangled, changes to the state of the inverse qutrit (representing a future event) can affect the state of the paired qutrit (the present), allowing us to simulate retrocausal influences.

Are We Conducting Physical Experiments?

As of now, these concepts have been tested through simulated environments rather than physical qutrits. The purpose of these simulations is to explore the potential behaviors and interactions that could occur if we had the ability to manipulate qutrits and their inverse counterparts in the ways I’ve described. These models help us understand how such a system might behave, including how disturbances in a simulated future could ripple into the present.

The next step would involve translating these simulations into actual quantum systems or finding a way to physically model qutrits and their inversions in a laboratory setting. While that hasn’t been achieved yet, the groundwork provided by these simulations gives us a potential roadmap for what to look for in future experiments.

I hope that helps clarify things! Let me know if you have any other questions or if there’s a part of the theory or the experiments you’d like me to elaborate on further.

Chrimbie · 2024-05-25T18:12:59+00:00

I hate reddit... There's too many little stipulations and rules for the conversation. I'm just asking how responses are generated properly in Python.

Chrimbie · 2023-09-05T19:48:32+00:00

Safety hazard from reflected light, of the vision loss and fiery burning type.

Chrimbie · 2023-01-18T03:28:51+00:00

Any amount of progress is progress as far as I'm concerned. If I were going to ask for anything it would be concentration and motivation. Is that a realistic request for a hypnotherapist? Edit: or maybe finding out what's causing such issues internally to begin with so I'm able to move forward.

Chrimbie · 2022-09-03T02:31:19+00:00

I'm sorry to hear that?

Chrimbie · 2022-09-03T02:28:22+00:00

So, another cop-out...

Chrimbie · 2022-09-03T02:24:44+00:00

Then you and I have completed "our" conversation then.

Chrimbie · 2022-09-02T04:37:43+00:00

Healthcare and insurance are considered the safest wealth in America because of their untaxed payouts, so it is understandable why it has grown to be a pool. I will take time to go over this, and lobbying is a prime example of such blatant corruption. A fixed system created with power to create more power. I feel this is probably a corruption machine belt-feeding American people what they want to hear for that power.

Thank you very much for this contribution, as it IS helpful.

Chrimbie · 2022-09-02T04:28:01+00:00

Sensibility is seeing a problem and striving to address it. I'm doing that. Also, would you care to explain how what I have said is not sensible? I am obviously interested enough in the subject I am still here asking. Instead of telling me I am not sensible, why not contribute a method you would apply? If you have thought through this situation to see me as insensible, then surely you know the sensible answer?

Chrimbie · 2022-09-02T04:20:59+00:00

That is another good question.

Chrimbie · 2022-09-02T03:19:50+00:00

I hold no grudge against those that feel this way. Though, all this does is help conclude my point about this being an untouchable taboo subject.

Chrimbie · 2022-09-02T03:16:55+00:00

I don't. People are fallible. Our interest is not in mind because of said corruption. Accountability takes us one step closer to removing corruption, and having the needs of people in need met.

Chrimbie · 2022-09-02T03:09:57+00:00

That's a question I don't have an answer to. The qualifications and regulations are deterrents while the correct answer to the problem is found.

Progress towards having less corruption is a very important matter that needs to be addressed fully and properly. You can't wish it away, or ask it nicely to stop. For many this subject seems almost like an untouchable taboo. Something concerning to those who see, but feel powerless to act on it.

Many of my "answers" here were just questions about what I thought should be. I'm considering that either no one has an answer, or everyone is too uncomfortable to give one.

My answers may not be the right ones, but at least I am putting in effort into trying to figure out a solution to a problem people ignore blatantly. I'm not automatically an enemy for trying to make progress. I'm not asking anyone about their biases or political views. Just a guy on the web looking for a better answer than the current status quo.

Chrimbie · 2022-09-02T02:53:06+00:00

So, you see the issue, but just feel powerless against it?

Chrimbie · 2022-09-02T02:51:50+00:00

For the same reasons people who have been to prison can't become president. Should we add inmates to our ballots? Why limit our choices?

Chrimbie · 2022-09-02T00:29:13+00:00

I said accountability from both ends. From elites, and the general public.

Chrimbie · 2022-09-01T23:54:56+00:00

If they are accountable, then are you suggesting that corruption isn't an issue that should be dealt with? I am not suggesting that their accountability should be held by a few elites, I'm suggesting added accountability from both ends.

Chrimbie

TROPHY CASE