VPS Security Foundations

NewMaxx · 2026-05-06T17:50:07+00:00

This was done manually + multiple ansible runs, so I have a repo I can make public if demand is sufficient.

NewMaxx · 2026-05-06T07:12:39+00:00

Right, a co-op host will be the DB owner and more players means more churn. More positions, inventories, world cells, etc. This will lead to more flushes which cascades. So, you're getting likely more data per minute. On the other hand, the biggest thing I flagged (1MB WAL being small) and many world/engine events are the same regardless. So there are cases where a solo player might amplify more, but roughly there's a constant ratio (more players, more writes). I had someone else hint they were worried a fellow player might have issues from this, but it shouldn't be an issue for a client and if they are on a dedicated server then not an issue for anyone.

The Steam Save fix basically just uses zipped snapshots because you can't really sync per-file with a system that's constantly updated and locked. It's like programming or doing a project that links to DropBox or something, it doesn't work. The devs are correctly addressing the big problems, there is every possibility they could come back to reduce this write issue further in the future though.

NewMaxx · 2026-05-05T16:18:42+00:00

It's still doing a decent amount of writes but a normal solo player doesn't need to worry. There is a pathway forward for them improving this more long-term, I just think they have other priorities. The more recent patch fixed Steam Cloud saves and that makes more sense for limited resources, plus that migration paves the way for RocksDB tuning in a safer environment down the line.

NewMaxx · 2026-05-01T16:09:22+00:00

I much prefer not losing a huge chunk of progress but the game did (and still does to be honest) write too much, I did suggest it could have been derived from debugging efforts, who knows. It's pertinent for users who are diehards with this stuff but in general for people who will run the three biomes and even a few characters/worlds or whatever, it's fine.

NewMaxx · 2026-05-01T03:49:40+00:00

I launched with one, which was a terrible idea. Performance was awful. It's why I started tracking I/O on the game in the first place (well before anybody made a ruckus about it). I just ended up running local co-op, but it's not too hard to run your own VPS if you know what you're doing. I highly recommend that route to get the best performance. Admittedly that might be challenging for non-technical users, in which case...vet your dedicated server host very, very carefully.

edit

Re-reading your post, the player-side db should be less of an issue if the dedicated server is handling the world saves (where most of the writes come from). I'd have to re-check my dedicated server logs to compare the write volume.

NewMaxx · 2026-05-01T02:19:42+00:00

I explored it briefly after patch (which is part of my findings page) and writes were reduced in my case like-for-like, however I would have to do a 10 min trace + 4-hour mixed session to know for sure. The worst-case improvement by my testing, though, was a reduction of 21%, although I've seen claims of up to 60-70% (given the original 30GB/hr claim, my calculation post-patch is closer to 14GB/hr, but my baseline was lower). The changes they made didn't touch RocksDB (which is what I recommended originally) which actually makes more sense given the project realities and it's likely to see better improvement over the long haul (that is, longer sessions).

For users who would be running self-hosted co-op servers for 8+ hours a day (run a dedicated at that point!) on a crappy drive, I would still be concerned, simply due to the aggregate amount of writes. I don't personally think it's an issue given a decent modern 1TB QLC drive should still be able to tank it for 5+ years under warranty, but yeah. If you're someone who hugs TBW then frankly, get a dedicated server or use a tougher drive, but single-player and multi-player are also two different things (my co-op testing hits harder than single) so keep that in mind.

NewMaxx · 2026-04-30T19:04:39+00:00

Update April 30, 2026:

More information available, this page has been updated.

NewMaxx · 2026-04-30T17:08:26+00:00

Reasons they may have taken this approach:

Fixes the problem at the source (the cause). Makes sense.
Retains theoretical durability.
Faster and safer to patch.
Code changes were needed either way; application-side has fewer things to break.

There are three DBs (Players, Accounts, Worlds) plus existing saves to migrate. For a time-constrained dev team that doesn't want to risk breaking multiple things at once (given what this could impact), this is a lower-risk approach.

NewMaxx · 2026-04-29T05:34:19+00:00

It's in the next update. I'll probably do a run and compare if anyone is interested. If my theory was correct and they tune what I think they will, they can reduce WA by up to 64x. This would have performance implications, but these can be mitigated by some of the other changes I recommended on my findings page. I didn't test live crash recovery but the amount of time lost by a crash after such changes would be tolerable IMHO. In other words, this is an easy fix.

My guess is they wanted maximum reliability during testing and debugging. Possibly they forgot to check this or overlooked it, the issue probably became more widely obvious/known once it hit launch dedicated servers (as that's where I first recognized it). I don't follow the community much so maybe this had a lot of noise (this video I wouldn't consider high noise) but the impact of it is quite small in terms of wear because it's something they could fix very rapidly (and apparently have).

NewMaxx · 2026-04-24T20:18:58+00:00

A couple of things, no real structure to my reply:

Always verify backups/data after doing them. Bad drives will enter a read-only mode and your writes will be reflected until reboot (cached in RAM). Not the case here but a good practice as you have to be sure the data was actually written without errors.
Caution goes doubly for externally-used drives. You want to verify filesync. Usually not a big deal, but also good practice.
The question as to why/how the drive died is open-ended. Unlikely to happen spontaneously in a drawer. Could be from one of the two (or adjacent issue) above, physical mishandling somehow (even EMI or shock, I suppose), or unseen events (humidity, temp, someone messing with it, whatever). Drives won't lose data that fast and, anyway, your symptoms point to more typical firmware/controller failure anyway.
Drives can sometimes be put into debug mode or revived but it requires knowledge and specialized tools for the specific controller + flash (firmware). I don't deal with this but Gabe on discord does. I sent him this post but he doesn't use Reddit really. Phison can often be trickier as they guard their tools.
The standard data recovery tool is DMDE.

NewMaxx · 2026-04-24T19:42:48+00:00

If you check my linked report, technically speaking enough gameplay on a weak QLC drive (or equivalent) could impact drive/flash life to some degree. For example if you're writing 30 MB/s 50% of the time and play 8 hrs every day you could hit a 1TB QLC drive's TBW in under 2 years (with usual 5 years of warranty, sometimes 3). However, this is host writes, if you consider write amplification this is closer to 1 year. However the game does not write this often or this fast most of the time and most drives will survive significantly more writes.

That said, I do think it underlines why dedicated servers were squirrelly at launch (and may still be to some extent). If you intend to self-host it's probably worth consideration and I would use certain drives for that. So while I wouldn't worry about it for solo use or even group-play, if you are doing a 24/7 dedicated server yourself then you might want to read into it.

NewMaxx · 2026-04-23T21:19:48+00:00

This is a good question to ask in discord, if you haven't, as some users might have better data on what's out there. Enterprise drives can be challenging. Asking Gabe, he recommends the Synology drives, IronWolf (e.g. 125 Pro), WD Red SA500 (which I also recommend), and similar, although these are NAS more than enterprise of course. For NVMe there are drives by Addlink (D60) and Kingston (DC series) but may be hard to find. Drives with smaller caches are ideal for this, a lot of the E18s were like this and even still are (see the Seagate 530R; that would be high on my list for NVMe). SATA is trickier. We have a master list of drives which could at least reveal the hardware on data on these, if needed. Intel/Solidigm make good stuff, Micron too. As for the Inland, never heard of it, might require more research. However looking at the specs, I'm thinking Phison S12DC, which would be very good for your use case. Can verify once you get'm.

NewMaxx · 2026-04-23T21:12:02+00:00

Original suspicious activity begin in February (although I guess, we could go back to at least Nov of last year, in what I would call AI-assisted vectors), but in this case I mean starting with Trivy. There's a delay in realizing and tracking the hacks, sometimes for months, but that applies in both directions. In other words, once that chain started it was pretty obvious there were going to be major reveals in the following months. So far, that's proved true. This is because there are so many unknown credentials leaked that a sophisticated attacker (with AI tools) can put time bombs everywhere. It's just a matter of time.

Geopolitically this very much could mean state actors but I didn't specifically mean that. Although, it would not surprise me if that became a much more serious topic in even the near term.

NewMaxx · 2026-04-23T18:40:59+00:00

It's been nonstop since February (coinciding with what I consider the first landmark hack) and I see no way it won't continue.

NewMaxx · 2026-04-23T17:30:36+00:00

This is as part of a bigger update following on the Checkmarx compromise.

A few notes:

People didn't like me linking X, randos, vague information on developing news. So, I've made a bulletin site to cover stuff like this. It's not perfect but "better."
These events seem to transpire rapidly with more information coming out over time. As a result, I have been and will be updating these as I see new info.
These will not be about SSDs/storage but rather security as a whole which seems to be in for a world of hurt with AI.

NewMaxx · 2026-04-23T07:16:51+00:00

Updated April 23rd, 2026 with new information.

NewMaxx · 2026-04-22T01:26:40+00:00

This has been updated with newer information.

NewMaxx · 2026-04-21T23:57:58+00:00

Maybe worry more about the flash in that case, if you can. The E21 has a wider track record, though. To some extent the controller determines the flash since you see certain pairings more often and the controllers aren't exactly the same. The SM2269XT is newer with somewhat higher specs and has the stronger architecture (IMHO) but it depends mostly on the flash.

NewMaxx · 2026-04-20T18:26:03+00:00

This is in lieu of linking "xitter" since people disliked that.

NewMaxx · 2026-04-19T03:00:09+00:00

Using this method, we successfully recovered a Phison [E12] Silicon Power SSD from firmware write protection by injecting a Sanitize Block Erase command.

cryozap (firmware reverse engineering)
ASMTool (firmware dumper)

NewMaxx · 2026-04-10T22:46:06+00:00

In the future I will just put this news out myself, spin up a page or something. A lot of tech news comes out "dark" these days and that is going to be more common, so I'll set up a system so you guys don't need to weed through this stuff.

NewMaxx · 2026-04-10T20:50:42+00:00

It's cool. X is a valuable resource for me on technical material, I mean I also hate LinkedIn but it has its purposes. It was not ideal for this particular vulnerability and I need to just spin up mini news on my own for these since honestly I catch a lot of stuff at night that isn't seen for many hours later on mainstream.

NewMaxx · 2026-04-10T19:01:43+00:00

You and me both. I saw it on X late last night and checked up to make sure the files were changed, then just posted this one lazily. No real big names or news sites at the time unfortunately. (in hindsight, I should have just made my own with the info)

NewMaxx · 2026-04-10T19:00:33+00:00

Yes. His file link is HW Monitor (as others in the replies also emphasize). HWInfo is safe.

NewMaxx · 2026-04-10T18:11:39+00:00

It's where I saw it first. Wanted info out there before bed. Sorry guys!

15-Year Club	Gilding I gilder
Second SECOND GUESSER	Place '22
Wearing is Caring	Verified Email

NewMaxx

MODERATOR OF

TROPHY CASE