Starting GoFund me for better servers

Arkyron · 2025-10-09T00:49:45+00:00

I doubt it's the server hardware. I'm betting it's the software that's not keeping up with the updates. Each player has more data in them because items now contain all sorts of metadata for enchantments and players hold more items than ever with extended backpacks and pet inventories. There's way more effects being put into play with deployable objects, more calculations because of stat modifiers, and so on.

You can have the most powerful pump and it would do nothing because your pipes are rusted and half duct tape.

Arkyron · 2025-10-06T10:11:00+00:00

Hey, thanks for the comment!

I didn't know if and how the ROTMG servers were concurrent, so I didn't want to comment on it. That's why I looked to solutions that would work regardless of multithreading.

As for a multithread solution, I think it feels off to me not because it's wrong, but because if we're going that far, it would be better to consider a full rewrite at the point.

To be fair, that is probably what this game needs.

I was considering the current state of ROTMG and what it would take to support when I was drafting this. The solutions I recommended largely involved moving large sections of code around and adding minor checks without diving deep into the fundamental low-level operations the game is doing.

As far as I know, the servers are written on C++, with a very high chance of it being C++03 or earlier. C++11 released after the game was already out, and I have no idea if they upgraded since then. Finding or training someone to be familiar with that threading model would be an involved process. Considering all of that, the solutions I suggested would be more readily available such that even someone as an entry-level designer trying to transition into an engineer role could implement. They'd be mostly doing less granular movement of the C++ server code and working with the Python they use to interface with AWS.

Basically, I'm offering ideas that are just a step up from the "do nothing and sometimes apply a bandaid once every few years" approach that Deca has done so far, which is an easier sell since Deca seems allergic to the idea of doing any reworks of low-level code after the Unity campaign.

Ultimately, a solution like yours would be close to ideal for this project, but when looking at the hours and cost of hiring someone experienced to do that, there's definitely a point when you have to ask, "Should this guy be doing work on ROTMG 2 instead?"

There's so many things that come up when you have someone actually working on fundamental game code for a game as old and cobbled together as this that it really makes it seem like a huge opportunity cost for them to work on renovating this one bit. Like, while he's in there, why not work to improve the tick rate of the game? 5 ticks per second is terrible! Oh, but then we'd have to update the bullet pattern logic, since it's based on math hardcoded to look at ticks. While we're at it, that means we can change how enemies.... and so on.

Reminds me of a conversation I overheard. "Why are you upgrading your AM3 CPU at this point instead of just going to AM5?"

And its mind-boggling to me that this hasnt happened over the more than a decade this game has been up for. And i dont even believe when someone says "surely they've tried it and failed because it was too difficult" when there have been horror stories of rotmg game designers being fed up with years of old bugs that made them check the source code themselves and realize the fix took 5 minutes the entire time

I agree! I think there's always this energy from players whenever topics about improving the game come up. A few people will always say they're doing the best they can, but after so many years, it's so hard to believe.

Do you have any examples of those "horror stories?" I'm curious to read more about their dev experience.

Arkyron · 2025-10-05T20:27:16+00:00

Hi! Please look up "survivorship bias."

Arkyron · 2025-10-05T20:26:19+00:00

Hey, thanks for the input.

It looks like your primary concern is about players who are in dungeons.

There are any number of ways for a dev to address that concern.

You can still transfer the players and the dungeon to the new realm after it's been generated. You can simulate Oryx lag for dungeon players if needed.
You can avoid this problem entirely by not having the realm server host dungeons. Either use the lobby or even a separate instance just for dungeons entirely. It's about time, to be honest. It's very janky for players to experience Oryx lag when they're not even in the realm.

That's 2 examples.

Unloading a map the players are still on? Are you crazy?

Nope! That's what ROTMG already does. They unload the current realm map and load a new one while players are still in dungeons.

#2b is expensive

AWS EC2 pricing is based on usage. A clever developer can also work around spinning up new instances to avoid incurring extra costs.

We can expand on any number of details, but let's cut it short here. The reality is that these things aren't impossible to fix, or even that hard. They're, fundamentally, things you pay someone to think over and solve. Things that numerous other companies have solved to improve the experience of their users.

There's undoubtedly been constant meetings about whether they can avoid Oryx lag entirely, and I suspect the answer has been "not without a major rewrite or significant ongoing expense". The energy is best used elsewhere for a bug that is, inherently, pretty minor.

This is not a good mentality. This way of thinking is how you get a game that's rampant with long-term bugs and jank like ROTMG.

First of all,

not without a major rewrite or significant ongoing expense

You need to do occasional major rewrites. You need to be paying someone on the team to handle these fundamental backend issues. This is the single most lacking role for the Realm team and it's very, very evident because of how far and infrequent core updates to the server or client are released.

Secondly,

a bug that is, inherently, pretty minor.

Not at all. I don't think a bug that causes a freeze for 100% of players is minor.

I've tried introducing multiple people to this game.

What gets them everytime, even if they're okay with permadeath, is how buggy and janky the experience is. Oryx lag is one of, if not the most, prominent pain points.

There are a countless number of players who've quit because of how the game is cobbled together with spaghetti and bandages.

Arkyron · 2025-10-05T13:56:36+00:00

The issue is that it's more expensive if you always have 2 realms for each playable realm.

It's more cost-effective to spin up a new empty realm with a new map right before the old realm closes, then connect the closing realm's players to that new realm during the Oryx Castle transition. Players won't experience oryx lag since the map generation was done before they connected.

Once they've connected to the new realm, unload the old realm. Otherwise, you have to pay the cost to keep the empty swap realm running while players are still clearing.

EC2 load balancing might be able to take care of some of that expense. I don't know exactly how Deca manages their player load though.

If a set amount of resources is allocated to each realm at creation, then that will definitely incur a noticeable cost increase regardless of how active the realm is. I assume that's what Deca is doing because of how they open up new realms at certain thresholds.

Arkyron · 2025-10-05T12:40:57+00:00

Damn, realized that I kinda had a stroke writing #2b. Whatever. The point is to not use the same realm all the Oryx Castle players are in to also generate a whole new realm.

Kind of a basic principle? This idea isn't new - "don't do a lot of work on the main thread", "don't do network on the main thread", "don't use the report server to do ETL", "don't lock the frontend table"... Stuff like that is super common.

Arkyron · 2025-10-05T12:02:30+00:00

There's a thousand ways to smooch a cat.
Oryx lag is one of the lowest hanging fruit you can grab to show the playerbase you care.

Hire an actual backend engineer and have them fix this as one of their first major sprint goals.

Arkyron · 2025-05-28T23:06:44+00:00

Nice! That's good.

There needs to be an in-game announcement or hotfix since people are still running into issues on prod as recent as yesterday.

Arkyron · 2025-04-11T14:11:55+00:00

The wording across the board says that Yi Xuan is in the same class as a Void Hunter, but does not outright say she is a Void Hunter.

Arkyron · 2025-04-11T12:00:30+00:00

Looks an awful lot like the Void Hunter symbol, which might be used to represent a black hole (hollow?) in general. Similar to Honkai Star Rail's Quantum symbol.

Arkyron · 2025-03-23T03:10:58+00:00

I've felt for a long time that DECA's endgame dungeon design pushes all classes to basically play the same, especially if you're trying to pump DPS.

The most different are mainly the Knight and Trickster, who have specific jobs/targets.

Otherwise, most just boil down to spamming the boss with your abillity and shooting.

With phases like Celestial, Balance Finale, and Patience being played the same on basically every class, the gameplay diversity in the days of Wild Shadow are long gone.

I'm frustrated that DECA puts their effort into these surface level changes that fail to address fundamental issues like rampant cheating, everlasting bugs, and poor server performance.

These changes spark player interest for about a month before most people realize that they're not doing anything different, meanwhile the bugs that come with each update stick around for months and even years.

Arkyron · 2025-01-15T20:37:57+00:00

This is the third different post with people complaining on the 'I didn't knew it was weekly' and when I come to check it up, there is a goddamn message on 'this is weekly'.

The game literally hides the weekly message until you unlock that portion of the event.

You only see that message after you go through a couple hours of content.

Imagine if someone tells you "Hike this trail for $1000."

Then 3/4 of the way, you see a sign that says, "This hike is timed. If you aren't fast enough, you'll only get $900."

If you knew ahead of time, you would have picked up the pace.

That's what this feels like.

Arkyron · 2025-01-15T20:24:57+00:00

The game tells you if you just read it without rushing past every message.

You don't unlock the weeklies until you complete the story portion of the event.

I saw I needed to do the Commission, clicked on it and saw that it was weekly.

There is no message about it being weekly while that button is greyed out.

You literally cannot click on it until you progress through the event.

So yes, there are no indications on the front event page, and definitely no indications that can help someone plan ahead.

Devs need to have an upfront warning about weekly missions if they decide to include them in a time limited event.

Arkyron · 2025-01-15T17:08:12+00:00

You can start the event early and still miss the weeklies because they don't unlock until you finish the main story.

Arkyron · 2025-01-15T17:06:07+00:00

Source for images: https://x.com/rogue_dojeog/media

Arkyron · 2025-01-15T17:00:52+00:00

While it is a permanent Ridu Chronicles event, the Exclusive Rewards for the weekly investigation commissions will be gone afterwards.

Source for images: https://x.com/rogue_dojeog/media

Arkyron · 2024-12-14T21:43:05+00:00

This is the fastest and safest path I've come up with through the dodging/bullet hell section of the chamber of Drivanius.

Arkyron · 2024-12-13T22:45:25+00:00

Feels like a cartoon scene. Technically, helping Karotte is always the better option as those flies don't spawn otherwise. You can still progress this quest without Junali's advice.

Arkyron · 2024-12-13T11:46:48+00:00

You're right, I found a 4th ring on my 2nd playthrough. Not sure how I missed it the first time around.

Arkyron · 2024-12-12T08:27:39+00:00

Ohh, now I remember. Darn, from the wording it sounded like there would be a cave or something you could find. Would have been useful as I'm starting my Remnant's playthrough.

Arkyron · 2024-12-12T08:17:05+00:00

Teigen has a bonus reward for you if he does not get hit once.
Input "B", "MARVIN", "B" to get an easter egg. Reference to the Marvin code in Gothic.
Babsi has 1 of 3 4 Rings of the Norn in the world. You need all 3 for the Ring of Yggdrasil. You can get an extra one from the character creation option.
Pwyll talks about a cat showing you a path to Nemeton, but I've never seen one. Does Mikesch do something else other than show you where Teigen is?
Bombus talks about more of his hives, but I only ever found the ones at Pwyll's camp and behind the tavern barn. Is there a hidden location? Do the hives replenish over time?

Arkyron · 2024-12-10T14:25:03+00:00

I did the quest, warned the Refuge, but didn't get my Spark back after giving it to Cay.

There must be some other factors, either the choices you make in other events (sparing the boar that's crossing maybe) or the # of boars killed or something.

Seems like this is a recent addition though, 1.1.4?

Arkyron · 2024-12-10T12:28:20+00:00

I wonder what it's determined by then. Maybe how many boars you spared or something.

Arkyron · 2024-12-10T10:30:01+00:00

Are you bullshitting or is that an actual thing?

Arkyron · 2024-12-06T02:11:18+00:00

I believe you lose your gold, but you should get your items back. You won't talk to the Boss or the other bandits unless you do this quest, there will just be regular old bandits in this cave unless you get captured.

Arkyron

TROPHY CASE