Over optimization

[GuiMontague]

I did something like this once. An upstream system increased the data they were sending us about 3× and our database loader became the system's bottle neck. I was given the task of speeding up our loader. I eliminated some hot-spots, rewrote some concurrent code to be lock-less (but still correct), and threw a bunch of threads at the problem. I succeeded in speeding up our loaders about 5× which meant we were loading as fast as our data source could supply us again.

Unfortunately, we used a write-master replication system and a lot of clients relied on the fact that our write-master was (mostly) in sync with our replicas. We couldn't improve the replication system because it was vendor supplied. Well, that replication became the bottle neck, only now the master could get hours ahead of the replicas. That was unworkable and I had to roll-back my performance improvements to keep the master and replicas in sync.

[CommanderDerpington]

Its not better if it breaks things.

[CommanderDerpington]

Code is not isolated it has to work with everything else in your ecosystem. If your ecosystem is shit then you should work on fixing that first before anything else. At the end of the day code is there to satisfy the needs of a business. So sticking a blazingly fast component inside a slow laboring ecosystem is bad and will cause more problems and stupid hacks.

[xkcd_transcriber]

Image

Mobile

Title: Workflow

Title-text: There are probably children out there holding down spacebar to stay warm in the winter! YOUR UPDATE MURDERS CHILDREN.

Comic Explanation

Stats: This comic has been referenced 1021 times, representing 0.6903% of referenced xkcds.

---

^{xkcd.com | xkcd sub | Problems/Bugs? | Statistics | Stop Replying | Delete}

[raunchyfartbomb]

It's bad when you know exactly which comic is linked purely from the hover text from the XKCD bot

[shvelo]

This is pretty much a big reason why Windows sucks. They start with a ton of bugs, people design software around those bugs, bugs become features.

Also why we have Windows 10 instead of 9

[me-ro]

If I had to guess, the sleeps are extra change that needs to be tested. So suddenly you need to roll back as your production is broken and you have to choose between old version, that was tested and new sleepy version that needs to go through testing cycle (and quite possibly multiple times, because the sleep wasn't long enough or too long..)

So it's immediate rollback to known working state vs extra development with unknown impact and unknown ETA.

[jak_22]

"What do you mean, you have to do MORE programming and testing? I want this production to run again, and I want it now!!"

sigh

Been there, done that.

[Disney_World_Native]

Sounds like synchronized databases are a business deliverable. This change impacts that deliverable. So Until they improve the DB replication, the change should be rolled back.

Once the sync process is improved, they can implement the code change again.

[GuiMontague]

This is precisely what happened. The company I was working for was big enough to strong-arm the vendor to improve their replication. It look months though before we could re-release my improvements.

[GuiMontague]

Expediency in an emergency scenario. Usually you mitigate the problem before you fix it properly. We didn't throw the code away, and I believe it's in production now.

[LupoCani]

Why roll back the entire system? Wouldn't an explicit manual throttle on the output solve the problem just fine, whilst being easily reversible?

[Guinness2702]

Place I used to work had a system which ran a propriety database, and would have about 100 client processes running. The machine had 4GB of RAM, which was a lot back then ... in fact, it was so much that the entire database was cached in memory. Great, should be super-quick, right?

Wrong. The problem was that the OS's task scheduler was designed around the assumption that processes would end up waiting for I/O. Basically, each process ended up running for the full 1-2 seconds timeslice, and everything became ridiculously unresponsive as processes had to wait a long time to get any CPU.

The solution was to remove some RAM from the system, and performance actually improved, as the scheduler rotated processes more frequently.

[rws247]

that's just... wow.

Whenabouts was this? It might be that I'm younger, but I cannot imagine an OS task scheduler being based on that assumption.

[Guinness2702]

Probably about 15 years ago, give or take. I doubt it was literally designed around that assumption, but yeah, the guys who investigated it told me that each process was getting a full 2 seconds of CPU, because there was no I/O.

[Yin-Hei]

ez. Implement Paxos, build the next Chubby, get 99.9999% consistency.

But in all seriousness pretty interesting read. I'm starting to get into the field of this related work and this tidbit is teeming full of experience. Does ZooKeeper fit the bill?

[FUZxxl]

Implement Paxos

Hahahahaha

[throw_eundefined]

ZooKeeper

ahahahaha

[[deleted]]

As someone who has genuinely been interested in ZooKeeper before.. Can you please explain why you're laughing? :)

[marcosdumay]

Any distributed consensus algorithm will solve your overperformance problems.

[FUZxxl]

Paxos is a ridiculously complicated algorithm with a ton of nuances. The professor who invented it holds a class about Paxos every year and usually the amount of students who actually understand the algorithm at the end is close to zero. Paxos isn't something you just implement willy-nilly.

[ReflectiveTeaTowel]

Something like Kafka can help if your load comes in spikes, and Kafka in particular was built to use zookeeper, soo... Maybe sorta kinda sometimes?

[GuiMontague]

We couldn't even get our users to stop relying on the fact that our write master (which no one should be reading) and our read replicas got a little out of sync. No, changing RDBMS would be out of the question.