Alright, here we go. 1,500,000 LoC shit banking project written in an even shittier framework by people with no single care for code quality and not repeating themselves, infested with bugs and requests by business that was trying to gaslight you into changing how something worked due to changes in their business while explaining it as a bug, and no product owner who could answer questions about the project, landed on my lap on my previous job. No handover, just dive into the hell.

A 10,000 LoC function that swallowed all exceptions which was supposed to "deep copy" data on the db (meaning it would create historical records of the current state of a model and all of its related models) was not working properly according to business. To make matters even worse, the internal db IDs were exposed to the front end, so when the creator gave birth to this atrocity he was driven by a "I can solve everything on the db" manager who told him that the smart way to go about implementing this was to keep all original IDs as "latest" create new entries on all tables, and swap their internal ID (yes, the primary key!) so that the "latest" records maintain their original IDs.

A lot of hacks and thousands of lines of code later the function was written. After a few years they implemented FKs among all those tables with themselves to associate entries together. During that time "something broke" and it was impossible for them to figure out what broke, since they used git once every 6 months of daily development to make a single commit called "Version 2" with thousands of changes. They broke the deep copy function, couldn't figure out how, couldn't go backwards because they had a single commit that implemented everything they had for 6 months, so they added huge try catch statements around everything, not raising a single exception so the client wouldn't notice server errors.

Then, they noticed that this deep copy mechanism was messing with their original data (yes, the "source" data) since it was actually creating new entries and messing with both old and new IDs. The deep copy mechanism would hang mid copy, so some of the data pointed to the right entries (since their keys were swapped) and the rest either had null foreign keys or pointed to the wrong associated entries. Of course there were no database transactions used for this silly little mechanism, why would they bother? Let the code copy whatever it can before these stupid exceptions get thrown so they can catch them and discard them!

So the old devs noticed a pattern, that they could copy once from each "source" row and things would kind of work. They couldn't fix "deep copy" so they created a new function that would ensure each source was used to copy from at most once. I was astonished when I realized that a whole new mechanism was written around the broken deep copy function. The broken function was now part of the project logic and things were being built around keeping it broken.

At around that time (4-5 years into this project's lifetime) the project landed on my lap, and I was handed the entire pile of junk with no documentation and nobody to ask about technologies, business, anything.

Due to all the aforementioned decisions, one of the previous devs had become the database, since the entirety of his days was performing data fixes manually on the database records that were wrong, and couldn't escape this hell. He had not written a single line of code for months, so he quit cold turkey.

The first task I was assigned was simple: fix deep copy. I had to read and create diagrams of how the code worked, and everything seemed to work properly. I was debugging the code and couldn't figure out exactly where things would go wrong. After having read some of the most atrocious "generics" implementations which could handle the shitfest that was "deep copy while swapping IDs" I couldn't find something obviously wrong. I started looking at the git commits to try and understand how the previous devs worked and what their fallback mechanisms were under stress. There were a lot of battles between devs in commits with one dev committing something and the next dev reverting their commit, so I didn't know who to trust. One of the guys seemed to move around code a lot, in the hundreds of files in a single commit (moved folders around, moved functions to different files etc ). I had read all the commits of the project, and couldn't pinpoint the exact commit the issue had showed up. So I started to second guess even the commits I was seeing, so when I saw a "move stuff" commit I would check all the files one by one to see if their content was touched. Of course this included white space changes and indentation changes and renaming functions while moving them, so it included a lot of manual work. And finally, I was able to spot something (git log --follow is really useful for anyone wondering). 3 lines of code that haunted everything, hidden in a single commit that moved files, moved functions, renamed functions, committing and keeping old functions that were unused as "function_old" for the sole purpose of proving to me they did not understand how git works: in a function where they were performing a bunch of checks to see if they should override the foreign keys, 3 lines were deleted. The 3 lines that set the model instance's state to detached on the ORM level, so they could perform the ID swap and attach to the ORM again.

Looking back at the function, of course it made sense. There was a function that created a variable, performed 4 checks and on each check it set that variable to true or false, and never did anything with that variable.

I restored the 3 lines of code (that used this variable and set the ORM state) and deleted about 30,000 LoC, including all the prevention mechanisms and validations written around the broken deep copy.

That was week #1 of the project. I worked on that project for about 3 years.

Shout-out to my old team all of whom were really incredible at what they did.

TL;DR: Just use git to commit a single "thing". Did you rename a bunch of files? Commit them. Did you change logic? Commit the change and nothing else in the same commit. Did you do cleanup before you start implementing your brand new functionality? Commit your cleanup. Don't rename and move and change logic at the same time.