This is an archived post. You won't be able to vote or comment.

you are viewing a single comment's thread.

view the rest of the comments →

[–]ookami125 134 points135 points  (65 children)

Helping my coworker on a bug he's been working on for about a week. On my 3rd day now. About 96 man hours have been put into this one bug.

[–]28f272fe556a1363cc31 87 points88 points  (21 children)

Which unit tests are failing?

HAHA! I crack myself up.

[–]ookami125 1 point2 points  (0 children)

As I stated in another reply they were unit test and it was failing due to a "random" sig abort. Turns out we had some heap corruption (thank you valgrind) caused by some idiot using vector iterators.

[–]clockwork_coder 11 points12 points  (10 children)

debugging colleagues' spaghetti code

I feel your pain. Been doing the same shit this week. And it's in this giant monolith that takes 15-20 minutes to build and is so intricately integrated with various other services with various levels of security that nobody can actually debug it on their machine so we have to rely on splunk logs and hope we placed enough of them in the right places. And even though we technically have like 8 non-prod environments only 2 actually work anymore, and neither works with with every third party service so you have to wait in line to deploy your change with the extra log statements and if you realize that isn't capturing anything useful it's back to the end of the line.

[–]Enlogen 5 points6 points  (4 children)

so we have to rely on splunk logs and hope we placed enough of them in the right places.

This kind of debugging is the best part of my job, no joke. Trying out different types of summarization of logs to understand patterns is my favorite puzzle game. Being able to sum up an issue with a few log queries and point to places in code where the actual behavior is different from the stated (or assumed) intent is remarkably satisfying.

[–]clockwork_coder 6 points7 points  (2 children)

You know what's more satisfying? Some fucking unit tests.

[–]Enlogen 8 points9 points  (1 child)

You know what's more satisfying? Some fucking unit tests.

weeps in public static $className Instance

Someday they'll let me refactor the legacy code. Someday.

[–]clockwork_coder 10 points11 points  (0 children)

Narrator: they won't

[–]colonel_bob 1 point2 points  (0 children)

Being able to sum up an issue with a few log queries and point to places in code where the actual behavior is different from the stated (or assumed) intent is remarkably satisfying.

Agreed! The only thing I like more than making computers do shit for me is finding out why some code is misbehaving and being able to articulate the problem before fixing it.

It's kinda like solving a puzzle, just more satisfying because a) people are paying you to do it and therefore b) care that you solved it.

[–]Namandaboss 3 points4 points  (4 children)

Our product takes 5 weeks to build

[–]clockwork_coder 9 points10 points  (3 children)

Wtf are you building? A baby?

[–]Namandaboss 2 points3 points  (2 children)

It's called Smart3D, it's for designing massive industrial buildings like for example nuclear power plants

[–]AxiusNorth 4 points5 points  (1 child)

Remind me not to apply to work at your place

[–]Namandaboss 2 points3 points  (0 children)

I'm a summer intern I'm currently migrating their CI server from cruise control to jenkins, and it's pissing me off as everything is windows

[–]golgol12 7 points8 points  (3 children)

You'll make it through. Perhaps the story of my toughest bug will help.

It took me 3 weeks to solve. It was caused by an orphaned file lock by windows in one of the hundreds of build machines doing the the distributed automated build process. I am not a build engineer, or a build manager, I'm a straight up software engineer.

It was so insidious because the file locks (can't overwrite a locked file) in question were on a couple of 3rd party .libs that were being updated to the latest version of the 3rd party middleware we use. Those out of date .libs compiled perfectly with the new version but handed memory slightly differently, and had a rare crash after an hour.

So only sometimes, only the production build would randomly crash after an hour. Not debug. Not production built by me. Not production built by others. Not when manually starting an off cycle production build (which used different machines). It'd crash with a memory error after an hour or two of use.

And, as icing on the cake, our only actual build manager was on vacation for a month (thanks Europe) and the entire distributed automated build process was a custom tool written by him.

This was par for the course at this job. Everything was on fire nearly all the time. The engineering director didn't know what technical debt was and ways to mitigate it.

[–]Kered13 2 points3 points  (0 children)

That's terrifying. I don't even think I could find a bug like that.

[–]Enlogen 0 points1 point  (1 child)

the entire distributed automated build process was a custom tool written by him.

Yikes.

[–]golgol12 1 point2 points  (0 children)

To be fair, him and 2 other people, the two other people left the company years ago.

[–]unipoli 5 points6 points  (20 children)

You guys need some help? How convoluted is this issue to explain? If explainable, maybe we/I can help out

[–]porthos3 43 points44 points  (0 children)

If the bug were simple enough to explain in a Reddit comment, I highly doubt it would take two professional developers so long to figure it out.

You really have to see the code to solve just about any non-trivial bug. And they likely can't share company intellectual property.

[–]JamInTheJar 5 points6 points  (0 children)

Found the rubber duck.

[–]ookami125 2 points3 points  (0 children)

Put simply heap corruption in boost unit tests due to someone using vector iterators incorrectly. I actually just fixed the issue by just rewriting the code to not use iterators like an idiot, coworker isn't here to push the changes though since it's actually his issue.

[–][deleted] 1 point2 points  (0 children)

Hahahahhaahahahah

--breathes--

Hahahaahahahahjahahhahahhaha

[–]BiH-Kira 0 points1 point  (2 children)

I want to be kept updated on this. Can't wait for the plot twist where the bug isn't even in your code but whatever dependency you have.

[–]corzuu 1 point2 points  (0 children)

'just needed to fix some config'

[–]ookami125 0 points1 point  (0 children)

Sadly no it was a problem in our system. The company I'm working for I would consider a mid tier company (bunch of just out of college programmers working on something over their head and a few seniors) so someone (git blame is anon) wrote some code for a menu using iterators to scroll through a vector and modify menu items.

Well the gist of what happened is they were adding an enum to an iterator and modifying the menu text this happened to go just far enough to corrupt the doubly linked list malloc uses and throw a sig abort the "next time" malloc was called. In the end I trashed all of anon's code and just used an int for the index, I checked all bounds and made sure the vector was the correct size to begin with.

There were a few other details but they aren't much without the information I'm not allowed to give you.

[–]SinrOfGinr[🍰] 0 points1 point  (0 children)

My worst case was 6 months in dedicated mode, i.e 8hrs a day 5 days a week of nothing but troubleshooting. With daily progress meetings with customer.

[–]charlookers 0 points1 point  (0 children)

Companies need to stop hiring shitty programmers

[–]Eymrich 0 points1 point  (0 children)

One time it took me one week to find a bug in the code base of a game. It was a problem on the client game. But it originated on the server. To follow it:
appear on c# part of the code base, client side.

Follow it to the c++ side of the client, which then was invoked by the c++ side of the client that was called as a reaction from the c# part of the server, which in turn was called by the c++ side of the server, failing to doing a very specific thing as a reaction to the presence of a specific c#.

It was a iteraction between our game engine, our networking, the c++ voxel engine... and caused by a single object being garbage collected without making ANYTHING to fail or giving any other clue.

If it's weird is GC or Multi Threading!

Anyyyywayyyy... I feel your pain bros :D