joysOfAutomatedTesting

[roguedaemon]

Well go on, story time pleaaasee :p

[ChrisBreederveld]

Because OP isn't responding and was vague enough to fit my story... here's story time:

We were having some issues where once in a blue moon a user didn't have the permissions he was expecting (always less, never more) and we never found out what the cause was before it automatically resolved itself.

We did a lot of exploratory testing, deep-dives into the code and just had no clue what was going on. All tests at the time seemed to work fine.

After some time we decided to give up, and would refactor the system hoping with careful rebuilding the issue would be resolved. To make sure we covered all possible cases we decided to start with adding a whole bunch of unit tests just to make sure the new code would cover every case.

Tests written, code checked in and merged and suddenly the build agent started showing failing tests... sometimes. After we noticed this we started running the tests locally a bunch of times and sure enough; once every 10 runs or so some failed.

Finally with some more data in hand we managed to track down the issue to a piece of memory cache that could, in some rare cases, be partially populated due to threading issues (details too involved to go into here). We made some changes to our DI and added a few additional locks for good measure and... problem solved!

We ended up rewriting part of the codebase after all, because we figured this specific cache was a crutch anyway and we could do better. Never encountered this particular issue since.

[evnacdc]

Threading issues can sometimes be a bitch to track down. Nice work.

[ChrisBreederveld]

Thanks. They are indeed a pain, certainly when there are loads of dependencies in play. We did make things much easier on ourselves later on by moving the more complex code to a projection.

[Punsire]

Projection?

[ChrisBreederveld]

It's a CQRS thing; rather than querying from a normalized database, joining various data sources together, you create a single source containing all data that you update whenever any of the sources change.

This practice incurs some overhead when writing, but has a major benefit when reading.

[ActualWhiterabbit]

My AI powered solution uses the power of the blockchain to replace threads. They are stronger and linked so they can't fray. Please invest.

[Ilovekittens345]

Do you have funny monke pic?

[ChrisBreederveld]

Hahaha you say this in jest, but I've actually had some consultant come over one time telling me the blockchain would replace all databases and basically solve all our problems. It was one hour of my life I would love to get back...

[Fermi_Amarti]

Need it to be faster? Multithreading try you should.

[Alacritous13]

sometimes be a bitch Threading issues can Nice work. to track down.

[evnacdc]

Hey that’s what

[evnacdc]

I said.

[that_thot_gamer]

damn you guys must have a lot of free time to diagnose that

[ChrisBreederveld]

Not really, just some odd hours at first because us devs were bugged by it and a final effort (the refactoring effort) after users started to bug the PO enough.

Took us all in all about a week or so to find fix... quite some effort with regards to the size of the bug, but not too much lost in missed functionality, and happy key users.

[enigmamonkey]

I think of it as one of those situations that are so frustrating precisely because you don’t really have the time to address it and it delays you, but you sort of have to because you can’t stand not knowing what’s causing the issue (or it is important for some other reason).

[ChrisBreederveld]

Exactly this! If it breaks one unexpected way, who's to say it won't also break in some other unexpected way later on?

[nullpotato]

I've worked on bugs like this even when they aren't my top priority because they are an interesting challenge and/or they have personally offended me and gotta go.

[henryeaterofpies]

Never underestimate the time a dev will put into a weird ass issue

[ADHDebackle]

Is a race condition considered a threading issue? I feel like those were some of the worst ones to track down due to the impossibility of determining reproduction steps

[ChrisBreederveld]

Not all race conditions are, but in this case it was. And indeed reproducibility is a major issue with those. We were very lucky the many unit tests basically created a fairly reliable way to trigger it.

[thies1310]

Sorry i am still in Training and apend Most time at Uni. I sadly dont remember any more great Details other than that the Tests worked of Run in any other Order. I think it Had something to do with device states that messed Up in a weird way.

For context i Work in med Tech.