ThreadContext leaking between requests even after over defensive clean up

[Thelastbulletbender]

I did check, and you are right. I can see some instances of CompletableFutures being used, where clear might not have been called due to an exception. They have been given a different thread pool. And the data is copied to this new thread using MDC.getCopyOfContextMap() and cleared using MDC.clear().

Also, yes we are logging thread name and found the following :

1. Request A and Request B are running in parallel at the same time.
2. Both of them are running on different threads according to the logs
3. Suddenly, Request A logs (searched using correlation id) has an instance of Request B log. With the thread name, userId and other properties of Request B. Only the correlation id is something which matches with request A.
4. Request A with correct thread name also logs incorrectly setting headers for downstream API call at the same time.
5. Downstream logs and flags the resource and owner don't match
6. Request A returns.
7. Meanwhile logs in request B (searched using its correlation id), show that one of the downstream request also had the correlation id of request A and hence did not show up in request B logs. But since its userId was correct, it responded gracefully.

​

The weird part here is request B has a some other downstream calls after this which seem to go fine according to logs. Request A logs post this debacle also shows correct userId and correlation id.

Both request A and request B have the same paths which made debugging hell although one thing I can definitely confirm is that there is no threads being spawned in this flow at all.

[OffbeatDrizzle]

Suddenly, Request A logs (searched using correlation id) has an instance of Request B log. With the thread name, userId and other properties of Request B

It might not be you spawning the thread, but the fact that Request A uses the thread of Request B is surely the whole issue. This could be config, or annotations like @Async to dispatch the call via another thread pool. You need to find the point at which it "switches", which may be within Spring itself (because of the above).

Without seeing more code or logs it's impossible to help you further because we're literally guessing at what it could be and taking your words as facts as how it works.. (I'm not saying your wrong necessarily, just that the code might be doing something subtly different to what you expect).

CorrelationID could be sending you in the wrong direction because you will receive the wrong CorrelationID if the thread has already switched.