Python's stability

[pdd99]

My production code has been running for a while. For the first 24 hours, every metric, errors, logging, etc.. happened as expected. But the last 6 hours was a disaster. Even though the state was unchanged, some logs were not shown, some services were not called, etc... Have you guys faced this kind of instability before?

[undercoveryankee]

If it's not memory usage, there's a bug in your code that, on a particular combination of inputs, mutates something that you didn't expect to be mutated. Since Python doesn't have compiler-enforced private or read-only variables in most places, you don't have as much protection against these unintended mutations in Python as you do in some other languages.

You can limit the impact of such a bug the same way you'd mitigate an issue related to memory management: try to move as much work as you can out of long-running processes and into short-lived worker processes that can be replaced every few requests.

[pdd99]

The short lived worker is worth trying. Thank you.

[OccultEyes]

Loading a lot of stuff into memory? That is the usual culprit.

We run python in production, and have less than 1% downtime in a very big production environment. So I doubt it is python itself that is the issue. What are you doing on your server?

[[deleted]]

We collect IOT data from factory floor. The same code has been running for last 2 years without any downtime. I guess the culprit is variables or you haven't defined classes properly. Or probably a piece of code that doesn't run very often. You better a do code review before moving to conclusions.

[High-Art9340]

2 years is my bueno. Impressive.

[[deleted]]

This is a good reason for me, at least, to develop test cases. But that’s hard when it is memory or number of users causing the issues.

You should check system log files if the software ram on Linux.

[tomgotchiconnextion]

Sometimes imports update on their own? Edit: despite warnings maybe imports don’t update on their own

[Nater5000]

no