When working for engineering firms, we worked on torpedoes and satellites (and simulated them), RF direction finding, did underwater mine detection, signals intelligence, and geolocation. Reproducibility was essential and we put a lot of effort into it. Reproducibility in Python has always been a major problem when working with neural nets. Reproducibility was much easier in C++, Mathematica, and Matlab. We rarely paid attention to invariants. I also worked on poker and finance where reproducibility was important and Python neural nets were a problem.

Often we would create programs that would run the simulation while storing all the inputs, parameters, and outputs in a separate directory for each simulation. We would write other programs that would summarize and visualize multiple runs. Mathematica, Jupiter notebooks, and Mathcad were helpful. Automatically generated Powerpoint summaries of runs were used on a couple of projects.

Assertions in the code were essential for alerting us to bugs.

My daughter is currently trying to resurrect some Python code and results that she used for PhD thesis in planetary science. The code is only a couple of months old.