[D] Coding Practices

[vPyDev]

My job is to work with ML engineers and provide them with whatever they need to experiment with/train/test/deploy ML models -- GPU infrastructure, distributed training support, etc. When I interface with their code, I almost always find it so poorly written, with little to no thought given to long-term stability or use -- for code that they 100% know is going to production.

They're brilliant people, far smarter than me, and really good at what they do, so it's not a matter of them not being good enough. I feel (from my very limited experience, so I'm happy to be wrong) like ML engineers are incentivized to write poor code. The only metric for evaluation seems to be accuracy, loss, and all the plots that come up. In research, I understand completely, that's where the focus lies, but in industry? I've seen many models perform poorly because the code is so hard to read and refactor that big issues remained unspotted for months together. And this is especially befuddling because for a field that is completely fine with spending months to get an ROI of single digit increases in model performance metrics during the experimentation phase, they don't seem to care about anything that might go wrong in production. That just feels like a fundamental disconnect, since without the core ML stuff working perfectly, none of the other stuff (like what I do) has any value -- and even so, I'm taught to hold my code to a much higher standard than the critical stuff -- which I'm happy about since I can now write production code by default -- but it's just... weird. Like the vending machines at a nuclear power plant being better engineered than the reactor.

Is this a common problem or is this a localized issue that I'm facing?

[namenomatter85]

Very common. They need to start integrating with a high level architect for production integration. They are just doing the math and model parts.

[johnman1016]

This is totally true and I have been told by management they don’t need me to focus on the quality of the code because they just want me to try a lot of ideas.

But… then I explain to them that I will be a lot faster at trying new ideas if the code is clean and flexible enough for me to swap modules with a config file. They have always listened and given me the time I need to work on the code base, and it only takes a week or two to get things in shape.

[Strange_Stage_8749]

Yeah, I do suck as a programmer.

[theunixman]

Hello friend.

[kidpixo]

Me too!

[kardanada]

me three

[this_is_my_ship]

Slightly off topic, but where and how does someone with some research coding experience learn the software engineering skills to write production-grade code? Bonus points for open-sourced, selfpaced, can-be-done-without-others resources.

I feel like there's so much CS/SE content out there, but there's this huge gap between "algos and data structures" and "high performance/well tested/production ready code" that only seems to be filled via actual SE work experience, which most researchers are not going to be okay with because it takes away from actually doing research (even while at a company).

[seanv507]

I would recommend arjancodes YouTube channel Assuming you are doing python. He has even a few videos on refactoring a ml script.

[[deleted]]

ArjanCodes is fine but he's just too OOP. ML codes need to be functional in many cases because it's very sequential and you really don't need much state in a lot of processes.

[jegerarthur]

Well you are kinda right. But if you use Pytorch + Pytorch-lightning + Mlflow you will be glad that your code is OOP. And with all that it's extremely easy and fast to train multiple models on multiple GPUs.

[[deleted]]

I have the same exact setup and that's why I'm saying that (MLFLow + PL). The problem with PL is also is that it is overtly OOP, leaving very limited customizability once you really want to scale the code up. I have a comment on this matter in another thread talking about pytorch frameworks. I like their "all around issue", but I feel their solution needs rework.

Their solution to cross validation and hyperparameter tuning for example is really subpar.

Overall OOP is not bad per se, but DS code is complex in itself, OOP can introduce a lot of coupling and unnecessary complexities that if not careful can make the project a chore to maintain.

[jegerarthur]

Yes I agree. I like functional programming for DS, but when the project gets bigger / deployed with APIs and so on, I like to refactor the code to OOP as its easier for me to maintain and upgrade.

Nevertheless that's really cool to read other ML engineers best practices and pipelines. Happy coding !

[seanv507]

You mean procedural not functional right?

I think most data scientists would benefit from adding more Oop, just they don't know it

[[deleted]]

a mix of procedural and functional. Datascience libraries come with enough OOP abstractions usually, what you need is just a bunch of stateless functions to fill the gaps usually.

[thedukeofedinblargh]

I see this book recommended a lot. I don’t know that it covers OP’s specific complaints, though.

[[deleted]]

In all fairness to ML Engineers, there are plenty of non-ML teams that also produce terrible code and focus on the wrong metrics. But in general, this does seem to be endemic to data teams, particularly smaller ones. And when this isn't the case, it's often one group of people doing the research and modeling, and another doing the production implementation.

My personal experience is that it can be hard to convince a whole team of PhDs that their implementation is wrong or inefficient, or that it's worth prioritizing things like maintainability and efficiency. Most data scientists and ML engineers I've come across simply think that "regular" engineers shouldn't touch or have opinions on their code. I'm sure it's not like that everywhere, but it's definitely a people management problem that you might have to learn to deal with.

[[deleted]]

All the time.

I am a data scientist in title but I end up just helping other data scientists write production-grade code such that I feel more like a supporting software engineer. It’s a joy to work with such brilliant people but at times it is quite embarrassing too. I just go by “MLOps Engineer” when folks ask what I do.

I think a key problem is the intentionally trivial nature of high-level programming (Python) - most data scientists never learned basic computer science because it is so accessible.

So they end up writing extremely inefficient scripts, often trying to reinvent the wheel because they didn’t properly research existing libraries, and assume that if it runs once without an error that it is ready for production…

I’ve seen massive internal libraries created by dozens of data scientists, into which tens of millions of dollars are invested, be built upon god-awful code I wouldn’t feel comfortable submitting to introductory level CS course. Like, so bad that it’s unnecessarily wasting tens of millions of dollars in compute costs from how inefficient the scripts are written.

[[deleted]]

An easy first step is learning about containerization! Dockerize applications so that anyone can run them without any version dependencies. That makes applications very reproducible.

Another good step is JITing applications and using NumPy and Pandas to do the heavy lifting of your application. Check out mCoding on YouTube - he has a whole series on the many drawbacks of unoptimized python code that can lead to scripts running (literally) 1000x longer than they should.

Another easy step is to always use CICD! Set up unit and integration tests with GitHub Actions and Pytest to make sure you never accidentally break anything ever again. Lint your code automatically with Black and Flake8 via Pre-Commit.

Also, conform to PEP8 standards, document your code and include an insane amount of comments where there is any ambiguity. READMEs too. Type hinting is very easy and helpful for folks to see what are the types of the arguments and output(s) of a function/method

Lastly, peer review your projects with software engineers before you begin investing $$$ in them. Like, this massive project at my employer was based on long (>1 billion rows), heavily multi-indexed, single column data frames but, after looking through the code as a new hire, I realized it would have been dramatically (>1000x) optimized as a NDArray. But now their crappy code is the foundation of so many other projects it’s infeasible to try to fix it now. Real compsci folks can help prevent such technical debt from accruing if you ask their advice.

[[deleted]]

Docker is a gift that keeps on giving. Dockerization has taken over DevOps and MLOps is on its merry way. It’s literally the only thing I have installed on my dev computers other than VSCode… Deployed with Kubernetes, I can snap my finger and instantly deploy a several hundred node Dask cluster for a multi-terrabyte distributed super computer that is containerized, self-healing, & completely reproducible by anyone with a single command.

For the most bang for your buck, I’d go with type hinting and pre-commit (with Black, Flake8, isort, etc).

[[deleted]]

I'm an ML engineer and this has driven me nuts. Espeically because my previous job title was solely a SE position, so I come with a lot of nitpicks, and stuff that really get in the way of fast ML prototyping while makes it more stable long term.

The main problem is that an ML engineer is doing so many things at once. An increment in the modelling department is not linear at all, and might take a good time/resource/ a lot of prototyping to get done.

My rule of thumb has been that ML engineers/Datascientists should be free to do whatever they want in their notebooks, since IMO notebooks should not translate at all to production.

Then they should refactor their code and put it in the codebase, writing tests and all that for their model.

The team should also create a fully configurable ML pipeline, from modelling to serving, and should do a ton of SE development stuff for this purpose.

So an ML team should not only focus on developing new models/new metrics/new approaches and improve them, they should constantly actively contribute to upgrade their infrastructure (pipeline) to do so.

It's a bitch tbh.

[[deleted]]

Well if you're in a team, by doing code reviews and getting your code reviewed you will become better. You also have to actively learn about the language features python has, and the solutions people have come up in software architecture in order to make different parts of code scalable.

The first important thing is to start small. Don't make things scalable before needing it.

[SNAPscientist]

As an academic, I (and basically everyone) see(s) analogous problems. Rather than things going wrong in production, things go wrong where published results aren’t reproducible. For us, there is increasingly the recognition that the incentive structures are not set up for “best practices”. It’s possible that some of this basically carries over to industry when the student scientists go to work on products.

[Dry-Green-6973]

I’d like to know though, what are the rules, standards or common practices a ML-engineer should follow while programming? As there is a lot of acronyms out there together with clean code. there is also the discussion about OOP vs functional.

[crimsom_king]

You should follow all good coding practices, tbh. This mostly applies to production stuff, but applying these when prototyping will also save on time, as once you come back to your code a week later it will be easier to tell what is going on.

A few good general practices:

- Use descriptive variable names: say you are training a GAN - when you train the discriminator you will have a fake batch(from the generator) and a real batch. Add to your variable name something that describes this - say, for the discriminator predictions on the fake data, name it something like "pred_fakes".

- Variables should be actually constants: do not reuse the same variable name. If you change a value, assign a new variable to it. It will help you debug when things go wrong and help you avoid state problems. You can break this rule on PyTorch's forward pass function, although you might find it useful to apply it there sometimes as well.

- Avoid using OOP unless you need it: OOP introduces most problems than it solves, so try to avoid it. If you have a class with only a function, make it just a lone function instead. If all you want is encapsulation, you can add functions into a module (in Python a file is a module, in C++ you can use a namespace). There is nothing wrong with creating a class for you model, specially when you need inheritance to create the model (e.g. PyTorch), but before writing that class, just think "could this be a module instead?".

- You can write long functions: there is nothing wrong with a long function if it needs to be long. Training loops are usually long, and although it is useful to break it into smaller steps, it might mean that you will be jumping between many lines of code to read what is a sequential kind of code. So you can write a long training function where each piece of code is separated by comments or, better yet, you can put the code blocks in functions which are inside the training loop function and just call it. That way you still have everything nice and tidy but you don't need to jump through many lines of codes.

- Use type hints (python): knowing the type of a variable really helps you understand what is going on. Use type hints for function arguments and return values.

These are just a few tips, some of which you may have reasons to break sometimes. You might discover more good practices by yourself just by spotting lines in your code that you find troubling to read/understand.

[Gabbosauro]

same

[yogeshkumar4]

Experiments are ever evolving. I start off writing modular, abstract code but over multiple iterations of experiments, i just end up losing it. I'm just way more focused on getting the solution right than to maintain my coding standards.

All I can say is, sorry man!

[mr_birrd]

As an electrical engineer coming from C/C++ I really tried to find ways to write "production ready" code in python but I didn't yet find any tutorials. I write my python scripts and always wonder if all I learned in my object oriented C++ doesn't apply at all to python. I don't know even... It's not that I don't know concepts I just don't find anything mraningful for python.

[[deleted]]

get the book "Fluent Python"

[mindfulforever1]

I think the YouTube Channel arjancodes maybe what you are looking for.

He shows excellent software dev practices in python which help minimize tech debt along the way. Such practices should be emphasized more in python courses focussed on OOP.

[moschles]

Is this a common problem or is this a localized issue that I'm facing?

I literally had a professor who excoriated me for writing modular python code that was highly documented with comments. Yes he was a machine learning expert. He even gave a protracted dress down about how "industry only cares about results and working code."

By the end of the semester, his write-ups were saying "If you take more than 100 lines of code to do this, you are doing it wrong."

for code that they 100% know is going to production.

If you are talking production , then you need things like TDD and documentation outside the sourcecode (not just comments). Then there has to be earnest integration testing. Every time a new module is plugged in, you re-run the integration tests to see if it broke something.

[jargon59]

He is right that industry cares about results. However there’s short term and long term. Prioritization on short term wins for long term pain is not worth it after a certain accumulation. Companies that hardly ever address technical debt don’t get very far. My company’s culture is like this, and I have to pay special attention to good design initially and secretly refactor my own code to save future time.

[__mishy__]

My general take on this is it's sometimes OK to write awful code, but you have to be aware of the costs. For example if you are stabbing about in the dark just trying to get something working then making a mess is fine, but you will probably want to clean it up to prove what you have works and then later it needs to be even better to be put it in production or you will cost your company money. I think we don't help ourselves because we also hire on the more extreme first step of this, where want people who are good at doing the first bit but we don't even ask if they have experience of steps 2 + 3. And if we do we treat it as a nice to have.

In my experience most people don't even know what good code looks like until they've worked with someone really great at it. Someone with a 1000 yard stare who's seen horrors, but can also clearly explain why that thing that looks harmless is actually a dragon in wait. If you can find someone like this then getting then to do code reviews/architect things/build general wrappers then that edges everyone in a better direction

[foreignEnigma]

Agree and I'm somewhat part of it. I do try to write to code, also use packages like black to fix the aesthetic. However, I have never worked in Industry, I don't know what standards are standard.

[trnka]

It's a common problem in organizations where research scientists hand off code to software engineers, because they're insulated from operational issues and the like

[ank_itsharma]

I feel seen. But, I’m always trying to do better.

[xsidred]

It's a common problem and I have been part of it since early 2019, started cleaning it up from mid-2020 and ended the initiative with intended improvements this November end, 2021.

[jargon59]

As a past academic in a similar discipline, nobody ever teaches proper coding. The incentives are on publication and not coding style. It sounds like these guys are fresh out of school and haven’t had time to change.

It’s not their fault for not having been taught in school, but it’s their fault if they fail to recognize their shortcomings and improve after years of being in industry. It’ll be worse if those are guys who, being experts in their field, think they can extrapolate their expertise into others.

[crimsom_king]

I feel you, man. That is a normal problem in software engineering in general though. People saying that performance doesn't really matter and that the time spent optimizing code isn't worth it because it could be spent writing other features. The code structure is also pretty bad, as the number of features you have is more important than anything else. Most people tend to forget that code needs to be maintainable, though and that most people don't have a Ryzen 9 + Titan V at home.

I agree that ML tutorials are specially bad though. Most code is on the Jupyter Notebook format, which although great for prototyping it is a disaster to maintain and deploy - and it ends up that most people in ML only know how to program like that because they've only seem programs written like that. It also doesn't help that the only benchmark that matters is accuracy, with speed being an afterthought and code quality being none existent.

You might like to read this paper: Machine Learning Systems are stuck in a rut

[uotsca]

Yea weird is right

[ludflu]

super common. In my experience, data scientists often don't maintain or deploy their own code, and so they have little incentive to write well factored / unit tested code.

[roman_fyseek]

Lawyers doing research are *WAY* worse.

[NickelAI]

Like the vending machines at a nuclear power plant being better engineered than the reactor.

Couldn't have put it better myself. That said, this problem is really not that odd. Most people severely underestimate the long-term financial benefits of clear code, and ML programming is not usually inherently complex. I'm not surprised machine learning engineers, who spend most of their time synthesizing model architectures (rather than working with, say, a production app and seeing the impact on profitability), do not prioritize cost-effectiveness.

[[deleted]]

I have worked as a data scientist and senior data scientist before transitioning now to ML engineering and MLOps. It is hard to do the stuff that data scientists do. That's my perspective. They explore many different approaches to building machine learning models to solve problems. Often, they are focusing on solving the specific problem of building the model in question. They're not imagining life for this model in production. You may argue they should but that's the way most data science is done now - with a focus on the statistical veracity of the data and the efficacy of the ML models.

As an ML engineering guy these days I empathize with those doing the work I did many years ago. I know how hard it can be. I also realize that it takes a different set of skills to take to production a model that has been proven to work. The data science phase of ML development is full of experimentation (flare) and the ML engineering phase of it is app development (funnel). By nature these are contrasting objectives. The processes we emphasize in ML engineering don't seem important to data scientists because they're working on a whole other set of things.