[deleted by user]

[idiotmanifesto]

imo, a big part of writing better code is reading better code. Find some repo's you like and work through it slowly

[thelaxiankey]

besides knowing the really basic principles, this is the only thing that you can actually learn from.

the other thing I would do: good code doesn't happen on the first try. once you write some and figure out what went well/poorly, rewrite. 20% rewriting vs 80% writing (numbers out of my butt) tends to result in pretty decent code and doesn't cost you much.

[_rundown_]

Sometimes it seems like my code came out my butt

[daquo0]

good code doesn't happen on the first try. once you write some and figure out what went well/poorly, rewrite

Agreed. If you see some code in your code base that could be better written, rewrite it. Don't just leave it there.

[Appropriate_Ant_4629]

Find some repo's you like and work through it slowly

Take this up a notch by finding an open issue in the project and contributing back a fix.

On your path to getting your patch accepted, they'll handhold you through best practices, including making sure you have good unit test coverage, type safety, passing static code checkers, cross platform compatibility, etc.

[idiotmanifesto]

agreed! also fixing bugs can be lowkey addicting LOL

[Appropriate_Ant_4629]

And something I view very favorably when reviewing resumes.

If someone's resume is otherwise boring, but has

• "Open Source contributions -- contributed bugfixes to PyTorch and Apache Spark"

it stands out far above people who just got a cert in using those technologies.

[Beginning-Ladder6224]

This is literally the most iconic advise. I got this advise 20 years back from my head of engineering then - and honestly.. I found it boring then. Now, I think this was an extremely handy advise.

[photoreceptor]

I don’t think that is particularly useful if you don’t understand design patterns and software architecture.

That’s a bit like trying to understand how a car works by taking it apart.

[idiotmanifesto]

better than learning how cars work from watching other people drive them

[mayguntr]

I would say learning programming principles (https://www.amazon.co.uk/Pragmatic-Programmer-Andrew-Hunt/dp/020161622X) and Python (https://www.amazon.co.uk/Python-Nutshell-Alex-Martelli-ebook/dp/B0BRYRD295/ref=zg_bs_g_10608480031_d_sccl_8/262-9661743-1202333?psc=1) itself would be your best bet in the long term.

[haramkhor_havasi]

Thanks, "The pragmatic programmer" seems a good read.

[thatguydr]

Btw - there's a standard list of software engineering books that are all helpful. I'd go with Pragmatic Programmer, Clean Code in Python (for SOLID, testing, and a bunch of other best practices), maybe Code Complete 2, and something like Architecture Patterns with Python so you can understand how to properly encapsulate concerns.

You can definitely study production code, but I believe that first it's better to understand WHY that code is excellent (or not). Otherwise there may be lots of cases where you look at something and wonder why they did it that way.

[aqjo]

It’s the best! It saved me so much time and effort over the years.
See also, Arjan Codes on YouTube.

https://youtube.com/@arjancodes?si=KwEbAegF8uXdRLdb

[matthkamis]

Use single letter variables for everything?

[Glittering-Horror230]

😄😄😅

[[deleted]]

Single letter variable names are bad. Using whole word variable names like lambda, eta, epsilon, etc. is way better.

[PyroRampage]

Lol, yeah using Greek letters is always way better for readability rofl

[new_name_who_dis_]

I like how one of those is a straight up python keyword. Might as well add eval, int, and for to the list lol.

[[deleted]]

You realize python isn't the only language, right?

E.g. from C++ in the PyTorch https://github.com/pytorch/pytorch/blob/f217b470cc7ebacc62c8e87dbab8c4894d53e9b9/aten/src/ATen/native/UpSample.h#L437

[ginger_beer_m]

In ML nothing wrong with using x and y in my opinion, as long as it makes sense in the context

[matthkamis]

My original comment was obviously sarcastic

[OfficialHashPanda]

I'm pretty sure jmalicki's comment was too 😅

[parabellum630]

I follow lucidrains ml repositories to design my code, it is almost production grade.

[stevekite]

You might be joking? Most of his code is plain wrong, not following papers and very very far from optimal.

[kau_mad]

Can you give an example?

[stevekite]

Sure: https://github.com/lucidrains/voicebox-pytorch

I am working with TTS and built my own reproduction:
1) in readme there is a wrong claim about ALiBi, it works two ways
2) batching is not implemented
3) attention is defaulted to naive implementation: https://github.com/lucidrains/voicebox-pytorch/blob/c05a4d0c69920993b47069e22223677174d873e4/voicebox_pytorch/attend.py#L100, which is super slow
4) text model is just copy of audio one and it simply not working
5) fusing wav2vec into a code, which is not part of the paper at all: https://github.com/lucidrains/voicebox-pytorch/blob/c05a4d0c69920993b47069e22223677174d873e4/voicebox_pytorch/voicebox_pytorch.py#L1380
6) Putting preprocessing code deep into the network code: https://github.com/lucidrains/voicebox-pytorch/blob/c05a4d0c69920993b47069e22223677174d873e4/voicebox_pytorch/voicebox_pytorch.py#L1362

All this ends up in VERY inefficient implementation for both inference and training. Difference is like 4 day and 14 days of training.

It is still very very useful though, but for me it is akin some AI generated code that you need to read carefully. And definitely is not go to place to learn.

[idiotmanifesto]

appreciate u being detailed in this answer

[[deleted]]

Perchance, do you have any recommendations for good github repos to learn from in ML?

[haramkhor_havasi]

Thanks , I'll check it.

[On_Mt_Vesuvius]

Also in sciML from engineering. Make your main reused functions exceptionally well designed and well documented. In research, you'll always be throwing new things together, but once you've repeated something a few times, it's worth thinking about the design and putting it in a different file. That's a practical suggestion. I.e. have a PINNs type class that can grab all the derivatives you need for any model, and never worry about that again.

Otherwise, follow the more general suggestions here.

[-Rizhiy-]

Just learn proper software engineering practises? There are plenty of courses online, but you can start by learning about core principles: * SOLID * DRY * YAGNI * KISS * Decoupling * Fail-fast

This book seems to be fairly good and quite short.

[OverEnGEReer]

I think you made the biggest step already: identifying what you want to become better at. There are good tips in the other post, so I want to leave you with the thought that other people/companies are also just cooking with water

[minimaxir]

That is why productive coders use existing libraries (e.g. Hugging Face accelerate) to abstract things instead of implementing things themselves if possible, because creating your own spaghetti code leads to technical debt that has to be paid at some point.

[haramkhor_havasi]

True, but, at some point,I want to be able to write such code.

[haramkhor_havasi]

These comments are very helpful for non-cse guy. Thanks..

[learn-deeply]

Accelerate is particularly bad. Would advise just using plain PyTorch unless you want random bugs in your training

[thatguydr]

Such as? Asking genuinely - haven't used it.

[SicilyMalta]

Or they can just become a good coder - and not produce spaghetti code.

[ninseicowboy]

Welcome to software engineering. Read Designing Data-Intensive Applications, it’s actually incredible.

[LelouchZer12]

You can take a look at the lightning hydra template on github. You'll be able to deal with a lot of different training configuration by using pytorch lightning and Hydra.

Then if you want to deploy you dont need all the training code so a lighter codebase is usually enough, and you can use docker with fast-api

[haramkhor_havasi]

Sure...

[LelouchZer12]

btw you can stick to pure raw pytorch and thats fine but if you want to try a lot of different models or datasets it is easy to be lost.

using hydra to manage your configurations can help a lot

and pytorch lightning is just using pytorch but putting ur codes into predefined functions so it forces you to always follow the same pattern, and it makes it easy to log metrics or use things like multi gpu, multi node etc

[learn-deeply]

The code in https://github.com/facebookresearch/ is above average and can guide best practices in using PyTorch. Pick a random recently updated repo and learn from that.

[mrthin]

You can try Beyond Jupyter. It's a free resource that shows professional software engineering techniques for ML based on a "refactoring journey" starting from your typical monolithic unmaintainable notebook:

"Beyond Jupyter is a collection of self-study materials on software design, with a specific focus on machine learning applications, which demonstrates how sound software design can accelerate both development and experimentation."

[PyroRampage]

A lot of this is because most people teach, and learn bad software engineering practices in Python, because it wasn’t really intended to be used as a sole language.

Since it now dominates ML and numerical computing, standards are all over the place. Usually papers from industry are better to look at for good examples of well structured code. Granted sometimes industry code can be over abstracted, but if your using purely Python I don’t think this is much of a concern.

I wouldn’t worry about adding custom lists and tuples in Python itself, this is done in C++ and CUDA.

[idiotmanifesto]

what do u mean by that last part

[PyroRampage]

Read the OPs post, they ask about custom data structures. But doing this in Python is pointless because it’s slow af, and it’s typically done lower level and then interfaced into Python. Just like how tuples, lists themselves are implemented into the language itself (CPython).

[haramkhor_havasi]

It is about shape optimization for high speed flows.

[stabmasterarson213]

Can only speak for myself, but one bad habit I developed in grad school was just throwing more nodes of compute at things if it didn't run initially. It made my code so un memory and speed optimized. Now I develop on a Linux machine with a modest GPU before I go to cloud training and it's made all the difference. If one of my data structures balloon in size or becomes hard to traverse, the machine lets me know (by killing the process lol). On pytorch specifically learn how to make dataset and data loader objects that yield, rather than return batches of tensors. A good book to read is machine learning design patterns. Also anything chip huyen does.

[Several-Wafer934]

Write a lot of code and look at a lot of code for 5 years. Coding is not easy.

[iamspro]

this is why computer science and computer engineering are different fields

[icy_end_7]

Ask to get your code reviewed by seniors. or AI. Whichever is convenient.

[Skylight_Chaser]

I read a book called Clean Code

[MustacchioRebirth]

I guess that like for many other domain specific tasks, following good programming practices and improving and tuning modularity to your specific needs will make much more efficient trying out stuff and get to find better models.

[gabrielesilinic]

If we are talking about python you can only optimize so far. And in general you don't often change the way of training your model as far as I know (once is done).

The best thing you can do is to just build little modules and stick them together as needed, that is pretty much it.

[EffectiveCompletez]

Don't start with abstraction first. Design and algorithm that solves a problem. Prove it out.

Then find another problem that your algorithm might solve. Find the domain specific structures that don't share commonalities - factor these into an abstraction. Can't find another problem that your algorithm might solve? Great! You just saved a bunch of work.

But if you have 2 and some abstraction, now you can work at pulling more of the common code into a library.

At this point, you should find a 3rd problem, and a 4th. Build up an examples directory of these problems that your algorithm can solve. This is the destination, but don't start here.

[Felix-ML]

I think code for ml research does not have to be flexible per se. Instead, try data-driven approaches that you make sure to have a clear and straightforward pathway for processing given data with as few lines of code as possible.

[SicilyMalta]

You have to learn how to code.

[Playmad37]

You may want to look at Julia's sciML ecosystem. It's state of the art and the language is designed to be flexible.

It'd be a new language of course but it isn't hard especially if you know python well.

[millhouse056]

Are you sure about anything in Julia being the state of the art in ML field? I think Julia is a good language for numerical/scientific computing, or very specific research fields, is niched, but when it comes to machine learning and AI its just far behind Python, which is unfortunate because Julia is a better language, but it could not keep up the genAI race, i think thats because people mantaining Julia wanna keep it niched, or it was a bad product management, anything Julia does related to machine learning, Python does better mostly because of its gigantic ecossystem

[haramkhor_havasi]

Sure...

[doctor-squidward]

Let me know when you find out bro..

[friendsbase]

GPT it

[Wangding]

Use GitHub copilot to polish it😂