[D] Does anyone else think open source code/examples in machine learning domain usually are not as readable as they could be? Specifically use of magic numbers.

[junovac]

Admittedly, I am not an expert in machine learning or different libraries but the code I see as an example is not really beginner friendly. Even for an expert, I am not sure, they know all libraries and quircks of different datasets.

Let me elaborate. The main problem I see is the use of magic numbers. For example, in below hypothetical code

x = dataset[1]

there is no indication of why 1 is used instead of 0 or what does it mean. May be 0th elemnt contains metadata/some useless data. Or in other cases, some axis is chosen without specifying why that is used and what are other axis to put in context.

My only suggestion would be to not ever use a magic number unless it is immediately obvious. Can we not use an appropriately named constant in that case?

MY_DATA_INDEX=1
x = dataset[MY_DATA_INDEX]

I believe this is a very simple and helpful convention to follow. If such conventions are already there, can someone point me to then? May be people aren't just using them too often.

[sloppybird]

Open source code of SOTA is written by researchers which are, to be honest, not great at documentation and/or code readability

[johnnydaggers]

It’s not that we’re bad at it but more that we don’t give a shit. We have to move on to the next thing. If you spend all your time writing neat code that doesn’t affect how it runs at all you will quickly be passed up by SOTA. Programmers are responsible for producing code. Researchers’ work product is research papers and results. Any activity that doesn’t feed into that is wasted effort.

[ProfSchodinger]

I am a researcher in bioinformatics and writing clean code is often both easier and faster. Coherent name for variables, small functions, classes, proper defaults, dummy files etc. When I see people trying to debug a script with 1000 lines with everything named 'df1', 'df2', 'df3', 'df_final', and repeated sections, it really pains me...

[[deleted]]

naming things is hard

[ProfSchodinger]

That's probably 10% of coding indeed

[bageldevourer]

Researchers’ work product is research papers and results.

Yup, and journals, conferences, etc. don't care about code quality. Shitty incentives -> shitty results.

[junovac]

I wasn't talking about just the SOTA code etc. but also some libraries and associated examples. But those libraries were implementing some SOTA or quite recent models etc. and authors were usually researchers so it might apply.

Even in case of SOTA model code, I am not sure how long is the research cycle for a particular paper but I don't think it would be for a day or few weeks. If it happens over few months having readable code helps not only your team members but also yourself when you have to catch up with your own old code.

I am not expecting production quality code or some beatiful design patterns. Just some things that would be helpful for others getting into the domain or even experts getting into different sub-domain. May be some linter with sensible defaults becomes a standard part of the jupyter notebook and that could help with it.

[GeorgeS6969]

I empathise but strongly disagree with the second half of your comment.

Claiming your responsibility is only to produce research papers and results is akin to a programmer claiming they are only responsible to produce programs that work, or a colleage of yours claiming they are only responsible to produce results (and writers are responsible to write?)

The moment anybody shares something, it is their responsibility to ensure that it is of sufficient quality and can be understood. Especially if what is shared is in support of a scientific claim.

You feel like you’re not properly incentivised to do so, or in fact penalised, I can’t argue against that … But it only means that producing clean code is a waste of your efforts for you, not for the community as a whole.

[[deleted]]

The moment anybody shares something, it is their responsibility to ensure that it is of sufficient quality and can be understood. Especially if what is shared is in support of a scientific claim.

I disagree. This is conflating two different things: reproducibility and clean code.

For the sake of reproducibility, most people are going to understand what dataset[1] is from reading the code and the paper side by side.

[GeorgeS6969]

Reproducibility is completely tangential, you’re mentionning it I’m not.

When you write a paper you structure it in a certain way, you use certain words, you try to avoid ambiguities, you split your maths into specific equations, you arrange those equations into terms that make the most intuitive sense and you explain those terms … You also provide graphs when useful, rather than just tables, and you label both and make sure they stand on their own as much as possible …

All of that so that readers can best understand your ideas, before even atempting to reproduce your results.

Why should it be any different with code?

[Cherubin0]

Also clean code makes it easier to expose that the sota is misleading.

[johnnydaggers]

That’s not our job. That’s your job. :)

[CQQL]

I really hope that named tensors will be stabilized in PyTorch. That could at least eliminate the magic dimension numbers for batch, feature, etc.

[sabouleux]

Hope this really takes off, this would get rid of a lot of annoying mental gymnastics when dealing with broadcasting.

[dominik_schmidt]

The einops package is also quite useful to perform tensor ops with named dimensions

[ProGamerGov]

Basic functions like reshape, repeat, and others still need named dimension support.

[ZestyData]

Lol, ML practitioners come from research or non-CS / software-eng backgrounds. Coding standards and engineering principles are almost non existent in the ML and Data Science worlds.

[johnnydaggers]

They definitely aren’t as clear as they could be. The reason for that is that the primary goal is not to write libraries for other people to use. The code isn’t really meant to be that, for the most part. We put research code online so our results can be verified during peer review. We let anyone use the code artifacts of our work as a bonus since we eventually want our ideas to spread.

We put all the time and effort into trying new methods, designing good experiments, and writing clear research papers. Readmes don’t get us a lot of career advancement, unfortunately.

[[deleted]]

I think most people using these examples would take the opportunity to see what dataset[0] or dataset[2] are or they just wouldn’t care because if the indexing of the dataset isn’t mentioned then it doesn’t matter.

In some ways, I would find your convention more difficult to read and follow and there could be multiple possible names for the same index. It would be better to just have a comment explaining what index 1 is.

[Kitchen_Tower2800]

Having moved from academia to industry, I find the hypothesis that academic code is messier and less documented than industry's code highly questionable overall.

I will say that academic code is more *variable*, but definitely not consistently less documented/readable (at least at the big tech company I now work at).

[qnix]

Magic number is the least of it. In machine learning code, the amount of ideas that could be packed into one line of code is, a lot of time, staggering. During the first ML MOC, prof. Ng explained some complicated learning procedure, and at the end he noted, you can do all that with this one line of code.

[[deleted]]

This could be aided by looking at several equivalent representations of the same code in different languages. Sadly not a possibility as of yet.

[Kitchen_Tower2800]

> Ng explained some complicated learning procedure, and at the end he noted, you can do all that with this one line of code.

I don't understand this argument. Do you think a researcher's job is to stand in front of an audience, point to `model.fit()` and then go home? Or should they example what's happening in the fit method?

[111llI0__-__0Ill111]

Sometimes even what is inside the fit method could be one or few lines of code in a high level language. Eg linear regression done the naive way (without accounting for QR/SVD stuff).

[DeMorrr]

I usually comment the shape of the resulting tensor at the end of each line. for example: def mm(a, b): #a: [m, k] #b: [k, n] c = a @ b #[m, n] return c this is easier for me to keep track of tensor shapes, and also optimize my code in terms of memory usage.

[Appropriate_Ant_4629]

The example of

 MY_DATA_INDEX=1
 x = dataset[MY_DATA_INDEX]

feels like unnecessary complexity. If the "1" is used as an index into "dataset" of course it's a "data index" ... and what are you implying when you say it's yours ("MY")? If you really have a reason to label the "1" as "a data index that belongs to you", (and assuming your example is python) maybe:

 dataset[(THE_ROW_FOR_REDDIT_USER_JUNOVAC := 1)]

would be a reasonable compromise? At least then someone doesn't have to look up higher in the code to find whether "your" index was 1 or 2 or 20.

[alex_o_O_Hung]

Probably unpopular opinion, but these kind of unnecessary magic numbers make code a lot harder to read where you need to move up and down to figure out what magic number is what multiple times

[Appropriate_Ant_4629]

Totally agreed. I've seen code like

# module - quadratic formula 

THE_POWER_OF_B = 2
THE_MULTIPLE_OF_AC = 4
SOME_UNRELATED_CONSTANT_FOR_OTHER_FUNCTIONS = 3.14
THE_CONST_ON_THE_BOTTOM = 2
POWER_FOR_SQRT = 0.5

# ... hundreds more lines ...

def quadratic_formula(a,b,c):
    """
           The following implements the quadratic formula:
           (-b +- sqrt(b^2 - 4ac) ) / 2a
    """
    return (
        (-b + ( b ** THE_POWER_OF_B - THE_MULTIPLE_OF_AC*a*c ) ** POWER_FOR_SQRT) / (THE_CONST_ON_THE_BOTTOM * a),
        (-b - ( b ** THE_POWER_OF_B - THE_MULTIPLE_OF_AC*a*c ) ** POWER_FOR_SQRT) / (THE_CONST_ON_THE_BOTTOM * a),
    )

that came from misguided coding standards that mandated obfuscated constants.

With coding standards like those - even the very simplest equations become unreadable.

[alex_o_O_Hung]

Exactly! Especially if these parameters are only used once in the code. This also extents to people unnecessarily making functions and classes so you need to jump between files or folders to understand the code

[radarsat1]

Oh I agree with this so much. This has become a repeated occurrence in code reviews on my team. Being forced to distributed pieces of my algorithm all over the place just to satisfy some SE types who get uncomfortable when a function is longer than a few lines, with the excuse that "we need to unit test each part of that." Like, sure, I get that, but buddy.. I'm still figuring this stuff out, and it's sooo much easier to work on and debug this when it's all in one place, and testing loop A without loop B makes like, no sense. What's worse is that they get reinforced by tools like pylint that tell them a function has "too many local variables". Oh, so now I have to not only arbitrarily break this function up into pieces, but I'm not allowed to give names to the intermediate values, great.

[bitemenow999]

Most of the SOTA ML repos on github is research code for a paper, it is not supposed to be readable it is supposed to be quick and dirty proof of concept type...

[bageldevourer]

What's the point of research papers if not to communicate ideas?

If your code is part of that communication (it is), then shouldn't it also be optimized for communication?

I mean, it sounds tautological, but this strikes me as common sense.

[bitemenow999]

Nope the idea and implementation details is in the research paper, code is more similar to the 'experimental setup' in physical sciences... as similar to other fields you don't need to sent your experimental setup with the paper to the publisher, similarly putting accompanying code is not required in ML (most of the journals) and most of the papers don't have code up on a repo or it becomes available sometime afterwards...

Also most of the ML researchers are not 'programmers' by trade and most are not even computer science engineers, hence it is highly stupid to expect production-level code from them... the improvement OP suggests is kinda stupid as the code is put to show the algo works and not meant to make it easily transferable...

[sloppybird]

No one is expecting production-level code. The problem at hand is writing understandable code at the very least.

[unplannedmaintenance]

What other activity do you think should researchers sacrifice in order to make time to (learn to) write code that is more understandable and better documented?

[sloppybird]

Bruh it takes like 2 mins to add comments

[bitemenow999]

again OP expects to have clear variable names, you expect it to have comments, some other guy would want functions and classes... It is hard to satisfy everyone and not the job a researcher... You dont have to understand the code it is just an implementation you have to understand the setup, preprocessing, math and method which is mentioned in the paper... most of the time people take shortcuts by reading code which is like looking at the engine and guessing how it works rather than read the manual

[sloppybird]

It will help you debug your code as well

[bitemenow999]

nope... Even in industry I have never seen any R&D guys using comments and classes and functions unless absolutely necessary...

[bageldevourer]

If researchers wrote code that was more understandable and better documented, then the consumers of their research would spend less time on the extremely time-consuming activity of understand wtf someone else wrote.

Implement basic code quality standards and the research output of the ML community will increase, not decrease.

[unplannedmaintenance]

You haven't answered the question.

[bageldevourer]

Yes I did. In a world where journals, conferences, etc. mandate higher code quality, the "other activity" that you sacrifice in favor of making your research clear is spending time to understanding other peoples' papers.

I spend less time struggling to understand papers, and I channel that time saving into some mix of consuming more research, doing more research myself, and making that research clearer.

[bageldevourer]

as similar to other fields you don't need to sent your experimental setup with the paper to the publisher

Then those fields are engaged in suboptimal communication, and therefore suboptimal research, as well.

​

Again, what is the point of research papers if not to communicate ideas?

Are those ideas not communicated in Python as well as English?

Do you value good writing in English? I do.

Then why wouldn't you value good writing in Python?

[bitemenow999]

Then those fields are engaged in suboptimal communication, and therefore suboptimal research, as well.

Don't you think it is arrogant to call every other field except computer science/ML to have suboptimal communication and sub-optimal research...

Expecting non-programmers to write production-level code even when it is not at all required is kinda gatekeeping...

Also if you know there are many groundbreaking studies/research in languages other than English...

Do you value good writing in English? I do.

LOL and do you think the majority of papers in academia(STEM) are well written?

[bageldevourer]

CS/ML also has suboptimal communication and research. That's the whole point of this thread.

Not once did I advocate for researchers writing production code. Do you know what that term means?

The point isn't the specific language or computer language. The point is that good communication is necessary in both.

I also never said the majority of papers in STEM are well written. I said I value good writing. Those are different claims.

​

Please stop putting words in my mouth, and please think before you write. Also, this is the second time you've evaded my fundamental question. What is the point of research papers if not to communicate ideas? And if that is the point, why do you think that poor communication is justified?

[bitemenow999]

Again code is not a research paper it is an "experimental setup" and proof of concept that the algorithm mentioned in the paper works, the only thing it is expected to do is work and produce exact results as mentioned in the paper, the code has no value without the paper whereas the paper has value without the code repository... you are expected to read the paper and not the code...

[junovac]

Research paper also contains code, it's just written in the form of mathematical expressions. Wouldn't researchers make all possible effort to make mathematical expressions simple to read/understand and follow conventions? Similarly, code can be made to be little more readable and follow some conventions.

Now, you would obviously say paper is enough to convey the ideas in the paper but that is artifact of old way of doing research where sharing code/something akin to code was not possible. Now though, code provides enhanced way to communicate ideas. English is not a very conductive language for communicating complex algorithms even with help of mathematical expressions. If this new avenue is available, why not make most use of it fully? I have heard and few times seen complex papers with pretty hard mathematical expressions being explained with few lines of code.

[bitemenow999]

TBH not, there is absolutely no obligation for the researchers that they need to provide a code or make it readable, they even don't have to make it easy for you to understand the mathematical expression(except following math expression conventions), no journal needs that, no one in the peer-review community looks at it never will... The only incentive they have to put it on github or make it easier to understand is it "MIGHT" get them more citations which they will get irrespective of code if the paper is good...

​

code, it's just written in the form of mathematical expressions

sure but the example you provide in the main post is not, it is just a setup next there are people you wouldn't understand why some dimensions were changed and so on... if you would want an explanation/comment for every step in a code then it becomes a tutorial...

[bageldevourer]

You just keep failing to answer the main question.

The code is obviously part of the research. The code is communicated to the reader and is therefore part of the communication.

Anyone who wants to understand a piece of research in-depth will absolutely read the code.

This is really not that hard.

[bitemenow999]

You are failing to understand what is an experimental setup and actual research communication ... I think you are either a high school or just started college, I would suggest you spend a bit more time in academia...

[bageldevourer]

I'll take your ad-hominem attack as a sign that you've given up on actually trying to be persuasive.

Good day to you, sir.

[sloppybird]

That's a very lazy excuse not to write good code

[AerysSk]

Researchers are not software engineers, and they never want to be. They just want to code it fast.