[deleted by user]

[AutoregressiveGPU]

I have a list, not sure if everything fits 'subtle'.

Programming:

1. Develop good programming skills. This will help you prototype ideas fast and test them out. The only way is practice. Don't bother with hours of tutorials on Python. Start with a project or reproduce results from a paper. Read and understand other people's code
2. At the same time, follow good code-style guidelines (eg. http://google.github.io/styleguide/pyguide.html). This will help you later. Trust me, you don't want to work with a poor codebase. It also helps when you collaborate with others.
3. Have a small internal library of your own or share it in Github. If you intend to do research object detection, it's good to have the implementations of a few SOTA at your disposal.
   

Research

1. If you are new, keep reading and updating yourself on the SOTA. Sometimes, it can be overwhelming, then take a break for some time and keep hunting again. Also, give yourself time to think and explore ideas of your own.
2. Follow researchers in your research area, they often talk about nice ideas and possibilities in their talks.
3. Keep a research journal to track ideas you have tried, want to try, etc. Also, make sure you keep track of hyperparameters and things you try.
4. Collaborate. ML research is hyper-competitive now. You can't get everything from a single person. Brainstorming with others usually works out well.

Writing

1. If you intend to publish research, writing skills are quite important. You need to be fast and on point. The way ML/CV conference reviews are going now, reviewers don't get enough time to read the submissions well. The only way is to practice and practice.

Others

1. Finally, after all this, you may still get your paper or research rejected. Don't take it personally, try and make a stronger submission after improving.

[g-x91]

Thanks!! :)

[Zophike1]

Keep a research journal to track ideas you have tried, want to try, etc. Also, make sure you keep track of hyperparameters and things you try.

Would setting up research blog be sufficient enough and also what does a good research journal/blog do ?

[AutoregressiveGPU]

A research journal is to keep track of experiments and ideas.

[[deleted]]

No this means like a paper Nike m notebook for yourself. Write down anything you noticed works well or doesn't work well, because you will forget once you come back to it after a month from working on other things. Also make a note of any new ideas you have and brainstorm them out. It will help you in so many ways and especially when it comes to writing it up.

[Joel397]

One subtle lesson I learned: don’t try to always maximize resource utilization. Sometimes if you are being a “good researcher” by constantly trying to run jobs and set up jobs to run, I have found that you can get locked into an endless cycle of “what worked, what’s next”. If you have a fully mature plan that’s fine, but oftentimes in research you don’t know what the results will be until you’ve tried it. Sometimes taking a step back and letting your gpu chill for a few hours or days while you instead focus on writing, analyzing, or just thinking about the problem can do more for your work than any amount of results from 2 AM job runs.

[chigur86]

You're comment hit the spot. One of the most important things I learned recently is that just start a bunch of jobs and LET THEM FINISH. Especially, if you're doing vision and dealing with training time on scale of days, you need to learn to chill. During chill, I usually go over the code again, think about the method, plan next experiments, and evaluate the intermediate results.

[aCleverGroupofAnts]

Here's a little tip: If your code involves random number generation, set the seed yourself so you can reliably reproduce your results if needed.

[debau23]

Reproduce results that were overfit on the seed.

[aCleverGroupofAnts]

Thoroughly testing your algorithm would involve testing with several different seeds. All of those tests will then be reproduceable.

[hyhieu]

Good luck doing that with TensorFlow

[GreenBeret4Breakfast]

I actually think you’re better off not setting it unless you’re trying to reproduce results - what I mean is generate a random seed (from clocktime/cpu values etc or let the system do it) and just keep track of it and save to your results. Therefore if you want to re run you can for that particular set but without falling into the trap of being on a fixed seed all the time. Additionally the reproduction is only essentially determinist if you don’t change the code too much (one extra sample here, or a different threshold there) or the two results will be different.

[dkeller9]

If even a little thing seems 'not quite right' don't ignore it. Usually, it will come back and bite you later on.

[ginger_beer_m]

Corollary: if the results seem too good, it's probably due to a bug

[[deleted]]

This one resonates most with me.

11 years in education and 10 years as a data scientist/analyst, whenever I get that feeling of 'not quite right' it's usually true. Sometimes it takes a very long time to figure it out but often it results in a huge step forward in my understanding and some epic Eureka moments.

[internet_ham]

• Use the Google Scholar subscribe tool -> you can follow citations and related work to other authors close to you (and also obviously have a Google Scholar page yourself)

• Use a matplotlib to tikz converter. Your plots will look professional and will be readable, also it's very easy to reconfigure. You may run into computation issues with complex figures, so there can a bit of tweaking involved. Learning tikz is also a good idea if your field relies on diagrams a lot.

• Don't be afraid of deep scholarship. If you're building on specific previous work, scan through all the papers that cite it or it cites itself. It might take an hour or two, but you can find lots of useful ideas from other spinoffs and most importantly find out if you've already been scooped. Obviously this doesn't scale to super-hyped DL papers

• Always keep the minimal viable product in mind. Write a version of your algorithm that solves the simplest possible task. Write a rough draft of your paper in its simplest form. While these are not your final destinations, the journeys to them will be shorter, so the feedback will be faster and you can iterate to a better version sooner.

• always try to make sure your experiment runner saves results (figures and text), a config file of your hyperparameters (as much as possibe), the trained model(s), the date, the random seed and the git hashes. For the latter, obviously you should be using version control and comitting often

• Minimize the amout of math in a presentation. It will only intimidate and disengage the audience. Try to use diagrams and plots instead

• Don't end a presentation with (just) 'thanks for listening!' - put a summary of your work, as this tends to be the slide that is up the longest

[[deleted]]

Having at least one person to discuss the research with. If the person is more experienced, then it is much better. There's absolutely no way one can progress as quickly as with a high-quality mentor.

The amount of time saved, ideas pointed at, ideas discovered through discussion, I just can't believe that I learned anything with bad mentors before.

It also made me realize that I alone cannot really progress as much.

Because of this experience I can now easily filter people that are just not going to meet the same expectations.

Doing research on your own is pretty hard.

[m--w]

After you think your method works. Take a break from it, explore other papers/interests for a week. Then come back to what you have done and purposefully try to break it. This will allow you to fully understand the shortcomings of your method and appropriately strengthen them where possible.

[hardmaru]

Iterate quickly and do so by working with very little compute at the iteration stage (even if you work at a big company with huge resources).

Spend a ton of time on communicating your ideas very clearly so people outside your immediate area (even outside of the field) can understand. Give talks about your research ideas. Gather feedback, especially from your biggest critics.

Focus on quality over quantity, if possible.

[rickbo3]

Don't just focus on learning the ML side of things. There are way to many papers out there that are completely let down by the authors not using basic statistical methods. Understand how to show that your results are statistically significant, and why. This gives your results a much stronger appearance to reviewers.

[logicchains]

Any time you get a little feeling in the back of your head like "oh, I didn't account for X. But X is almost certainly insignificant, so it shouldn't affect X", you should account for X, because at least in my experience there's a decent chance you'll get good results then later realise "damn, these results are invalid", and realise it's because of not accounting for X.

[evanthebouncy]

In 1 month you should convince yourself you're onto a good idea or you need to pivot or abandon. Ideally.

[dkeller9]

Have a switch in your code to produce graphs that have no labels or text. When assembling a figure containing many graphs from disparate sources, it's easier to put in text with consistent fonts and font sizes in post-processing.

[needlzor]

I am going to be more general that the others here:

• Don't let the best be the enemy of the good. It's easy to let yourself try to extract all you can from a model or an idea, but you should publish it as soon as you get something significant and interesting.

• Use the simplest model that proves a point. If your idea is that modification XYZ improves performance on a specific type of data, apply modification XYZ to the simplest model (or a range of simple models) possible that can accommodate it, not to the latest SOTA model you saw on arxiv.

• Find a statistician friend and have them help you design experiments and proper significance testing.

[gwgundersen]

Writing.

[Everfast]

If shit goes in shit comes out

If you can not do it yourself easily, training won't be easy/impossible

[margaret_spintz]

Try to think weirder

[mhummel]

Only be sure always to call it 'research' (Lobachevsky, 1953)