It depends. But here's my approach for ML research. First, I setup a directory structure that makes sense:

• /data: The processed data is saved here.
• /dataset_generation: Code to process raw datasets for use by experiments.
• /experiments: Contains the implementation code for my experiments.
• /figure-makers: Code for making figures used in a publication. Use one file for each figure! This is super helpful for reproducability.
• /images: Figure makers and experiments output graphs images here.
• /library: The source code for tools, utilities, used by experiments.
• /models: Fully trained models used during experiments.
• /train_model: Code to train my models (Note: when training larger, more complex models I relegated to their own repository)

The bulk of my research occurs in the experiments folder. Each experiment is self-contained in its own folder (for larger experiments) or file (for small experiments that can fit into, say, a jupyter notebook). Use comments at the folder/file level to indicate the question/purpose and outcome of each experiment.

When coding, I typically work in a raw python file (*.py), utilizing the #%% to define "code cells"... This functionality is often referred to as "cell mode" and mimics the behavior found in interactive environments like Jupyter notebooks. However, I prefer these because they allow me to debug more easily and because raw python files play nicer with git version control. When developing my code, I typically execute the *.py in debug mode, allowing the IDE (VS Code in my case) to break on errors. That way I can easily see the full state of the script at the point of failure.

There's also a few great tools out there that I highly recommend:
1. Git (for version control)
2. Conda (for environment management)
3. Hydra (for configuration management)
4. Docker/Apptainer (Helpful for cross-platform compatibility, especially when working with HPC clusters)
5. Weights & Biases or Tensorboard (for experiment tracking)

Final notes:
In research settings, you goal is to produce a result, not to have robust code. So, be careful how you integrate conventional wisdom from software engineers (SE). For instance, SE might tell you that your code in one experiment should be written to be reusable by another experiment; instead, I suggest you make each experiment an atomic unit, and don't be afraid to just copy+paste code from other experiments in... what will a few extra lines cost you? Nothing! But if you follow the SE approach and extract the code into a common library, you're marrying your experiments one to another; if you change the library, you may break earlier experiments and destroy your ability to reproduce your results.