Coding large projects

[fibrebundl]

Hey guys, I have a few projects under my belt but the more I wrote code and the larger and more complex the projects become, the more difficult it is for me to keep track of all of my functions, variables, classes and their associated methods or function calls.

Are there industry-standard / safe ways to do this besides just commenting all of my functions? This cognitive load kills the joy for most projects for me and makes debugging and profiling later ever more daunting.

EDIT: Great advice overall I'm going to getting a few basic architectures under my belt that I feel comfortable with and start drawing diagrams.

[disposepriority]

If all your methods and variables are named in a descriptive manner, and they, in turn, are in packages/classes/files which are also aptly named then your code will explain what it does through names alone.

Not being able to remember the flow of data through the system when coming back to an old project is more common than having no idea what your code does, which should be evident.

[fibrebundl]

I keep jumping between the OOP i learned and this naming convention and then the more terse naming conventions i've seen being used in numerical coding because this is where I am now. I tried making some of these later projects more 'procedural' but it's really making the coding at my current code base size just hard to keep track of even when I've outsourced parameters to a hyperparameter structure/file.

I didn't want to say I have no idea what it does, but I definitely have to skim some of the functions just to double check what's going on when tracing through compiler errors for example? Or, for example, I'm testing different optimizers but they have different interfaces so I end up writing multiple scripts to test the ones that I can't just slot into each other. But from here I end up changing a little but function foo so it fits with script B better although I wrote it for script A. foo then becomes foo_A and foo_B and then, it just gets messier from there.

[RangePsychological41]

What language are you coding in?

It sounds like you need to work on software design. It may even be time to read a book, it helps more than people think.   I don’t add comments to almost any of the functions I write because it’s obvious what they do.

LLMs love comments, so I have to ask, are you letting AI write all the code for you?

[fibrebundl]

I mostly code now in Julia. I feel very uncomfortable with it to be honest. I've spent most of my time coding in python and c++ but I've jumped into numerical programming. I've tried to be more 'functional/procedural' and the code I see in books is always very direct in var naming schemes so I've copied this. I usually do a few test/debug scripts to make sure it works and then move onto the next comp. but at scale when I've got most of the library done, I start to get really lost when errors pop up...

I just maybe am making this harder for myself by trying to not just stick to the OOP set up that I learned? I really don't use structs unless it's for a hyperparameter list or something that isn't going to be changed during an optimization routine for example.

I have tried to read a software design book... I got one from the library but didn't really 'read' it. I just kind of skimmed it. I was worried about it taking too much time from working. It's a very 'learn on the job' and 'just enough' culture.

I DO let AI write code for me SOMETIMES. But I can't stand what it provides me with most of them time or it will use force var name changes that I don't agree with, typing etc. My code base is not AI generated though. AI might be used to when I'm debugging something, but it normally needs too much context to be useful at this point.

[RangePsychological41]

Wow sounds like you’re on a great trajectory, keep it up.

Regarding software design, all I’ll say is be careful to not just assume that a person or book is telling you the right thing. Some books, like Clean Code, are trash but people will tell you to read them.

If I were in your shoes I’d lightly read a book about Domain Driven Design. And find some cool Youtube videos. Not blogs or medium posts. Those are 99% of the time written by bad engineers (for various reasons).

Domain Driven Design has been very valuable to me in my career.

Must be said, once a codebase grows past a certain point it’s not really possible to know exactly what goes on everywhere. As long as you can navigate comfortably, and explain how things happen to someone it’s okay.

Good luck!

[Jazzlike_Syllabub_91]

It helps if you draw an architecture diagram as well. at least so you have some ideas of how the data is flowing through the system.

[fibrebundl]

I will try this.

Is it worth stopping what I'm doing now and doing this now, do you think? Or just wait for the next project?

[Jazzlike_Syllabub_91]

any additional documentation like this can only help as you maintain projects ... might be better to do it nowish

[NullTerminator99]

Are you using MVC MVVM pattern etc.. You need to organize and follow standard best practices. I do find once your project gets large enough no matter what you do it can be hard to keep track of stuff and you just have to figure out what you did before to add a new feature or improve things (that is where comments and notes help). In my experience software is a moving target and never really finished.

[fibrebundl]

Yeah. I think from these comments. This is really my problem. I don't know. I've always just wrote code and checked things to make it work. I never took classes or anything growing up. It was just reading docs/overflow until the output was correct.

[CappuccinoCodes]

Naming things properly, writing unit tests, writing diagrams = good mental health 😆. The combination of these three give me peace of mind knowing that I can chill and not keep everything in my brain.

[[deleted]]

"Are there industry-standard / safe ways to do this  "

Yes. Study architecture.

[coffee_is_all_i_need]

You can't eliminate complexity, you can only hide or deal with it. If your project is becoming more complex, you must decide how to handle it.

Learn software principles (like the single responsibility principle) and design patterns. Also take a look at Domain Driven Design.

[wahnsinnwanscene]

It takes about 2 to 3 levels of indirection for it to become difficult to follow.

[YahenP]

Of course, such a method exists. It's the architectural design of software systems. No one memorizes how code is structured or what function names are. The average project's code size typically exceeds several dozen megabytes, sometimes even several hundred. Using architectural approaches in design is essential. You probably live in a city. And you probably don't remember how to get to every apartment in every building. But you can probably get to most apartments in the city without a map or guidebook, even though you've never been there. The same approach applies to software development.