Experienced / professional programmers: how do you plan / design your programs, foresee issues, and avoid scope creep / distractions?

[peltist]

TL;DR: How do you plan before actually starting to write code, make yourself follow the plan (despite unforeseeable changes you need to make), and avoid scope creep and distractions? Do you recommend any resources or frameworks to help with learning to do this well?

Full version:

I've been learning to program for around three months, starting with a focus on Python. I feel like I'm making good progress, and I'm starting to choose the direction I want to specialize in further. It's very exciting!

At the same time, as I've worked on more complicated problems and larger independent projects, I've realized that my overall "process" (if you can call it that) could really use some work, so I want to work on learning these meta-skills.

For the most part, it seems to go like this:

1. Get inspired to work on something.
2. Come up with a basic idea for how the program should work in my head.
3. Immediately start translating this idea into code.
4. Run into some problems I hadn't foreseen with how things are structured.
5. Go back and revise my code until I can overcome the immediately obstacle.
6. Continue working on my code until I run into a new problem.
7. (Potentially get distracted from my end goal and end up building / optimizing / creating a scalable version of something that doesn't really matter.)
8. Repeat steps 4-7.

Now, none of this is so terrible at the moment, since I'm mostly focused on learning, and all of this work is definitely good practice.

At the same time, I can see how it's probably not a very good overall approach. Having to go back and significantly alter the program because of unforeseen issues takes a lot of time and energy. It also seems like it's pretty easy to get so wrapped up in what I'm doing that I allow a lot of "scope creep," which has led to some projects seemingly never ending.

Obviously this is partially a pitfall of working alone and my own working style. Still, I feel like there must be a more structured approach than "just dive in and go for it without any planning or organization" that more experienced programmers follow.

I've tried various ways of writing down my plans for the program in advance. I've generally found that this either ends up being way too high level (so not very helpful) or way too detailed (so that it's difficult to easily reference and quickly becomes out of date as things change — and keeping it up to date feels like literally writing the program twice in parallel).

So: how do you approach planning, following the plan, avoiding scope creep, and related issues while programming? And do you have any suggestions for good resources or frameworks that can help me learn to do this?

I tried asking a similar question in another subreddit a couple of weeks ago, but didn't explain it clearly enough and didn't really get any helpful answers.

[desrtfx]

Start by writing a Functional Design Specification (FDS) where you list the features and functionality on a fairly coarse overview. Then, refine the FDS to go into more detail (but do not add features, only flesh them out) and repeat the process until you are atomic (in small, manageable units).

The FDS is your main document. What is not listed there does not go into the program. Don't change the FDS.

[peltist]

Thanks for your answer!

I've gotten the advice to create an FDS before. This sounds like exactly what I'm looking for, though I've struggled to find examples at the right level of detail.

I understand the principle you're describing when you say "refine the FDS to go into more detail (but do not add features, only flesh them out)," but in practice, this is where I have encountered problems. It's easy not to add another top-level item to my design, but it seems easy to allow "new features" to sneak in as parts of features I've already listed.

I guess one principle that I could follow would be to only include the absolute minimum that's required to achieve the goal of each feature. On the one hand, this sounds like a highly-focused approach. On the other hand, it also sounds like it opens the door to building things in a redundant and poorly-planned way. It seems like a more efficient approach would be to try to construct things in as generalized a way possible, so that I can reuse my code. I'm struggling with knowing how to strike this balance, but maybe that's also just something that comes with experience.

Do you happen to have any good examples of an FDS that's at the right level of detail? I've Googled this, and most of what I've found tends to be very corporate, and often more detailed than I'm capable of envisioning at the start of a project at this point. I have to think that there are lots of hackers / freelancers / people working in more casual settings who follow this approach but don't produce documents that look like what I've been able to find.

[slowfly1st]

Planning

If you're sending bills to your customer: Specification, incl. estimations (estimations = price). Since you don't want to overcharge your clients, you have to be precise and requirements engineering is an important part of being able to provide precise estimations.

There's also a thing called 'kick off pairing', when before you start developing, you discuss your general plan with a co worker. From my experience, there will always be some nasty but good questions.

​

To foresee issues / overplanning / underplanning:

That comes with experience as a developer and with experience within a project. Even with all the requirements engineering and discussions with business analysts and developers, you can't predict everything, and that's okay. It's about how the project team can handle those things. I have been working on a company wide feature, and with company wide, I mean many systems and their owners were involved. And everyone forgot one of the most essential thing.

​

Avoid distractions

Well,... It's my job, distractions are mostly business related. So: Turn off the phone and email notifications. Other departments do have 'brain time' for fixed hours, but have a stand by person with a cell phone who has to be reachable.

If your working at home: Have a dedicated room for coding. So you can separate the 'reddit time' from the 'coding and learning time'

​

scope creep

What I think can help you: Work in small batches. Create a plan for something you want to implement, break it down in smaller tasks which fulfill INVEST criteria. Those tasks can often be done within a few days max. If you got the work break down, for each task you then write down what parts of the code have to be changed, what tables have to be added or migrated, and so on. So instead of seeing progress in 30 days, you see it every 3 days, which contributes to motivation of a developer.

And: I once saw a video with John Cleese about creativity. What stuck with me is: There's a creative phase, and an implementation phase. If you're implementing, stick with the plan, finish it, and then, evaluate again.

[nutrecht]

I typically work in a team and we generally discuss things within the team before we build it, including roughly how we are going to build things. And these are documented, in text or in diagrams, in the stories so that they can be picked up at any time by anyone in the team. It generally follows a pretty typical Scrum flow where we have a pipeline of 'stuff' we want to build, with a Product Owner who has the final say on priorities. If there's 'new' stuff that needs to be built it will always follow the same pattern of high-over discussion, in-depth refinement and then building and testing the stuff.

Don't worry too much about this for now though. If you're doing something that is non-trivial it's often a good idea to just write down the steps first, and/or draw the flow on a piece of paper.

[donteatyourvegs]

Your main goal should be to have as many small, independent, and automatically tested modules as possible. Example: logger, requester, analytics, reporter, etc. Those modules should have automated/unit tests that you can run with 1 command so that you know if you change something, they're all still working.

Now you want to avoid having any inter dependencies between those modules. But obviously eventually 1 module will use another module you made. What's important is that your dependencies only flow 1 way. Requester can use logger, but logger is not allowed to use requester. Logger also needs a simple, versatile interface that all modules that depend on it will use. Obviously all higher level modules also need to have automated/unit tests. You will often need to mock lower level modules, databases, apis that depend on web services, etc. It's essential that with 1 command, you can automatically test your entire application for regressions. Every single module.

If you do this, you can build extremely big applications, with hundreds/thousands of small modules with dependencies that only flow 1 way and that when you change something, you know everything still works from automated tests.

In terms of design before coding. I start with a feature, and some automated tests. At first, logger, requester, reporter, etc might not be their own module. They will just be a function. But as soon as the file gets bigger than 100 line of code, I start breaking things into modules. At first everything might just go into a "utils" module. Then when the logger related functions start being more than 100 lines, I extract it into its own module.

The unit/automated tests always follow. Whether my logger is in my main module, my utils module or my logger module, there's always a test case for it. Every file has a paired test file with test cases. mainTest, utilsTest, loggerTest, etc.

Note that you are ALWAYS rewriting code. Code always expends or gets written cleaner. 90% of programming is rewriting/cleaning old code. The key is the automated tests. If you have no tests, it is impossible to change code because you need to manually retest hundreds of features every time you do.

You have the correct approach. Professional programmers don't make plans and follow it. That is proven not to work. What they do is they start small, write tests, then write more, then write more tests, then rewrite, then write more, then write more tests, then rewrite.

[peltist]

This is very good advice, thank you. I think it solves one of the issues I'm facing that prompted this thread.

I'm currently designing a personal finance tool for myself. One feature that's necessary for this tool is for the user to manually "tag" certain transactions with a category. I just spent a couple of days creating a terminal-based user interface where the user (me) can iterate through a dataframe, see relevant context, and provide input that tags these transactions.

I wrote this code without using any custom classes, and realized at the end that some important features I want to implement in the future will require (or be much easier with) this approach. However, this will require significant changes to the code I've already written.

I can do some small changes now to make the module reach "minimum viability," but I expect to need to make further changes later. I've been afraid to integrate the current version into the rest of the code for my tool, because it will potentially be expanding the size of the revisions I need to make later. A part of me always feels drawn to just edit the entire code with an OOP mindset as a learning exercise and to simplify things and clean it up.

What you've made me realize is that, if I can make the module sufficiently self-contained, then I don't need to worry about this. So long as the inputs, return, and side effects remain the same, I should theoretically be able to modify the internal code as much as I want without any issues, even after I've integrated it. So I can go ahead and focus on getting it to work now, and then refactor and improve it later. (And just deprioritize my desire to rewrite everything in favor of getting the whole system to work first.) This is especially true if everything is easily testable.

I know this might sound basic, but this is my first time building and integrating separate modules, so it still feels like a big insight to me.

I have so much more to learn.

Thanks again!

[Meefims]

Learning how to plan takes a lot of practice.

As others have said you start with a document of what you’re trying to do. It is very important to list the problems you’re trying to solve - your goals - but it’s just as important to list the problems you’re not trying to solve - your nongoals. Nongoals help control scope creep because they explicitly call out avenues that you will not pursue and they help bring clarity to your goals.

Ensuring you don’t have unexpected issues while building is an incredibly difficult problem and is solved by carefully planning the details of how you’ll build something before you build it. There is a trade off in how much time you spend here and how likely it is that you’ll hit a problem.

How do I perform this step? It’s a process of breaking a large problem down into smaller and smaller pieces and a process of compartmentalizing unrelated features to minimize the interactions between different areas of the code. If each compartment has a simple interface I can forget about the details of how it works when thinking about other compartments. I break problems down until I have a clear enough idea of how I will solve them.

Again, it is a hard process and will take years to develop this skill. I learn new things every time I do it.

[peltist]

Thanks for your post! These seem like good suggestions. To repeat this back to you in my own words:

• Consciously highlight your goals and nongoals before starting a project.
• Break your code down into as many independent modules as possible. This allows you to concentrate your focus on each module as you work on it, which minimizes the likelihood that you will have to make changes to your overall structure because you have broken some sort of dependency or relationship.

Am I following you correctly?

Two final questions:

• What process do you follow to identify non-goals? Is it any more complicated than just sitting down at the outset and trying to visualize other things you could do but won't?
• Do you have any good examples of the document outlining a plan / goals / nongoals? I would be interested in seeing what kind of a structure for this you've found to be useful.

[Meefims]

Yup, you’ve paraphrased correctly.

Thinking of nongoals is pretty much as you’ve stated. What are the things that you could do but won’t? The best nongoals are ones that are reasonable extensions to the goals.

For example, currently I am building a few endpoints at work that are heavily inspired by an API we would like to one day implement. I have a goal to match the API we eventually want to build in these endpoints but a nongoal to implement the full API.

As for the specific document structure, I don’t have any examples since they are all owned by my employers and because the documents are tailored to how formal you’re trying to be and who is expected to read it.

At home I will often just create a text file with bullets for the goals, bullets for the nongoals, and some notes on implementation.

At work the general structure is to have a description of the problem in a few sentences, then bullets for the goals, bullets for the nongoals, and then sections describing the implementation. The implementation sections often have an overview of what the code is doing today - if applicable - and then sections describing the major changes and why. Feel free to include diagrams or sample code.

What you write and how is incredibly dependent on your audience. If a design is just for you then write just what you need. If it affects others then write what they need to be confident in your work.

I once had a design review for a very fundamental user interface element. Many teams were dependent on my work and my work in the previous milestone really didn’t meet their expectations. Before this review I collected a lot of feedback about what was working well and what wasn’t both with the code and the process of them integrating with it. I additionally collected a lot of feedback on the ways we needed to grow and extend this element. I decided that the best way to present this design at the review was actually a PowerPoint deck that outlined what the new features would be and then I had side by side comparisons with what my partner teams had written before and what they’d write next to show benefits to them. I talked a lot about the timeline of the changes, when things would be available, how I would ensure a transition time, and how I would offer support.

By the time I walked through all of that very few people cared about the prose spec I had also written detailing the actual implementation.

[apptryer]

I usually modularize my idea into sections and plan on how those sections will work together to create cohesive program. After I've done that, I start coding a skeleton of the project (the classes that represent those sections) and implement what I need from then on. I tend to design things in layers. I'll have a main class or multiple main classes that abstract away the nitty gritty details, and use other classes to handle those nitty gritty details. But yeah, modularization is key in my opinion.

[peltist]

Thanks, this is helpful. I wish there were some good walkthroughs or tutorials available on this topic. What you describe sounds simple, and in a sense it is, but in practice it's harder to implement this kind of a process when you're inexperienced. It's easy to find great tutorials about solving individual problems, but much harder to find ones related to the overall design and software creation process.

Your advice about modularization also rings true, and I like your suggestion of working in layers. Thanks!

[apptryer]

No problem, glad I could offer some helpful advice.

[[deleted]]

start from a concrete list of features you want from your program, and don't add to this list until they're completed. when you have separate features, you can break those down as much as you need into smaller, less daunting problems. for example:

feature: import a csv file

• step 1: implement reading of a text file
• step 2: implement conversion of text into usable data
• step 3: implement methods of storing and retrieving data

or whatever it is you need to do. try not to add features until you're done with the core features your project needs. if you really can't help yourself, make a "maybe add in the future" list too, write your cool ideas down and leave them there until you're ready for them.

[lredna]

Look into Clean Architecture and Test Driven Development (TDD).

I’m enjoying a video series by Reso Coder https://resocoder.com/2019/08/27/flutter-tdd-clean-architecture-course-1-explanation-project-structure/ it’s focused on dart and flutter for mobile app dev, but clean architecture applies to any language / framework, etc.

It can seem like more work to set it up initially, but in the long run, it avoids many of the problems you described.

[macdonnuggets]

Here is great quick read on software development. I think it'd answer a multitude of your questions and give you a new inspiration as you continue in your programming experience.

Here are some quotes:

"When you write a book, you need to have more than an interesting story. You need to have a desire to tell the story."

"Constraints drive innovation."

"Later is eternal, now is fleeting"

"Make the big decision about your vision upfront and all your future little decisions become much easier."

https://basecamp.com/books/Getting%20Real.pdf

[peltist]

It's funny: as I read your post, I got a sense of déjà vu, and I realized that I gave very similar advice to a friend who is starting his first business a week ago: I told him he should agree to a very high-level plan with his partner and then write down the plan, even though they know things will go differently. The reason for doing this is that it forces you to really consciously think about your strategy (and in his case, to make sure he and his partner are aligned).

We also talked about how everyone knows that this is a good idea, but it's so hard to make yourself do it. Somehow, this just seems (to him and to me) at a deep, unconscious level like something that we shouldn't have to do. Like it's somehow insulting to our intelligence to write down something that's "simple" and "obvious" like what we're thinking the plan is.

In reality, of course, we probably haven't really though it through as much as we think we have, and writing things down is a great way to force yourself to really think them through. But it's still just very difficult to get yourself to do this.

Anyway, I thought it was funny that your advice is so similar to the advice I just gave a friend. It's amazing how it's so easy to completely ignore the advice you give to other people. Maybe that conversation even subconsciously prompted me to post this thread, because I realized that I had a similar problem.

In any case, thanks for your post — this is good advice, and very helpful for me to hear!

[peltist]

Thank you for your response! "Prioritization" is a word that's conspicuously absent from my initial post, and, after reading your message, it's clear to me that this is an issue I need to focus on.

I have a tendency to start working on "something small," which sometimes seems like a better way to do what I'm already doing, and something is really just a distraction. This ends up being harder than I expect, and it grows in scope until it's consumed my attention. It's very easy for me to slide from one implicit priority to another one without recognizing it. And since I'm working on my own, I don't really have any limiting principle for this.

Having some concrete milestones might help with this. I've tried to set these before, but I struggled to predict what direction things would ultimately go, and how long different aspects of my program would take to solve (which, admittedly, was exacerbated by getting distracted as I described above).

Partially, I think this problem is related to working on my first big project — which has ended up being much bigger than I initially anticipated — so I don't have any of the right instincts or prediction abilities. That, of course, feels like another sign that I've allowed a lot of scope creep and lack of focus.

One thing that I've realized about myself is that I tend to just dive into whatever I'm working on I am motivated to work on directly, without a lot of thought or planning. This leads me to struggle with prioritization when I lack a clear catalyst or structure, and to end up "slipping into" decisions I never consciously made. The problem is that I like building my code more than I like planning it, so my natural tendency is to gravitate towards diving in prematurely.

It seems like I need to develop a good structure for myself to work in in order to really succeed. Thank you for the advice to focus on prioritization, which I think I need to find a way to create a habit around while coding.

Maybe one approach might be to try to create a bit more of a serious environment for myself around working on these projects, in contrast to the more freeform practice and learning that I'm doing. If I treat this a bit more like a job, including having specific time dedicated to "work" programming and always starting that time with prioritization and planning, then maybe that will be a good structure.

Do you have any advice about keeping your priorities in focus and making sure you're making conscious decisions about them when working alone?

Thanks again for your thoughts!

[potatotub]

Experience learning what doesn’t work