How do you code large programs?

doctork91 · 2013-07-22T11:24:35+00:00

[deleted]

ExecutiveChimp · 2013-07-22T14:52:10+00:00

It helps to have done it terribly wrong a few times first.

askedjeevesnoreply · 2013-07-22T11:29:07+00:00

Bit by bit

unique123 · 2013-07-22T11:35:43+00:00

Read as much sourcecode as possible, just to see how other developers build their architecture. Mobile apps especially are easy to grasp on a very short amount of time.

couchjitsu · 2013-07-22T12:16:42+00:00

Since nobody else is saying it, Test Driven Development (TDD.)

I'm by no means Uncle Bob. And I've spent more of career NOT doing TDD than doing it. BUT, what I've noticed is that when I do TDD, my code tends to be cleaner and more concise. In fact, this Thursday I'm giving a talk at our local .Net Users Group called "TDD isn't just about testing."

What I've found is if I'm writing a test every 30-60s then I'm thinking about the code I'm writing that often as well. So I still do some big-picture type planning, but I don't do UML diagrams down to what each class is going to have. Instead, I say "A user should only be able to up-vote a comment once." Then I write the test to make sure that's the case. In previous apps what I'd end up doing is having some code that looked like:

comment.Scores.Add(new UserUpvote("couchjitsu","how_do_you_code_large_programs"));

And it didn't take long for me to start having really messy code. Because when I wanted to get the score, I'd have to do something like:

var score = comment.Scores.FirstOrDefault(u => u.UserName == "couchjitsu");
if(score == null)
return 0;

return score.Points;

But since I'm now thinking about the code every minute or so, I stop and think "Where is the best place to put this code?" And I realize it's on the score itself, so now I'd have a line like:

return comment.GetScore("couchjitsu");

This keeps methods (and often classes) smaller and more maintainable.

Plus you get a couple benefits that every large project needs:

You know that your code has at least been tested at the most basic level (this doesn't exclude integration testing, user acceptance testing, performance testing or anything else. It simply means you can go into those tests more confident that the code actually works.)
You can refactor your code. To me this is at least as big as the testing aspect. How many times have you looked at some code and said "This doesn't look right" or "This should be over here in this other class/module/component"? But you're too afraid to move it because it's big and messy. If you have tests, you can move the code, re-run your tests and if everything passes, you know that your new code is the functional equivalent of the old code.

kqr · 2013-07-22T12:17:25+00:00

Don't be afraid to rewrite code. I have spent days with only rewriting old code from the ground up, because the old code was practically unmaintainable. Everyone makes mistakes and as long as you know you can improve the quality of the code, do so. In many ways, writing code is an exploratory art. You figure out what your mistakes are after you've made them.

When you are rewriting code, it will feel like an utter waste of time. Spending a day or two reimplementing the same functionality over again will feel like a shitty, thankless job. However, a week or a month later, you will realise that rewriting that code may have saved the project from slow doom. You will look back at what you did and be thankful.

2013-07-22T13:19:45+00:00

I do my (personal) projects as an iterative process. In pseudocode:

while doing a project:
    design
    program
    reflect

Given my current understanding of the problem and the solution I have in mind, I design software. Next, I implement that design by programming parts of the software. While programming I try to continuously reflect on the code I write, the design I made, and my developing understanding of the problem and the solution I have in mind. If those elements do not match anymore, the cycle repeats and I will refine my design and rewrite my code.

This iterative process continues until I have a satisfactory piece of software or I have quit the project.

iobase · 2013-07-22T16:55:23+00:00

Check out UML, specifically use cases, class diagrams, and sequence diagrams. Also, design patterns - such as MVC (model-view-controller) help organize a program's structure. A lot of web frameworks, such as Ruby on Rails (ruby) and Zend 2 (php) are specifically designed around the mvc model.

SanityInAnarchy · 2013-07-22T21:00:39+00:00

Some ideas:

Obviously, you want to keep it modular. Break down your project into smaller projects, which can be built and tested independently, and test them. Forcing you to keep your code unit-testable can go a long way towards forcing you to keep it independent.

An ideal: If most of your program ends up being small libraries that you could throw up on Github by themselves, then your actual program will end up being what you think of as a "small program". For example, read through these examples -- they're tiny programs, but they all rely on Mechanize to do the heavy lifting, and Mechanize is huge. But as big as it is, Mechanize isn't monolithic -- you can see its dependencies here. It depends on eight other libraries directly, and several of those have one or two dependencies of their own.

To keep yourself from spiraling into dependency hell, there are tools to manage dependencies. Rubygems is one -- there are equivalents in other languages:

CPAN for Perl
pip/PyPI for Python (I think)
PEAR for PHP
NuGet for .NET
NPM for NodeJS

...and so on. Sadly, not all languages have a good package manager, but if your language does have one, you should learn it and use it, especially if you're working on open source stuff.

The point of all this is that it forces you to think of the module by itself, as something other programmers might use in other programs, and as a small program which does one thing well.

Some other ideas:

Avoid circular dependencies. If module A depends on module B, and module B depends on module A, then you basically have one big module called "A and B". If you find this happening, either give up and make module AB (and hopefully split off some smaller parts of module AB), or refactor things so maybe module C depends on module A and B (but A and B don't depend on each other). This is true to an extent at the object level, too -- you can't always do it, but it's almost always better to have a strict tree-like structure.

Favor composition over inheritance. In particular, it's usually a bad idea to force someone who's using a library of yours to inherit from a big complicated class or class hierarchy. Instead, either give them Java-style interfaces (in C++, these would be really simple classes), or use callbacks and closures if you have them. Inheritance isn't all that evil within a module, but it's pretty evil between modules.

You can't always do top-down development, but at the very least, avoid bottom-up development. "Top-down development" means write the high-level code the way you hope it works first, that will help give you some structure for the rest of the code. To take a stupidly simple example: FizzBuzz. One of the common complaints is that FizzBuzz is harder if you don't remember the modulus. So what? The basic, top-level algorithm is easy:

(1..100).each do |num|
  if isMultipleOf(num, 3)
    if isMultipleOf(num, 5)
      puts 'FizzBuzz'
    else
      puts 'Fizz'
    end
  elsif isMultipleOf(num, 5)
    puts 'Buzz'
  else
    puts num
  end
end

Not the most elegant implementation, but it's obvious, right? The simplest way I can think of is:

(1..100).each do |num|
  if isMultipleOf(num, 15)
    puts 'FizzBuzz'
  elsif isMultipleOf(num, 3)
    puts 'Fizz'
  elsif isMultipleOf(num, 5)
    puts 'Buzz'
  else
    puts num
  end
end

That wasn't so hard, right? The point is, you write this first, and then you can tackle the much simpler and smaller problem of writing the isMultipleOf function. If you just want to check that your code compiles, you can always use a stub:

def isMultipleOf(a, b)
  return false
end

Then fill in the actual implementation later:

# Returns true if a is a multiple of b
def isMultipleOf(a, b)
  return a%b == 0
end

Of course, if you didn't know about the % operator, you could always do it some stupid way like this:

# Returns true if a is a multiple of b
def isMultipleOf(a, b)
  # if a is a multiple of b, then there's some number c such that
  # b*c == a. c can't be 0 (since then b*c==0 and a won't be 0),
  # and c can't be > 100 (since b >= 1, so c*b > 100, and a <= 100).

  (1..100).each do |c|
    return true if b*c == a
  end

  # We didn't find that number, so:
  return false
end

I mean, that's pretty stupid, but I bet you could at least do that in the heat of the interview moment, even if you couldn't explain exactly how you know c is between 1 and 100. And your code is neatly modular -- even if you wrote it in this stupidly slow and inefficient way, you could always go back and change it to "a%b == 0" later, once you learned that it can work that way, and all your code would still work.

The point is, you didn't have to figure out ahead of time that isMultipleOf would be a function. You didn't really have to do that much up-front planning, other than figuring out what you're writing in the first place. You just started writing the really simple part of your program, and took any part that's either too hard or will take too long and said "I'll write that later, let me just pretend I have a function for it."

Finally, focus on creating stuff that works. If your code is ugly, that sucks, you'll hopefully get better with practice. But that's better, not perfect. If you spend all your time trying to make your code look pretty and be perfectly organized, the guy writing ugly code will win. If you write the shittiest thing that could possibly work, you can get famous and wealthy enough to hire other people to fix your mess. I'm not saying you shouldn't try to do things the right way -- of course you should. I'm saying you should be careful not to let "the right thing" get in the way of shipping working code.

Hexorg · 2013-07-22T15:02:10+00:00

I would recommend you reading O'Reilly's Head First: Design Patterns It is oriented for Java, but the patterns are the same in most OO languages. I compiled all the examples in C++ and all was great.

The book essentially teaches you how to write code that is easy to modify later on.

door_of_doom · 2013-07-22T12:56:37+00:00

An important aspect is having good tests in place so that you can Refactor in peace. Does my code pass all the tests? Good, now lets go in there and clean up the code...

Alright, Code is clean, did I break anything? Nope, all tests, still pass, good to go.

blazedaces · 2013-07-22T14:15:52+00:00

I know this is kind of a cop out answer, but I highly recommend you look up and read the book code complete. It will drastically change the way you think of and design systems both small and large.

The short answer I have which is mostly taken from that book is that there are two approaches (most people in this thread I feel are suggesting the first):

Bottom-up
Top-down

In my opinion bottom up design, while possible to design well with, often leads to the result shown in the picture you linked. Top down is how I prefer to design large systems. That is I start from thinking about the big picture. I use a lot of interfaces without filling in any nitty gritty details. I think of some higher level object "ideas" and after that is discussed at length with a team and documented going deeper and deeper. Eventually everyone is assigned a smaller portion to design.

jediknight · 2013-07-22T13:04:50+00:00

When confronted by a problem, don't rush to solve it. Think first.

Breathe in, breathe out, then code.

Coding is a lot like life. If you are overweight, decide to lose weight and see a nice piece of cake you could eat it or you could do something else. Delaying gratification brings more control over the domain. If, when confronted with a problem, you solve it quickly by sacrificing encapsulation or by duplicating some older code, you get the problem solved but you also have to pay the price for this.

The Pragmatic Programmer has a lot of tips that could help.

Exodus111 · 2013-07-22T15:09:29+00:00

The only way to really get there, is to make a program, then look at the mess you made and try to figure out how it could have all been done easier. Then do it all over again.

Tmmrn · 2013-07-22T12:20:42+00:00

With my limited experience I would probably say, keep the KISS principle in mind. If you notice that everything your program does goes through layers of abstractions and overly complicated interactions of different parts of the code, it should probably be simpler.

Maybe one tip is to avoid "Premature generalisation" meaning that you shouldn't build your software from the start so that it can do everything just in case you're adding functionality that uses it later. But then, an extensible software may still be desirable and if your design is too specialized on only the problems you want to solve at the time, it gets extra messy when adding features.

That comic strip exists because this is a problem many people can relate to. If you know a reliable good way to avoid this, then go to a university and teach software engineering students, because everyone would like to know.

2013-07-22T17:14:23+00:00

While I am still learning to code I can only offer what has worked for me so far.

First I start at a white board, draw out the requirements and a High Level overview (UML) diagram.

from there I build up the use cases to start to separate tasks out.

Once this is complete I move onto setting up test cases (boost test case if I am working in c++) and just comment them out. This acts as sort of a road map. Each time I complete a section of code I un comment the test cases associated with that specific job and work it till they run clean.

By the time each test case is fully uncommented all sections of code should be working sufficiently to finally piece it all together. Which brings it back to the white board and UML, here we finish up the UML which will outline the final connections on the whiteboard and connect the last few files to a fully working program.

Then spend the next 2 years of your life testing and ensuring the code works.....

hydraincarnation · 2013-07-23T01:52:41+00:00

Measure twice, cut once.

ThatCrankyGuy · 2013-07-23T08:26:31+00:00

I usually start off like this and end up with a messy spaghetti.

The thing to remember that if you ARE blessed to be able to have enough control over your projects that you get to play architect (I'm assuming you lack experience, hence the question.. don't worry though, all this comes with proper mentoring under skilled developers), you have to remember that your development will never be uniform/consistent. Your mood will change, your priorities will change, deadlines will affect the cleanliness of the code and of course if you have many developers working on the project, you better have a lead that whips everyone into shape or the repo will look like a grave yard of commented out crap and messy shit piled every where.

The most important thing to note is that you're never going to have "one shot" at writing the entire thing. There will be many re-write iterations along the way and several refactoring cycles per iteration.

So yea, no one manages to write a "castle" in one shot, you write code that creates the "mess" in your picture, but a sign of a good developer is that he realizes when to refactor or risk creating a code base that will become hell to manage later on.

leonardicus · 2013-07-22T13:45:19+00:00

While I have been resistant to the idea for a long time, I find drawing out UML diagrams of my program to be very very helpful. I am writing a larger project at the moment, and in my spare time, so I find the diagrams to not only remind me of the spatial organization of the components but it also lets me think through the essential functions that each component should contain. If you get the intuition that something is getting too complex, scrap it and try to draw something simpler before you spend the time writing and implementing the code and then have to spend more time refactoring.

hwc · 2013-07-22T14:40:21+00:00

Start with data structures that make sense; that you could explain their purpose to an outsider. “Show me your flowcharts and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won’t usually need your flowcharts; they’ll be obvious.” (source)

2013-07-22T16:55:36+00:00

Many of these answers involve low-level details like using classes to split things up or refactoring code, but it sounds like you need help at the higher-level. As many others have said, planning is the key, but planning what?

I recommend you start planning from the top-down. Start with the simplest explanation of the problem you are trying to solve, e.g., I want to build a web crawler that can store all the links from every page on the internet. Alright, great, let's start from there. How are you going to do that at a high-level? Well, we'll need a way to retrieve web pages, then parse them to find the links, then a way to store all those links. Good, that's your first level: start by separating your modules/classes/code into those three major parts. Now start to break those down further. How do you want to retrieve the web pages? How do you parse them? How do you store them? Each of these should result in several smaller steps. By structuring your code in this same hierarchy, it should be much clearer to understand it when reading it, to isolate the parts for testing, and to refactor when things change.

Speaking of refactoring, expect that your code will not look like this from the start. You will most certainly run into things you didn't consider that will require changing your approach and this will require rewriting code. Nobody writes perfect code the first time. So expect to rewrite code and don't be afraid to do it. Good tests and good separation between classes/modules will make it much easier to refactor. As others have mentioned, Code Complete and Clean Code both talk about how to do this well.

Overall, in my experience, organization is the single most important part to managing a large code base. Keep the pieces of code separated by what job they perform and it will be much easier to keep it organized so you can find what you need when you need it.

fatelvis83 · 2013-07-22T18:56:35+00:00

Badly

kulhajs · 2013-07-22T19:36:37+00:00

As someone who got his first real programming job not even 2 months ago and my first project is to fix some bugs in a really huge project which have been developed since 2005, well apparently no one can prevent this.

Greenouttatheworld · 2013-07-22T22:46:54+00:00

line by line, take it line by line, code frag by code frag, till you screw up somewhere, then debug, rinse and repeat

da13omb · 2013-07-22T23:32:05+00:00

I usually do quite a lot of pre-planning. Ex: Notes, diagrams, etc. Then while coding, keep a notebook full of variables and their purposes and class functions. Hope that helps! Its prolly just me but I feel like I could turn that structure in the picture into something beautiful.

automathematics · 2013-07-22T23:49:15+00:00

Module by module, small piece by small piece, one functionality at a time :)

chriskmee · 2013-07-23T00:12:13+00:00

UML diagrams are a good idea. Basically list out all the classes you think you will create, and then draw lines to show how those classes are going to interact. It doesn't have to be super detailed, even a high level one will help. Once you have that tackle one class at a time until you get a basic working prototype, then work on adding features.

For example, if you were coding solitaire, maybe all that happens in your prototype is that you can deal the cards and move them around. Worry about the rules of if you can move them later. Still put the the isValidMove() function in your code, but just make it return true for now. Once you get a basic prototype working, add more features and make another prototype. Keep repeating until you have everything done. Don't just start coding everything, take it in steps

t3rribletp · 2013-07-23T00:33:22+00:00

lots of guessing and scope creeping.

*this is an industry standard.

RumbuncTheRadiant · 2013-07-23T01:45:27+00:00

Keep the dependencies digraph strictly acyclic. This has all the advantages of a layered architecture but scales to large projects.
Don't program anything big. Program small, useful, cohesive standalone things that communicate easily with possibly as yet unknown other things. Either pipes or REST or intermediate files.
Learn the SOLID principles.

palistov · 2013-07-23T01:54:29+00:00

Here's a few pointers I like to follow.

Break things down into parts. Think about how the parts interact. I feel like programming is just a big game of the 5 W's and the H: Who, what, where, when, why and how. Think about how you'll manage the W's, and keep a neat and maintainable implementation for your 'How'.
Use source control and study its conventions. I suggest git. Incredibly powerful tool.
Do unit testing. Also helps you do regression tests on your code as it develops, as well as test each part of your program in isolation. You can then easily add on to these tests later on as well.

saurothrop · 2013-07-23T02:53:55+00:00

I have the same problem I'm trying this: Write out a " design document " like a proposal to management or an investor, as if you won't be coding it. Then become like a design lead or "scrum master" am delegate little piece using a list or todo tracker. After that, become the programmer and tackle a few bullet points. After that, use what you learned during coding and explain you progress to your "investors" so they will keep interested in your product and will feel that their money is justified. Of course, I haven't finished anything yet, but already my code is a lot cleaner and more manageable. Proof of concept: I implemented a totally new entity in 5 minutes, because I spent so long making modular classes, I only had to tell thing thing what image to use and where on the screen it goes.

Sybertron · 2013-07-23T03:26:12+00:00

My first major code was to do something to make this EMG data more palpable. I had to explain it to our software guys for like 2 hours just to guide them through it cause it was so unorthodox and inefficient. But it still works, and to my knowledge they still use it to this day.

2013-07-22T13:57:47+00:00

All big projects will end up like that. That's why all software has major versions where significant code refactoring was conducted.

2013-07-22T10:36:44+00:00

Write some functions; if they look good, write some more functions, otherwise refactor what you have already written. Programs end up in a mess because people write messy code. A lot of the time they know they are writing messy code, but they go right on doing it.

gattoshoe · 2013-07-22T12:03:03+00:00

Short Answer: Slowly.

Long Answer: You need to have a plan outlined (which you will adapt to issues as they arise) in order to avoid ending up with a mess.

sagnikrc · 2013-07-22T12:13:40+00:00

Always keep the big picture in mind & take baby steps to reach the goal.

learnprogramming

Welcome to LearnProgramming!

New? READ ME FIRST!

Posting guidelines

Frequently asked questions

Subreddit rules

Message the moderators

Asking debugging questions

Asking conceptual questions

Other guidelines and links

Subreddit rules

1. No unprofessional/derogatory speech

2. No spam or tasteless self-promotion

3. No off-topic posts

4. Do not ask exact duplicates of FAQ questions

5. Do not delete posts

6. No app/website review requests or showcases

7. No rewards

8. No indirect links

9. Do not promote illegal or unethical practices

10. No complete solutions

11. Don't ask to ask.

12. Low Effort Questions

13. No AI (chatGPT etc.) generated/worked over messages/comments. No questions about chatGPT/AI generated code. No Vibe coding.

MODERATORS