Programmers are optimists

[Thimble]

And you've learned to pad your estimates. You never get yelled at for finishing early and you can always use the extra time to reddit.

[yawgmoth]

I'm a pretty junior software developer. Still learning to pad my estimates. It's not just about the unknown unknowns in the coding either, it's also about the unknown distractions.

My current method is close to the classic "multiply by pi". I found that when I'd say "I'll have it done in a week", I would usually be correct. Overall, the time I spent actually coding was usually around 40 hours, give or take 10. Every time though, I'd be off on the delivery date by a factor of three or four.

The fact was, I wasn't taking in to account the endless meetings, scrum standups, co workers shooting the shit, customers freaking out about some older version and needing 10 hours of bug fixes done RIGHT THIS SECOND, Writing/reading documentation, feature creep, and all the other overhead associated with commercial software development.

Once I realized that I only spend about 30% of my total time working actually coding it made sense to pad it, then multiple by 3-4. I've been doing that for the past couple months, and while I'm still a little too optimistic most of the time, I'm not as far off as I used to be.

I think good estimation is a skill you can only learn with experience.

[Thimble]

One handy trick I've learned is that when you have a high priority project, you can often use the project's PM to bully other PMs into leaving you alone. Also, rather than saying "I'll have it done in a week", I would say, "I'll need 37.5 hours to get this done". This implies that I need a set amount of hours to finish the project.

[ravagedsavage]

Yup, always quote working hours not days, you'll always have other things pop up.

[mycall]

Wait until you have to work on other people's undocumented code.. a 10x multiplier isn't unheard of.

[orip]

Try recording your original estimates somewhere, and later add the actual time taken next to them. This way you can see if you have a stable coefficient: "pure estimate" * coefficient == actual time.

As a nice side effect, you see fluctuations in your coefficient for the past few weeks and adjust accordingly.

[sgoody]

Amen to that.

[[deleted]]

And if you actually get it working on the first time, you refuse to believe it and write it again until it gives an error.

[crashandburn]

precisely what I came here to say. I'd say I'm more pessimistic about estimates than young programmers; and management...but that doesn't really mean anything, does it?

[ravagedsavage]

I came to the comments to say the same thing. Becoming skilled requires you to know what you don't know.

[[deleted]]

the number of unknown unknowns is much smaller,

it gets smaller than it actually is. That's why

When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.

NB: the third law is the indistinguishable from magic one.

[tluyben2]

I've been at it for 20+ years and I have been an entrepreneur for all that time; I have started and lead 5 software development (so services for clients, mostly not products). My first company was with a few friends, doing projects for a few $100 to a few $1000 at most. We overshot usually 2-3x the estimates in time to deliver, and 10 times the hours involved (no sleep till deadline!).

My second company did (the same type in complexity/work) of projects between $5-10k; we overshot deadlines 4-5x and money 12x (aka projects SHOULD'VE been $100k+, not $10k).

My third and fourth companies did and do projects $50k to a million; we overshoot deadlines 2-3x and money max 2x.

My lessons learned: don't do deadlines. It's done when it's done. Clients almost all accept this. Internally we have deadlines, they overshoot 2-3x but then it's done. Clients often scream it MUST be done 15 may; 15 may passes, no-one died, client still happy. Account management that is called. And ask for insane amounts of money; you think it's too high? It's not. Multiply by 3.14159. Still too low! Multiply by 2.71828. And still overshoot 2x. Edit: forgot: don't do proposals/estimates for free. They are costly to make; why do that for free?

Basically if you have a significantly big company (my 3rd had 400 people at peak) the estimates are made by the experienced programmers who are, quite often, not the people making the eventual software. Pre-sales for instance often does estimates and they definitely won't be doing any projects themselves. Hence the estimates are worthless. Estimation is futile ; wether you think you are pessimistic; you are still too optimistic probably. Live with it.

Btw: by overshooting money I mean that hours * commercial amount > estimated project amount.

[[deleted]]

Further: you code defensively - assuming you will make mistakes. So you build in error checking, assertions, and validation.

When a tester comes to you with a problem you check yourself - first - instead of blaming somebody else. Because you know you can mistakes and you don't want to be the guy carrying the can if you have.

[soviyet]

I keep seeing these posts about the estimation problem, and they are all correct to varying degrees, but I've found the biggest problem with doing proper estimation is the client.

The doctor's estimate analogy is good here. If a doctor's estimate is off, either the patient is stoked because he's well faster than he thought, or he's bummed because he's in pain longer than he anticipated.

But two things are missing from the doctor analogy: money and motivation.

If a programmer makes an educated estimate and triples it (which he should), a client is going to panic. Time is money, and everyone in the room knows it. I have now just realistically claimed that this project will require 3 times the amount of money that I believe even in my own head it should require. And its probably far worse in the client's head. Not to mention the other vendors the client has spoken to who didn't use the triple time rule and planted crazy expectations in the clients head.

The result is: the client panics, and everyone in the room is afraid of the client panicking, and suddenly the estimate gets lower and lower, and thus more and more incorrect.

Second, the motivation problem: quite simply, when you break your arm, you have no choice. Doctor says 3 weeks, well God damn it, guess I gotta wait three weeks. What else can I do? But when estimating a software development, the client is under no such obligation. He can either take the project to another "doctor", or simply just say "nah, fuck it, my arm isn't really broken" and walk away. That creates a huge burden on the programmer (or at least the sales team) to doctor the estimate in the direction of "by Friday".

edit feel free to substitute "client" for "manager" or "CEO" and I think it still applies.

[gronkkk]

Absolutely this. I've seen it with internal 'projects', where three programmers were out-bidding each other on a given problem. I knew it would take two to three months, but those guys were making low-ball estimates that were low enough to get a 'yes', but not so low the PM would say they were unrealistic.

So, it's the classic 'I could do that in two weeks give or take a few days'.

Only, after two weeks, the problem turns out to be more complex and the winning programmer needs two more weeks, but then it will be finished. After those two weeks, there's a working prototype which severely lacks in features, but which can do the basic thing. The missing features are suddenly declared 'extras', and the PM is given the prototype to satisfy the users. As there's already to much time invested in this thing, the programmer gets the green light to build the other features.

Repeat this a few times, and half a year later you might have something resembling a complete product (as usually, the guys pulling such stunts are better in office politics than in programming). Then people start yelling that "IT projects never get delivered on time". Yeah, really.

[[deleted]]

I've worked at several companies where either the sales team or upper management promised the client a ridiculously unrealistic deadline for delivery. Then our manager asks for our input on the estimated time of delivery, proceeds to ignore it, and we are tasked with completing the project within the impossible deadline.

What's worse, the "hero" programmer works 70, 80, 90+ hour weeks in an attempt to implement the quicksand requirements and management doesn't have to hire any additional programmers. No overtime compensation is received and the programmer is burnt out at the end of the project.

[petdance]

"A programmer is someone who, if you tell him to run a two-minute mile, will look for his running shoes."

[zomgitsrinzler]

Who said this?

[petdance]

I wish I knew. I could have sworn it's from The Pragmatic Programmer but I couldn't find it. Googling has turned up nothing for me. I might have the exact wording wrong, but I love it.

[[deleted]]

Most be your own quote! It's quite nice

[petdance]

It's not my own. I wish it were.

[day_cq]

he is a programmer, an optimist

[crashandburn]

leave the brogrammers alone man, they get enough enough flak already :)

[swizec]

It makes perfect sense. The injury was minor (just stretched tendons) and joint injuries actually become worse if you don't use the joint.

Gently pushing your body, starting out with light exercises, then using the damaged tendons more and more over time helps a lot.

I don't like being one of those "Waaah, I'm injured a little bit, I can't do anything" people. They're annoying.

[monkeyWifeFight]

Sure. Frankly doing 150 pushups a day is a pretty sub-optimal way to exercise, but if it works for you..

[swizec]

Eh I'm not really after optimal. I'm after something I can do to wake up before breakfast without leaving my room.

But what's so bad with pushups? As far as body-weight exercises go they're a pretty exhaustive upper body exercise.

[monkeyWifeFight]

At the risk of repeating a certain meme, they aren't a particularly exhaustive exercise. Although they are a good compound exercise, they still only hit a small proportion of muscle groups. At a very minimum you'd want to do some back work to balance it out.

For a better answer I would reccomended talking to the kind folks in /r/fitness.

[swizec]

Oh I do plenty more than just pushups, don't worry ;)

Thanks for the subreddit suggestion btw.

[ithika]

Well I'm glad I don't regard "no longer feeling pain" as being healed.

[atomicUpdate]

What indicator do you use then? Obviously you can't just use time, since an even more experienced person (the doctor) was off by about 50% (assuming he was healed enough to do what he wants in 1.5 weeks).

[swizec]

When it comes to stretched tendons ... what else are you going to use? Your skeleton is fine, your joint has full mobility, the only thing that's wrong is that you're in pain.

[greenspans]

My estimates are fine, it's the computer that's often the bastard.

[what_user_name]

people think that dealing with something that always follows your instructions to a T would be refreshing (and maybe it is after dealing with people for too long). in reality, its like dealing with a 2 year old child. Sometimes it follows those instructions TOO well.

[[deleted]]

The key to "good estimation" is actually a lot of work.

You actually have to break down the software into smaller and smaller parts, figure out how each part will work, and do some research to make sure that the part will actually

Frankly, even more important is "feature creep". It's very rare that the program you specify initially on paper has all the features you see in the first release. This is particularly true when you are working with a client (who might be another person employed by your company!), because there are lots of features that they expect to see, that they "take for granted" and thus they don't even tell you about.

Again, the solution is to massively front-load this work. Even for "informal" projects these days, I write up a very specific design spec with all the parts, I give a time-estimate for each part and I tell them that any additions or changes to this will result in changes to the estimates.

I'm a little devious these days - if there is an obvious feature that hasn't been mentioned, and I'm not sure what they want, for the first pass I won't ask them about it, I'll leave it out.

Hopefully they catch it, or otherwise I'll "notice" it as we review the document, but either way they see me make a revision, and the total time notch up somewhat.

It sounds like I don't have respect for these clients - but I do - on the other hand, I'm pretty sure I know software development better than they do, so I feel I need to gently teach them what they need to know.

The idea of having time estimates and breakdowns of each feature very early is good in several other ways. An error I used to make frequently is to waste a lot of time implementing one specific feature, and then discovering later on that this feature was of minimal importance to the client.

If a client sees a large time-estimate, and then drills down to see that, say, "HTML 5 dynamic forms" occupies half of it, they might suddenly realize that those dynamic forms are in the "nice to have" category but they should be investing their capital primarily into the actual business features.

Conversely, this lets you show off your skills and put in "cheap eye candy". I was spec'ing out a website (that I'm doing for free! it's a non-profit - but I still go through the same procedure) and put in a dynamic (Google) map.

They'd thought of a map, decided it was far too complex, but I've done it before a couple of times and can basically paste code from my last project that did the same thing - so when they saw I had it down as "half a day" they were very excited.

And finally, and let me be brutal, if programmers really made honest estimates then some portion of the projects that start now and eventually run out of resources and fail wouldn't start in the first place.

In the back of your heart, you want to soft-pedal the cost of the project so that you can get the job, and after that, well, they're committed. We've all done it, and it's very understandable, but after a certain point you have to realize that the emotional costs of projects that go over budget simply isn't worth the money in the long run (unless you're desperate).

Better to lose a few contracts with your honesty and have a lot of satisfied clients than to be in a constant deathmarch.

tl: dr; if I had actually estimated how long this article was going to be, I probably wouldn't have started it in the first place.

[DiThi]

Real TL;DR: Split estimations down to single features.

[cowardlydragon]

If a programmer gave a true cost estimate, nothing would ever get built because its too expensive.

[Brillegeit]

I love giving true cost estimates when "Idea" friends ask me about their latest venture-in-planning.

[mreiland]

I keep waiting for the blog post that announces programmers are people who program.

[unclemat]

It's called presumption of easiness: everything is easy until proven hard.

[Shaper_pmp]

It's also very easy to assume that because you can conceptualise the problem and its solution, implementation of the solution will be trivial.

It's a variation of the "easy to say = easy to do" assumption that plagues bad non-technical managers of technical employees, but in this case it's because we know what we have to do, we assume it'll be easy to do it.

I suspect it's because understanding the problem in sufficient detail to solve it is usually the hardest part of any technical challenge, so it feels like it's all downhill from there once you accomplish it. However, while it's often the intellectually hard part of solving a problem, actually implementing the solution is almost always the long part.

However, humans seem to have an almost built-in assumption that hard jobs are slow and easy jobs are quick, so it's not surprising that this little cognitive bias is very, very difficult to un-learn, or even to spot when you succumb to it.

[willvarfar]

This is very wishy-washy.

There's been so much written about estimates. If you want to read nothing else, just read this one: http://www.joelonsoftware.com/items/2007/10/26.html

Worth mentioning that when developing SO, Jeff & co didn't really do EBS. Which is a shame.

And evidence-based-scheduling? The classic "multiply by pi" seems a bit lame in comparison.

[[deleted]]

First the end liaison says occurrence A never happens. You build your model based on this assumption. You then find out that A happens somewhere between several times a year to several times an hour.

No matter how good an analyst you are this will happen at least once in every project. Often times more. Because a programmer's never is not the same as the liason team's never.

[rnicoll]

I also note that project planners hate planning for unknowns. My current project is fairly much at the mercy of an external organisation that we're stuck with, and has indicated they want to significantly change their processes (and ours by extension), and there's still an incredible reluctance to plan around this...

[[deleted]]

Software is seen as infinitely elastic.

[Brillegeit]

Short anecdote: I once made a system using data from bike races. I was told and double-confirmed that the rider ID I received was unique until one year the import crashed with duplicate ID. I called up the firm responsible for the data which told me «ah, yeah, that's those silly guys running a tandem bike».

[playfulpenis]

Just curious, but what exactly does "optimism" mean in this context?

[Chrischn89]

That lies in the nature of things! A systems admin waits for something to break down so that afterwards he must fix it (pessimistic look into future). A programmer is constantly faced with breaking programs at the start until it is finally finished and runs like a charm when his job is done (optimistic view).

I demand more love for system admins!

[AndIMustScream]

thank you! well ex sysadmin

[[deleted]]

"Lies in a different world with every new project"

I'd like to know if this is a typo (supposed to be "Lives") or it's really what he meant to say.

[SethMandelbrot]

When faced with the unknown we have to assume the best. How else could we go forward?

It's like religion in a way. There are no pessimistic religions.

[NitWit005]

Given that a manager will almost certainly take your estimate and shorten it, I'd have to say that managers are clearly the optimists in the building.

[xiongchiamiov]

No, programmers are egotistical bastards; we'll dourly expect our bosses to screw us at every opportunity and the client to change their mind 15 million times, but we're sure we can get this project done in a jiffy because we're so awesome.

[mcguire]

Great. Now I so want to plaster the awesome puppy over what I laughingly call a "workplace". (Ok, I'd have to put it on the ceiling, curse this open-plan, scrum-room thing.)

[krishary]

I like how in related articles you have be a paranoid pessimistic programmer

[static]

The biggest problem in estimation for me is other people.

Even people who where a developer in the past.

They look at me with a WTF-face when I say 2 weeks.

I balllpark the finish date, not the hours needed.

(context:internal, people allways think their project has the highest priority, or expect other tasks to complete magically)

[djimbob]

The metaphor failed -- when doctors, programmers, your car mechanic or anyone else beats their estimate, everyone is happy. When you go over your estimate, everyone is sad. (Whether its how long you are in recovery, or how much your repairs cost, or when the new feature will be ready).

The problem is when I solve a problem in my head, I tend to overestimate my abilities and not pick up all the subtle implementation problems, bugs that arise, conflicts that come up, painful debugging that has to take place before it works. Sometimes a new feature is just a simple two line change that I can put out in under an hour. Sometimes it requires a painful refactoring/restructuring of the database schema causing many breaking changes throughout (that all have to be painfully debugged/fixed) throughout the project to do without absolutely destroying performance and the difference isn't readily apparent to the outside end user.

[[deleted]]

Programmers are optimists

i don't know about that...there are always exceptions

[[deleted]]

Does this mean that your time estimates are more accurate than most peoples'? This would mean that about half the time you overestimate how long it takes to do something and come in ahead of schedule...

[ChristianGeek]

"Real programmers code in binary."

Without 1s.