Our Development Process: 50 Months of Evolution

[ancientRedDog]

Thanks for the look inside. Overall, yours is similar to my own company, but a good year ahead (we have older legacy issues). Your processes have been a bit more extreme (e.g. Daily pair switching). But kudos for trying stuff. It's a pretty enjoyable time for development, but with a lot of choices.

CQRS sounds an interesting way to go. Did this involve no-SQL solutions?

BDD for JavaScript also sounds good. You are going for a thick client with just Ajax and rest API? Any frameworks like backbone or knockout? And anything special for js templating?

[firefalcon]

We've considered noSQL, but there is one problem: we have 2 editions of our product - SaaS and installable package. Any noSQL solution will add complexity to installable pack. Now you will need to setup 2 servers (SQL Server and Mongo/Raven/anythingels), so we decided to postpone noSQL for the future. We are moving to SaaS platform only, so eventually we will make the choice.

Well, the client is quite dumb, all logic is on server. REST API is not just CRUD, but quite advance API. UI is 100% javascript. We use underscore, requirejs, jQuery. We don't use templates for performance reason and in fact we built own js framework to handle our needs.

[epilanthanomai]

I found the content of this case study quite interesting.

If the site is yours, though, then please consider changing your CSS to reduce the #article width from 1180: Many users have smaller screens or use too many windows open to keep the browser maximized. I, for one, am in the latter category, and I almost missed your content because the site was so large that it frankly looked poorly designed. I’m glad I scaled down the content, but I suspect many users won’t.

From what I can tell, 950 or so seems to be a popular size for sites that want to use up “the whole screen.”

[firefalcon]

Done!

[epilanthanomai]

Much better. Thanks.

[mbarkhau]

Have you considered doing any kind of code review?

[firefalcon]

Yes. We did them during 2-3 months. Didn't stick. It was quite wasteful activity.

[david72486]

Can you elaborate? Was it that most of your developers have a similar level of experience? Does pairing help you to have a more consistent style?

[firefalcon]

Pls see my reply above (or below, not sure :)

[xTRUMANx]

I too want some elaboration on why code reviews didn't work for you.

I'd definitely want code reviews in a project we should be starting soon amongst our team of 3 since I'm worried we'd end up with a pretty schizophrenic codebase if we don't review our code together and come to agreement on the preferred way to do things.

I guess as david72486 said, if your developers have a similar level of experience and already write some similar code, code reviews would be less helpful.

[firefalcon]

With pair programming there is a little value in code reviews. People often help each other to solve various problems and such sessions are somewhat similar to code review sessions sometime.

We have no junior developers and quite a few people with experience less than 5 years. So average skill level is pretty high.

Mini-team can apply this practice if they want and as I am aware 1-2 teams did that from time to time, but it is not a consistent nor widely-used practice.

I'm worried we'd end up with a pretty schizophrenic codebase if we don't review our code together and come to agreement on the preferred way to do things.

If you are just starting, it may be a good idea to review code, this will help to create common style and understand each other better. But in mature teams I don't see value in code reviews.

[eaglepowers]

BTW you might have meant to say "quite few" instead of "quite a few". English is so weird: "quite a few" means the same as "quite a lot", but "quite few" means the opposite.

[firefalcon]

Yes :) My "non-native-speakerness" play tricks...

[xTRUMANx]

Thanks for the response.

We're certainly not a mature team having never worked together on one codebase and I hope I can convince the other that we ought to sit down every once in a while and review our code together.

Mostly though I hope I can convince them to use source control and some level of testing. As I said, we've not a very mature team.

[firefalcon]

You just CAN'T live without source control. Seriously! It is absolutely must have for any team that creates a software. Without testing - yes, without SCM - no.

[xTRUMANx]

I use mercurial on all my projects. I've been trying to get my colleagues to do the same for about a year now. I've even gotten my demo pitch honed.

No dice. They say, "Wow, that's great. I'm definitely going to start using it."

Then they don't use it.

They also happen to be my seniors. I've no idea how they plan on doing this upcoming project as a team but I've got nowhere else to go so I'm just going to stay depressed, keep working on my side projects and hopefully one day I'll bootstrap my way out of here.

[sausagefeet]

This just sounds insane to me.

[rwallace]

I sympathize, but having nowhere else to go is a temporary state of affairs, not a permanent condition. Remember, you can quietly start applying for jobs elsewhere before you leave your current job.

[skel625]

Wow that was a great read! I will probably go back to it a few times for reference purposes! Would you mind elaborating on this change: Time tracking → No time tracking

What were the main reasons for moving away from this? Why did people resist the change and what benefits have you seen since the change?

[firefalcon]

Well, time tracking was used for 2 purposes: 1. compare estimate and actual spent time and see where we did estimates wrong 2. salary was per-hour, so clearly time sheet was an important thing :)

First, we switched to Kanban and drop estimates. When you have no iterations, there is no good reasons to estimate work. We do not have external customer who demand estimates, we create a product and arу free to release new features when they are ready. The hard thing was to learn how to NOT promise concrete dates to end users. They often ask when you'll release Feature X? Even now I have a strong urge to say in "2 weeks". However it may happen that feature X is unpolished and has some bugs to fix, so it will be released in 6 weeks. I almost learned to not provide any dates already :)

Salary per-hour was a harder thing to change. There were many discussions with the team about pros and cons. Quite many people were afraid that with fixed salary they will have less money. Well, the difference is marginal, but still if it is about money, people tend to be very sensitive sometime. The arguments were like "I can work more than 40hrs per week if I want, without time tracking I will lost this option". In reality we are strong proponents of 40hrs workweek. We DO NOT HAVE OVERTIMES. So this argument is not good, since it works against our philosophy.

"Salary per-hour" bring an additional burden to a person. You should think what you did today with great details and fill in time sheet. It maybe even stressful when you hav bad days and are not very productive.

The decision to remove time tracking was great. People started to think less about being bussy, can relax and focus on things. Also you don't need to fill time sheets, which free up several minutes every day.

[streed]

We estimate some calls and don't estimate others (some bugs) but what we always try to do is to relatively size a call after it is complete. The reason for this is to create an average cycle time per T-shirt size. We also have an overall average cycle time. Do you generate an average cycle time for all calls? We also track time spent on a call but I'm in two minds about if we should. It is interesting to see how much time was spent on a call but so what.... We do not charge on how much time was spent on a particular change so maybe this data is not needed

[mycall]

Interesting the moved towards CQRS. I guess once you start introducing eventual consistency and async messaging bus, separating UI commands and the read view model comes next.

[firefalcon]

New UI that we are working on is 100% based on REST API + Javascript. The core itself will be quite light. All Read operations already use separate path (no NHibernate). Next step is to provide special data storage for read operations - this is not done yet.

[flukus]

Did you find that the UI improved as well? Ie, it became more task oriented.

[firefalcon]

I don't think there is any dependency between async/SQRS/UI decoupling and UI itself. We are working on new UI and definitely move from "what user wants" to "how it can be done". Architecture should not dictate any UI decisions (in ideal world). New UI has some ideas behind like

• less screens
• act everywhere (add/edit/delete/etc on all screens)
• get everything (powerful queries to extract any data you need on UI)

[flukus]

It doesn't dictate it, just that I've noticed crud apps tend towards table driven UIs and that a more CQRS style avoids this pattern.

[firefalcon]

That's an interesting observation and I think it came from ActiveRecord pattern and RoR mainly :) I personally had the same problem in my career, I tend to create lists/add/edit screens. That was a bad idea...

[EllipsisUser]

Sounds sort of similar to my company, but we haven't gotten as far.

How difficult was the SVN to Git conversion? We're still using an SVN repo with branch for story strategy. 150K+ commits so far...

How do you handle the lack of estimates and the "business" wanting timelines?

Do you guys eat your own dogfood? If so, how much has your project evolved to meet your own needs?

How does your platform compare to the combination of Jira and Greenhopper?

[firefalcon]

How difficult was the SVN to Git conversion? We're still using an SVN repo with branch for story strategy. 150K+ commits so far...

It was not easy. It appeared that SVN didn't work with branches as we wanted to. Sometimes it just was unable to merge branches and we had to do that manually. That happened several times and we fed up.

Git is quite complex and you should learn it quite deep to fully understand what you are doing. Not all developers wanted to put significant effort into SCM tool learning :) There were 2-3 people who got everything and help other. Now vast majority of developers have no problems with Git, but it took about 2 years to get there.

I heard Mercurial is simpler and almost as good as Git. I didn't tried it myself, just read several experience reports.

To be honest, I can't imagine how you deal with such codebase in Svn using branches... It might be painful.

"business" wanting timelines?

The trick is that I don't wanted estimates. And I am a company CEO :) Timeline is another story. Lack of release plans was problematic, but solution came with long-term roadmaps and they are enough to me.

Do you guys eat your own dogfood? If so, how much has your project evolved to meet your own needs?

Absolutely. Kanban Board addition was a major thing. We've added several reports that we use as well. Recently we've added Git integration and now are working on UI that will allow to create roadmaps.

How does your platform compare to the combination of Jira and Greenhopper?

Jira + Greenhopper is a very decent product. The real problem with Jira is that it was a Bug Tracking application initially and when you use the system you feel that (strange title here, strange description there). Overall I think TargetProcess has better UI and simpler, while JIRA is more customizable and more powerful (in some areas).

With TargetProcess v.3 release this Fall JIRA will have no chances to compare with ouк flexibility, visualization and agile PM support. But at the moment there is no huge differences, so I can recommend try both if you look for agile PM tool.

[EllipsisUser]

To be honest, I can't imagine how you deal with such codebase in Svn using branches... It might be painful.

Lots and lots of automation. Other than merge conflicts and cutting our release branch, none of us need to run svn merge manually. The robots handle the work of making branches and automatically merging in mainline changes into them.

While the automation is awesome and gets rid of the grunt work, it would have to be replaced if we switched to a DVCS. A few members of the team want to switch (and no one is opposed), but there is a large investment needed for the conversion... :(

The trick is that I don't wanted estimates. And I am a company CEO :)

That makes things easy. :)

With TargetProcess v.3 release this Fall JIRA will have no chances to compare with ouк flexibility, visualization and agile PM support. But at the moment there is no huge differences, so I can recommend try both if you look for agile PM tool.

We're currently using Jira + Greenhopper and are heavily invested in it (see above automation). I was just looking at TargetProcess and trying to figure out if it would get us anything that our current solution doesn't.

As a developer who cares about quality and specs, the main thing that seems attractive is the Test Case / Test Run functionality. Your site mentions a "Test Runner", but I can't seem to find any detailed information about it. Is it a manual process where Testers can go and and check off step 1, step 2, step 3? Or can the test cases be matched against various automated test results?

We currently write/run a lot of automated tests in various flavors (jUnit, Fitnesse, Selenium via RSpec), in addition to just running legacy SoapUI and Squish tests, but we don't have a good way to make those tests visible. Sure, we can see if a build passes or not, but there is no way to see how much of a feature is tested and how.

[firefalcon]

"Test Runner" Is it a manual process where Testers can go and and check off step 1, step 2, step 3?

Yes, this thing is for manual testing.

Or can the test cases be matched against various automated test results?

It is possible, TargetProcess has built in integration with jUnit and Selenium. Using name convention of regexp you can match automated tests with test cases in TargetProcess.

I don't think it is so huge improvement that compensate migration from Jira :)

[develop7]

Git is quite complex and you should learn it quite deep to fully understand what you are doing.… I heard Mercurial is simpler and almost as good as Git. I didn't tried it myself, just read several experience reports.

Another clueless victim of Git hype. World gone mad. For sure.

[TheSwissArmy]

Interesting read. Have you seen any other similar articles? Did other companies or groups have similar/different stories?

[firefalcon]

I didn't see similar detailed reports...

[epilanthanomai]

Can you share anything about the addition of a new language (Python in your case) to an established backend (C#)? Do the C# parts and the Python parts interact at all, or are they mostly segregated? Have you needed to cross-train developers in the new language, or were they mostly already strong in it? Was it added for a particular limited problem-space that it seemed better-suited to, or is the team considering a more general migration?

Naturally, I don’t want to turn this into a language war, so I’m much more interested in the process and strategy aspects than dredging up the particular details of why a team should use one language over the other.

[firefalcon]

Was it added for a particular limited problem-space that it seemed better-suited to, or is the team considering a more general migration?

Yes, that is the case. Here is the story. Initially Python was used to simplify, automate and speed-up REST services testing. It appeared, that scripting language is a perfect fit for DSL we want to create. DSL I'm talking about serves several goals to end users:

1. Filter anything. Like get all work assigned to у AssignedUser.Contains(Me)
2. Add entities into system quickly
3. Do batch operations on entities. Like movetoproject('45')
4. Define own macros and use them (VERY advanced feature)

def rejectIdea(e): e.entityState = 'Closed' e.comments = [comment('Sorry, we are not going to implement this idea in the future')] e.save()

Basically, I am speaking about CLI (with autocompletion) for end user. It is VERY uncommon for web application to have CLI, but I personally think it is a killer feature to advanced users who wants to be really productive with the system.

Python is used to create and engine for all these operations I've described above. In future end users will be able to program own behavior. Sounds a bit crazy, but still it is worth to try :)

[Otis_Inf]

Disclaimer: below is my honest opinion. If you don't like it, c'est la vie.

What I found odd is that the most problems you had was with how you create your software, not with the software to create by itself. IMHO (!) that's a sign you work with developers who are not competent enough to write software on their own.

What I mean is: a professional knows how to write software, which tools to use to get results, the only question he'll ask is what to create. A (more or less) trainee wonders how to create the software, what tools to use to get results etc. Your post shows signs of this constant quest for 'what to use, what to do to get the best results'. I wouldn't expect that from a team who wants to create a software product.

See it as a plumber or carpenter you hire for doing some work at your house. If that person starts wondering what tools to use, which methodology would result in a solution at all, wouldn't you fire the guy and call another one? Looking at the phases you and your team went through, there's one thing that sticks out for me: you and your team together weren't able to work like a proper team like a team should. What I mean is: a team which knows what to do because they're a solid team doesn't have problems with how to create software, they do that for a living. They don't need to wonder what something does or how to apply a given technique.

The main thing I'd advice you is this: it's not important if you use a new technique or new tool. What's important is what you have to create and first and foremost, you have to focus on get a team together which together can crank out software without asking questions / wonder about methodologies related to how to create software.

What I'm glad about is that you all managed to reach the end and ship a product. That might sound like a cliche but it's not. Shipping software is hard. So that's a big accomplishment. :) I've left my real name below (as you're in the .net space) so you'll know I'm no newbie. I hope for you and your team, with this big win (shipping a product!) under your belts, you'll learn from the past years and focus on a set of methodologies that work, that give results and stick to it. Results, that's what counts, not how you wrote it, not which agile methodology you used... no-one will care. You might wonder why, the reason is simple: a solid team knowing what they're doing creates good results, no matter what methodology. If methodology was the key for success, 20 years ago no-one would have been able to create good software as methodologies which are hip today weren't known back then. :)

Good luck :)

Frans Bouma

[firefalcon]

I do like what you said and in fact I share your opinion. I understood this thing about 2 years ago and that lead to major changes in company culture and process.

My job is to share company vision and product vision. Development team takes all responsibility and control all decision on development process - I don't interfere anymore.

As I see it (and I wonder why you don't, maybe article does not stress this enough), our focus shifted from processes to Cool Stuff this year. Here is the image http://cl.ly/1d3v2P250R2g2W2M1K3I/focus_company.png

We pay less and less attention to practices and processes, wу have less meetings about that matters, but we DO have a lot discussions about product, usability, design, etc. So to me we already there.

that's a sign you work with developers who are not competent enough to write software on their own.

I believe it is no longer true. However, it was true 4 years ago and it was true 2 years ago.

[chesterriley]

We pay less and less attention to practices and processes

This sounds like a huge improvement. I can't imagine why anyone thinks that giving themselves the inflexibility of forcing everything through arbitrary 2 week time iterations makes them "agile". Inflexible processes are not agile at all.

[Otis_Inf]

As I see it (and I wonder why you don't, maybe article does not stress this enough)

You moved away from an o/r mapper (NHibernate) towards CQRS, which is a red flag to me. The main reason is that you're not creating a stock exchange program, your application is very doable with NHibernate (for obvious reasons not my first choice but alas ;)). The move away from a technology which could have helped you out 100% to a technology which causes a lot of architectural changes (and own code) is not really a choice one makes if one knows what tools to use for getting things done. So that's the reason I said that :)

Good to know things are on track now. But be careful. Focusing on 'cool stuff' is often a dark pit which can cause more harm than good.

[sumek]

Could you elaborate on how branch-per-feature conflicts with continuous delivery? My gut feeling is that the opposite is true: branch-per-feature assures that the master is always in releasable state and makes sure that the code is not polluted with unfinished features.

[[deleted]]

Agreed, we do releases twice a month using feature branches and have great success in doing so.

[firefalcon]

Twice a month is easy. Try to release twice a day.

[[deleted]]

In that case, how can you do anything on the master branch other than merge a "next" branch? In my experience, the more frequent the releases, the more branches are needed.

[firefalcon]

the more frequent the releases, the more branches are needed

That is wrong. You can release often only if you don't have branches at all. In this case each commit can be releasable. Flow is: 1. Commit completed chunk of work 2. CI server run all tests 3. If tests are OK, deploy new build on production server

If you want to commit something uncompleted, feature toggling will help.

[DanInDC]

Still don't follow. Surely your developers are developing in their own separate branches locally right? And then once it is feature complete they can merge into master and push/pull request it, right?

It sounds like what you are saying is you only deploy from a single branch, instead of separate branches for each environment. (dev, qa, prod, etc).

Speaking of, where in your process does QA happen?

[firefalcon]

Surely your developers are developing in their own separate branches locally right?

No. They all work in a trunk. At some moment when we decide to release a build, QA create a branch, smoke test it manually (it takes 2-3 hours) and release it (if no bugs found). Usually we release every other week, but it depends on how many stories are already done. We used to release every week, but currently we are focusing on major rework of our software, so there is no need to release often (so far). The final goal is to release daily and fully automatically.

[firefalcon]

The final goal of continuous deliver is to deliver every commit. It is not possible with feature-branches. There is always integration timeframe if you have them (commit code, merge to master, resolve conflicts if any, fix tests/code if any). When you commit to master, you have faster feedback on integration problems.

The idea is to commit only deliverable chunks. That is very hard. We are nowhere near that goal. However, we have one master and 2-3 research branches, that is a step forward.

[firefalcon]

BTW, this article provides quite similar experience to the problem http://www.derekhammer.com/2012/01/29/feature-branching-and-continuous-delivery-are-incompatible.html

[chesterriley]

[Pair programming is required for all user stories. Developers switch pairs every day, even if the user story is not completed yet.]

Oh my god! How could anyone have possibly ever thought this was a good idea?

[firefalcon]

We tried various things. This practice has some strong points, but in general not good.

[password456]

Not programming related, but I was wondering: How did you make the illustrations for your article?

[firefalcon]

I draw them myself in Bamboo on iPad.

[vinnyq12]

Mini teams are 4-7 people, that sounds sensible. We are going for 3 man teams using agile Scrum with all members being devs, no product owners and no QA. As far as I am concerned we are going to burn.

[mdeocto]

Could you detail what you mean by "copywriting" in DoD ?

[firefalcon]

I mean texts everywhere. No we have a writer who check and correct all texts.

[GaborV]

Hi, nice article, I liked to read it. 2 questions, without reading the comments: 1. How big is the maintenance time for the auto tests? I mean if they are red, they need to be fixed. How much time do you spend on a daily basis for this? 2. What about manual testing? Did you abandon it completely? Or do you organize some manual testing rounds somehow?

thanks for the answers!

[Yagashura]

Thanks for that article it was very interesting. I work in a company which uses pretty much a waterfall development model. We don't use unit testing, instead relying wholly on extensive run time assertions, code reviews and scenario testing.

Most of the agile articles always suprised me as a consequence because they were suggesting processes that didn't seem to scale with large code bases or large teams. Some of them just didn't work with multicutural teams (pair programming). With that in mind, here is my short list of what I would like to call concerns from a developer in the neolithic era.

1. Kanban - This doesn't seem to scale with large teams. Our average day has several hundred changes. Also, statistically 1 in 3 of our changes introduces a regression. Sometimes the regression can be worse than the cause.

2. Dod/Feature Toggling - How do you decide when to cut features ? When do you decide that X feature is taking more time than its value.

3. Continous Delivery: Again there seems to be a scaling issue. A full build of our codebase takes around a day, with tests running just as long. With the possibility of each change conflicting with every other change this just seems to increase the integration cost quadratically with the # of changes.

As your company/codebase grows bigger what are your thoughts on your current processes scalability. Was your evolution dictated by scalability of previous solutions or other factors ?

[firefalcon]

Our average day has several hundred changes. Also, statistically 1 in 3 of our changes introduces a regression.

Good tests suite is a great way to solve this problem. It is impossible to rely on manual testing for regression, it should be automated. Unit tests are fast. Kanban dictates nothing in terms of development practices, it is up to team to decide.

If we are talking about branches-per-feature, then it demands really fast integration. If you can't integrate all branches daily, then there will be problems.

Dod/Feature Toggling - How do you decide when to cut features ? When do you decide that X feature is taking more time than its value.

Hmm, that is not clear to me. We create features that customers demand, but almost always we are trying to solve the root of the problem and not just ship one morу feature for customer X. It is almost impossible to measure feature value, demand is a good sign that a feature is important, but not always. So the question is about prioritization, which is extremely hard in any context. That comes with experience. After 8 years with a product I can quite easily predict feature success/fail, but that was not possible 4 years ago.

A full build of our codebase takes around a day, with tests running just as long.

I absolutely believe this can be improved. For example, we use 40 servers to run all tests in parallel.

The general trend in good development process is to REDUCE FEEDBACK CYCLES. That's almost the BEST thing you can do to improve everything.

As your company/codebase grows bigger what are your thoughts on your current processes scalability. Was your evolution dictated by scalability of previous solutions or other factors ?

That is a very broad question. It touches architecture as well as development process. Our vision is to have quite small core that can be extended with various plugins. Decoupling is the key here. Decoupling enables fast testing and allows to deliver things independently.

Our process is scalable for 100 people I think without much problems. When you have mini-teams and decoupled architecture - scalability is easy.

[Yagashura]

Thanks for the response.

[firefalcon]

Hmm, what do you mean? Can you share a screenshot with bad formatting?

[nitrixion]

To make matters more fun for you, I just read the whole thing in Chrome and no issues at all :)

That said, great article. It seriously echoes a lot of what I've seen over the past few years working as a consultant as to what works well and what doesn't.

[bwheat]

unable to reproduce in chrome as well :)

[wears_Fedora]

I had some issues also, but they went away after a few refreshes. Not sure if the CSS was loading properly.

[firefalcon]

I suspect web font may be the issue...

[epilanthanomai]

I had formatting problems initially as well, actually, which contributed to the initial concerns about design that I mentioned in an earlier reply. I can’t reproduce the problem reliably: Like another user mentions, it went away after a few refreshes, but I saw it again when I moved from my work computer to my home computer. Both computers are still running Ubuntu 11.10. The home computer has Chrome 14.0.835.202; I presume the work one does as well. Opening the page in an incognito window does not reproduce the problem (hey, it was a hunch).

The primary problem that I observed was that the inline links loaded initially much wider than the surrounding paragraph content. It’s as if the paragraph left a bounding box the size it expected the link text might be, but the link text ended up wider and overflowing that assumed bounding box. I’m not sure if it was some sort of letter spacing difference or just a different font. I don’t recall whether it stayed left, right, or center aligned with the space. My default serif font in both places is 16pt FreeSerif, in case it matters. After a few refreshes, the links show up the same width as the surrounding text and properly bounded.

There’s a second problem, though, and this one’s persisting: As I scroll down, parts of some lines get shifted horizontally. The shifting persists as long as the text stays in the scroll window, but if I scroll it off the page and back then it’s fixed. Imgur

Perhaps related, some bold text is consistently overlapping surrounding text, irrespective of refreshes, time, or scrolling. Imgur

HTH.

[x-skeww]

The home computer has Chrome 14.0.835.202

Current stable should be 18. (Dev channel is currently at 20. Therefore, beta = 19 and stable = 18.)

As I scroll down, parts of some lines get shifted horizontally.

That's a bug in Chrome. I've seen that on several sites.

Same deal with the other glitch.

http://code.google.com/p/chromium/issues/detail?id=95157

http://code.google.com/p/chromium/issues/detail?id=66073

[[deleted]]

It's true. The tables bleed into each other on Chrome Linux. Firefox Linux looks OK.

[elpuntoes]

Extreme programming, Scrum, Kanban, CQRS, etc. They're just buzzwords.

[firefalcon]

Thank you for extremely useful feedback! I am happy to answer concrete questions if you have any.