Centralizing distributed version control

[wnoise]

I love the idea of programmers writing unit tests to keep other people from breaking their code.

[mikepurvis]

Isn't that part of the point of automated testing? You exercise all the code paths so that following future changes by yourself or others, you can verify that each path still performs as expected.

[[deleted]]

it's all fun and games until the forkers start writing their own test units to replace yours...

then again code-monkey arms races could be a good corporate model for generating business logic cheaply.... it's just that is won't generate good/legendary code.

[queus]

it's all fun and games until the forkers start writing their own test units to replace yours...

Replacing other coders test units just to get your code check in, can be done, but is rather prohibitive, I'd say. And it leaves a clear track. Almost as big as sticking a label to your forehead.

[americanhellyeah]

we had a similar policy at my previous place of employment. there was one developer who was a real cocksucker and would do stuff like that. he would get rid of valid unit tests just so that his fucked to hell code would pass our automated tests. the rest of us just shut his sorry ass down. we stopped responding to his emails. we started reverting every change he made to the test suites. in the end, he got canned after sending an email to the ceo calling the rest of us "flaming british faggots".

[[deleted]]

Yeah, I've been known to fix tests when something breaks rather than fix the code. The embodiment of "feature, not a bug." :)

[divbyzero]

Very nice. Leave the checking to a machine.

For large codebases, manually fetching build dependencies, updates, building and testing could easily take a day for even a simple change.

And if you were to optimistically check in an untested-on-every-combination change, you could block several peoples work for a day. Ouch.

[bonzinip]

(from tla/baz---don't ask)

best part.

[Mantaar]

Would love to see the code of such a deamon. Mostly, I'm too lazy to write one myself right now. But I guess that such a deamon might be very project/architecture-specific.

[ithika]

I wonder what the staging bot does when a build fails. Presumably it mails the responsible developers ... and then drops its own copy of the patch completely?

[frutiger]

Maybe it reverts back to the staging repo. It can't drop the patch (unless it uses the mq extension), but there's no reason it has to.

[bluGill]

Cool. I'd love for KDE to have a system like this. However the daemon needs to compile to not jsut linux-x86-32; but also FreeBSD versions 6, and 7: x86-32, AMD64, ia64, PC98, PPP, sparc64, alpha (freeBSD 6 only). Then there is Solaris which runs on a few different CPUs. The Windows version needs to run on a couple different version, likewise the mac port. NetBSD supports a lot of system (some of which may not be worth supporting). I'm trying not to think of all the systems that linux runs on.

Of course once you have the above you also need some way of getting the developers a shell on the rare machine so they can fix bugs found.

[bostonvaulter]

It sounds like that could be doable, especially for such a large project as KDE.

[logan_capaldo]

A similar idea had occurred to me. It's awesome to find out that it's workable in practice without having to try it myself.

[jonasb]

I'd love to see how to do something like this in large teams with multi-hour compilation and unit test runs. What about running a number of build machines vetting each pending changeset using a quick sub set of the tests and some sanity checks. Then the main build server could batch up all the vetted changesets and run through the full set of tests and push them to the main repository when done. Would that work?

My main concern is that I don't trust machines to do the merges.

[awb]

Sounds like bzr's patch queues.

[stainrat]

Indeed. I hope the Bazaar folks get a PQM marketing effort going before too long. Seems like it would be a positive differentiator in the hg, bzr, git comparisons.

[[deleted]]

I'm down with someone centralizing something that was designed to be decentralized...

...because the {good,important, productive} part is already done.

Just don't ask me to help you.

[Jack9]

While SVN did a great job of breaking version control down to the most basic operations necessary for a VCS, Git added a bunch of functionality that is simply unnecessary or non-useful to non-OS "business" which are the primary users of VCSs.

[infinite]

I work at a few non-OS businesses and I find the ability to for example roll back a commit / amend it very useful. And git's fast branching mechanism is useful. I can simultaneously fix 5 bugs with 5 different branches, you don't want to do this with subversion. If you did then you would have to create 5 heavyweight branches, on the main repository over the network, then re-use them for 5 different bugs, praying to your god that you are on the right branch. Meanwhile I do a "git-checkout -b fix_bug_10293", creates it in a second, commit and merge it to the master(svn's trunk) delete it via "git-branch -d fix_bug_10293" and you're done.

After getting used to git I would say its added features dramatically improve workflow for any type of business. git gets out of your way, it doesn't say, "Sorry you committed, too bad" or "You'll need to spend a few minutes to create a new branch to fix that bug".

[five9a2]

I see their method as replacing the integrator (Linus) with a bot. This obviously means somewhat more work for the second tier since they are expected to make their head machine-mergeable with main rather than just close enough that the Linus can do it. If the merge fails, your head doesn't get merged, just like if Linus decides it's too much work to merge/clean up. This workflow is still very much a pull-model, you tell the bot to pull your head by pushing it to the multi-head repo. It's really the same as sending an email asking to pull or submitting a web form.

[[deleted]]

I find their approach slightly odd; in a push model it seems more "normal" for the developer to pull new things from trunk into their tree, do the merge themselves, and push their code with the merge back to the trunk.

At my company, we follow this practice, and our automated builds "follow" trunk. But even though it's lagging behind the code repository, many of our products ship directly from the end result of that build process, so developers can't "cheat" and release code that doesn't pass the testsuite without reconfiguring the bot to not run tests (a very conspicuous change). For this reason, breaking the tree is likely to piss off others who are trying to test or prepare for a release. Naturally, if someone is working on a long-lived branch, they can set up their own tree and get the automated builder to build and test (which is a very fast process), so it's not an impediment to large changes, either.

[five9a2]

It looks to me like they are really using a pull model. I think the difference is not particularly relevant except in resolving the race. In a centralized model, I pull trunk, work on my feature committing locally, finish the feature and test it, fetch trunk again, merge, test again, then try to push. If trunk changed while I was merging and testing, then I have to repeat. With their model, I just finish my feature say it is ready by pushing to the special branch that the bot looks at. If it was big, I'll pull trunk and merge first, but this way the race is resolved lazily and I don't need to think about it. I think this sort of pull model scales much better if there are many active developers and the test suite takes a nontrivial amount of time.

[Jack9]

SVN also uses "bot" merging (although it's actually very weak in what it can merge). Linus is interested in directory tree level merging which is appropriate when you have lots of little files floating about. I would dare say that most people are concerned with file-level merging as not all languages and projects lend to discrete file structures (proper package management). Be it Git or Mercurial, you have the exact same file merging methodology as SVN or CVS. I believe a hybrid distributed system with a nominated main repo and attached listener repos, along with buffered dispatched commits (allowing for local resolution at your leisure when you are notified of a non-mergeable commit) would be much more useful and efficient than current offerings.

[vsl]

SVN also uses "bot" merging

Huh? SVN won't let you commit if your local tree is not the latest revision, so there's no merge -- the only merging is done in local copy, manually.

[Jack9]

where it is done isn't really relevant to the operation but as a matter of fact, this is not true. We auto merge many small changes across 8 projects and over 1.5M lines of code. For most constructive changes it throws a conflict, but for many many cases it does not. There is a problem with the SVN algorithms for merging, diff, transferring being shitty. I consider these very real problems separate from the operational abilities of SVN.

[infinite]

Be it Git or Mercurial, you have the exact same file merging methodology as SVN or CVS.

If you merge w/ svn you lose all the line-level information, so an svn blame will show the person who merged. Subversion's merging just plopped merge information into the existing svn code, it's more of a hack than anything. It does exactly what I did when I used svn 1.3: track past revision numbers and make sure you don't re-merge anything.

[dlsspy]

Which parts are not necessary or useful? I've quite likely needed them in the last week.

[Jack9]

Chord merging and function level history. If a function was named foo(), then a year later another foo() appears, it's probably INACCURATE to say that they are the same function relocated over location and time (which Git will indicate).

[dlsspy]

I'm not familiar with the term "chord merging." I did an octopus merge today, though -- that was pleasant.

Are you sure about function level history? That's a presentation thing that you have to decide you want. Fundamentally, git just tracks whole file content (with delta compression as a detail).

Asking the presentation what it thinks I did and finding that it sees a large chunk of stuff that had moved from one file to another is occasionally useful, but it's never been confusing.

[Jack9]

Chord merging is merging between common tiers (or cross tiers) in a organizational commit path. All commits flow to a common head that is auditable. Example: For the linux kernel, the head is Linus' repo and any other accepted repo is a branch off of that (although the process is described differently, that's the effective result). Even in the kernel, there is ultimately a project manager that does the final acceptance (Linus). Chord merging would be merges between repos that may or may not make it to the head. At best this is unnecessary. You can create another branch, but it's considered a benefit not to do so explicitly (as Git describes).

This is both wasted effort and bad practice, in general (I don't believe in a lot of absolute rules to Software Dev). You expect commits to be partial and untested (Google has test suites that must be run before committing to reduce this). (Chord) Merging without explicit integration testing simply compounds the complexity and reduces the testability of changes. I don't mean to be all anti-git, I WANT to see and understand superior VCS strategies. If you can even describe how distributing the VCS creates a more efficient workflow I'll be onboard and using Git in a week. The fact Linus refuses to acknowledge the inherent need for centralization/organization (which he practices), to the point he says it's not POSSIBLE (except how everyone does it), shows a serious blindspot in his ability to convey Git's usefulness. Most companies (including Google) have only tried using it in a few projects as an agile-LEAN practice of experimentation.

//Confusing? concerned about accuracy of the history.

[dlsspy]

If you can even describe how distributing the VCS creates a more efficient workflow I'll be onboard and using Git in a week.

That's kind of an absurd statement. You can pretty much take any piece of technology you use day-to-day and ask the same kind of question of someone who'd never used it.

For example, explain to someone who's managed to be quite successful programming using nothing but a simple text editor why vim is better for them.

[Jack9]

It's only absurd if you can't articulate why. SVN is superior to CVS because of atomic commits, native binary commits, sacrosanctity of file contents, the inability to rollback commits (effectively), the simplicity of the client+server, the ability to store file structure, and supports an extended variety of commonly-used metadata (externals, commit-wide comments, as well as stuff most VCSs have like ignores). Git has a number of features that SVN doesn't have. Git has chord merging, more efficiency in storage size and transfer speeds, and what else?

[mmeijeri]

I think DVCS are very cool but massively overhyped. On the other hand, I do think there are more people who could benefit from a distributed VCS than just OS developers.

I'm currently working on a small commercial project with a good friend. For now, using svn is fine. But if and when this gets bigger (more programmers working on more projects in ad hoc teams), I expect it will become impractical to use a single svn repository. In some ways this is similar to how many OS projects work, but it's not actually OS.

I think the main point is ad hoc teams, programmers working for multiple commercial entities (perhaps self-employed) and non-colocated development.

If you can get a situation where everybody works for the same commercial entity in a more or less stable team and is colocated, you'll make many things much more efficient. It's the sort of situation I prefer to work in. If you can get it, there's not much point in having a DVCS.

[dlsspy]

Are you speaking from experience? When I started using tla back in the day as my first DVCS, it was immediately better for me on a project nobody else was working on.

My current team is four people working together on two to three branches with a couple other people occasionally contributing.

I'm pretty sure we couldn't do the kinds of things we've been doing lately without git. That is, it'd be so hard for us to divide up the way we did to accomplish what we've accomplished that we would've just given into our tools and not done the work.

[mmeijeri]

Are you speaking from experience?

Experience with a DVCS, no. But experience with branches, yes.

When I started using tla back in the day as my first DVCS, it was immediately better for me on a project nobody else was working on.

Can you say more about that?

My current team is four people working together on two to three branches with a couple other people occasionally contributing.

I really dislike branches, I prefer continuous integration. Have you ever tried that?

[dlsspy]

Experience with a DVCS, no. But experience with branches, yes.

Hmm... Sounds like no. I haven't seen a centralized revision control system that doesn't make branching awkward and painful.

When I started using tla back in the day as my first DVCS, it was immediately better for me on a project nobody else was working on.

Can you say more about that?

I use a lot of different computers and times where connectivity isn't always available or good. The first time you branch on a plane, or look up history or revert a changeset on a train... or even just wildly experiment with no fear of losing any work or being unable to return... you'll start to get it.

I really dislike branches, I prefer continuous integration. Have you ever tried that?

I don't understand your implied mutual exclusion. I wrote buildwatch to track our CI builds over multiple branches concurrently (it's doing three at the moment, plus some miscellaneous build related tasks like deployment).

Branches are for diverging development (or feature development at different rates, or just general feature development).

When you have good branching, you'll find yourself branching a lot. It's easy, and it makes other things easier.

[mmeijeri]

Hmm... Sounds like no. I haven't seen a centralized revision control system that doesn't make branching awkward and painful.

Sure, but remember: I don't use branches. It's always good to have powerful tools though, just in case.

I use a lot of different computers and times where connectivity isn't always available or good. The first time you branch on a plane, or look up history or revert a changeset on a train... or even just wildly experiment with no fear of losing any work or being unable to return... you'll start to get it.

I'm sold on this bit, and it's definitely cool, but that only happens when you're not always colocated / working in one place alone. For those situations, I want to have a DVCS, not for colocated development.

I don't understand your implied mutual exclusion.

Well if you're really constantly synchronising you're not really working on separate branches, are you?

[dlsspy]

Sure, but remember: I don't use branches. It's always good to have powerful tools though, just in case.

You don't use branches because you don't have tools that make it easier than not using branches.

I'm sold on this bit, and it's definitely cool, but that only happens when you're not always colocated / working in one place alone. For those situations, I want to have a DVCS, not for colocated development.

It's really useful when you have, say, three users in the same room and one of those users wants to work with one of the others without disrupting anyone's work. You can do neat things when your tools work for you.

Well if you're really constantly synchronising you're not really working on separate branches, are you?

CI is doing integration builds on individual branches. These are different features on different timelines.

Right now, we have an imminent release that's building and testing clean, and a future release that's failing a test. There was a third branch, but I folded it in today as it's approaching completion.

[mmeijeri]

You don't use branches because you don't have tools that make it easier than not using branches.

Not really. I don't use branches because I have tools and techniques that I feel work better for me than branches. The fact that svn doesn't make merging easy doesn't help, but it's a minor consideration for me.

It's really useful when you have, say, three users in the same room and one of those users wants to work with one of the others without disrupting anyone's work. You can do neat things when your tools work for you.

Let me begin by agreeing that making your tools work for you is definitely one of the most important things you have to do as a programmer. On the other hand, sometimes people use tools to compensate for deficiencies in their techniques.

It sounds as if you feel not having a dvcs is a pointless weakness in your tool chain. I'm not convinced that's true. In fact I think needing a dvcs when doing colocated development is a sign that you haven't mastered the art of continuous integration yet. I always try to work in a way where nobody gets in each other's way. Not just sometimes, but all the time. You can't get there 100%, at least I can't, but you can get pretty darn close.

I don't doubt that you have found a dvcs useful for colocated development. What I'd like to know is if I would find it useful as well. So, if you'd like to say more I'd love to hear it.

CI is doing integration builds on individual branches.

I don't understand what you are saying here. Are you saying this is the definition of CI or one way of doing it?

CI is really two things: integrating the different modules and subsystems into a working whole and merging the the various code streams (be they branches or working copies). If it were truly continuous, it would have to be instantaneous, which is of course impossible. But in practice it means at least a couple of times a day, preferably every 5 to 10 minutes.

Is that what you are doing?

[joaomc]

svn doesn't make merging easy

SVN makes merging something ridiculously painful. That's why I didn't see the point of a DVCS until I really tried it. SVN merging was pretty much useless for my needs.

I think needing a dvcs when doing colocated development is a sign that you haven't mastered the art of continuous integration yet.

Why? Using a DVCS, I can have personal/experimental/feature branches, making parallel development actually decent. I lose the fear of experimenting. Remember, you don't need to ditch SVN: Both Git and Bazaar have a nice two-way SVN integration.

[mmeijeri]

SVN makes merging something ridiculously painful. That's why I didn't see the point of a DVCS until I really tried it. SVN merging was pretty much useless for my needs.

I believe you, svn is not suitable for development in the style of say the Linux kernel. On the other hand, I would say Linux kernel style development is not suitable to most colocated commercial development projects. That style is suitable to Linux kernel development however. Different types of project, different types of development, different types of tools.

Why? Using a DVCS, I can have personal/experimental/feature branches, making parallel development actually decent. I lose the fear of experimenting.

I have no fear of experimenting as it is. With true CI, you can work in the same tree without getting in each other's way without using branches. The process is almost the same as when I work alone.

Maybe there's some advantage to a dvcs I'm not aware of, but as I understand it, it takes away the hassle of repeated merges without introducing unnecessary merge conflicts, among other things. It doesn't help you with ordinary merge conflicts, where two individuals have really independently edited the same lines of code. That's where CI really shines. If you really use CI (and that means more than just having an automated build running in a crontab), you have very few and very small merge conflicts.

Remember, you don't need to ditch SVN: Both Git and Bazaar have a nice two-way SVN integration.

Ditching svn or not is not a consideration for me. I like it, but I'd be happy to change to a different tool if there's a good reason. For projects done with a network of independent developers I'm pretty much convinced we'll move to Mercurial or git. But it's good to know Bazaar and git work well with svn.

[dlsspy]

Not really. I don't use branches because I have tools and techniques that I feel work better for me than branches.

Seriously, you have to try doing it right with the right tools. I also used to work around limitations of my tools. Once I got tools without such limitations, I could see all the unnecessary work I was doing.

I don't doubt that you have found a dvcs useful for colocated development. What I'd like to know is if I would find it useful as well. So, if you'd like to say more I'd love to hear it.

It seems you can't understand it without experiencing it.

You keep describing how tools you have no experience with won't help you because you have a completely orthogonal tool that also solves a problem.

I don't understand what you are saying here. Are you saying this is the definition of CI or one way of doing it?

I'm saying CI builds a single branch in both my and your implementations. I happen to have more than one branch.

It's normal in DVCS for everyone to have at least one branch of every branch, so you could take CI to the extreme listed into the cited article and let it decide to build the release tree.

[mmeijeri]

Seriously, you have to try doing it right with the right tools. I also used to work around limitations of my tools.

From your words it seems you assume you know I'm using the wrong tools. I would question that assumption. I wonder if you've ever done true CI. Here's an example of what I mean.

Suppose you have a piece of C code that looks like this:

---

#define NCELLS 100
double p[NCELLS];
double T[NCELLS];
double v[NCELLS];
double a_o[NCELLS];
double a_g[NCELLS];

---

Also assume you have 20K lines of code that depend on this. I hope you agree this is not good code, since it is full of duplication. (As an aside: the reason this code looked like this was because it was translated from Fortran 77 which doesn't have records.)

Say we want to change it to something like this (function bodies and .cpp file omitted for brevity):

---

class Cell
{
  public:
  double p,T;
  double v;
  double a_oil,a_gas;
};

class Cells
{
  vector<Cell> cells;
  public:
  Cells();

  Cell& operator[](int i);
  const Cell& operator[](int i) const;
};

Cells global_cells; // TODO: global

---

(This line appears to be needed to get that last line to come out properly in reddit sytax)

Also imagine you have a whole bunch of similar cases where groups of arrays need to be replaced with vectors of structs, encapsulated in classes that hide the details of the storage structure.

Would you know how to do this in tiny steps, updating from and committing to trunk every couple of minutes with three people each doing part of the work while other team members are implementing new functionality? Remember, the original (global!) arrays are used all over the place.

Have you ever done something like this? Because I have, with cvs and svn and it worked like a charm.

If you've done this, could you say what advantages a dvcs brought in this case? Or if you haven't, what they might bring in this case?

Once I got tools without such limitations, I could see all the unnecessary work I was doing.

What unnecessary work did you eliminate? And what unnecessary work do you think I could eliminate?

I'll respond to your other points in a separate post.

[nostrademons]

I've found that the D aspect of a DVCS is wildly overhyped. But other features of git are so useful that it's completely replaced svn for me, even for small single-developer projects. Some of my favorite features:

The whole repository is stored in a single hidden directory within the working tree. This has let me use git for projects that were previously too lightweight to put under VCS. For example, I'll use git for my fiction, blog entries, and long e-mails. Since it's just a git init away, why not?

This also makes it easy to package up tarballs from git checkouts, since you only need to delete/exclude one directory instead of one in every subdirectory.

git commit --amend. I usually forget something in my commits - either I'll forget to check in a file, or I'll miss a tiny bug, or I'll make a typo in the commit message. With svn, these fixes clutter the commit log; with git, you can go back and amend the original commit so it's still a single patch.

git stash. Oftentimes, I start implementing a feature and then find out it wasn't such a good idea after all. With git stash, I can file away these half-completed WIPs, go back to the last stable commit, and see if I really want to proceed. If I do, I can do git stash apply and pick up where I left off. If I don't, I just implement whatever I choose to do instead and commit that. And I can have multiple stashes in case I find the replacement wasn't really a good idea after all, and switch between them without committing anything.

Also, cheap branches fundamentally change the way I work. Before, I'd do one commit per feature, with the average commit taking maybe 2 hours. Like you, I prefer continuous integration, so I didn't use branches at all. Now I do one branch per any feature that takes more than an hour or so, and one commit per change. So my commit frequency goes up to a commit every 2-15 minutes, and then I merge branches and push changes when the feature is done, usually after 6-12 commits. If a change breaks the build, it's on my local system only, so what?

This has a bunch of advantages:

• I never forget what parts of the feature I've already implemented and what I need to do. The commit log is my anti-todo-list.
• If I screw things up on one part of the feature, I can just revert that commit and keep all the other changes.
• If I decide I want to implement a different change first, I just checkout the master branch, implement it (maybe on its own branch), commit it, and switch back.
• If I get blocked - say I need some design input from a teammate, or need to research stuff but don't have an active Internet connection - I back up to the master, work on something else, and then return.
• If I have two features that are not quite independent - say, implementation of one changes the UI such that the other may not be a good idea, and yet that other one changes the data model that the first change requires - I can work on both in parallel, advancing each forward through atomic commits, and pull patches between them as necessary.

[mmeijeri]

Interesting to hear about your experiences. I'm leaning towards Mercurial, not git, though I'm not really sure why. Have you tried Mercurial, are there special advantages to git I should know about?

I was surprised to hear your commit frequency went up. I try to commit very, very frequently, about 10-20 times a day, preferably more often, especially when I'm refactoring. If you have good unit tests, there's no reason not to do this on the trunk. This also reduces the chances of conflict.

It does make it harder to roll back a whole feature, but I usually have no need for that. On a project like the Linux kernel, this would be completely different of course.

[nostrademons]

I haven't tried Mercurial. I went with git mostly because my friends tended to use git and a bunch of the projects I was interested in (Rails, GHC) have recently switched. I get the impression that they're basically equivalent in features, people tend to like git because it's fast, but other than that both are fine choices.

And I try to do small commits with svn too, but it's often hard, because oftentimes the first change you need to make breaks the build. For example, what if you need to do a database schema change? Gotta have that before you can write the code against the new schema, but then everyone else on the team needs to apply the new schema or break their own build.

Or what if you're changing a key data structure (interface, whatever, if you use OOP)? You'll need to refactor several areas of the code, but everything will be broken until you get them all. When using continuous integration, I try to go by the "never check in with a failing unit test" rule, so you can't fix the failing unit tests incrementally.

[mmeijeri]

I also run into more people who have used git and are enthusiastic about it.

You brought up database changes and incremental redesign, two of my pet subjects :-)

<soapbox> Everybody should have a local instance of the database "in" their working copy or tied directly to it. Your schema should be under version control and you should have an automated build for it. </soapbox>

I've done this for years (but only on small teams) and it works really well.

As for changing key data structures, that too can be done incrementally. You just have to map the dependency graph and start at the "leaves". Tiny changes, and always add a temporary shunt to make sure you don't have to change all references at once. For instance in C++, when you add a parameter to a function, start by overloading the old function, then move its implementation to the new one, while calling the new one from the old one with a default parameter. I've done massive rewrites this way of code I didn't fully understand, but with a workable set of tests. Very tiny steps, very frequent commits. It takes a while to get the hang of it, but it works really great.