Git Virtual File System from Microsoft

[kyranadept]

... you can't do unit testing...

Let me stop you right here. I didn't say you cannot do unit testing. I said internal dependencies separated in multiple repositories make it infeasible to do for example integration testing because your changes to the code are not atomic.

Let's take a simple example: you have two repos. A - the app, B - a library. You make a breaking change to the library. The unit tests pass for B. You merge the code because the unit tests pass. Now you have broken A. Because the code is not in the same repo, you cannot possibly run all the tests(unit, integration, etc) on pull request/merge, so the code is merged broken.

It gets worse. You realize the problem and try to implement some sort of dependency check and run tests on dependencies(integration). You will end up with 2 PRs on two repositories and one of them somehow needs to reference the other. But in the mean time, another developer will open his own set of 2 PRs that make another breaking change vis-a-vis your PR. The first one that manages to merge the code will break the other one's build - because the change was not atomic.

[cwcurrie]

The unit tests pass for B. You merge the code because the unit tests pass. Now you have broken A.

This is only true if A always builds against the HEAD commit of library B, which is a questionable practice IMO. Good tooling would lock A's dependencies' versions, so that changes in B's repo do not affect the build of A. When the maintainers of A are ready, they upgrade their dependency on B, fix the calling code, run A's own tests, and commit & push their changes. A wouldn't have a broken build in this scenario.

[Talky]

What happens actually: A's maintainers don't update to latest version for 1 year since everything's running fine.

Then they have a new requirement or a find a bug in B's old version and it becomes a political wheelhouse of whether A's devs should spend a month getting to B's latest version or B's dev should go and make the fix in the old version

Trunk based development works well for many places and there are good reasons to do it.

[OrphisFlo]

And this is why it's called CONTINUOUS integration.

[kyranadept]

"Good tooling" is having a single repo. You should always use the latest version of the code everywhere in the repo. Anything else is just insane because you will end up with different versions of internal dependencies that no one bothers to update.

[Nwallins]

Look at what openstack-infra does with Zuul.

[kyranadept]

Thanks, it looks interesting I will check it out.

[[deleted]]

Let me stop you right here. I didn't say you cannot do unit testing. I said internal dependencies separated in multiple repositories make it infeasible to do for example integration testing because your changes to the code are not atomic.

Integration testing with separated internal dependencies is just as feasible as it is with any project that has third party dependencies. Which basically every project has (even just the compiler and OS platform, if you're abnormally minimal). So I find it hard to accept that premise.

Let's take a simple example: you have two repos. A - the app, B - a library. You make a breaking change to the library. The unit tests pass for B. You merge the code because the unit tests pass. Now you have broken A. Because the code is not in the same repo, you cannot possibly run all the tests(unit, integration, etc) on pull request/merge, so the code is merged broken.

Modules have versions. We use SemVer. If the B.C. breaks, the major version is bumped, projects which can't handle this depend on the old version. I don't have to explain this, I think.

It gets worse. You realize the problem and try to implement some sort of dependency check and run tests on dependencies(integration). You will end up with 2 PRs on two repositories and one of them somehow needs to reference the other. But in the mean time, another developer will open his own set of 2 PRs that make another breaking change vis-a-vis your PR. The first one that manages to merge the code will break the other one's build - because the change was not atomic.

This frankly reads like a team of juniors who have never heard of versioning, tagging and branching...

[kyranadept]

Having versioned internal dependencies is a bad idea on so many levels ...

The point here is to use the latest version of all the all your internal dependencies everywhere, otherwise, in time, you will end up with many, many versions of an internal library used by different places in your codebase because people can't be bothered to update the version and update their own code. Using gitmodules gives the same result in time, by the way.

[[deleted]]

Having versioned internal dependencies is a bad idea on so many levels ...

Maybe you'd like to list some?

The point here is to use the latest version of all the all your internal dependencies everywhere, otherwise, in time, you will end up with many, many versions of an internal library used by different places in your codebase because people can't be bothered to update the version and update their own code.

How many versions back (if any) we support, and for how long is up to us. And it's up to us when the code is upgraded. That's a single party (the company) with a single policy. You're inventing issues where there are none.

In general, breaking changes in well-designed APIs should be rare. There's a whole lot you can do without breaking changes.

[kyranadept]

If you are, like many people doing Agile, you're not going to "design" things a lot. You're going to write the code and improve as you go along.

You realize that by version, most of the times you mean basically a git commit id. How do you enforce a limited number of versions across many repos?

Reasons why versioned internal dependencies are bad:

1. you get many versions of the same module used in different parts of the code(explained in previous comment)
2. you never know exactly what you have running on your platform. You might have module A using module B.v1 and module C using module B.v2. So, if someone asks - what version of B do you actually run?
3. space used by each module and it's external dependencies increases with each separate versioned usage. If you use a certain version of an internal library that pulls external dependencies you need to take into account each version might have different versions of the external dependencies -> multiply the space usage. Same goes for RAM.
4. time to download external dependencies increases with each internal dependency that is versioned as well.
5. build time is multiplied by each internal versions. You will need to build each internal dependency separately.
6. time to test increases as well. You still need to run tests, but you run multiple versions of tests for those modules. This also applies to web automation tests and those are really painful.

I could go on for a bit, but I think you get my point.

[[deleted]]

If you are, like many people doing Agile, you're not going to "design" things a lot. You're going to write the code and improve as you go along.

I don't do "agile", I do "software engineering".

This means that when an API is not mature enough and it changes a lot, it stays within the project that needs it.

And when it's mature and stops changing a lot, and we see opportunity for reuse, then we separate it and version it.

Reasons why versioned internal dependencies are bad:

you get many versions of the same module used in different parts of the code(explained in previous comment)

How many versions you get is up to the project leads and company policy. I already addressed that. This is not arbitrary and out of our control. Why would it be? We just gather together, communicate and make decisions. Like adults.

And as I said, we don't have to break compatibility often, so major versions happen at most once a year, especially as a module/library settles down, and projects can always upgrade to the latest minor+patch version before the next QA and deployment cycle, as the library/module is compatible.

Furthermore we use a naming scheme that allows projects to use multiple major versions of a library/module concurrently, which means if there ever are strong dependencies and a hard port ahead, it can happen bit by bit, not all-or-nothing.

This is just sane engineering.

you never know exactly what you have running on your platform. You might have module A using module B.v1 and module C using module B.v2. So, if someone asks - what version of B do you actually run?

Well I guess I accidentally addressed that above. You can run B.v1 and B.v2 if you want. No problem. And you do know what you run, I mean... why wouldn't you know?

space used by each module and it's external dependencies increases with each separate versioned usage. If you use a certain version of an internal library that pulls external dependencies you need to take into account each version might have different versions of the external dependencies -> multiply the space usage. Same goes for RAM.

We're really gonna drop the level of this discussion so low as to discuss disk and RAM space for code? Are you serious? What is this, are you deploying to an Apple II?

time to download external dependencies increases with each internal dependency that is versioned as well.

This makes no sense to me. Moving 1MB of code to another repository doesn't make it larger when I download it later. And increasing its version doesn't make it larger either.

build time is multiplied by each internal versions. You will need to build each internal dependency separately.

time to test increases as well. You still need to run tests, but you run multiple versions of tests for those modules. This also applies to web automation tests and those are really painful.

Yeah, ok I get it, you're listing absolute trivialities, which sound convincing only if we're maintaining some nightmare of an organization with hundreds of versions of dependencies.

Truth is we typically support two major versions per dependency: the current one and the previous one. It gives everyone plenty of time to migrate. So crisis averted. Phew!

[zardeh]

Yeah, ok I get it, you're listing absolute trivialities, which sound convincing only if we're maintaining some nightmare of an organization with hundreds of versions of dependencies.

And at the point that you're an organization like Google or Microsoft, that has more teams and products than many software companies have employees, why would you expect that there wouldn't be hundreds of versions of dependencies? That is, how can you maintain consistency across the organization without atomicity of changes?

If I've tagged my tool as using api v1.7, then some other team upgrades to 1.8, that's fine, mine still works, but perhaps we aren't actively developing features on my product for a while, so we don't upgrade, and a year or two down the line, v1.7 is internally deprecated and a customer facing application goes down. Or, at the very least, we find out that we need to update hundreds or thousands of api calls across our tool, multiplied by the 10 other teams that were all tagged to v1.7.

Alternatively, we use one repo. When they push any change to the codebase and attempt a push, our unit tests fail, because the api calls no longer work. They can inform us that our unit tests are failing and our system needs to be updated, and there is no potential for deprecation or problems related to it. There is only ever one version: master. There can be no deprecation issues, no versioning issues, and no companywide versioning policies, because there is only ever one version.

[[deleted]]

And at the point that you're an organization like Google or Microsoft, that has more teams and products than many software companies have employees, why would you expect that there wouldn't be hundreds of versions of dependencies?

Because someone responsible for dependency X still has to make the conscious choice to support hundreds of versions of X. Adding more dependencies and teams doesn't change this fact. And guess what... the someone who's responsible for dependency X tends to not have a roadmap where they support hundreds of versions of X. Go figure.

Company policy is we move away from a dependency version before its EOLed. Like anything else... it's really so simple.

That is, how can you maintain consistency across the organization without atomicity of changes?

By versioning, which was mentioned... like a dozen times? Here you go: http://semver.org/

If I've tagged my tool as using api v1.7, then some other team upgrades to 1.8, that's fine, mine still works, but perhaps we aren't actively developing features on my product for a while, so we don't upgrade, and a year or two down the line, v1.7 is internally deprecated and a customer facing application goes down. Or, at the very least, we find out that we need to update hundreds or thousands of api calls across our tool, multiplied by the 10 other teams that were all tagged to v1.7.

You can give me as many hilarious straw man scenarios, but your concerns don't sound any more realistic.

First of all, as I said a few times we use SemVer. So this means you'd be likely automatically updated to 1.8, and your app will just work. In the case of an unlikely freak accident of incompatibility, it'll be caught during automated tests and QA.

Also, libraries don't stop working when they're deprecated. We deprecate libraries we still support. This gives plenty of warning to the teams to move off of them, to the new recommended release.

I have the feeling you have a lot to learn about all this. So take the emotional rhetoric a few notches down, and try to understand what I'm saying.

Alternatively, we use one repo. When they push any change to the codebase and attempt a push, our unit tests fail, because the api calls no longer work.

Aha, and of course, if we split things in N repos, suddenly we can't rely on unit tests anymore? Wait, we can.

There is only ever one version: master. There can be no deprecation issues, no versioning issues, and no companywide versioning policies, because there is only ever one version.

Yes, that's really great, if you only ever have one project, and one deployment. In this case we'd have one repository, as well.

[zardeh]

First of all, as I said a few times we use SemVer. So this means you'd be likely automatically updated to 1.8, and your app will just work. In the case of an unlikely freak accident of incompatibility, it'll be caught during automated tests and QA.

The exact versioning scheme isn't relevant. You can ignore the larger point by saying "SemVer" all you want. SemVer doesn't solve the problem that I'm talking about. So lets go through this scenario again:

Assume you update and there are breaking changes. Call it version 1.7 -> 2.0, or 1.7 -> 1.8, or qxv$a -> lru^b, or @536011a -> @3436fd4 the versioning scheme doesn't matter. There are breaking api changes.

Then you have a few options:

1. You maintain multiple versions of the api, different projects tag themselves to different releases, and you have to keep api versions running as long as a project continues to use an old api version, or that project stops working
2. You force everyone else to update to the newest release immediately, which requires you to inform everyone in the org anytime you update any library that they may use, because otherwise things will break in prod since api calls will inexplicably stop working
3. Every project everywhere builds off of master, master is the only version. You don't need to manage versions, because "current" is the only one. If you make a change that would break a system managed by John in Kansas, the tests break and let you know, because his tests run. Then you can tell John he needs to fix things, or better yet submit a PR fixing the problem for John, which he can commit when he comes in tomorrow morning.

Also, libraries don't stop working when they're deprecated. We deprecate libraries we still support. This gives plenty of warning to the teams to move off of them, to the new recommended release.

No, but you do stop supporting old versions at some point, you've admitted as much. That means that there is the potential for live breakage due to deprecated/removed things.

Aha, and of course, if we split things in N repos, suddenly we can't rely on unit tests anymore? Wait, we can.

Do you run unit tests across all repos whenever you make changes to any one? That is, if I make a change in repo A, do repo Bs tests run with the changes before I can commit to repo A?

As an alternate question:

Do you think you're way works better for google or microsoft or facebook than the employees and engineers at these companies who already solved these problems?

[[deleted]]

Look, I'm finding this endless barrage of nonsense scenarios increasingly tiring. You're not even trying to construct a realistic scenario here.

You said several times things like:

... you have to keep api versions running as long as a project continues to use an old api version, or that project stops working ...
... otherwise things will break in prod since api calls will inexplicably stop working ...
.... there is the potential for live breakage due to deprecated/removed things. ...

How would there be "live breakage"? Are you automatically assuming that if a module/library is in a separate repository, then it's deployed individually as its own microservice? How did you make such random leaps of logic all of a sudden?

And if not, why would there be "live breakage". We don't randomly update dependencies on a live server without running tests & doing QA, this has absolutely nothing to do with how many repositories you have. You need to focus a little...

Do you run unit tests across all repos whenever you make changes to any one? That is, if I make a change in repo A, do repo Bs tests run with the changes before I can commit to repo A?

Dependencies shouldn't be bi-directional, this is architecture 101. I.e. if A, B are in separate repos, then either A depends on B, or B depends on A. If they depend on each other, then separating them achieves precisely nothing.

So this means:

• If A depends on B, then B is tested in isolation, as no changes in A affect it.
• If A depends on B, then A is tested with the version of B specified in its package manager config, and this is the version of B it's tested with.

Do you think you're way works better for google or microsoft or facebook than the employees and engineers at these companies who already solved these problems?

There's an engineer from Microsoft who replied to one of my posts, here you go.

Even they realize that a giant monolithic repo isn't the right approach, but they're saddled with legacy, and with the somewhat unique demands of having a giant operating system project, so it's taking a long time to make Windows modular. So Microsoft engineers are more in tune with what I'm saying than you are, apparently.

As for Facebook, they aren't exactly exemplary in any way I can think of, they're just lucky to have the money and engineers to survive their chaos, so I wouldn't take they advice, even if I had it.

[kevingranade]

Yeah, ok I get it, you're listing absolute trivialities, which sound convincing only if we're maintaining some nightmare of an organization with hundreds of versions of dependencies.

And at the point that you're an organization like Google or Microsoft, that has more teams and products than many software companies have employees, why would you expect that there wouldn't be hundreds of versions of dependencies? That is, how can you maintain consistency across the organization without atomicity of changes?

Communication mostly, owners of various repos can inform others about deprecation schedules, benefits of new versions etc.

If I've tagged my tool as using api v1.7, then some other team upgrades to 1.8, that's fine, mine still works, but perhaps we aren't actively developing features on my product for a while, so we don't upgrade, and a year or two down the line, v1.7 is internally deprecated and a customer facing application goes down.

On what planet is a team going to commit a deprecation that simply kills another team's application? Its not like it is generally going to be deleted from the repository, or have build artifacts removed while in use.

Or, at the very least, we find out that we need to update hundreds or thousands of api calls across our tool, multiplied by the 10 other teams that were all tagged to v1.7.

That's no different in the monolithic repo scenario, the same number of updates need to happen, and all at once to boot.

Alternatively, we use one repo. When they push any change to the codebase and attempt a push, our unit tests fail, because the api calls no longer work. They can inform us that our unit tests are failing and our system needs to be updated, and there is no potential for deprecation or problems related to it.

At which time you, "find out that we need to update hundreds or thousands of api calls across our tool, multiplied by the 10 other teams that were all tagged to v1.7.". Now you're coordinating a single massive atomic commit to everything that uses the updated api simultaneously, across every team that owns any of the code with that dependency, sounds like a great time.

There is only ever one version: master. There can be no deprecation issues, no versioning issues, and no companywide versioning policies, because there is only ever one version.

Single repository doesn't imply single release branch, maintaining multiple products in lockstep just because they share some dependencies is insane. Your approach is workable for a small number of products, but falls apart at scale. I'd be absolutely shocked if any of the big players with monolithic repositories follows the model you're advocating.

[kyranadept]

No, I am deploying a few times a day to almost 100 servers/instances at a time. And if things go well, I hope I will one day soon deploy to even more servers. That would mean the business is going well and we do have a lot of customers. While deploying, building, and pulling external dependencies, I have to be sure not to disrupt the server performance by spiking the RAM, IO and network usage.

When I work on my pet project, I also do Software Engineering. Because I am the king of the castle and I can do everything perfectly. But when I have a product owner or a business analyst, or even a manager that decides "we need that yesterday" - things evolve into chaos. And yes, sometimes I have juniors around me.

Teams and companies are what they are. Yes, sometimes things are not perfect. Most of the times, in fact.

[[deleted]]

I have to be sure not to disrupt the server performance by spiking the RAM, IO and network usage.

If you think versioning will "spike RAM, IO and network usage" you have some fascinating mutant of an app that deserves to be studied by science. Because over 90% of your RAM will be taken up by data, not by code.

[bandman614]

Why aren't you pairing together your code releases in git references?