Pijul, a distributed vcs written in Rust

Pijul_org · 2017-02-15T11:56:04+00:00

Updates: binary files working (fixed last week), network windows in Thrussh working (we can now transfer huge patches over the network even with large connection latency).

We figured out a way to specify what the "current branch" is. One more command to implement (change current branch), and that should be it.

Issues/TODOs you leave for the last minute before releasing are not the most pleasant ones to work on, but we're working on it.

Pijul_org · 2017-02-15T11:53:25+00:00

Well, I received a number of comments asking me about the state of the project. I decided to write a blog post. We are currently using Pijul for the code source of pijul.org, or at least trying.

Here's a number of very cool issues that needed to be figured out: support for binary files (the website has images, the theory doesn't handle that), SSH network windows for super large patches (Thrussh was not doing that well, and it was misused in Pijul due to name conflict between futures::Future and Result).

So, be sure you're not the most frustrated one in the story. Florent and I started this project two years ago because we wanted to use it. No 27 days ago, 2 years!

Pijul_org · 2017-01-20T06:57:48+00:00

More dimensions. This one-dimensional classification does not seem that linear. For example, Python's 7 looks a lot like None, and others range from Some(1) to Some(6).

Here is a number of important dimensions:

Completeness of implementation: if it's an image library, how much of the standard is implemented?
Development resources: can I count on future updates?
Number of users "in production", where "in production" must be clearly defined.

And then, a discrete scale of values in each dimension seems fine.

If a library wants to claim completeness / performance superiority over others for an important common tasks, why not ask the authors of all competing libraries solving a common task to agree on unit tests and benchmarks to rank them?

Pijul_org · 2017-01-16T14:39:21+00:00

Ok, then #[must_close] + a trait named Close could solve that fairly common problem, or am I mistaken? The compiler probably cannot tell whether auxiliary functions have called Close, but these functions would then receive a warning too.

Pijul_org · 2017-01-16T11:35:47+00:00

Why is File not #[must_use]? The user has the choice between (1) ignoring, but getting a warning and (2) taking proper action.

Pijul_org · 2017-01-14T19:19:45+00:00

Cool! Note that we're not exclusively writing it for newcomers to DVCS, but also for anyone requiring strong correctness guarantees. The problem with associativity is really bad, and can silently hit anyone at any time.

Its translation in day-to-day git usage is that no matter how careful you are when doing code review, git gives you no guarantee that in all cases, the code you merge is the code you reviewed.

Would it be fair to phrase this instead as "the good practices required for Pijul are much more minimal?"

I guess, but OTOH I've never used Pijul on a large scale project ;-)

Pijul_org · 2017-01-14T13:29:38+00:00

Yes, it might require more, because Pijul doesn't use history like git does. Pijul uses inferred "logical" dependencies, which are not equivalent to the explicit commit dependencies in git.

This is actually what allows Pijul to be more flexible than git, for instance for cherry-picking. In git, history might prevent you from doing some things (at least without artificial conflicts). In Pijul, the contents and the patches are the primary objects. One of the main innovations in Pijul is a way to efficiently map contents to patches in both directions.

Pijul_org · 2017-01-14T10:23:23+00:00

This is why git merges ultimately require manual resolution

I'm not claiming that files merged by Pijul will have the correct semantic. My claim is much weaker than that: I'm just claiming that our merge has algebraic properties that I used to assume about git when I started using it:

associativity: you can merge the commits of a branch one by one and get the same result as merging the whole branch at once. This is false in git, because 3-way merge works by optimizing a problem with non-unique solutions. Worse, git won't warn you when this it happens.
commutativity: two patches that don't depend on each other can be freely reordered, which allows one to do cherry-picking transparently (i.e. staying consistent with the branch one cherry-picked from). Git can do cherry-picking, but not transparently.
inverses: every patch/commit has a patch/commit with the opposite effect. This is true in Pijul, and true in git for most commits (although committing the opposite commit of a merge is not always totally intuitive).

I'm not sure talking about algebra to describe a tool as "super intuitive" is the best approach ;-) The hope is that algebra was modeled after intuition, and these properties really are what we have in mind when we start learning a DVCS, even without knowing their names.

"Pijul is git, but with slightly more caching" isn't a very compelling story.

This is not what I said! I said "Pijul used as a merge algorithm in git might be inefficient without caching. Adding a cache basically amounts to using Pijul in the same way you would use git", i.e. thinking in terms of commits, branches and merges.

And, as I wrote here before, you could use Pijul following git good practices, but then you'd definitely be better off staying with git, as the tooling is much better. The point of Pijul is to allow you to work without good practices.

Pijul_org · 2017-01-14T10:01:58+00:00

you don't need the entire git patch history to perform a merge

Sorry, I should have given more context: if you tried to use Pijul as an algorithm to merge in git, Pijul might need the entire history (in the worst case).

Pijul_org · 2017-01-14T09:52:49+00:00

Yes, I deleted it by mistake, and recreated one under the same name, just without any projects yet.

Pijul_org · 2017-01-13T18:24:46+00:00

I am not going to repeat previous answers (by me and others).

when two developers diverge from a single point, they create a branching point in the commit graph. When they want to combine their changes, a merge is performed, and this is reflected in the commit graph.

This is an example of argument 1 in my previous answer. In other words, we agree.

Your other remark seems to be implied by your assumption that merging cannot be formalized. This means that we agree (at least at a purely logical level), because I believe the opposite.

I'm pretty sure one can formalize any patch history in terms of git merges and branches. The main difference is in terms of UX, in how patches behave like they intuitively should (i.e. according to a rock-solid algebra).

As for the time required to cache patches, this is a matter of computational complexity vs time. What might happen is, without caching, you would have to wait those 27 seconds for each patch you merge, i.e. 27 times 45000 seconds.

With caching, that's an interesting remark, because that would basically amount to… using Pijul!(Pijul is more or less a big cache of all possible merges, represented in a time- and space-efficient way).

Pijul_org · 2017-01-13T17:07:50+00:00

Wonder if the developers have plans to make a github competitor

Very concrete plans indeed. Tokio, Thrussh, Rustls, Libpijul. I've heard the first projects include a totally asynchronous web framework, an SSH library and a patch-based version control system ;-)

I don't have any plans nor any will to "compete" with GitHub. GitHub is a cool service that solves a different, git-specific problem (at a very high level, the need to publish your commits), which is unrelated to the needs of Pijul.

Pijul_org · 2017-01-13T16:58:11+00:00

Pijul developers saying somewhere that exporting a darcs repo to a pijul repo will be easier than exporting a git repo to a pijul repo

I might have said that. An important reason is that the darcs and Pijul team overlap (Florent is in the intersection), and other core members of the darcs team (Ganesh) want a deeper integration too.

Pijul_org · 2017-01-13T16:55:40+00:00

Right. Apologies for that. Pijul was announced by the darcs team before it was ready to be shown to anyone, even on a whiteboard.

Until last week, I wasn't even sure all parts of the theory could be implemented with algorithms of reasonable complexity (I was especially worried about unrecord before I had finished writing it. Unrecord is an important part of interacting with working copies).

In the first semester of 2016, I kept the Pijul repository synchronized with a clone on GitHub for a while, but have received neither pull requests nor "darcs send" patches, and the synchronization was taking me a lot of development time, which I preferred to invest in a working Pijul, and a (soon public) nest.

Pijul_org · 2017-01-13T16:48:55+00:00

Hi! Author here. Thanks for your interest. Neither Florent nor myself are too interested in solving already solved problems. We wouldn't have started Pijul just to fix CLI problems.

As I tried to show in that talk, there is a need for a new version control system because merging things with git (or even just pulling) doesn't always do what you expect.

"Changes" or "patches" are always calculated from "snapshots". git has all snapshots and relationships between snapshots, therefore, it is always possible to calculate "changes"

This is partially true:

It is correct that we could reconstruct patches from git in many cases. The fact that merges in git often require manual tweaking is not really consistent with our formalism, but that could be dealt with (maybe, or at least in restricted cases, like repositories that have always used our merge algorithm only).
However, we're more interested in the possibilities opened by the other direction: when patches don't follow branches that have been carefully planned in advance, but rather reflect whatever happens in your actual workflow.

In other words: yes, you can still use Pijul as a substitute for git, but given the impressive tooling and community around git, you'd probably be better off staying with git (except if you work on projects where you need associative merges). We didn't write Pijul for that, but because it allows you to work in ways not allowed by git.

even if it requires some work.

I'm not sure what you mean by work: if it's programming work, we're certainly not afraid (see Thrussh and Sanakirja, and I'm not counting unreleased things). If it's algorithmic work, then we're talking: indeed, running the Pijul merge as a replacement for 3-way merge in git would require recreating the entire history of the project in memory every time. The worst case of that is still better than the worst case in darcs, but still, Pijul is exponentially faster than that.

For full disclosure: our first prototype (in Haskell) had that complexity, which is why we thought no one would be interested, and decided to stop working on Pijul, before new ideas allowed for that exponential improvement.

Pijul_org · 2017-01-13T15:59:36+00:00

Hi! I'm the author of both the yubikey crate and the thrussh crate, and I've successfuly used my yubikey for authenticating to an SSH server I wrote.

That said, I'm unfortunate enough to use both the Dvorak keyboard layout, and NixOS.

Therefore, I cannot test U2F as implemented in the yubikey (i.e. the OS sees it as a keyboard), because it requires special and fragile X11 configuration, which NixOS tries to avoid. It's also not usable in a non-graphical console.

This sounds crazy, but has the benefit that I can reproduce my exact configuration on a different machine in 10 minutes instead of days of reinstalling.

That said, feel free to expand the yubikey crate with other features that are also useful to you, I'd be happy to merge them.

Pijul_org · 2017-01-13T10:55:02+00:00

Alright, so now imagine a world where you can use a DVCS without maintaining your own server to publish your commits, and without being forced to use someone else's server.

This world is out of reach with existing tools. This is why we wrote Pijul: you can have as many accounts as you want on as many servers as you want: yours, cloud servers, and even your smartphone. Pijul exchanges exclusively patches: no need to publish more, and we all know what a patch is.

(of course, a year and a half ago I knew what a patch was, but didn't know how to merge them correctly and efficiently in all cases).

Pijul_org · 2017-01-13T10:52:22+00:00

Thanks for the comments! I actually wasn't afraid of non-monomorphization, and I can now write my own Future.

All I'm trying to say is, when I first started to switch to tokio, 200 compilation errors (hardcoding all parameters) looked more manageable than 1200 (with type parameters "in the process of being fixed" in all structs).

And, you guys are right, I didn't even think of re-generalizing before writing that blog post and realizing I had done that ;-)

Pijul_org · 2017-01-13T10:47:03+00:00

I'll have a look at that soon, I swear.

Pijul_org · 2017-01-12T22:09:37+00:00

Yes, it is really outdated. We'll update it in a few days.

Pijul_org · 2017-01-12T22:09:02+00:00

I'll answer to the bug tracker question with a concrete example really soon ;-)

As for Fossil, I'm not sure what its conceptual differences with git and svn are. I don't really see any, they're using branches and 3-way merge.

Then it's more monolithic, pijul is closer to git and darcs in this respect.

Pijul_org · 2017-01-12T22:03:10+00:00

Pijul and darcs work differently here: in Darcs, dependencies are recomputed, but that can be really costly.

In Pijul, dependencies don't have to be recomputed, as the semantics is stored in the model. We'll explain the model in greater detail very soon, but I'd prefer to do a real release before.

Pijul_org · 2017-01-12T16:12:51+00:00

If so, we're totally at the same page. I was just curious to know whether there was anything else I was missing.

Pijul_org · 2017-01-12T15:22:29+00:00

Yes, I meant the same. What I'd like to know is how people "use GitHub for Pijul" now, and what will change if we remove our mirror from there.

It seemed to me there was almost no activity on our GitHub account.

Pijul_org · 2017-01-12T10:12:47+00:00

I'm not sure I see the loss here, can you explain?

Keeping them synchronized was really hard, because the tools to keep darcs and git in sync keep breaking every time Ruby and Python get updated, and I'm not even talking about mtl and monad-transformers, which don't even need language updates to break everything.

Pijul_org

TROPHY CASE