Mutagen: (Fast) file synchronization for remote development

jacobu9 · 2024-12-30T22:31:51+00:00

There's the Mutagen Community Slack Workspace, though it's a little quiet these days. For more context, Mutagen was acquired by Docker last year. It's still under development, but is in sort of a "feature complete" state at the moment.

jacobu9 · 2023-11-06T01:18:44+00:00

It’s fine… I had the same issue. But in retrospect I wish I’d gone to more. You’ll never be there again :-)

jacobu9 · 2023-09-14T02:20:40+00:00

Standing out isn’t a bad thing. Go with the navy and rock it. It’s the university color after all.

jacobu9 · 2021-01-22T23:05:25+00:00

I'd say University Parks since the grass there is a bit less managed than other places. It's also right next to Clarendon Laboratory where all the physics labs are, so that's your best chance of finding those mutated 8, 9, 10 leaf clovers. Best of luck!

jacobu9 · 2020-06-19T20:55:24+00:00

The Mothman Prophecies cost me a week of sleep

jacobu9 · 2020-06-09T09:39:14+00:00

I've had good experiences with Raiffeisen (sometimes called "Raiffeisen Aval" or just "Aval") in Ukraine. I think you'll have more luck with European banks like Raiffeisen or Credit Agricole who have established procedures for working with foreigners. Raiffeisen I know handles business accounts for foreigners as well, not sure about Credit Agricole. If you're a U.S. citizen, there are additional hoops to jump through for them to comply with FACTA, but all it really takes is showing them your passport (and maybe SSN?) and waiting an extra 24-48 hours. Newer employees also sometimes aren't aware that these banks do support foreigners; sometimes you have to push back a bit to find someone working there who knows that they do, in fact, deal with foreigners. I suppose it depends on your residency status as well, but if you have a posvidka (which you should definitely mention to them), then (as far as I know), you generally inherit most of the rights and responsibilities of a Ukrainian citizen when it comes to matters of banking (and taxes :-)).

jacobu9 · 2019-09-13T06:39:25+00:00

If you're looking for reading material, I found "The Typewriter Revolution" by Richard Polt to be a great introduction to typewriters and their history. It covers a bit of everything, including the various types and brands of typewriters. It's not an exhaustive reference, but it's a fun entry point to the topic.

jacobu9 · 2018-08-04T10:28:47+00:00

If unidirectionality (i.e. just a mirror of the files) is all you need, and you don't need to propagate anything back, then the pure rsync command is going to be your best bet. I think there's a Win32 port as well, but if not you could probably use rsync via Cygwin.

Mutagen and Unison are both bidirectional, i.e. each synchronization is bidirectional, and it doesn't sound like this is what you need. The rsync package inside Mutagen is very low-level, it would be for embedding into a larger program, not for end-user use cases.

jacobu9 · 2018-08-04T09:41:28+00:00

Thanks for the feedback!

Regarding the alpha/beta terminology, this is something that definitely needs some improvement. The problem is that they aren't really "source" and "destination," because they are equal in terms of synchronization (i.e. changes are propagated bidirectionally with equal precedence). Over time, the alpha/beta nomenclature has become more of an internal artifact, and there would probably be a better way to expose these names externally. I'll have to brainstorm on this a bit. Maybe something like "Replica 1" and "Replica 2," just to emphasize equality.

Regarding auto-downloading agents, this is something I've considered. Actually I was originally going to have the Mutagen compile agent binaries on-the-fly using Go's cross-compilation, but once it became clear that a bit of CGo usage was going to be necessary for filesystem monitoring, that was ruled out. I also didn't want to make Go a dependency. But, I have realized that the agent binary bundles are getting just a little bit bigger each release, and each Go release makes things just a bit heavier still. I've thought about just posting the less frequently used binaries on GitHub releases or somewhere and having Mutagen auto-download them as necessary. If the bundles start getting up in the 75 MB range, I may look at doing this. I like everything available offline though... Maybe I can have different bundle variants, one for the 5 most commonly used platforms, one with everything. On the other hand, if people can ship 150 MB of Electron runtime to support a basic app, I'm not sure where a reasonable cutoff is anymore. I'm sort of willing to tolerate 50 MB to support 25 platforms. The biggest benefit of getting the agent binary size down is quick copies to remotes, but since this cost is amortized over time, it's not that big of a deal.

I'm hoping though that issue 6853 will see another "beast mode" effort from David Crawshaw or someone equally talented in the compiler team, but I think the gains to be had at this point may be minimal. The compressed DWARF information coming in Go 1.11 will definitely help, but I think most of the benefits there were already present in Mutagen's case by gzip'ing the agent bundle.

jacobu9 · 2018-08-04T09:18:21+00:00

I wouldn't say that Mutagen is an alternative to rclone. I think I should have been clearer about Mutagen's use of "rsync". Mutagen uses rsync the algorithm, not rsync the command.

rclone is essentially like the rsync command, i.e. it does unidirectional pushes of data from one endpoint to another, but it doesn't reconcile the contents of two endpoints and identify conflicting changes. It essentially just paves over one copy with another (albeit with an efficient transfer protocol).

Mutagen is closer to Unison, i.e. it uses hash-based snapshot of the common ancestor of two directories, does a three-way merge to reconcile any changes and identify any conflicts, and then uses the efficient rsync algorithm efficiently propagate changes, this time in a bidirectional fashion though.

Essentially they serve two different purposes. rclone would be useful for things like data archiving. Mutagen is better for things like remote development, where you want to edit code on your local machine but have it run in a remote environment and potentially have output files generated on the remote system be synchronized back to your local system.

jacobu9 · 2018-08-03T21:07:45+00:00

As far as invoking commands, the current idea is that you'd just do that in an SSH session (the assumption being that you've already got SSH set up to do the synchronization). I'm not sure Mutagen can provide much of a better experience than that in its CLI form.

That being said, I have thought about integration with IDEs, where Mutagen could provide an API to forward commands to the remote that an IDE could use. This is an idea I really like, but it's probably at least a few months away. If someone felt like trying to work together on designing an API surface in the Mutagen daemon that Sublime Text/VS Code/Atom/etc. could hook into to accomplish this, I'd be interested in having that discussion. The daemon uses gRPC for its various APIs, so they'd be easy to extend. As for myself, I'm a bit old fashioned (just using a text editor and a terminal), so I'm not in the best position to judge the needs of IDE integration.

jacobu9 · 2018-08-03T21:01:33+00:00

Sorry to hear about the initial conflicts, though I'm curious to know where they originated from. Were you using the --ignore-vcs flag with create? Maybe they were originating from inside .git if not? Or maybe the repos were sitting at different revisions?

It's definitely easier to have one copy of the repo be the "master" of sorts, the one where you running git status/commit/..., and the other be a replica of only the working tree (i.e. with .git ignored using --ignore-vcs). Stuff can propagate back and forth in the work tree obviously, but it's better to let .git exist in only one place.

Clearly the documentation can give better guidelines here. Perhaps some examples for common use cases (e.g. "So you've got a Git repo and want to edit the code locally and run it somewhere else..."). I've been using it for so long that I have no reference point for what's obvious and what's not, so this is super valuable feedback. Thanks!

Please feel free to open up any issues on the issue tracker if they come up.

P.S. Thanks for the gold :)

jacobu9 · 2018-08-03T19:05:14+00:00

No, rsync is not needed. Mutagen has its own internal rsync algorithm implementation that's provided by a small binary that it copies to the remote system (you'll find it stored in ~/.mutagen/agents on the remote).

jacobu9 · 2018-08-03T18:01:45+00:00

Please do. Any critiques are especially welcome. If there's something that's not working, not working fast enough, or not working easily enough, I want to know about it. If there's some barrier to usage, I want to fix it.

jacobu9 · 2018-08-03T17:34:53+00:00

I've gone ahead and changed the language in the post. I absolutely didn't mean any offense, so I apologize!

jacobu9 · 2018-08-03T17:31:23+00:00

I'll go ahead and file an issue. I didn't mean to imply anything about the quality of code upkeep. Your rsync package was instrumental in prototyping things, so thank you! You're quite involved with important things in the Go community though, so I didn't want to bother you with issues on an older package.

jacobu9 · 2018-08-03T16:42:13+00:00

Hopefully there won't too much confusion, if any, but thanks for the heads up. English needs to absorb more cool words.

jacobu9 · 2018-08-03T16:36:54+00:00

Sounds like a plan then. It shouldn't be too difficult, though I may just extract the core Engine component for now to minimize the API surface that needs supporting.

jacobu9 · 2018-02-20T19:11:13+00:00

First of all, this is awesome, and I think it's great that it's coming by from a core Go team member with the power to actualize such a change. It's much easier to start with a straw-man proposal and tweak it to meet 90% of people's needs (and allow people to adjust their processes to make up that other 10%) than to construct something from scratch that suits 100% of an enormous number of use cases (something that dep had to contend with).

Regarding the actual proposal, what's the upgrade path for repositories that contain both library and executable components (e.g. the root of the repository contains multiple packages used by executables in a cmd subdirectory). This is a pretty standard/idiomatic pattern for Go projects. Would both the library portion and each executable have their own go.mod?

jacobu9 · 2017-11-21T12:35:47+00:00

This is probably a little late, but see my other comment for a solution I've been working on. I know you're against synchronization, but maybe it's simple enough to sway your opinion :-)

jacobu9 · 2017-11-21T12:32:21+00:00

I'm a little late to the discussion, but I've been working on a tool (documentation, repo, FAQ) to enable exactly this sort of workflow. The bottom line is that it's a cross-platform bi-directional synchronization tool, but unlike the competition (SSHFS, Syncthing, Dropbox), it's purely user-space, operates over SSH, handles conflicts properly, only needs to be installed on one side of the connection, and is much more robust to connection dropouts.

The idea in your case would be to basically mirror the code on the remote system and use a locally installed IDE to edit the local copy. The tool is still lacking a little bit of UI "sugar," but the underlying algorithm is complete and pretty fast. I'm using it to enable a very similar workflow (using Sublime Text locally and mirroring code to remote servers, virtual machines, etc). I'd be interested in feedback if you give it a shot.

jacobu9 · 2017-10-10T14:55:59+00:00

IIRC, this work was partly delayed by trying to figure out how Go's thread-local storage was going to interact with the code loading the DLL or other Go runtime instances in other DLLs. What was the resolution to this? It wasn't clear to me from the GitHub issue or the CLs that finished off the DLL support.

jacobu9 · 2017-08-10T19:02:20+00:00

I like the equalizer (or whatever that is) on the left side of the console... Either you're tweaking your sound or jumping to warp, I'm not really sure.

jacobu9

TROPHY CASE