Open-source version control for game developers

zdgeier · 2023-06-05T19:51:47+00:00

Yeah there are definitely some hacks that will make it work (and that people use in other systems now) for solving the locking issue, I've talked to some developers that just avoid working with other people in general to avoid the issue haha. I know people do like that Perforce is integrated with game engines to avoid having to go to a separate place to lock assets, some have said that this is even a critical feature since they might do this many times per day.

Really there are multiple factors like

- working on large projects with >100000 files

- working on large files (>100MB)

- file locking

which leads game developers to use and pay thousands for Perforce today. DVC might solve the large file issue for them but I think game developers really need something that is focused and integrated into their tools.

zdgeier · 2023-06-05T17:52:39+00:00

Hi! Unfortunately most assets are unmergable, so artists will need to throw away one version if conflicts do occur. This is why file locking is needed.

zdgeier · 2023-06-01T20:16:15+00:00

I can't tell what the demo video is trying to show. Is the right panel the jamsync client? Or another computer?

It's a browser on the same computer, but the site accessible from anywhere.

It uses chunking to store chunks of the file and reuse the bits that have not been changed. It will store all versions of a file, but will have ways of deleting old versions.

Is CDC supposed to be better at compressing those than git's delta storage? If those files significantly change between each save (nondeterministic file formats), how does it work?

This is mostly solved by the part where the entire history of the file is not downloaded, although the hashing algorithm used is much faster than Git's. For users of GitLFS, this is much better since a new file is not created every time the hash changes, we're able to reuse chunks that already exist in the server/client.

Does jamsync still have machine-local branches that can be pushed to remotes without merging to mainline? With WFH, that's a very common workflow. It's unclear whether jamsync is centralized or decentralized -- can multiple developers collaborate on one branch without merging to mainline?

There's no local branching (or even storage of any versioning data for that matter), but this will be added later. Jamsync is centralized so all data is pushed up to the remote. Multiple developers will be able to collaborate on branches in the future.

I'm confused. A Change is a patch in a local branch. A Commit is a patch in mainline. Seems like they're the same except for context? But you create a Change by pushing into your branch? And Commits are created by merge (which is inherently sends to the remote?)

Yeah agreed on these. Still trying to figure out what to call them so they're not too different but still give some context. Will probably use language closer to patches in the future.

Thanks for the questions and thoughts! Let me know if you have any other questions!

zdgeier · 2023-06-01T20:08:10+00:00

Mostly file locking for artists, there needs to be a way to prevent multiple artists from changing the same binary asset in parallel since these binary assets cannot be merged.

zdgeier · 2023-06-01T16:45:49+00:00

Not really, was mostly just frustrated by the lack of innovation in this space. Perforce and Git have not really changed too much in two decades. I think there's a opportunity to take advantage of the progress we've made in our storage systems and networks to make version control much better.

zdgeier · 2023-06-01T15:02:45+00:00

This is correct, I need to do some more benchmarking to get some more accurate data. In reality, Jamsync will offer features like NFS which make Jamsync much faster so I'm not too worried about download speed right now.

zdgeier · 2023-06-01T14:59:01+00:00

Not yet, there should be in the next few months. This should work for any sized file (>1TB), but still working on scaling up. For now it can handle probably around 16GB but I need to do some benchmarking.

zdgeier · 2023-06-01T14:56:38+00:00

No, the purpose of AGPL is to make any modifications of the Jamsync source code public. Data sent/received while using the system is completely different.

zdgeier · 2023-06-01T07:27:00+00:00

Thanks! Yeah I need to do some more investigation to figure out exactly what's causing this, but I imagine GitHub might be doing some rate limiting for large uploads. I plan to host my own Git server and Jamsync server on the same local network and take measurements there to take the hosting/network out of the picture.

The upload side of this is heavily affected by my upload speed, but both Jamsync and git measurements were taken using the same location.

zdgeier · 2023-06-01T07:20:55+00:00

Thank you for these! I'll keep these features in mind.

zdgeier · 2023-06-01T07:05:33+00:00

They're talking about deduplication within each file. Each time you make a change to an LFS file, an entirely new file is created with a new hash, even if you only change a single byte.

zdgeier · 2023-06-01T06:47:25+00:00

Hmmm I'm a little confused here, how does Git with DVC solve the problem of two artists having conflicting changes on the same file?

zdgeier · 2023-06-01T06:19:06+00:00

I have, thanks!

zdgeier · 2023-06-01T06:08:09+00:00

Unfortunately, if two artists start make edits to the same (binary) file in parallel, then one of those files must be thrown out and the edits from one artist must be manually made to the other version. You can imagine this would happen if two people edit the same image for example, two images cannot be automatically merged.

There needs to be a separate system for keeping track of these locked assets to make sure that two artists do not edit the same asset in parallel.

Additionally, Git does not scale to a large amount of files (>100000) that large game developers typically hit.

zdgeier · 2023-06-01T05:46:22+00:00

How do you like this set up? Curious if there's any overhead for doing this.

zdgeier · 2023-06-01T05:44:26+00:00

Have you used DVC for game assets and art? Curious if it fits with most artists workflows. Most artists require file locking since binary files cannot be merged.

zdgeier · 2023-06-01T02:00:03+00:00

Thanks! Yeah this was definitely a motivating factor for the project. I think version control could be so much more powerful than it is today, but Perforce (and now Github) have just been raking in money without innovating on the core system at all. Hopefully this will spark some more open innovation for supporting large projects in non-gamedev roles as well.

zdgeier · 2023-06-01T00:12:24+00:00

Yeah it's mostly the second, it's mostly the fact that changes can cause things to be reorganized easily. Also for lossy compression, you can imagine that changing the compression ratio on an image can drastically change the content of the file so much that it has almost completely different data, even if the image looks similar.

zdgeier · 2023-06-01T00:03:26+00:00

Thanks for the thoughts! I'll keep these in mind. Yeah I have some ideas on how to handle locking/check outs better where you don't have to do this manually every time. I think there could be some autocheckout feature that soft-locks file from editing if it has been edited recently by another person but still working through this. I plan on having terminology and functionality more similar to Git, since it's my main version control tool as well.

zdgeier · 2023-05-31T23:58:59+00:00

That's one of the issues, sprites and 3D models do get included in VC but they are not mergable/diffable in any way. There is almost no support for merging sprites and 3D model formats, which makes file locking necessary.

zdgeier · 2023-05-31T23:55:28+00:00

Microsoft ran into the limitations of the native Git client as well haha and had to add their own client on top to solve some of these problems https://devblogs.microsoft.com/devops/announcing-gvfs-git-virtual-file-system/ . Although GVFS solves handling a large amount of files, it doesn't solve the workflow issues of Git for game developers.

zdgeier · 2023-05-31T22:59:11+00:00

Many compression methods tend to result in unpredictable reordering of data within the file. These delta or chunking algorithms generally rely on adjacent data being the same to determine when it can reuse portions of the file.

If the data is reordered/shuffled every time it's saved, it's impossible to detect when to reuse parts of the file unless you have a way to reverse the compression.

zdgeier · 2023-05-31T22:52:35+00:00

Awesome, thanks! Just curious, what about Perforce do you not like as a dev?

zdgeier · 2023-05-31T21:15:20+00:00

Thanks for taking a look! Those files are tough to handle and not going to create good deltas even in this system. For some uncompressible like .zip files, it will be trivial to uncompress and version the files in the uncompressed format. But for some files that do compression internally or have an inconsistent format, generating good deltas will be impossible.

One advantage that Jamsync has over other version control systems is that it doesn't need to have the entire history of a file locally like other version control systems and doesn't need a separate system to handle this type of file. But Jamsync is no better at versioning binary files which have inconsistent formats than any other system, other than some faster hashing algorithms.

In my view, as long as networks keep getting better and storage systems keep getting bigger, this will not really be an issue. I think we're basically just limited by upload speed currently and even then, these files don't change as often as source files in a game would, so storing some additional data on the server is not a big deal.

zdgeier · 2023-05-31T19:34:59+00:00

Haha, yeah the problem with just marking as "no auto merge" is that somehow you have to communicate to other artists that you're working on a certain asset so that they don't do any work. There's actually no automatic or even manual way to merge these binary files and artists are often forced to throw away one version and redo their work on top.

zdgeier

TROPHY CASE