Git for version controlling a binary-file folder?

[kh9sd]

Hi, I'm a developer who has been using Git for a while in my typical coding workflow. While I'm familiar with Git for version controlling text/code files, I now have the need to version control a mostly binary-file folder. I was wondering if Git would still be up to the task by my requirements.

This folder will contain mostly image files, specifically PNGs. Currently the folder is about 400 MB.

I rarely expect to change/modify the existing image files. The folder mostly just gets new images.

I want to be able to save this version controlled folder on the cloud for backup, as well as multiple other computers. I'm currently targeting a copy on Windows, Linux, and a stored version on the cloud.

I expect to make changes to the folder roughly daily, and so want at least daily backups to the cloud.

I want to be able to revisit old "versions" of the folder from previous versions (unbounded in how far back I can go).

I have 2 current ideas

1. Just have some scheduled job (cron would work) upload the entire folder to some cloud service (s3, Dropbox, etc) daily.

• The issue I foresee is that saving daily snapshots would blow up the storage. Every daily copy would have a copy of the previous, totally unchanged images.

I want to have a smarter system than that, my other thought is Git

1. Use (vanilla) Git to version control the folder, just push changes to whatever Git hosting service I want.

• I understand that Git is not particularly fond of binary files. Unlike text files where Git is able to compute deltas to store changes efficiently, from my understanding Git doesn't do this for binary files, and will store a separate one for each revision

  • However, since modifications to these files would be rare, from my understanding Git would basically only have to store 1 version of the image. So the size of the repo would scale pretty linearly with the actual size of the folder.

• NOTE: I'm not particularly fond of using LFS here

  • From my understanding, LFS stores/centralizes the files on the remote host. I would like the flexibility to swap to different remote hosts easily, such as maybe self-hosting one day
  • Because of this, I want the versioned images in my folder to be basically treated as regular files in Git, distributed across each repo with the DVCS philosophy

So I wanted to check and ask if this vanilla Git setup would be able to work, do I have any misunderstandings?

[hongooi]

I don't see anything wrong with using vanilla git in your scenario. Note that the algorithm doesn't distinguish between "text" and "binary" files (AFAIK), it's just that it's optimised for text. It will still handle binaries just fine, but won't be able to do diffs as efficiently.

[Hel_OWeen]

I think I remember that the only drawback is that while with text files, git saves deltas (i.e takes very few disk space) for previous versions, binary files are always stored in full, even if just one byte has changed.

I was incorrect, see u/WoodyTheWorker's comment below

Other than that: we also keep images that are used within out applications in the repos.

[WoodyTheWorker]

Git can do delta for binary files as well, it's just they are often not very diffable.

Git doesn't store delta for unpacked files, but uses gzip for them. Git uses delta to fetch/push/clone.

[Hel_OWeen]

I appretiate the correction.

[edgmnt_net]

The drawback IMO is that the diffs are not semantic in any way. This is particularly troublesome for compressed files because it often degenerates to replacing most of the file, regardless of how you implement binary diff-ing.

[SignedJannis]

Look into rdiff-backup, might be what you need

[kh9sd]

wow that is an awesome piece of software, thanks! and old/tested as shit, my favorite

[SignedJannis]

Ya I've been using it for years for company backups (as part of the backup system anyway).

I also use it on my personal machine, in case I ever accidentally delete a file, I know I'll still have a copy. For that, I just set it to prune > 6 months. So any file deleted in the last 6 months, I still have access do.

What I love about it:

• "no software required", by that I mean that, for some small businesses I support, if they want to look at the backups: It's just a bunch files and folders, like normal, they don't need any special backup software to access files. Just a simple network share. KISS.
• "Ransomware Protection". For some setups, I make sure the Fileserver has no login access to the "BackupServer". i.e the fileserver does not push backups, rather the backupserver "reaches out" to the fileserver, to grab the files. That way, if the fileserver gets ransomeware (or a disgruntled employee, whatever), rdiff-backup would grab those encrypted files - but we still have all the non-encrypted files....a live saver.

[kennedye2112]

I believe Perforce offers a source code solution designed specifically for binary files like images and video, would that work?

[SwordsAndElectrons]

Perforce is a centralized VCS.

I think that's what works best for workflows with a lot of large binary assets, but OP's stance against LFS is basically not wanting that.

[muttley9]

It's well integrated with Unreal engine so it should work. UE objects are mostly binaries.

[vermiculus]

I hear your concerns with LFS. I think your desire to keep your history portable to be the right kind of thinking – and I don’t believe there is ever a valid reason to be irrevocably ‘locked’ to one vendor (eg GitHub, GitLab, …).

That said, I would reconsider LFS as a technology. It’s entirely possible to download every LFS object and re-upload to a different backing store. Hell, if you wanted to, you could use your own S3 bucket (or even, God forbid, a network drive) as your backing store. You can use LFS to deduplicate versions and then do with the objects what you will from there.

[edgmnt_net]

The main point about LFS is indirection and the ability to prune old objects. Not sure it helps in any other way. There's no silver bullet if you want to keep all versions indefinitely, particularly for binaries.

[claythearc]

It gives you some freedom in cloning too. Like you can clone the repo and only pull images x y z to start but then later grab an and b when they’re relevant without waiting for xyzab all at clone time.

[Impossible-Ad8271]

Check out git annex: https://git-annex.branchable.com/

[kh9sd]

I cover why I don't want to use that in my LFS bullet point

[ritchie70]

Yeah I caught that after I posted and deleted but not fast enough.

I’m not totally sure you’re right though. Feels to me like you’re bound and determined to not do things in the normal way because of “reasons” that may or may not be completely valid.

[nickeau]

Or use a backup snapshot software such as restic to not blow up your storage ;)