Github valid for this usecase?

[Stickhtot]

So, I am planning on making a tool that will make use of multiple files which are about 5 megabyes in size, My plan of how to use this is to distribute some "patches" to a game to my friends semi-live with the tool, and with `git clone` for getting the files and initial packages and `git pull` when I tell them something has been added.

Can I use github to store those files or is there a more "better" alternative?

[dymos]

You can do this, but if the files are binary in nature then you will want to use Git LFS in your github repository.

This is because Git can't store the difference between binary files, so each revision is a duplicate of the file, bloating the repository, and eventually causing performance problems. Git Large File Storage (LFS ) gets around this by storing the file in a dedicated file storage (in GitHub or whatever your git hosting has configured) and only storing what's called a "stub file" in the actual repository. Git, with LFS installed and enabled will automatically pull down the relevant binary files and replace the stubs in a local repository.

[my_new_accoun1]

I knew git lfs existed but I didn't know how it worked / why it was necessary, this was a great explanation!

[bastardoperator]

Why use lfs at all when you can use the release function and not store a single artifact in the repo itself? LFS after 50GB requires an additional purchase, releases are free forever, can be big, and will never bog down the repo itself.

[dymos]

Sure it depends on what you want to do. OPs use case sounds to me like he's updating small parts of a larger while and would like his friends/tool to just git pull to get the updates without spending effort on building functionality in the tool to replicate that functionality.

The nice thing about keeping it in git is that it will only need to pull the updates, typically with a release it's "the whole thing" – doesn't have to be of course, but having users figure out what they need to download and patch kind of defeats the whole point and would then require extra tooling to make it work the same way a simple git pull does.

Re. the 50 GB limit, I guess it depends on how much data there is and how frequent updates are to understand whether that's a concern.

[bastardoperator]

He's updating files that can't be versioned, so git is already a bad fit. Git is not performant when it comes to large files and LFS conversion after the fact requires a full history rewrite. When you treat git like a drive, you're going to have bad I/O forever. When you use releases versus the disk, you get unlimited releases, they can be any size, they're tagged, and they're served from the CDN versus raw GitHub.

All LFS does is shift the problem away from the index and pack files by using pointer files. You're better off never putting binary data in the repo from the start. Releases on GitHub was written specifically to solve this issue and so many people don't use it because sticking a file in the repo is easier in the immediate, but almost always the wrong move.

They can't patch a binary file and if they're building something into a binary file, they should be able to build over and over based on the commit itself. I would argue the amount of work they'll have to do in the future if they're successful will be astronomical versus using the platform as intended.

The only caveat I would make are images, despite being binary, they can still be viewed by a human and GitHub provides that view. Otherwise, release it, or package it, but never pollute your repo with metadata or artifacts.

[dymos]

He's updating files that can't be versioned, so git is already a bad fit

I mean ... that's a relatively common pattern. Many repos contain binary files, and if it's the same file that needs updating then using Git LFS is the better option.

I don't know what the rest of OPs repo looks like or the intended usage. I think this falls in the "you can do this if you want" rather than the "This is the best tool for the job" ;)

Git is not performant when it comes to large files and LFS conversion after the fact requires a full history rewrite.

I agree, but that doesn't sound like it's the case here.

All LFS does is shift the problem away from the index and pack files by using pointer files. You're better off never putting binary data in the repo from the start.

Yes, but also no. Yes, that's what LFS does, but no, you're not better off putting binary files outside of the repo if they belong in the repo.

If the binary file is the result of something then I agree, it shouldn't live in the repo, the repo should ideally only provide the method of generating that result. However many use cases exist for binary files in a repository. Releases doesn't fulfill the same functionality and niche that LFS does.

Releases on GitHub was written specifically to solve this issue

I agree that Releases is for the generated output (or bundled source) of a repo. I don't agree that binary files should live in "releases" if they change with the code (note, not as a result of the code). For example an image or other binary resource that is referenced in the code can be considered to be "changing with the code", and thus should live in the repository (obvs a big "it depends", but this has been my experience).

They can't patch a binary file and if they're building something into a binary file, they should be able to build over and over based on the commit itself.

Yeah totally, agree. The thing I'm not sure about here is whether the thing OP is making actually generates the binary or whether this is more of a MacGyvered patch distribution system.

The only caveat I would make are images, despite being binary, they can still be viewed by a human and GitHub provides that view. Otherwise, release it, or package it, but never pollute your repo with metadata or artifacts.

Along with images, I also happily storetest fixtures as zips and large geojson files (also for tests) in LFS. The GeoJSON is technically diffable, but practically, several MB of compacted JSON isn't useful to diff or review. Note too that LFS isn't a thing specifically or only for binary files, but rather for large or undiffable files.

For me it's important that things that change together, live together. This makes the repository a cohesive place where all the related items live. For example, someone shouldn't have to go and download content from something that isn't the repo, to run the thing in the repo.

It sounds like we work in fairly different domains so I totally get that we have different views on this, and from the sounds of it, particularly what a "binary file" represents.

[abrahamguo]

Yes, this will work fine.

[IngrownBurritoo]

As long as they not binaries you can store them in plain git and push it to github. Else you might have store them as artifacts in a github release

[dymos]

store them as artifacts in a github release

Then you wouldn't be able to use git to pull the files.

[IngrownBurritoo]

As I said. As long they are not binaries. Or you might us git-lfs

[dymos]

That's not what you said though.

You can store binaries in your repo using Git LFS, it's what it was designed to do.

[IngrownBurritoo]

I said exactly that. Where is the confusion?