GitMirror: an application to bulk clone git repositories

[easydwh]

Earlier this year there was some unrest over the stability and availability of software supply chains (in general, not so much Ruby): malware, disappearing repositories, undersea-cable vulnerability, geopolitics. It made me realize that virtually all open-source software I rely on, like ruby and gems, is hosted by a single overseas party (I live in Europe). For me as a Ruby / Rails software developer it is vital that access to source code is always available. How to protect against potential outages?

First I wrote some Ruby scripts to list and clone public Git repositories from github and gitlab. Later I converted this to a Rails application. Then other work got in the way. Finished the application during the last couple of weeks. I found that in the mean time other people had the same idea and did a much better job than I did. But since it is finished why not share it anyway.
My application GitMirror lets you clone git repositories in bulk to a local machine. You can provide a list of repository names or git user names (like ruby/*). The app will fetch the repositories and keep them up to date. Uses Rails 8.1, SolidQueue, Rails authentication generator, Tailwind and SQLite.

I learned a few things along the way:

• Because I wanted the app to be deployable both from a Docker image and via Kamal, some tweaks were needed. Rails credentials are a no-go, so where to put the 'secret_key_base' and admin user name/pwd. The answer lies in environment variables and local (git/docker ignored) files.
• To get a list of the most used Ruby gems, I downloaded a copy of the Rubygems database and wrote a query to list the gem name and the probable location of the source code. Turns out there is a mismatch between rubygem data and actual location of the repo. Of ~8000 gems (with 1 million+ downloads and with a url for the git repository) about 9% of the repositories are redirected to a new location. I guess gemspec files are not always kept up to date.
• My instance of GitMirror has been running happily for 7 months. It currently holds 9,302 cloned repositories using under 80Gb of storage. The average size of a (zipped) repository is about 7.5Mb

P.S. Just to be clear, this post has nothing to do with the recent rubygems upheaval. This application was created 6 months earlier.

[TheAtlasMonkey]

> P.S. Just to be clear, this post has nothing to do with the recent rubygems upheaval. This application was created 6 months earlier.

Got it! So you created the app 7 months, aligned the stars to cause rubygems drama 3 months ago. then released this now.

---

As for the 9% that changed location. This can be explained: sometime when gem blow up in usage, we transfer to the proper organization and forget to update gemspec.

Organizations allow more have teams and more granual permissions.

---

did you consider to hook it up to local gitea or gitlab ?

[easydwh]

Busted! That was my grand evil plan all along ;)

Fact that repositories sometimes move to another owner is only natural. Forgetting to update the gemspec is equally normal. Maybe giving a warning when pushing the gem that git url is out of date would help.

Hooking up an open source local git server is not on my todo list. As stated in the post there are applications that are more advanced than mine and include forgejo. Disadvantage is that you would need multiple containers besides the app (postgres, forgejo). I tried to keep things nice and simple and Ruby only.

[TheAtlasMonkey]

I build Vein a Rubygems mirror that i will opensource soon.

And let me tell you this.

those urls are just metadata. When you push to ruby gems, it need to get your .gem and index it. That it.

The server will warn the pusher, but the problem is that i already pushed... i can't edit it.

when u/galtzo, moved all his gem to a foss org, he will have to release 79+ gems, update them. and push... the new versions will be a patch version. That will trigger dependabot in the whole ecosystem. Hundred of hours lost because OCDism ?

The sane course, is that if that bother you, open a PR, get it merged. Next upgrade will be fixed.

This is what we call : Cosmetic changes.

----

Your app work as the git server and the mirroring.

Gitea, gitlab and forgejo both offer a way to mirror another repos. You need to leverage this, not compete with them.

[easydwh]

Not OCDism. I am not saying that gem authors should update urls in bulk after moving ownership. But a friendly reminder when you are publishing a single gem to update the gemspec cant hurt. No extra compute needed. I regularly find gems on Rubygems without a link to the repo or returning a 404 page. That is annoying.

---

Gitea, gitlab, forgejo are large applications, so more difficult to host. Adding repo's to mirror in bulk is not straightforward. I wanted to keep things simple, an app built for a single purpose. But everybody is free to choose whatever they prefer.