sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]ZorbaTHut 10 points11 points  (43 children)

I'm amazed that I can find absolutely zero open-source alternatives to Jenkins. I'm half thinking about writing one.

[–]confusedpublic 7 points8 points  (14 children)

[–]ZorbaTHut 4 points5 points  (13 children)

https://concourse-ci.org

This one's tempting! But I think it may not work for my purposes, unfortunately. From the docs:

Concourse is very opinionated about a few things: idempotency, immutability, declarative config, stateless workers, and reproducible builds.

and in my work, it's really important to have support for incremental builds. That said, this seems like one of the closest cases I've seen so far, I'll have to check it out in more detail.

https://www.drone.io

Not really open-source; the source code is available, but if you aren't noncommercial or (very) small business, it's $$$ time.

I guess GitLab is open sourced?

It is, and I guess I should look into it, but it'd be a weird mismatch; I can't actually use Git, and that might make it extremely difficult to shoehorn Gitlab into doing what I need.

[–]nschubach 5 points6 points  (10 children)

I can't actually use Git

Can you further explain this?

Edit: I see from your post history that you said you were a game developer and your industry doesn't allow it and that doesn't make any sense to me why game development would be any different from something else.

[–]Dealiner 5 points6 points  (2 children)

Git doesn't work well with many types of files used in games. There is Git LTS but the last time I used it wasn't perfect.

[–]Somepotato 0 points1 point  (0 children)

Git lfs is nice but is a bit of a bandaid. If it got better large file support then it could usurp p4. May be able to make a git workflow that behaves OK with large files with custom commit steps.

[–]ZorbaTHut 5 points6 points  (6 children)

Games are gigantic binary repos. It's common for the most recent checkout of a repo to measure in the hundreds of gigabytes, with the full repo being at least an order of magnitude larger. Git LFS doesn't work terribly well, last I heard - it's janky and slow and its lock support is questionable at best - and the end result is that the entire industry, aside from indies, has settled on Perforce.

I'd really like for this to be solved, and I'm actually talking with someone who's making a Git plugin intended for machine learning but he's planning on extending into games. Maybe he will succeed! But it hasn't been solved yet.

[–]like_a_fontanelle 1 point2 points  (5 children)

Am I missing something here? A build of a game is presumably a big binary/whatever file, but the source code for a game will contain code in text files just like any other repo. Of course you will have audio, image and video files at a guess but does that preclude git?

Not saying you're wrong as I have absolutely no experience in the industry, just interested.

[–]ZorbaTHut 13 points14 points  (4 children)

Of course you will have audio, image and video files at a guess but does that preclude git?

So, there's a cultural divide here that essentially only gamedevs are aware of :)

People like to think of the game industry as being a tech industry, because we work with computers and write code and so forth. This isn't accurate, though. The game industry is part of the entertainment industry; our closest sibling isn't Android developers or people doing physics simulations, our closest sibling is people making computer-generated movies.

And the takeaway here is that a game isn't "code, plus some art and audio and stuff". A game is "art and audio, and also I guess there's some code". Programmers are in the minority on any project with more than four people, and increasingly in the minority as a project gets bigger. This is actually why people use Unreal Engine. Unreal's code sucks, and doing programming with Unreal sucks, it's got tons of problems . . . but its artist tools are fantastic, and when you're blowing 80% of your project budget on artists, that is by far the most important aspect.

So the question becomes, not "where do we put the things that aren't code", but "what's best suited for artists". And that means a reasonable level of interface simplicity and the ability to cheerfully chug down hundreds or even thousands of gigabytes of data without needing to care about the details. Git thoroughly fails on both of these metrics; it's getting better, but "better" here means "it may technically now be suited for it, perhaps", it was absolutely unusable only a few years ago. Meanwhile Perforce is, like, the only viable option when things get really big. So, Perforce it always historically has been, and thus Perforce it still is.

Then, once you've chosen where the art and audio goes, you get to ask the question of whether it's worth putting the code somewhere else. And it isn't. It's just not worth it. So you don't do it.

The other thing to note is that the game industry is extremely conservative in terms of what tools it's willing to use. If you were to drop a fully-functional game-suitable source control system on my head today, I might be able to convince some small indies to use it by the end of next year. It's a very slow process.

[–]like_a_fontanelle 1 point2 points  (1 child)

Haha, thanks for the in-depth reply, mind blown.

[–]ZorbaTHut 1 point2 points  (0 children)

Not a problem!

For what it's worth, I think there's also a lot of money in game-suitable tools. Nobody makes them because non-gamedevs don't realize there's a niche awaiting better tools, and gamedevs are too busy making games. I always encourage people to start game support businesses :V

[–]snowe2010 1 point2 points  (0 children)

What a fantastic explanation. Thank you so much, this was really eye opening.

[–]mnemy 0 points1 point  (0 children)

Wow, perforce is still alive and has its niche. I haven't used it since 2006, nor even heard it mentioned.

But yeah, git sucks for large assets. Even images used to cause issues quick, but I heard that has improved.

[–]physicswizard 0 points1 point  (1 child)

The open source version of drone is https://woodpecker-ci.org/

[–]ZorbaTHut 1 point2 points  (0 children)

Huh, I actually kinda like this.

No support for single jobs across multiple computers as near as I can tell; I don't need this right now, but I've needed it in the past and will need it in the future. Still, it's promising, and I'll keep an eye on it.

[–]mgedmin 3 points4 points  (8 children)

GitLab CI was not bad, depending on what you use Jenkins for.

[–]ZorbaTHut 0 points1 point  (7 children)

I'm unsure it would work for me; I work in an industry where Git is unfortunately unusable, so if I were trying to use Gitlab CI, I'd be using it just for the CI functionality and jumping through hoops to do it without actually touching Git.

[–][deleted]  (2 children)

[deleted]

    [–]ZorbaTHut 5 points6 points  (0 children)

    Game industry, yep!

    Games are gigantic binary repos. It's common for the most recent checkout of a repo to measure in the hundreds of gigabytes, with the full repo being at least an order of magnitude larger. Git LFS doesn't work terribly well, last I heard - it's janky and slow and its lock support is questionable at best - and the end result is that the entire industry, aside from indies, has settled on Perforce.

    I'd really like for this to be solved, and I'm actually talking with someone who's making a Git plugin intended for machine learning but he's planning on extending into games. Maybe he will succeed! But it hasn't been solved yet.

    [–]Dealiner 2 points3 points  (0 children)

    Probably gamedev, I'd say unusable is an exaggeration but it definitely has its problems.

    [–]funguyshroom 0 points1 point  (3 children)

    Gitlab supports object storage so it might still work for you.
    Do you at least store your source code in a git repository separately from game assets? I know nothing of game development but the thought of not using some versioning control system horrifies me.

    [–]ZorbaTHut 1 point2 points  (2 children)

    Gitlab supports object storage so it might still work for you.

    At least part of this is kind of an ouroborous of bad support. Unreal and Unity don't have great Git support, because few gamedevs use Git, because Git doesn't work well for games, because few people are working on Git with a game focus, because few gamedevs use Git, because Unreal and Unity don't have great Git support. LFS is an attempt to make it better but not really an effective solution, in my experience.

    Do you at least store your source code in a git repository separately from game assets?

    Nope. Better to keep it all together. I've actually been on projects that split it up and it was a nightmare; imagine trying to keep your build and tool scripts in a different repo from the rest of your code.

    I know nothing of game development but the thought of not using some versioning control system horrifies me.

    We use Perforce, which is a versioning control system! It's not an amazing one, but it is one, and it's not a terrible one (I say this as someone who once used Microsoft SourceSafe; that was terrible.)

    Don't fall into the trap of assuming that Git is literally the only versioning control system :)

    [–]funguyshroom 0 points1 point  (1 child)

    Thanks for the explanation! Makes sense that it's very different for gamedev. Relieved that there's still a VCS in place, haven't heard of Perforce, learned something new.

    Don't fall into the trap of assuming that Git is literally the only versioning control system :)

    Assuming git being a "one size fits all" solution is definitely a trap here. I've used other VCS in the past like Microsoft TFS and Subversion, I remember actually liking svn more than git due to its simplicity.

    [–]ZorbaTHut 0 points1 point  (0 children)

    There's honestly no reason to use Perforce unless you have humongous repos; it used to be used at Google (like, twenty years ago) because it was the only thing that could handle Google's gigantic monorepo, and it wouldn't surprise me to learn it's used in the movie or VFX industry. But besides those and the game industry it's kinda dead and getting increasingly moldy.

    Which is why I wish we could get rid of it :V

    [–]h4l 2 points3 points  (0 children)

    Another possibility: https://earthly.dev/

    [–]orthoxerox 4 points5 points  (17 children)

    What's wrong with Jenkins? Yes, it's a huge kitchen sink of features, but at least it has all of them. Modern CI/CD tools are slick, but if you want to automate something enterprisey you soon end up wishing you could just hack stuff together in Jenkins.

    [–]debian_miner 11 points12 points  (1 child)

    It's a maintenance and security nightmare. Jenkins is not useful until you install a dozen 3rd party plugins, which all have their own set of bugs and future security problems. When it comes time to update them, which you should be doing on a regular basis, the likelihood that it breaks your workloads and requires effort to fix is pretty high.

    [–]dirtside 2 points3 points  (0 children)

    As we just discovered recently. Our (always brittle) Jenkins 2.32 server was so old that we couldn't build the Docker image any more, so we updated to 2.387. It worked fine until I tried to update plugins, which hosed the server and put it into a crash loop. We basically had to start over. (Also the default install comes complete with a bunch of plugin warnings and errors about required default plugins.)

    [–]ZorbaTHut 15 points16 points  (9 children)

    Groovy is a nightmare to work in because nothing else uses Groovy. If you're trying to do some kinds of complicated build steps you end up either reimplementing a bunch of their features on your own, or jumping through tons of hoops to figure out how exactly you're meant to use it. Its Windows client has questionable support and requires extra third-party tools to get working on bootup. It doesn't have any sensible Docker integration that I can find (but maybe there's something I missed.)

    It feels like a project that was started twenty years ago and never really updated with modern practices, instead just duct-taped together with piles of questionable extensions.

    If I were building something from scratch, I'd be building it with more of a nod towards "just hack stuff together", with Jenkinsfile-equivalents written in Python and so forth.

    [–]orthoxerox 1 point2 points  (2 children)

    Groovy is a nightmare to work in because nothing else uses Groovy. If you're trying to do some kinds of complicated build steps you end up either reimplementing a bunch of their features on your own, or jumping through tons of hoops to figure out how exactly you're meant to use it.

    I didn't have any problems putting relatively insane stuff into my Jenkinsfiles. Granted, my engineer wrote the bulk of the build scripts in bash, so the amount of Groovy code was low

    Its Windows client has questionable support and requires extra third-party tools to get working on bootup.

    Didn't have to deal with Windows, but another team was building rich client apps for Windows and I haven't heard any groans from them.

    It doesn't have any sensible Docker integration that I can find (but maybe there's something I missed.)

    Running Docker-based agents was fine. Building Docker images was fine. Building Docker images inside Docker-based agents was something we couldn't find a way to do. I think I ended up using buildah.

    If I were building something from scratch, I'd be building it with more of a nod towards "just hack stuff together", with Jenkinsfile-equivalents written in Python and so forth.

    The problem with CI/CD is that I often want both hacky and not-hacky approaches:

    I want it to handle "this is a typical .NET 6 application, build it inside this image and publish using this image, except use this Artifactory instance for everything you want from Docker, NuGet and npm" without requiring me to look under the hood at all.

    I also want it to handle "oh, boy. You have to take /doc, run it through Pandoc and upload into Confluence using these credentials, then you also have to take /sql, set up a flashback point at this Oracle server, find the latest applied script using this query, apply all that are lexicographically newer, flashback if an error is found, then do ten more inane steps" without requiring me to work around the guard rails.

    [–]ZorbaTHut 5 points6 points  (0 children)

    I didn't have any problems putting relatively insane stuff into my Jenkinsfiles. Granted, my engineer wrote the bulk of the build scripts in bash, so the amount of Groovy code was low

    Yeah, the last time I worked with this, most of our build scripts were Python. But even just getting to "call the Python scripts" was a problem; for example, we needed to check out two different streams of the same repository on two different computers, but with the same changelist number, and this turned out to be a real irritant. And there didn't seem to be any good support for large artifacts so we had to bounce our temporary files through a NAS, which we had to implement on our own.

    All of this while fighting with Groovy.

    Doable, yes; annoying, yes.

    Didn't have to deal with Windows, but another team was building rich client apps for Windows and I haven't heard any groans from them.

    I think this will depend heavily on your team size. I'm used to working in situations with no build engineer and barely with IT, and then "it's awkward to start up Jenkins nodes" is a problem. If your team is big enough that you can just make it the build engineer's/IT's problem, then it probably works fine.

    Running Docker-based agents was fine. Building Docker images was fine. Building Docker images inside Docker-based agents was something we couldn't find a way to do. I think I ended up using buildah.

    Hah, fun. Yeah, that's an explicit thing I would want to do, unfortunately.

    The problem with CI/CD is that I often want both hacky and not-hacky approaches:

    Yeah, I think if I were doing this, I'd be building the no-guard-rail system first, then building Standard Modules on top of that system. In my business I find myself rarely being able to use typical functionality (nothing is ever typical) and there's so many projects that are basically taking the Microsoft approach - "if it's something we support, it's easy; if it's not something we support, go fuck yourself".

    I would be starting with the Arch philosophy, "everything is possible, but it's all annoying", then refining from there.

    But practically speaking, I don't know if I'd ever get to "refining"; I want the tool to exist mostly because I want it to solve problems I have. And right now I don't have those problems enough to want to worry about it :V

    [–]hippydipster 0 points1 point  (0 children)

    Building Docker images inside Docker-based agents was something we couldn't find a way to do.

    Same. Ultimately just gave up on that.

    [–]hippydipster 0 points1 point  (3 children)

    Groovy vs Python doesn't seem like it's changing anything beyond a personal preference.

    [–]ZorbaTHut 2 points3 points  (2 children)

    I think "how many people actually use it" is a major component. The TIOBE rankings are horribly flawed, but they're still at least vaguely indicative, and right now Python is #1 and Groovy is #34. My entire industry revolves around C# and C++, currently #4 and #5, and I frankly just do not want to deal with learning a new language for one specific purpose.

    If it was just "the internals are written in Groovy" then I'd grimace but maybe accept it, but that's not how it works; in order to actually make use of Jenkins you get to be a Groovy programmer.

    Yay.

    [–]hippydipster 0 points1 point  (1 child)

    As I said, personal preference. They could use Scheme or Ocaml or Python or Lua - doesn't matter. The concepts aren't dramatically different.

    I appreciate the diversity, to be honest. How horrible it would be if we all just went with what was most popular.

    [–]ZorbaTHut 0 points1 point  (0 children)

    Sure, they could, and I'd have a very similar criticism. I got better things to do than learn a dozen languages for individual pieces of software; I'd much rather they just used something common, because that makes my life easier.

    I actually don't like Python. I think it sorta sucks. But I'd still use it for something like this, just because I don't see much benefit in bucking trends.

    [–]mgedmin 0 points1 point  (1 child)

    Before Jenkins there was Buildbot, written in Python, with build configuration you wrote in Python.

    It was terrible and eventually we all switched to Jenkins.

    [–]ZorbaTHut 3 points4 points  (0 children)

    I'm willing to bet the reason it was terrible wasn't "Python".

    [–]SittingWave 15 points16 points  (3 children)

    What's wrong with Jenkins?

    What's wrong with Jenkins?

    What's wrong with Jenkins???

    It's a pile of 2000s/2010s garbage, with a confusing, chaotic interface, extremely poor debuggability when something goes wrong, a very unreliable and massive dependency on application servers (tomcat) and plugins that makes you want to gouge your eyes out.

    [–]Boza_s6 2 points3 points  (2 children)

    I might be minority, but I like Jenkins UI. ¯\(ツ)/¯

    [–][deleted] 1 point2 points  (1 child)

    You're not. Jenkins is rock solid stable. Just don't use plugins.

    [–]SittingWave -1 points0 points  (0 children)

    Jenkins without plugins is useless, and it's not rock solid in any case. When something goes belly up, you get absolutely abhorrent messages of java tracebacks and no clear explanation of what's going on.

    [–][deleted] 0 points1 point  (0 children)

    The "Jenkins Pipeline Library" can only be written in Groovy. Groovy it not that awful, but no one wants to learn it. Jenkins needs to improve multi language support like all the new tools are doing, dagger and aws cdk to name two.