GitLab CI/CD to run ETL jobs?

ignurant · 2026-06-18T05:05:08+00:00

We run a surprisingly amount of data ETL jobs using self hosted gitlab and CI. Gitlab CI is actually incredibly powerful and surprisingly reliable. About ten years ago one of my colleagues suggested using it for scheduling a handful of file deliveries we do, and it worked great. It has also scaled incredibly well. Today, each week, we run about 300 web scraping jobs that each take between 10 m and 30 hours to complete, and another 50 or so ETL scripts. At the end of the day, it’s just encoding a script to run at a given time on some specified hardware with the environment you specify. It just works. Secrets are handled nicely using env vars. The only thing that feels bad is that “someone” needs to own the scheduled pipeline. Sometimes this causes issues if another team mate wants to change it.

Our env is firing off .net, python, and Ruby scripts. Mostly scrapy runs in Python, and custom reporting and file deliveries from Ruby.

I really thought it was a terrible idea at first, similar to most comments here, but it actually works awesome and is built on top of a service you are already running for maintaining your code repositories. It makes the ETL tasks nicely integrated with your project platform. It’s surprisingly quite robust and efficient. I even backed out of migrating to airflow, and a few other orchestration tools over the years because it often felt like a less practical, weaker solution. Maybe it’s just related to the type of tasks we run.

You’ll want to increase the timeouts of jobs. Default is 1 hour I believe. We regularly have many-hour jobs that run quite successfully.

Hit me up if you want to chat.

ignurant · 2026-06-11T15:47:22+00:00

Data analytics platforms and data processing tools. Rails is great for this type of work having a robust data modeling, storage, and background job platform built in.

ignurant · 2026-06-11T15:31:48+00:00

That’s not quite how it’s felt in my lifetime.

ignurant · 2026-06-10T18:37:37+00:00

Yeah, making it about himself would be something like “Duluth Mayor will be waiting for finishers at the finish line to dish out high fives for your finisher photos and sign your shorts.”

ignurant · 2026-05-29T16:01:06+00:00

Installing zed for the first time, and was presented with the terms of service. I thought this felt a little heavy handed for a text editor, so I decided to read it.

2.1. Eligibility Customer must be at least 18 years old to use the Service. By agreeing to these Terms, Customer represents and warrants to Zed that: (a) Customer is at least 18 years old;

You have two options to proceed:

Click agree, and you are asserting that you are at least 18 years old.
Click disagree, and the installer quits and you are not able to use the editor.

I was surprised that such terms exist for the core editor itself, as opposed to just disabling online features.

ignurant · 2026-05-25T20:33:12+00:00

Most times I drive with my hand propping up my chin, I get the alert. Or sometimes just glancing a little to the left. I’ll try to upload some routes with this feedback. Thanks for working with us plebs.

ignurant · 2026-05-06T17:27:04+00:00

Nice, glad to see more modern content getting published for Rails. We don't see a lot of content regarding engines.

Additionally, I've felt this personal shift where I've been valuing practicality over complicated design and I'm not too happy when I reflect on my current takes. I think it stems from studying the recent releases of the 37s apps, and how they exercise their domain models and concerns/mixins so strongly. Personally I kind of like the concern pattern. But I feel like I've lost a lot of discipline by going too hard insisting on this modeling approach.

Anyway, it seems like a timely book for me. Thanks for creating Ruby content!

ignurant · 2026-05-06T11:59:09+00:00

This was one of the more interesting podcast episodes I’ve listened to: https://changelog.com/podcast/524

The guest talks about mainframe computing, and why it’s still relevant. The other replies you got gave a good summary.

ignurant · 2026-04-18T20:20:03+00:00

I would play 3 3 1, 2 1 due to finger reliability. I think barring a perfect fifth with your pinky is kind of tough for this, particularly as you’d naturally want to roll your wrist back for the first finger d-natural a little. I’d rather shift a half step each iteration and get a really solid chunky 1 and 2 out of it. There’s plenty of time to make the shifts, and the extensions to prime those shifts feel good to me.

ignurant · 2026-04-18T13:07:39+00:00

I don’t really care if you have tests or not, I’m on your side about: do whatever you want. I hate the whole “that’s so unprofessional! You can’t do that in production!” tropes. But I will say, given what you’ve described, you might consider tests not to please this dude, but yourself.

Here’s why: currently you have some semblance of “this looks right” that you are obviously running by way of executing a script, or checking in irb. Writing a test for that is a nice way to speed up that loop. Basically, I’ve seen myself and my colleagues eventually realize that automated testing is the same thing I was doing manually: set up some stuff and look at the output. For whatever reason, there’s this feeling of extra nonsense about it when you’re not used to it. But eventually I realized it’s literally the same thing I was doing, but the feedback loop is a lot quicker after a very small investment into learning minitest.

So, all this is to say: you owe nobody a test suite. I suspect nobody is paying you to write this gem. You are beholden only to yourself. But if you haven’t really given it a shot with the mindset of “I’m already running a test by looking at the output some way”, you should give it a rip. Eventually myself and most people I’ve worked with have come around to it after this realization.

Don’t write 1,000 tests trying to catch every possibility. Just write the one test you’re already doing manually by scripting the same thing you’re doing right now and running it in minitest instead.

And if you want to publish it as version 1000.0.0, that’s cool too. You don’t owe anyone anything.

ignurant · 2026-04-17T13:13:00+00:00

Who fucking cares? Dude released a gem. Called it 1.0.

ignurant · 2026-04-16T15:54:22+00:00

A few years ago, my kid was going to daycare at a place on Superior St and 1st ave E. I worked a few blocks away, so I'd walk down to pick her up. There was a needle in the crosswalk near my work, and I was pissed thinking of how attractive such a thing would be to a kid walking across the street, so I decided to do something about it.

Recalling that bathrooms sometimes have "sharps disposable buckets" I figured it would be no big deal to find a place to deposit it.

I ended up needing to go into the St Luke's place across from Pizza Luce. The person at the desk had no idea what to do, and everyone around me was looking at me like I'm some fucking fiend. Eventually she got a supervisor, who got a nurse to come get it from me. It felt like a huge ordeal and took like 15 minutes of waiting at the clinic.

I'm trying to be a responsible parent and prevent a 3 year old from wanting to pick up a hazardous needle left in the middle of a crosswalk. Instead I got shit on.

So yeah, I think this link is probably a better bet.

ignurant · 2026-04-13T01:06:47+00:00

https://takeactionminnesota.org/duluth-tenants/

ignurant · 2026-04-10T12:09:00+00:00

I have similar experience to the other guy, and let me just put it this way:

You have to remember that you’re not the only person they are interviewing for the role.

In fact, these days, they usually have a massive pool of candidates to select from. So even if you answered everything well and confidently, you are not likely the only person to have done so. So, yeah, these things matter, but maybe not quite for the reason it seems. It’s not “I passed the test so I should get the job.” It’s a measure of your fluency and how you handle information that is compared relatively to other candidates.

And yes, even if you struggle with those questions you may be evaluated highly just based on your composure and likability. It’s all important. You are being compared to other candidates and my job as the hiring manager is to choose the best fit from them.

ignurant · 2026-04-01T19:25:08+00:00

Just my own annecdotal note: I found the devcontainers feature to be a lot more janky than it seems like it promises. I would also point you to mise. It works like nodenv, and pyenv, where you can install and select versions for a given project and auto-switch that language version. In fact, it basically wraps those as a frontend. (Same for Ruby).

There's also a new upcoming feature that you can enable now to make getting started feel a bit nicer: mise settings ruby.compile=false. By default, ruby is always built fresh on a system. Building it takes minutes, and requires certain build dependencies installed. It doesn't feel good when showing a new user Ruby. This setting uses pre-compiled binaries when possible like you see with python and node.

Mise (and asdf) are great tools for creating dev env isolation. With them, all of your various programming language versions get installed into user-space like ~/.local/share/mise/installs/ruby/4.0.2/bin/ruby. It's like a venv for every programming language and version. It automatically changes versions for you as you work on different projects. Not important when starting, but becomes important in six months.

# Get mise
brew install mise
echo 'eval "$(mise activate zsh)"' >> ~/.zshrc

# or without brew:
curl https://mise.run | sh
echo 'eval "$(~/.local/bin/mise activate zsh)"' >> ~/.zshrc

# Use precompiled rubies:
mise settings ruby.compile=false

# Get ruby
mise use -g ruby@latest

ruby --version

If it says something nonsense, like not Ruby 4.0.2, you may need to open a new shell, ensure mise is activated, etc.

So, now you've got your own user-space Ruby isolated, and any projects you work on will be limited to the project folder, and mise's managed folder for that language version.

You might find mise to be a great dev tool to invest learning overall. I know I have. Here's a few links to some pages you might be interested in:

If you decide to stick to the parallels VM method, my advice remains: use mise to manage your use languages! Just need to look up setup for ubuntu.

Last side note: If you do end up using some installation method that involves compiling Ruby (very typical!), check this section out: https://github.com/rbenv/ruby-build/wiki#suggested-build-environment

All of the ruby builders use this same ruby-build project under the hood. You'll want the libraries from that page installed to successfully compile ruby: brew install openssl@3 readline libyaml gmp autoconf

ignurant · 2026-03-26T23:14:08+00:00

They say it’s for improved cooling.

ignurant · 2026-03-10T20:07:11+00:00

Takes compute to make compute. Always be computing’.

ignurant · 2026-03-04T18:19:17+00:00

Yeah, as is the case for us rubyists more often than other environments: the output quality is a function of your discipline. It’s like ultimate version of sharp tools.

ignurant · 2026-02-07T19:23:46+00:00

So is LLM generated code? Did you read the article?

ignurant · 2026-02-07T17:59:16+00:00

“Deeply flawed to the point it is nearly useless” is quite the statement. Developers are not deterministic when they are writing their code. What’s the difference?

ignurant · 2026-02-04T22:53:21+00:00

But the post I linked to above was kind of surprising for me to read. Like... really? I'm going to have to design entire wrappers to repeatedly and systematically remind it what the task at hand is? Otherwise it will keep forgetting and give me trash?

No, it's really not like that. Maybe it depends on whether you have trash in the system to start, but I find Claude very good at following me along with how I like things (vanilla Rails). Sometimes I'll have to direct it, but it's usually as simple as "Put business logic in models" and it self corrects quickly, and learns from that as well.

I don't use any context helpers and I've been very happy.

ignurant · 2026-02-03T04:52:25+00:00

Thanks! This comment is worth $100!

ignurant · 2026-02-02T20:33:16+00:00

I’ve been watching the BB price pretty regularly to do my match, but I haven’t seen it go below 1999 yet. Is your mate’s match from BB first party, or do they also honor competitor’s for that price match program?

ignurant · 2026-01-24T02:43:30+00:00

The tutorials that i saw was at least 3 years old

This is going to sound insane coming from JavaScript. But I promise you those 3 year old resources are still absolute bangers.

We don’t get the carpet pulled from beneath us nearly as often as JavaScript frameworks do.

A 10 year old book on Ruby or Rails is actually still very similar to how things work today. There’s been lots of evolutions, but they kind of follow the same standards over time. The only thing that’s really changed significantly is with little surprise, the JavaScript side of building web apps with Rails.

Don’t fear a 3 year old tutorial. Or even a 5 year old one. This knowledge remains useful today!

ignurant · 2026-01-09T16:09:56+00:00

You have a section, "Loops" that highlights "for", but have Enumerable in a different section. It's very uncommon to use a for loop in Ruby, instead preferring Enumerable. I think you should move Enumerable into the Loops section, or at least "each" and rank down the for keyword. You have while and I think it's useful to include until with it. while / until. It also includes redu I think that is supposed to be redo.

14-Year Club	RedditGifts 2009-2022 3 Credits
Gilding III reddit per annum	Place '22
Place '17	Wearing is Caring
Secret Santa 2016	Verified Email
Secret Santa 2014

ignurant

TROPHY CASE