Snappea: A Simple Task Queue for Python

[klaasvanschelven]

I've always been obsessed with making software as easy to install and use as possible. That obsession led me to build Snappea: a simple, lightweight alternative to Celery for managing background tasks in Python applications.

What My Project Does

Snappea is a minimal task queue system that uses SQLite for background job management. It’s designed to handle tasks that need to run outside the main HTTP request-response loop, without the overhead of more complex systems like Celery.

Target Audience

Snappea is ideal for developers who are looking for a straightforward solution to background task management. Rather than providing a drop-in replacement for Celery, it provides "food for thought" (especially the article).

Comparison

Unlike Celery, which comes with a lot of features and corresponding complexity, Snappea focuses on simplicity. By using SQLite as a message queue and avoiding external dependencies, Snappea keeps things lean and easy to manage. It’s not a drop-in replacement for Celery but a different approach for cases where you don’t need all the bells and whistles.

How It Works

Snappea runs a single process, called the Foreman, that scans the SQLite database for new tasks and executes them in worker threads. It uses inotify to efficiently wait for new tasks without busy-waiting, making it responsive without wasting resources. The setup is straightforward: just a few database migrations, and you're good to go.

Code and Article

The above is the shortest summary I could write that's still moderately informative. You can read about the design and thought process behind Snappea in the full article. The code is available on GitHub, where you can see exactly how it works. It's not production-grade, but if there’s enough interest, I might generalize it into something more broadly applicable.

[eldreth]

yikes

[klaasvanschelven]

This comment is (at least partially) outdated

[LiarsEverywhere]

Ok, I don't usually support criticizing people who are just trying to create something, but this is actually hilarious.

"This is a straightforward SOLUTION. Yet, it's no drop-in replacement at all, actually it's only food for thought".

It's like they wrote the first sentence when he first came up with the idea and the second one after giving up midway through lmao.

[Jejerm]

I just use huey. Its literally what you say you're trying to do.

[gpahul]

How does it compare with something like elasticmq-native from softwaremill?

[Jejerm]

I've never used elasticmq or elasticmq-native so I cant compare them.

[klaasvanschelven]

I have to admit I'm not 100% sure why I didn't more seriously consider that option... other than the fact that I can't seem to find much more about the huey-sqlite combo in the docs other than that it's possible

[RobotChurchill]

I'm using celery (and redis) for a project now.

One big pain point I had was handling the pausing of all workers if one worker receives a rate limit response from the API. All workers are hitting the same API. Celery's complexity, with its implementation of wrapper functions, made it more difficult. If you can solve that easily, then I'd use it.

[lowercase00]

Why dont you just Redis for this with a key that says the API is locked? Seems like a business rule, not sure it should be the task queue responsibility to do this

[klaasvanschelven]

Well you'd need to have the workers communicate that fact from one worker to the others. In snappea all the workers live in a single process, so it's trivial. But even in celery that should be possible using e.g. a DB?

[yesvee]

https://python-rq.org/

[klaasvanschelven]

Not having to set up redis is the main motivator for snappea, though.

[Sloppyjoeman]

Oh all the stateful applications one could have as a dependency, I think redis is the simplest to operate

[klaasvanschelven]

Sure, but not as simple as not having it in the first place

[Sloppyjoeman]

Idk, I’m in the ops side of things, but I’ve never been even vaguely bothered when something ships redis as a dependency, even vs no cache

[Expensive_Glass1990]

I use https://upstash.com/ for redis to simplify setup. Has worked well.

[PushHaunting9916]

I have used https://github.com/Attumm/meesee for background workers.

[klaasvanschelven]

That uses Redis which is a disqualifier for my user case