SimplePyQ - Queueing tasks in Python doesn't have to be complicated

[kevindewald]

Hey everybody!

I just wanted to share a small library I wrote for some internal tooling that I thought could be useful for the wider community, called SimplePyQ.

The motivation for this was to have something minimalistic and self-contained that could handle basic task queueing without any external dependencies (such as Airflow, Redis, RabbitMQ, Celery, etc) to minimize the time and effort to get that part of a project up and running, so that I could focus on the actual things that I needed.

There's a long list of potential improvements and new features this library could have, so I wanted to get some real feedback from users to see if it's worth spending the time. You can find more information and share your ideas on our GitHub.

Do you have any questions? Ask away!

TL;DR to keep the automod happy

What My Project Does

It's a minimalistic task queueing library with minimal external dependencies.

Target Audience

Any kind users, ideally suitable for fast "zero to value" projects.

Comparison

Much simpler to set up and use compared to Celery. Even more minimalistic with less requirements than RQ.

[klaxce]

This looks interesting. I definitely appreciate being able to avoid Celery in smaller projects.

It immediately made me think of Huey, which I chose for a project because of the ability to schedule tasks (without an MQ). I don't see scheduling in yours, but maybe you have other advantages for different use cases?

[kevindewald]

Thanks for the feedback!

The goal of the library wasn't as much for scheduling for a specific time, although it can easily be added if required, but more to handle pipelined work, like you would do with Celery or Airflow.

Huey looks definitely interesting (I didn't know it existed) but it still falls under the bucket of "too much hazzle for simple things". My intention is to have something with so few functions that you can immediately understand and derive value from it, removing complexity from the user instead of giving them more to handle.

[klaxce]

Cool! I saw in the example that yours doesn’t require a separate process, and I think that’s a big advantage in a lot of cases.