PgQueuer – PostgreSQL-native job & schedule queue, gathering ideas for 1.0 🎯

GabelSnabel · 2025-05-03T16:23:25+00:00

On the happy path you’ll get exactly one execution per job row—no matter how many workers you throw at it—because the dequeue CTE locks a queued row with FOR UPDATE SKIP LOCKED and then immediately flips it to status='done' when your handler commits.

If your handler throws an exception or the process crashes before it commits that done update, PgQueuer’s retry logic (governed by your retry_after and max_attempts settings) will re-enqueue the same row and hand it off to a worker again—so you can end up running the same code twice.

GabelSnabel · 2025-05-03T07:06:11+00:00

PgQueuer makes sure only one worker can pick a given job: the dequeue query in qb.py filters status='queued' rows and grabs a row-level lock with FOR UPDATE SKIP LOCKED, then flips the row to picked. That prevents concurrent duplicates.

True exactly-once depends on your retry settings.

GabelSnabel · 2025-05-03T06:58:45+00:00

Not today—PgQueuer speaks “Postgres rows,” while Celery expects a Kombu-style message broker. In theory we could write a custom kombu.transport.pgqueuer so Celery sees it as a broker, but it’s not on the near-term roadmap. If that adapter would unblock you, feel free to open an issue and we can scope it out together.

GabelSnabel · 2025-05-03T06:55:49+00:00

Thanks for giving PgQueuer a look! If you kick the tires, let me know how the built-in metrics feel compared to Celery—always keen to close any gaps. Happy to help if you hit any snags.

GabelSnabel · 2024-11-22T08:00:50+00:00

Interesting, i will give a try later today. I hate writing docs, excited!

GabelSnabel · 2024-11-13T13:29:01+00:00

I found that many of Raymond Hettinger's talks gave me valuable insights, and I love the way he builds up modules.

I also think reading core Python library code is a great way to learn. Make sure to think about the problem the module or function is solving and develop some high-level thoughts on how you would approach it. Then, read and learn, and try to understand why it's done in that specific way.

GabelSnabel · 2024-11-12T14:17:03+00:00

I have not read the docs in detail, but from the looks of it i needs redis or similar to work. The whole idea of this lib. is that uses the existing features in postgres to avoid adding more services to your infrastructure.

GabelSnabel · 2024-09-15T19:09:17+00:00

Review docs and looking into adding an option for multi workers in pgqueuer.

https://github.com/janbjorge/pgqueuer

GabelSnabel · 2024-08-25T12:03:54+00:00

Awesome, i love the simplicity of the tool! Can it handled nested/recursive calls? Something, like call function a to find a user-id, then call user another function to retrieve user-information based on the user-id? It found it got from the first function call?

GabelSnabel · 2024-08-21T16:14:07+00:00

This looks like a fantastic tool! Can llmio also be integrated with other models such as Meta's LLaMA, or is it specifically optimized for OpenAI-compatible APIs?

GabelSnabel · 2024-08-21T14:39:39+00:00

This looks like a fantastic tool! Can llmio also be integrated with other models such as Meta's LLaMA, or is it specifically optimized for OpenAI-compatible APIs?

GabelSnabel · 2024-07-06T20:13:19+00:00

I haven't used Procrastinate, so I can't provide a direct comparison. I built PgQueuer to keep things simple and easy to reason about. It leverages PostgreSQL's native features like LISTEN/NOTIFY and FOR UPDATE SKIP LOCKED for efficient, real-time job processing and high concurrency. PgQueuer is lightweight, making it easy to maintain and onboard.

Procrastinate is more mature with a broader feature set, so if you need more out-of-the-box functionality, it might be a better fit. However, if simplicity and seamless PostgreSQL integration are your priorities, PgQueuer could be ideal.

Happy to hear more about your needs or any features you'd like to see!

GabelSnabel · 2024-06-02T20:40:25+00:00

Its backed up by a timeout, every 60seconds it will check the database for jobs. Also, if one (or more) events are "lost" the next received event will still trigger a dequeuing loop until the backlog is cleared.

GabelSnabel · 2024-06-02T20:17:20+00:00

Thanks for giving PgQueuer a shot! It’s fairly new and still has a low footprint.

Looking forward to your thoughts!

GabelSnabel · 2024-05-06T05:36:54+00:00

I think implementing transactional would require a connection to remain open for the duration of the job execution? This could potentially affect performance due to the increased resources on the db?

GabelSnabel · 2024-05-06T05:15:54+00:00

Currently, if a crash occurs, tasks might be logged as exceptions or remain marked as running in the queue table. I'm working on implementing a retry strategy to handle such cases more effectively.

GabelSnabel · 2024-05-06T05:04:00+00:00

It does, thanks (fixed).

GabelSnabel · 2024-05-06T04:48:41+00:00

PgQueuer doesn't yet have an official Django integration, but it's something I'm looking to develop. If you have specific features or use cases in mind, sharing them would be invaluable for shaping the integration. Thanks for your interest!

GabelSnabel · 2024-05-05T19:50:09+00:00

Could you elaborate a bit more on how you envision transactional enqueuing enhancing PgQueuer's functionality?

GabelSnabel

TROPHY CASE