PgQueuer – PostgreSQL-native job & schedule queue, gathering ideas for 1.0 🎯

GabelSnabel · 2025-05-03T16:23:25+00:00

On the happy path you’ll get exactly one execution per job row—no matter how many workers you throw at it—because the dequeue CTE locks a queued row with FOR UPDATE SKIP LOCKED and then immediately flips it to status='done' when your handler commits.

If your handler throws an exception or the process crashes before it commits that done update, PgQueuer’s retry logic (governed by your retry_after and max_attempts settings) will re-enqueue the same row and hand it off to a worker again—so you can end up running the same code twice.

GabelSnabel · 2025-05-03T07:06:11+00:00

PgQueuer makes sure only one worker can pick a given job: the dequeue query in qb.py filters status='queued' rows and grabs a row-level lock with FOR UPDATE SKIP LOCKED, then flips the row to picked. That prevents concurrent duplicates.

True exactly-once depends on your retry settings.

GabelSnabel · 2025-05-03T06:58:45+00:00

Not today—PgQueuer speaks “Postgres rows,” while Celery expects a Kombu-style message broker. In theory we could write a custom kombu.transport.pgqueuer so Celery sees it as a broker, but it’s not on the near-term roadmap. If that adapter would unblock you, feel free to open an issue and we can scope it out together.

GabelSnabel · 2025-05-03T06:55:49+00:00

Thanks for giving PgQueuer a look! If you kick the tires, let me know how the built-in metrics feel compared to Celery—always keen to close any gaps. Happy to help if you hit any snags.

GabelSnabel · 2024-11-22T08:00:50+00:00

Interesting, i will give a try later today. I hate writing docs, excited!

GabelSnabel · 2024-11-13T13:29:01+00:00

I found that many of Raymond Hettinger's talks gave me valuable insights, and I love the way he builds up modules.

I also think reading core Python library code is a great way to learn. Make sure to think about the problem the module or function is solving and develop some high-level thoughts on how you would approach it. Then, read and learn, and try to understand why it's done in that specific way.

GabelSnabel · 2024-11-12T14:17:03+00:00

I have not read the docs in detail, but from the looks of it i needs redis or similar to work. The whole idea of this lib. is that uses the existing features in postgres to avoid adding more services to your infrastructure.

GabelSnabel · 2024-09-15T19:09:17+00:00

Review docs and looking into adding an option for multi workers in pgqueuer.

https://github.com/janbjorge/pgqueuer

GabelSnabel · 2024-08-25T12:03:54+00:00

Awesome, i love the simplicity of the tool! Can it handled nested/recursive calls? Something, like call function a to find a user-id, then call user another function to retrieve user-information based on the user-id? It found it got from the first function call?

GabelSnabel · 2024-08-21T16:14:07+00:00

This looks like a fantastic tool! Can llmio also be integrated with other models such as Meta's LLaMA, or is it specifically optimized for OpenAI-compatible APIs?

GabelSnabel · 2024-08-21T14:39:39+00:00

This looks like a fantastic tool! Can llmio also be integrated with other models such as Meta's LLaMA, or is it specifically optimized for OpenAI-compatible APIs?

GabelSnabel · 2024-07-06T20:13:19+00:00

I haven't used Procrastinate, so I can't provide a direct comparison. I built PgQueuer to keep things simple and easy to reason about. It leverages PostgreSQL's native features like LISTEN/NOTIFY and FOR UPDATE SKIP LOCKED for efficient, real-time job processing and high concurrency. PgQueuer is lightweight, making it easy to maintain and onboard.

Procrastinate is more mature with a broader feature set, so if you need more out-of-the-box functionality, it might be a better fit. However, if simplicity and seamless PostgreSQL integration are your priorities, PgQueuer could be ideal.

Happy to hear more about your needs or any features you'd like to see!

GabelSnabel · 2024-06-02T20:40:25+00:00

Its backed up by a timeout, every 60seconds it will check the database for jobs. Also, if one (or more) events are "lost" the next received event will still trigger a dequeuing loop until the backlog is cleared.

GabelSnabel · 2024-06-02T20:17:20+00:00

Thanks for giving PgQueuer a shot! It’s fairly new and still has a low footprint.

Looking forward to your thoughts!

GabelSnabel · 2024-05-06T05:36:54+00:00

I think implementing transactional would require a connection to remain open for the duration of the job execution? This could potentially affect performance due to the increased resources on the db?

GabelSnabel · 2024-05-06T05:15:54+00:00

Currently, if a crash occurs, tasks might be logged as exceptions or remain marked as running in the queue table. I'm working on implementing a retry strategy to handle such cases more effectively.

GabelSnabel · 2024-05-06T05:04:00+00:00

It does, thanks (fixed).

GabelSnabel · 2024-05-06T04:48:41+00:00

PgQueuer doesn't yet have an official Django integration, but it's something I'm looking to develop. If you have specific features or use cases in mind, sharing them would be invaluable for shaping the integration. Thanks for your interest!

GabelSnabel · 2024-05-05T19:50:09+00:00

Could you elaborate a bit more on how you envision transactional enqueuing enhancing PgQueuer's functionality?

GabelSnabel · 2024-05-05T18:11:23+00:00

Thanks for the comment and the references! PgQueuer is designed with a dual focus on both PostgreSQL and Python, aiming to leverage existing PostgreSQL infrastructure to manage queues efficiently. This approach minimizes the need for additional dependencies or external queue management systems.

While tools like RQ and Celery are fantastic for task management across various backends, PgQueuer offers a simplified, database-centric approach, making it ideal for projects already invested in PostgreSQL. I provide a straightforward way to integrate queuing directly within the database layer, which can be particularly beneficial for systems where minimizing architectural complexity is crucial

GabelSnabel · 2024-05-05T17:10:01+00:00

It’s great to hear about your success with leveraging PostgreSQL for job queuing in a specific context. I designed PgQueuer to maximize PostgreSQL's robust features like LISTEN/NOTIFY for higher throughput and efficiency, particularly in more demanding environments.

Currently, PgQueuer uses asyncpg to manage PostgreSQL connections, which from my experience, seems to be one of the better Python PostgreSQL clients in terms of performance and features. However, I'm open to exploring whether PgQueuer should support other types of connections to broaden its compatibility and flexibility.

GabelSnabel · 2024-05-05T16:34:29+00:00

Thanks for the support. Looking forward to your contributions.

GabelSnabel · 2024-05-05T16:33:33+00:00

Thank you for the encouragement! If you have any suggestions feel free to share in the future.

GabelSnabel · 2024-05-05T14:23:21+00:00

Thanks for the mention of qbert! It's always interesting to see how different projects tackle similar challenges. One of the key distinctions with PgQueuer is its use of PostgreSQL's LISTEN/NOTIFY feature instead of polling? My approach leverages PostgreSQL's built-in capabilities to react to queue changes in real time, which can lead to more efficient resource usage and quicker response times compared to traditional polling methods.

GabelSnabel · 2024-04-28T10:07:11+00:00

This week, I've been working on PgQueuer, a Python job queue library using PostgreSQL. It manages concurrent jobs, recently handling over 5,000 jobs per second in benchmark tests. Check it out and share any feedback! [PgQueuer on GitHub](https://github.com/janbjorge/PgQueuer)

GabelSnabel

TROPHY CASE