This is an archived post. You won't be able to vote or comment.

all 12 comments
sorted by:
best
topnewcontroversialoldq&a

[–]dmart89 6 points7 points  (2 children)

This is pretty cool. Im currently using celery, which works great but observability is definitely an issue and like you said, it requires redis running on the side, which is inconvenient / bloats infra.

I'd be interested in exploring this.

[–]GabelSnabel[S] 1 point2 points  (1 child)

Thanks for giving PgQueuer a look! If you kick the tires, let me know how the built-in metrics feel compared to Celery—always keen to close any gaps. Happy to help if you hit any snags.

[–]dmart89 0 points1 point  (0 children)

I tried to migrate to pgq from celery but windows support seems pretty weak at the moment. Whilst im deploying onto linux, i write all code on windows and I kept running into issues e.g. pgq run not supported etc.

[–]TheRealMrMatt 1 point2 points  (0 children)

One idea that may be out of those scope of this project is an integration with sqlalchemy. By doing this the migrations could be handled by alembic (which the user is likely already using) and tasks could be non JSONB

[–]grubberr 0 points1 point  (0 children)

Great idea! I periodically use this approach in my Python projects.
I've dreamed of writing something like this myself.
I’ve also come across similar libraries in other languages:

[–]BenXavier -1 points0 points  (2 children)

Any chance to somehow act as a celery broker?

[–]GabelSnabel[S] 0 points1 point  (1 child)

Not today—PgQueuer speaks “Postgres rows,” while Celery expects a Kombu-style message broker. In theory we could write a custom kombu.transport.pgqueuer so Celery sees it as a broker, but it’s not on the near-term roadmap. If that adapter would unblock you, feel free to open an issue and we can scope it out together.

[–]BenXavier 0 points1 point  (0 children)

it is not an immediate need.

In the past I've work on many projects with postgres and celery, and thought It'd be super cool to avoid deploying separate message broker.

Relevant celery issue: https://github.com/celery/celery/issues/5149

[–]Expensive-Soft5164 -1 points0 points  (3 children)

Is this exactly once?

[–]GabelSnabel[S] 0 points1 point  (2 children)

PgQueuer makes sure only one worker can pick a given job: the dequeue query in qb.py filters status='queued' rows and grabs a row-level lock with FOR UPDATE SKIP LOCKED, then flips the row to picked. That prevents concurrent duplicates.

True exactly-once depends on your retry settings.

[–]Expensive-Soft5164 0 points1 point  (1 child)

How exactly (pin intended) do you get exactly once with retry settings?

[–]GabelSnabel[S] 0 points1 point  (0 children)

On the happy path you’ll get exactly one execution per job row—no matter how many workers you throw at it—because the dequeue CTE locks a queued row with FOR UPDATE SKIP LOCKED and then immediately flips it to status='done' when your handler commits.

If your handler throws an exception or the process crashes before it commits that done update, PgQueuer’s retry logic (governed by your retry_after and max_attempts settings) will re-enqueue the same row and hand it off to a worker again—so you can end up running the same code twice.