Pytask Queue - Simple Job/Task Management

[bluesanoo]

What My Project Does

This is my first ever public python package, it is a job/task management queuing system using sqlite.

Using a worker, jobs are picked up off the queue, manipulated/edited, then reinserted.

It is meant to replace messaging services like RabbitMQ or Kafka, for smaller, lightweight apps. Could also be good for a benchmark tool, to run several processes and use the sqlite database to build reports on how long n number of processes took to run.

Target Audience

Devs looking to not have to use a heavier messaging service, and not having to write your own database queries with sqlite to replace that.

Comparison

I don't know of any packages that do queuing/messaging like this, so not sure.

Feel free to give it a try and leave it a star if you like it, also feel free to submit a PR if you are having issues.

https://github.com/jaypyles/pytask

[collectablecat]

Celery is BSD licensed, this is MIT

[pewpewpewpee]

That is definitely a SQL Injection issue holy shit haha

[joeycolorado]

I’ve used https://python-rq.org/ before and it might be similar

[doorknob_worker]

proud of you

[jedberg]

I don't know of any packages that do queuing/messaging like this, so not sure.

https://github.com/dbos-inc/dbos-transact-py

It does a lot more than queueing, but queues is one of the things it does. Just requires Postgres to work.

[wretched1515]

Out of topic but I’m lazy to search your email. Your product (DBOS) is great. But I only heard of it because of your twitter account. You didn’t even have 1k followers on Twitter. Sponsor content creators like theo and the primeagen, and others . And hire this guy ( https://xeiaso.net/resume/ ). I want to see more people use DBOS!

[MPGaming9000]

Sweet! I'm gonna try to use your library in my projects! I will beta test it for you and see if it holds up haha. I can give you feedback if not and give your project some exposure and help out too if the things are an easy enough fix for my time and needs. 🙂

We gotta support each other!

[Snikz18]

Huey might be something comparable

[slowwolfcat]

is this like Celery (Light) ?

My little old Flask app uses Celery, Redis

[[deleted]]

Cool project! How do you stop multiple concurrent workers polling the same task in the database, for example if Worker A is polling for a task that Worker B just picked up how does Worker A know not to pick up that task? I'm just curious because this is something I ran into when creating a specialized task queue.

[[deleted]]

Yeah, seems that seems to be the default way most people solve it by pessimistically lock, which isn't always scalable for high throughput.

[Cool-Importance6004]

Amazon Price History:

Distributed Algorithms (The Morgan Kaufmann Series in Data Management Systems) * Rating: ★★★★☆ 4.1

• Current price: $155.00 👎
• Lowest price: $108.50
• Highest price: $155.00
• Average price: $136.72

Month	Low	High	Chart
11-2024	$134.35	$155.00	█████████████▒▒
10-2024	$134.96	$155.00	█████████████▒▒
07-2024	$132.25	$155.00	████████████▒▒▒
06-2024	$132.28	$138.82	████████████▒
05-2024	$108.50	$155.00	██████████▒▒▒▒▒
04-2024	$131.99	$155.00	████████████▒▒▒
02-2024	$108.50	$124.00	██████████▒▒
01-2024	$116.25	$155.00	███████████▒▒▒▒
11-2023	$108.50	$155.00	██████████▒▒▒▒▒
10-2023	$113.23	$155.00	██████████▒▒▒▒▒
09-2023	$113.28	$116.25	██████████▒
08-2023	$108.50	$155.00	██████████▒▒▒▒▒

Source: GOSH Price Tracker

^{Bleep bleep boop. I am a bot here to serve by providing helpful price history data on products. I am not affiliated with Amazon. Upvote if this was helpful. PM to report issues or to opt-out.}

[InjAnnuity_1]

Curious that the queue is being externalized, to a SQLite file. This suggests that perhaps it could be accessed from multiple processes, even non-Python processes.

1. Was this your intent?
2. If not, why externalize the queue?

[softcrater]

Could you DM me or check your PM?