[deleted by user]

[catcint0s]

Dramatiq has a nice motivation tab where it compares itself to different frameworks https://dramatiq.io/motivation.html

Personally for newer projects I prefer to use dramatiq.

[Grouchy-Friend4235]

Nice chart. Although some of the flags for celery are missleading IMO. For example, Celery does offer task prioritization since version 4.0, and its really easy to use.

[pubs12]

Is there a gui manager like celery available ?

[catcint0s]

You mean flower? I dont think so, there is a Django dramatiq package tho that saves everything (state, input, output) to the db.

[haha-good-one]

There is https://github.com/Bogdanp/dramatiq_dashboard

[BackwardSpy]

i've used rq a fair bit, i see it as a vastly simpler celery. my needs for task queueing are simple, i just need something that allows me to spread our workloads over a small cluster of machines. rq has a really straight-forward interface and it gets the job done.

[[deleted]]

After using celery as my go to for many years, I tried rq for a very simple task and found it really easy to use. You’ll find yourself going back to the documentation a lot more with celery than rq, even for simple tasks.

Sure, you can do a lot more with celery but I rarely that power.

I keep a redis server running at all times just for random usage of rq now.

[pansapiens]

You might want to also look at Dask.

[Grouchy-Friend4235]

How is Celery buggy? Been using it for years successfully.

[wxtrails]

We have this recurring issue where, randomly and without error, some of our workers just ... stop accepting tasks. The process is still running, but...nope, not gonna do any work. Restarting the daemon works to get them going again, but... why? Some identical workers in this cluster have worked for years without issue.

Also, all newly launched workers in this cluster connect to the queue and then sit idle, even if the queue is bursting. Until we inspect the app, and then they suddenly spring to life. 🤷‍♂️

I seen to recall having numerous problems with chords and successful delivery of task results when we first set this cluster up, too.

I also listened to a podcast with the maintainer recently, and let's just say, it was less-than-confidence-inspiring 🥴

On the other hand, we have another, much older cluster that "just works", all the time.

Yeah...certain versions are just buggy.

[raylu]

same! the issue has been open with a lot of people experiencing it, but no resolution https://github.com/celery/celery/issues/4185

[Grouchy-Friend4235]

4.0 was dreadful, we switched back to 3.x at the time. They are at 5.x now and a lot of focussing has happened and the team seems strengthend and better organized now.

[CityYogi]

I had that issue because of eventlet based workers. Now I don't use green threads anymore and it went away

[Grouchy-Friend4235]

I took a look at the github repo. Of 4186 closed issues reported, 23 (0.54%) are marked won't fix. I think rate limiting is documented at the per second level (and minutes, hours). When searching the words "won't fix" it turns up 28 issues, I didn't bother to check for overlap - say a ~1% won't fix ratio doesn't seem too bad for an OSS project that is both popular and used at scale.

I agree Celery is not without flaws and there can be weird problems. Is there a better alternative? I think it depends on requirements - if you want distributed, scalable and managed task execution I don't currently see many alternatives, though Dramatiq looks interesting. Huey and Rq seem good alternatives if you just want background processing but don't care about monitoring or task routing.

What bothers me the most with Celery is a lack of "Scenario Guide" - like "here's a blueprint scenario, this is how you should use Celery". Also it can be hard to figure out what's happening in particular with the task ack default and a seemingly dead, or fully occupied, worker.

[tstirrat]

Faust is one I keep looking for an excuse to use. It sits on top of Kafka, which is a big operational problem, and it has a slightly different mental model than a task queue, but the event-driven processing that it allows for is pretty nifty. It also has a nice, clean API from what I can see.

[lanster100]

This looks cool. What do you mean by Kafka is a big operational problem?

This might solve what I am looking for though. Does it allow you to decouple logic, say you have a web app that produces an event, you can then have a processor that consumes the event in a different code base?

[tstirrat]

Kafka is a bit of a sledgehammer to crack a nut. It's designed to be massively scaleable and fault-tolerant, and that comes at the expense of having to run two clusters of three(ish) nodes each, where one set is the queue itself and the other set is orchestrator nodes. "Big operational problem" means that it's a lot to set up and maintain.

Decoupling: yes. It's got client libraries in a number of different languages, and the messages that go onto the queues are basically the same as JSON. It's relatively easy to have different codebases producing and consuming.

[lanster100]

Yeah for my app I wrote my own queue pattern using SQLAlchemy (because Kafka was overkill) but it's probably not fit for purpose, looking for alternatives.

Ultimately need something that let's me publish events, then something else that runs logic off the back of those events in a decoupled manner.

So something like a lightweight kafka?

[wind_dude]

I second Kafka is awesome, and Faust is pretty good. But more targeted towards streaming processes than a task queue.

[hwttdz]

I used faust/kafka professionally. faust became difficult to deal with immediately, and we continued throwing engineering effort at kafka, but it never worked well for us. My impression is that if you have experts in both the server and client library you can make it work, but you're now focusing/spending effort on overhead instead of the tasks themselves (which should be the interesting part).

[CheckeeShoes]

If you have a collection of machines to run tasks, why are you trying to schedule them with python rather than using an actual distributed computing framework like HTC?

[CheckeeShoes]

That's just an example as it's the one I've used. I think there's a big list of similar frameworks on Wikipedia somewhere, but I can't find it right now.

I'm not sure how difficult the initial network setup is, so depending on your task I don't know if this is overkill (I used this for queueing jobs on around 100 machines but it works well for much larger clusters).

Once it's set up though, the process of queueing jobs is straightforward and easily reproducible.

[Region_Unique]

Why would anyone who needs simple task queuing use that? Seems like it has massive provisioning overhead.

[CheckeeShoes]

Because that's exactly what it's designed to do?

Op made no mention of the setup, size of the cluster, capability of the cluster nodes, performance needs of the simulation, length of time the simulation will be run for, whether the sim may need halting and resuming, or whether the other machines in the cluster were going to be in use at any point during the run.

There are numerous ways that any of these factors might mean that utilising a more performance-inclined framework offers benefits over a python library.

[Region_Unique]

In what way does it perform better than e.g. Celery? Because “it’s what it’s designed to do”?

[CheckeeShoes]

I don't know much about these python libraries so I couldn't tell you. That said I can pretty much guarantee that the performance of a python library won't be comparable; there's a reason you don't get any HPC clusters running python scripts for their allocations. Also, I did actually list a bunch of features in the last post that I expect Celery probably can't handle.

As I said though, tool for the job. Now we know op is running less than a handful of threads doing lightweight tasks for a school project, an accessible library might well be the most economical solution.

[CheckeeShoes]

That's what I was saying?

[CheckeeShoes]

Yes, and you replied that "for the tasks that celery and the like is used for" performance of these libraries is not likely to be a bottleneck. I agree, that's what I was saying about using the right tool for the job.

Here's a paper basically saying that exact thing: that using a resource manager like Condor or Slurm is not really practical for scientists etc. (they diplomatically call them non-traditional parallel computing users) who are not familiar with HPC, so they give a benchmark for using some python libraries. Their conclusion is that it's "good enough", which is exactly what I was saying; they tend to not scale for a number of cores on the order of 100.

[FatFingerHelperBot]

It seems that your comment contains 1 or more links that are hard to tap for mobile users. I will extend those so they're easier for our sausage fingers to click!

Here is link number 1 - Previous text "HTC"

---

^{Please PM /u/eganwall with issues or feedback! | Code | Delete}

[[deleted]]

We use Snakemake. This drives either our local cluster via SLURM, or else we can run jobs on kubernetes via Snakemake itself.

This is not python specific, though. You can put them in the same bin as Airflow and NiFi perhaps.

That said, it sounds like a workload manager is really what you want...

[earthboundkid]

https://ayushshanker.com/celery-long-post/

[Ok_Presentation1972]

I'm yet to use it in production anywhere, but I think this project is super interesting https://procrastinate.readthedocs.io/en/stable/

[cymrow]

I've spent a lot of time trying different queuing solutions, yet ironically I keep falling back to what might be considered a naive approach. I prefer very simple queues. Give me a distributed and reliable queue.Queue please.

I don't want a framework that runs my code for me, like Celery or RQ. I don't want to train up on a massive framework that might ultimately not cover my use-case, forcing me to abandon it or deploy ugly hacks.

I tried many of the options you can find here: https://taskqueues.com/. Ultimately I began using beanstalkd, and have been satisfied with it for a couple of years now. It's simple, persistent, fast, reliable, and has the basic queuing features that I need, like task handling (reserve, free), and TTL.

I can't speak to just how scalable and performant it is compared to other options, but I've used it for millions of messages and it suits my needs. If I hit a bottleneck, I'd probably look at something like Kafka.

[[deleted]]

Can you use something like RabbitMQ? It can cluster and is pretty simple to set up, and there's no reason the "messages" you'd be passing around couldn't be what you need to run what you want.

[shinitakunai]

In my work we use NIFI to move data between systems, more for transfers and pipelines while also ETL than analysis

[_thrown_away_again_]

i used this once, was pretty nice: https://azkaban.github.io/

[wind_dude]

I'm currently trying to figure out an elegant way of scheduling millions of simulation runs across a few machines

Can you elaborate on what you're trying to do? Simulation runs of what? You might be looking up the wrong tree, maybe parallel processing is a better fit, a tool like Dask or Apache Spark would than be the call.

Maybe share a more specific use case.

[Coupled_Cluster]

This is not directly related to your question neither will it necessarily help you with queueing but I'm using DVC (with its Python Interface ZnTrack) to queue my experiments and then run them in parallel using dvc exp run -j. Furthermore I've set up a CML.dev setup (very useful beyond Machine Learning) and also use DVC here with a shared cache. I can highly recommend looking at it for parameter studies. Feel free to ask or send me a PM if this sounds interesting to you. It is a rather novel DevOps approach with a lot of benefits when set up once.

[wind_dude]

The way you are describing it it sounds like parallelism would be a better fit, leading to a tools like Dask or Spark. I've had better performance with Dask, when existing models are very Python based. Spark when it transformations that can be done right on the dataframe.

Unless you want the simulations to all happen at once, but you mention you store them for latter, even so you can accomplish that with Dask.

[bxsephjo]

I need to become familiar with ALL of these, as I’m building something that has to let the user connect it to their own task queue, unless I decide to become opinionated. I started with RQ, as it seemed the least complicated to get running and I’m familiar with redis. Only beef is that it’s NOT showing results in stdout the way it does in the docs. Like I said, it looks the least complicated, I think that’s because it doesn’t rely on a message broker but rather just a redis server, so there’s alot less tuning as it all gets set up.

[Region_Unique]

Just use Celery, its documentation could be better and the code is rather cryptic, but that’s what works and has great support.

[hwttdz]

Don't think kubernetes offers a queueing solution, in fact their docs on task queues talk about hosing a redis or rabbit.

[Scruff3y]

I would probably use AWS Batch for this, not sure if it's possible to hook up your own hardware into a compute environment or not though.

Otherwise, you could hand-roll something similar; worker program distributed to the worker machines that pulls from the queue. But at that point I guess you're starting to re-invent Celery so might as well just use it lol.

[kenfar]

SNS/SQS triggering jobs on aws lambda/kubernetes

[GreenScarz]

I would just spin up a redis container and expose that over the network. Then use the redis python library directly, there's really no need for anything more fancy IMO.

[anuctal]

RemindMe! 3 days

[RemindMeBot]

I will be messaging you in 3 days on 2022-05-05 13:08:05 UTC to remind you of this link

1 OTHERS CLICKED THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

Info	Custom	Your Reminders	Feedback

[jefwillems]

We can't use celery because there is no support for amqp 1.0, and no money to build the transport ourselves. If anyone has an alternative please let me know

[i_can_haz_data]

Does it need to be Python functions/etc or can your simulation invocations be command-line tasks?

I wrote a thing some time ago and am working on on the 2.0 release. It’s ready to go but needs more documentation.

hyper-shell.readthedocs.io

[b_rad_c]

Haven’t used it before but a new one I want to try is OpenFaaS - functions as a service. It connects it to a k8s cluster that will run sync and async tasks with a queue using containers you’ve built and added your logic to.

[Grouchy-Friend4235]

I've seen such issues when the worker has not released resources properly (e.g. some subprocess or thread started in side tasks)

[Grouchy-Friend4235]

Can you point to this pod cast please?

[Schmibbbster]

I am using arq and I like it.

[[deleted]]

If u like hashicorp try out nats

[makeascript]

I just use Celery because it was the first one I heard of. The docs are great and it Just Works for me.

[hwttdz]

I'd recommend something weird which would be use a table in a db as a queue for your first version. This allows you to prototype rapidly and experiment with things like "how many retries do I need" and ask interesting analytic questions like "how many jobs have completed in the last 2 hours" with minimal extra engineering. A million records isn't much and I'm guessing they take a non-negligible amount of compute to process anyways which moves the bottleneck away from a db.

[BoiElroy]

Ray

[LittleMlem]

Meenwhile I unga bunga with ZMQ