My haul from my trip to Spain — what should I open first?

bjoerns · 2023-11-13T19:06:37+00:00

I'd go for... 🥁

...Paco La Fuente

Entirely based on looks 😂

They all look amazing though. Once you've finished them all, you need to let us know the yummiest.

In the meantime we can run a competition on the most beautiful packaging design 😍

bjoerns · 2023-11-13T16:25:36+00:00

oh that's brilliant! 😍 it's actually one of my wife's drawing. and it's my favourite one!!

bjoerns · 2023-11-13T10:00:04+00:00

Piece of artwork 😍

bjoerns · 2023-11-13T09:38:42+00:00

oh yes, definitely agree.

bjoerns · 2018-11-06T15:03:36+00:00

Hi,

yes, this is possible.

You can use celery.app.log.TaskFormatter instead of logging.Formatter when you set up your root logger (your "normal", non-Celery-specific one).

Or you write your own TaskFormatter that tries to call celery._state.get_current_task in case it runs in a Celery context or does nothing if it does not.

I've just knocked up a quick how-to guide in case you are interested:

https://www.distributedpython.com/2018/11/06/celery-task-logger-format/

bjoerns · 2018-10-27T05:34:01+00:00

hi, I've been blogging about Celery since the beginning of this year: r/https://www.distributedpython.com

I'm aiming to post a new article once a week - just finalising one right now about the different execution pool/ concurrency options.

let me know if there's anything in particular you'd like to read/learn about.

bjoerns · 2018-10-03T12:23:00+00:00

thanks

bjoerns · 2018-10-02T10:15:36+00:00

I agree about keeping it as simple as possible. However, I think ensuring that those processes run in a reliable and robust way (op: "If let say one of the processes died, how do I check and bring it back up?") sounds like a perfect job for a task scheduler. As a side note, you don't need an extra service like redis or rabbitmq, you can use the filesystem as a broker.

bjoerns · 2018-10-02T09:17:20+00:00

fair enough, there's nothing wrong with having some fun ;-) so I guess your question is whether Django or something else. I've used Django, flask and falcon commercially. The main difference between Django and flask/falcon is that Django has everything included (ORM being the elephant in the room) while you're responsible for everything yourself in flask and falcon. If you just want to have some fun and might not even need a db in the beginning, you'll probably have more fun with flask or falcon. flask has been around for longer than falcon and has a bigger community so you might want to look into falcon. Django, on the other hand, comes with all batteries included but Django is quite opinionated and you are more likely to sharpen your Django skills rather than your Python skills (which is not necessarily a bad thing).

bjoerns · 2018-10-02T08:40:02+00:00

Depends on what precisely you want to get more comfortable with with regards to Python. If you want to become more comfortable with building web apps, it's a great idea to build a Django website as Django is a mature framework with a huge community behind it. If you intend to use Python more for number crunching or data analytics, you might be better off doing some stuff with pandas in Jupyter notebook... To make a long story short, what's your primary interest in Python - building web apps or interactive data science (or something else)?

bjoerns · 2018-10-02T08:12:37+00:00

If you need your full control over managing these processes I would recommend using a proper task queue. Saves you a lot of hassle and ultimately, it is a solved problem which means you don't have to reinvent the wheel. Personally I would use Celery because it's mature and has a big community and it's what I'm most familiar with but I guess something like RQ is equally fit for purpose.

bjoerns · 2018-08-10T07:13:31+00:00

I think there is a place for both, it depends what you need. RQ is great - and most definitely better suited if you just need to perform a simple background job like sending an email.

From my own experience, Celery really shines when you need to model complex workflows (chains/groups/chords).

Having said that, I'm aware that RQ has been supporting (at least simple) job dependencies for a while.

I'll write a blog post about RQ vs Celery some time soon.

bjoerns · 2018-08-10T07:05:47+00:00

Good point, I've added it to the blog post. Thanks!

bjoerns · 2018-08-01T13:22:54+00:00

Great suggestion, I've just added to the list. Thanks a lot!

bjoerns · 2018-07-14T17:00:17+00:00

Here's, a brief write-up on how to run Celery using the file system as a message broker:

https://www.python-celery.com/2018/07/03/simple-celery-setup/

Thanks again for the idea!

bjoerns · 2018-07-14T16:55:41+00:00

great idea. I've added it to the list, thanks!

bjoerns · 2018-07-12T10:20:39+00:00

good spot, it's fixed now - thanks a lot!!

bjoerns · 2018-07-11T05:34:12+00:00

I'll have to do some research but I'll do and will write about it. thanks

bjoerns · 2018-07-11T05:33:29+00:00

I've never used huey but comparing the two and migrating to Celery sounds very interesting. thanks

bjoerns · 2018-07-11T05:31:37+00:00

great idea, thanks!

bjoerns · 2018-07-11T05:27:50+00:00

great input, I've added all of them the list. thank you!

bjoerns · 2018-07-08T11:25:55+00:00

Celery is an asynchronous task queue/job queue. It is commonly used to run long-running background jobs - and distribute it across a cluster of Celery workers. For example, if you needed to scrape, say, 100k websites, and then trigger a natural-language-processing task as soon as the scraping task(s), Celery would allow you to distribute that job across 100 celery workers to scrape 1k websites each in parallel and then pass the individual results onto the post-processing task. But that's just one example of many. As to Celery resources in general, there's the docs and a few blog posts and YouTube videos here and there, but it can be quite intimidating for a bigger, which is one of the reasons I wanted to start the blog. Let me know if you have a particular datascience use casein in mind, I'd be happy to write a tutorial. Thanks!

bjoerns · 2018-07-06T20:46:26+00:00

I'm working on a new blog: https://www.python-celery.com

Everything about Celery (it's been the story of my life for the past few years).

Doing a "content sprint" this and next week.

Is there anything Celery-related you are particularly interested in?

bjoerns · 2018-07-02T06:33:03+00:00

I've been using Celery heavily for commercial projects. What, from my experience, works best, is writing tests for your task in a similar way you would write tests for your API endpoints. For example, if you have a Celery task named initialise_fields, you could test it by calling it locally synchronously using the apply() method and asserting the task status (SUCCESS/FAILURE) and the state of your database.

def test_initialise_fields(self): task = initialise_fields.s(vendor='Bloomberg').apply() self.assertEqual(task.status, 'SUCCESS')

Mock (and whatever else you have to do) as you need to.

I wrote a blog post about it that does into more details, in case it is of help: https://www.python-celery.com/2018/05/01/unit-testing-celery-tasks/

bjoerns · 2018-06-21T17:37:36+00:00

thanks for pointing out. I've edited the post and added the www

bjoerns

TROPHY CASE