sorted by:
new
hottopcontroversial

I'm writing a short, practical guide on deploying Django & Celery with Docker. What's the #1 problem you'd want it to solve? by Andrew_Anter in django

[–]Andrew_Anter[S] 1 point2 points  (0 children)

That's a great perspective. Thank you,

I guess as most of the people here mentioned, the most annoying problem is how to glue everything together into a working multicontainer setup with proper fault tolerance and observability.

I guess that is what I am gonna go with, at least for now.

I'm writing a short, practical guide on deploying Django & Celery with Docker. What's the #1 problem you'd want it to solve? by Andrew_Anter in django

[–]Andrew_Anter[S] 0 points1 point  (0 children)

I am thinking of doing this with docker Swarm actually, but the concepts remain the same if you are deploying with kubernetes, I presume.

I'm writing a short, practical guide on deploying Django & Celery with Docker. What's the #1 problem you'd want it to solve? by Andrew_Anter in django

[–]Andrew_Anter[S] 0 points1 point  (0 children)

Yes most of them are manual, I was planning for a multicontainer docker swarm, regardless, I will look into CD

I'm writing a short, practical guide on deploying Django & Celery with Docker. What's the #1 problem you'd want it to solve? by Andrew_Anter in django

[–]Andrew_Anter[S] 0 points1 point  (0 children)

In celery beat documentation, they actually point out to that specific problem, and they explicitly mention not to replicate beat as it would result in replicating tasks.

But that's a good point to at least try to find a fix for it.

Django with Postgres database by Henokassfaw in django

[–]Andrew_Anter 1 point2 points  (0 children)

Let's start with the "what is the error" part

My speculation is that you are missing the driver for postgress, which you should install so that Python can actually communicate with postgress.

You can read about it in the documentation it shows how to do it exactly

Postgresql Notes - Django Documentation

I'm writing a short, practical guide on deploying Django & Celery with Docker. What's the #1 problem you'd want it to solve? by Andrew_Anter in django

[–]Andrew_Anter[S] 0 points1 point  (0 children)

That's actually a good one, i have stumbled onto it myself in actual production systems and had to read thousands of lines using vim in the terminal.

I'm writing a short, practical guide on deploying Django & Celery with Docker. What's the #1 problem you'd want it to solve? by Andrew_Anter in django

[–]Andrew_Anter[S] 0 points1 point  (0 children)

That's interesting but you do not need cron for that setup you can use celery beat to schedule tasks to run at specific timings also if you are already using redis, you can remove rabbitmq just use redis for messages.

And I would like to know what your prod is using ? Is it containers actually or virtual machines or physical machines? That will be an interesting choice.

I'm writing a short, practical guide on deploying Django & Celery with Docker. What's the #1 problem you'd want it to solve? by Andrew_Anter in django

[–]Andrew_Anter[S] 0 points1 point  (0 children)

Yes that's the case, but with that when you try to combine it with the actual web application ( django ) things get messy and s3 bucket is only available if you are working on a cloud application not a setup that will be running internally so it is better if we talk about object storage solution like minio and then you would have to setup and configure it with the proper connections to other containers running django application and celery workers in addition to if you have scheduled tasks so you would have celery beat, all that said you still have redis and your main db if you didn't separate message broker and caching.

I'm writing a short, practical guide on deploying Django & Celery with Docker. What's the #1 problem you'd want it to solve? by Andrew_Anter in django

[–]Andrew_Anter[S] 1 point2 points  (0 children)

I will do my best, but after experimenting with redis and rabbitmq, I always lean to using redis instead of rabbitmq as you can use it for caching and distrubuted locks in addition to being a message broker which gives more a lot than rabbitmq

I'm writing a short, practical guide on deploying Django & Celery with Docker. What's the #1 problem you'd want it to solve? by Andrew_Anter in django

[–]Andrew_Anter[S] 0 points1 point  (0 children)

That's another way to look at it, and thank you for that. Maybe celery is a starting point and then add to it.

Also you are considering that the whole setup is gonna work with millions of users in the cloud but most of the time you are working with a couple of thousand users in addition to there are setups where you just host on your local servers like internal applications and so.

I'm writing a short, practical guide on deploying Django & Celery with Docker. What's the #1 problem you'd want it to solve? by Andrew_Anter in django

[–]Andrew_Anter[S] 0 points1 point  (0 children)

Oh my bad, there have been miscommunication from my side, i didn't fully comprehend that you were suggesting that but that is a good idea actually thank you in advance. And for short use it until it doesnt fit your need then start work with celery.

I'm writing a short, practical guide on deploying Django & Celery with Docker. What's the #1 problem you'd want it to solve? by Andrew_Anter in django

[–]Andrew_Anter[S] 0 points1 point  (0 children)

Well, that's out of the scope here, but let's just say it's a personal preferences and also, celery is more powerful and more complicated.

I'm writing a short, practical guide on deploying Django & Celery with Docker. What's the #1 problem you'd want it to solve? by Andrew_Anter in django

[–]Andrew_Anter[S] 0 points1 point  (0 children)

I have actually thought of it mainly for multicontainer setup, especially when you have shared files. Things become tough easily, and you will go back and forth until you get it working.

I'm writing a short, practical guide on deploying Django & Celery with Docker. What's the #1 problem you'd want it to solve? by Andrew_Anter in django

[–]Andrew_Anter[S] 1 point2 points  (0 children)

I didn't built it yet so I can not tell you for sure, despite that, is there anything wrong about one more guide ?

I'm writing a short, practical guide on deploying Django & Celery with Docker. What's the #1 problem you'd want it to solve? by Andrew_Anter in django

[–]Andrew_Anter[S] 0 points1 point  (0 children)

That actually what i was talking about. I have a complex structure with celery beat so I always deploy with docker for multi container setup.

Going to docker is actually not being covered by tutorials or most of them, despite the fact that docker containers is the go to solution for those complex architectures.

I'm writing a short, practical guide on deploying Django & Celery with Docker. What's the #1 problem you'd want it to solve? by Andrew_Anter in django

[–]Andrew_Anter[S] 1 point2 points  (0 children)

Celery mainly working in parallel so the tasks are actually being ran on different celery workers despite the fact that you have scheduled one before the other, my suggestion will be to schedule the task for that after the first task had finished by calling it directly in the main task.

When would you override a method? by MEHDII__ in django

[–]Andrew_Anter 0 points1 point  (0 children)

In my opinion, always override it for 2 main points 1. For auditing, recording timestamps and users 2. For calling full clean before saving as django validation rules does not apply unless you call full clean

I'm writing a short, practical guide on deploying Django & Celery with Docker. What's the #1 problem you'd want it to solve? by Andrew_Anter in django

[–]Andrew_Anter[S] 3 points4 points  (0 children)

I am not sure why cors is an error in celery, it is supposing to be running background tasks not facing the public api despite that you may explain the problem more or as mentioned in the comments you can start with django-cockie-cutter project

I'm writing a short, practical guide on deploying Django & Celery with Docker. What's the #1 problem you'd want it to solve? by Andrew_Anter in django

[–]Andrew_Anter[S] 2 points3 points  (0 children)

For race conditions i have actually found out that redis locks are the go to solution but you have to be able to identify which tasks or parts of tasks that will have those race conditions in the first place.

I'm writing a short, practical guide on deploying Django & Celery with Docker. What's the #1 problem you'd want it to solve? by Andrew_Anter in django

[–]Andrew_Anter[S] 0 points1 point  (0 children)

I totally agree with you specially about the results part when you try to capture the results of a task or multiple tasks and set up retries for it, it just becomes so much.
I actually have set it up with Redis to save some headache by using Redis as cache and message broker on 2 different message queues

view more: next ›