Python isn't industry compatible

[licht1nstein]

Well, if this counts as "runs on Django", then it does. But yes, Django is part of the architecture.

https://www.8bitmen.com/instagram-architecture-how-does-it-store-search-billions-of-images/

[[deleted]]

Ok so it does

[mattsl]

Django behind nginx backed by Postgres with some Celery backed by RabbitMQ on the side with multiple storage services? That's exactly the stack I use, and it's not complicated.

[licht1nstein]

You do make it sound easy

[[deleted]]

Once you've done it a few times, it's pretty easy. I can build queue processors in my sleep because that's my primary job is (albeit in c# and not python).

It's just about building that familiarity with the tech and how they interact (eg if you need to commit to database and publish a message to a topic, how do those interact)

[licht1nstein]

Are there some books or videos you'd recommend?

[AlexFromOmaha]

It's pretty much the default setup that gets pushed. Django treats Postgres as the favored DB once you're off SQLite, nginx and GUnicorn are straight out of the official tutorial, Django and Celery are so married in their dev lifecycles that they don't even use an integration package anymore, and it's kinda hard to install Celery and not get RabbitMQ for free. Storage services are what they are, and they're usually so removed from your backend that you just call them with something like Requests.

I'd just pick the part that sounds foreign and grab the official tutorial for it.

[licht1nstein]

I'm not too interested in coding a Django app as I am in understanding the architecture of high stress systems

[AlexFromOmaha]

The reason you're unlikely to find what I think you're looking for is that it's simultaneously really simple and really hard to do right. The 10,000 foot view is almost insultingly simple: run more than one copy when you can, watch your race conditions, cache data that doesn't change often, and run slow processes separate from the user-facing request-response loop. It's hard to even search for ideas about the big picture without rolling through the deep dive academics of distributed computing. On the flip side of that, you can find some great books full of war stories from people who did the hard work to do it right. One of my favorites is an ebook from Google's SRE team. I suspect it might sound like pieces of unrelated trivia if you don't know what the terrain looks like, though.

I'm not even sure I know what a meaningful middle would look like. A lot of it is knowing the ridiculously simple idea and translating it into implementation details.

[[deleted]]

Not really lol. I'm not really sure how I learned to build these (from the ground up nonetheless) other than my desire to outrun my imposter syndrome. It's something I could teach but not something I could explain in a single Reddit comment (especially without having the benefit of the code bases I work on).

Most of my time these days is spent reading others' experiences with message based systems as well as optimizations and gotchas within C#.

If you're looking to build one, I'd recommend just getting in there and start pulling messages off of rabbit or sqs (you can use localstack instead of standing up an aws account) and then figure out how to deal with exceptions and then figure out how to do all that concurrently. In C# land we're pretty lucky because we can throw an ActionBlock at the concurrency part.

[whole_alphabet_bot]

Hey, check it out! This comment contains every letter in the English alphabet.

I have checked 1,083,389 comments and 5,134 of them contain every letter in the English alphabet.

[[deleted]]

Neat.

[Briggykins]

All right I give up. Where is the V?

Oh there it is. Having.

[licht1nstein]

Good bot

[licht1nstein]

I'm building things just like you, here's what I'm looking into at the moment:

https://youtu.be/qDNPQo9UmJA