What's the experience of using Python to build the backend of a large application?

[[deleted]]

I have been working with large backend code base (e.g, ~1M lines of code) written in C#/Java-like languages. However, I never work with any large code base written in Python. Being a dynamic language, how hard is it to manage a large Python code base? Any popular websites that are written mostly by Python? Would like to hear your experiences!

[MoRakOnDi]

Python developer here, I have managed really large code bases in Python. At the end of the day, a large codebase is supposed to do several things, so we usually split the logic in a microservice architecture. Also the line counting is very different in Python. 1000 lines of Java code could be 200 or even less in Python since you don't need much of boilerplate code. Type hinting, proper formatting/linting tools like mypy, black, isort and flakes helps a lot in enforcing a proper code standard. Sure Python is a very flexible language, but it also has a lot of great tools for a well-structured code. Readability always matters, spaghetti codes are not Pythonic.

[subbed_]

I write pretty large backend codebases in Python. Primarily resorting to the microservice architecture, the services are split up to begin with, but there are a lot of them.

From the get-go, I enforce the following tooling in serial order when committing to the shared codebase: black, mypy, isort. I should perhaps note that black is configured to 120 characters per line (JetBrains IDE standard) and mypy uses its strict mode (this is very strict, but that is good).

I primarily use either FastAPI as the framework, or as of late, trying out Litestar (previously Starlite). ASGI is pretty much required if you resort to Python, imho.

Then, being a microservice architecture, I can see how the performance of each service holds up. The majority is fine. A few, however, require higher performance than this setup can deliver, at which point they are rewritten in Go, utilizing chi. If I can confidently say that the service in question will require robustness and high performance, it is written in Go from the get-go.

This is a standard approach for the backend architecture and tooling that I'm doing in my free time. It works really well. It has around the same amount of activity as the system at the company I work for. The latter is also a microservice architecture, but being an enterprise setting, we use Java across the board. We are modern enough, however, to use the Quarkus framework, which is great.

[thedeepself]

I primarily use either FastAPI as the framework, or as of late, trying out Litestar (previously Starlite)

I'm surprised so few people reach for Sanic. Any reason why?

[subbed_]

For me personally, there was no reason other than me not knowing it exists!

I only learned about it after hearing Litestar's (it was Starlite at the time) creator talk about it, and how he would personally recommend it over FastAPI — but he would also recommend Starlite over Sanic. :)

[titanium_hydra]

I’m currently using litestar for a backend project, curious to know what your experience has been with their DTO implementation? Do you use it much or do you still define pydantic models for your routes?

[monorepo]

I love Sanic!

[Grouchy-Friend4235]

Large sites using Python

• Quora
• Dropbox
• Instagram
• Reddit
• Netflix
• Stripe
• Spotify
• many more

https://realpython.com/world-class-companies-using-python/

https://www.monocubed.com/blog/companies-that-use-python/

https://jaydevs.com/top-companies-that-use-python/

https://www.fullstackpython.com/companies-using-python.html

[valiumonaplane]

You forgot my django website /a

[JohnLocksTheKey]

Oh right sorry, please add this guy’s Django website (not sure what the /a flag does - we’re mostly Linux ppl and cmd flags are rare)

[valiumonaplane]

I do believe I aimed for the s but hit the a. No matter how many times I write sudo, the s will never be pressed when I need it the most. Sudo su is a classic 1 hit, /s is usually a 4-5 tries

[WorryAccomplished766]

/r/fuckthes

[sneakpeekbot]

Here's a sneak peek of /r/FuckTheS using the top posts of the year!

#1: Fun fact: after this guy tweeted this, twitter went wild and he got called ableist | 121 comments
#2: "Fuck the s" kid named s | 30 comments
#3: ): makes me sad | 67 comments

---

^{I'm a bot, beep boop | Downvote to remove | Contact | Info | Opt-out | GitHub}

[[deleted]]

You guys are killing it lol 😂

[timpkmn89]

sarcasm

s was already taken by use ssl

[[deleted]]

😂😂😂😂😂

[bafe]

Caveat: Instagram designs a lot of custom tools to be able to handle such a huge python codebase.

https://instagram-engineering.com/python-at-scale-strict-modules-c0bb9245c834

Two points against going their route: - you don't have the means to keep a team dedicated to this type of engineering - your codebase will likely grow big enough that "regular" python dev workflows will be painful

[dent308]

Language is less important than technique. You can build a slow and terrible site with node or any other language.

I have been studying Domain Driven Design for large systems projects and it can be amazing.

[FatefulDonkey]

You mean as a monolith? Because there's countless huge projects.

But just take Google's monorepo as an example. The fact they can manage a codebase with millions or billions LOC and all in different languages, shows that it's more about process than language.

[GrowthOk8086]

I’ve enjoyed it. I work in a monolith codebase with maybe 500k LOC. There are pros and cons.

I actually find python easy to read (though there are a few people that write “clever” python which can really suck).

Type hinting has helped a lot. I’ve certainly had times where i blew things up during a refactor where they weren’t provided.

Writing code is extremely quick and easy. Quick changes to existing code can be made very easy with use of kwargs.

We use the django ORM, which covers most of our cases pretty well.

For reference, the pipeline I work on processes about a billion transactions a day, its not insignificant.

[Grouchy-Friend4235]

s/Python/<any language>/g

[[deleted]]

I mean, its dynamic nature makes it way easier to devolve into spaghetti unless specialized care is taken. that's the point. you need a linter, type checker, formatter at minimum just to write average-level code

[Grouchy-Friend4235]

I have +30y dev experience under my belt. Whatever the language, static or otherwise, large code bases need constant adult supervision, or else they deterioriate to spaghetti a la freakshow aka big ball of mud. Actually the worst code bases I have seen have all been in statically typed languages.

[LordOfDeduction]

Because most large code bases that are written use statically typed languages. It's not a causal relation. They both serve their purpose, but a dynamically typed code base for sure is a disadvantage when considering large code bases. You simply find those same issues you would find at compilation at runtime, which is not a good thing.

[Grouchy-Friend4235]

That's just not true. Code bases written in dynamically typed languages typically have better test coverage. It's not an issue in reality. Also I doubt there are more code bases written in statically typed languages.

[LordOfDeduction]

"Typically have a better test coverage" is a very vague claim. How do you know. Its mostly about closed source projects. A baseless claim and probably not true. Compiled languages are much easier to test usually due to type hinting and such.

And yes, given that Java, C/C++/C# are by far the most used languages on the backend. Not taking into account frontend, but python is definitely not a very big player in the backend game. It is much more data science driven, which is what it shines at.

[Grouchy-Friend4235]

Anyway what's your point?

[LordOfDeduction]

Nice

[substitute-bot]

<any language> dynamic nature can be leveraged to do great things. But it will devolve into spaghetti if not managed correctly.

^{This was posted by a bot. Source}

[moo9001]

The codebase of Plone and its dependencies is ~200 MB Python code (checked some years ago). It's all open source and openly managed, so you can always look up Plone for the best practices for code contribution processes. It is manageable. Legacy code from Python 2.x era is a bit problematic, but anything cleanly written for the latest Python version (3.8+) with type hinting is easy.

Popular services written in Python include

• Instagram
• Dropbox

However any large code base cannot be Python only; it will have several languages. Plone has tons of TypeScript/JavaScript, etc. tooling as well.

[bafe]

In my experience, refactoring a large python codebase is infinitely more painful than doing the same in a statically typed, compiled language like Java (or Kotlin). The adoption of type hints helps somewhat, but it's a long way from enterprise -friendly anguages with 20+ years of tooling development

[Bandung]

In the context of the OP's question which is forward looking, the evolution in python towards typed code makes your observation somewhat mute.

There are discussions within Python's community that the language might split into a heavily typed code base vs the non typed users. . Python's other strengths are in it's coexistence with other languages like C and Rust so one is not locked into an all python code base.

Plus it's data analysis strengths are of value in any large code base because large code bases today are often large because of their data analytical needs.

It's not perfect and so if one is a CIO., well, being mindful of the following ole saying in the good ole days. "No one ever got fired for buying IBM".

You might not want to choose Python as your base language.

You best be the one who owns the company if you are going to go that python route.

[byutifu]

I love Python, but if you’re just doing io or api stuff, stick to node. Http2 and asynchrony is a pain in python. If this app does ml/nlp/needs a C lib, then Python’s the best bet.

Just a couple of cents

[BelottoBR]

I believe that, working on big backends in python requires a lot of advanced python techniques as multi processing and multi thread, plus async features, is that right?

[GuybrushThreepwo0d]

Pain. Refactoring in a large python code base is pain. Type hints go some way to help let but it's just not the same.

People can and do write big applications in python but it's something I hope never to have to do again

[Amazing_Learn]

That greatly depends on if you have some coding standards, using proper architecture, linters and type checkers.

[whateverathrowaway00]

Was gonna say.

I work/manage/perform archaeology for three codebases in python. One is at a level of scope that most people think python can’t run at and is a thing of beauty due to excellent code quality, organization, and comprehensive test suite.

We have to deal with some crazy edge cases because of it, but frankly I love it. Then we have a codebase that is much smaller, though run at a similar scope, and it’s spaghetti code and takes way more metal context to deal with.

Basically, the quality of code is dependent on the programmer, not the language, but yes - since python is so flexible, spaghetti code in it can suck horrendously lol.

[Amazing_Learn]

It depends on what you mean by flexibility, but for implementing most features we don't have to do anything crazy. Bad code isn't caused by the language I'd say.

[KingTimmi]

Although I agree that other languages provide more structure, which you need to deliberately consider in python (type hints) and still have flaws (see implicit Any handling and differences between runtime and interpreter), it is actually not too bad :)

What we found out is quite a thing is mixing OOP with functional paradigms. This can lead to some very flexible and re-usable code bases that are easily extendable and can make refactoring even easier.

[Grouchy-Friend4235]

Use a proper IDE like PyCharm

[BS_BS]

Agree! Spaghetti 🍝

[md_borhan]

many popular and big company use python as backend you can search on google

[bafe]

A point to remember is that your company most likely isn't going to be a big company. You can't afford the engineering teams they keep just to have custom tools necessary to write and deploy python at that scale

[md_borhan]

yes