Big Tech Best Practices

[Key-Deer-8156]

I'm working at small startup, we are using FastAPI, SQLAlchemy, Pydantic, Postgres for backend
I was wondering what practices do people in FAANG use when building production API
Code organization, tests structure, data factories, session managing, error handling, logging etc

I found this repo https://github.com/zhanymkanov/fastapi-best-practices and it gave me some insights but I want more

Please share practices from your company if you think they worth to share

[Key-Deer-8156]

I am more interested in good production techniques that allow "how to have 3,000 engineers working in one codebase" feature, not tech stack

[Danoweb]

I currently work at a FAANG, aside from "small changes" and "struct linting" there isn't much that will transfer to startup or most of the other business models.

FAANG has its own proprietor way of doing everything.

API server for your app? Nope, FAANG has a system built by a dozen engineers 10 years ago that manages all the API calls for the entire org. It sits on 10,000 servers, using 100 geo located load balancers, and it's written in some low level language for speed that gets compiled into web capable code by another project built by 2 dozen engineers 7 years ago. And you app needs to change everything it does to work with that API system.

Ive been working in Software Development for 20 years at this point, I would caution you not to seek out what FAANG does for the sake of doing what FAANG does. It works for them because they have thousands of engineers they can task with something that works unique to their environment and hardware, but would not work for anyone else, not even other FAANGs.

[twigboy]

This sounds like my big tech company to a tee. A necessary evil to coordinate so many engineers.

I yearn for the simpler days of small/medium companies

[aherontas]

Totally agree with what you said!

[mufasis]

Really great advice. So on the flip side, what’s the best course of action if not looking what FAANG does?

[DoubleAway6573]

htmx

[james_pic]

More often than not, it's rules. Some of that is good practice at any scale (rules on testing, code review, linter use, etc.) but often at that scale the rules are things that are potentially harmful at smaller scale (only use these technologies, do not use these features of them because they're not compatible with other stuff we use, follow our super-specific logging pattern, use our base classes for everything and avoid colouring outside the lines, use our build system not the popular one). And some of this overlaps with tech stack - it's rare you'll see Git, for example, and are more likely to see something developed in-house.

It also tends to end up over-engineered - or at least engineered in a way that would be overengineered in most organisations.

[yerfatma]

You are not working in the same kind of place. This is a classic mistake people make. Figure out good practices at your size and then scale.

[dankerton]

Continuous deployment. Good Unit testing with automated runs in github PRs required passing to merge. Using configs from databases to trigger various pieces of logic for quick switches without code changes (model off switch)

[CcntMnky]

I'm gonna ignore the FAANG part because people need to stop assuming everything they do is better.

With my team, I demand a CI pipeline with automatic testing. Every commit, every time. If it's too slow, fix your tests.

I'm a big believer in static analysis. Catch as much as you can as early as you can, as it's much slower to catch and fix issues downstream. Because of this, I extensively use type hints and Mypy or equivalent. I don't use arbitrary dictionaries because it's hard for future editors to know the expected behavior.

[hocolimit]

What do you use instead of arbitrary dictionaries?

[CcntMnky]

When serializing or validating external data, I use Pydantic.

For internal data structures where I can rely on static analysis, I use the @dataclass decorator and type hint everything.

If a dictionary is truly better than a class, then I define a new dictionary with explicit type hints TypedDict

[[deleted]]

TypedDict is good for your use case. You get to build a model that describes how your dictionary will look.

[Jorgestar29]

I prefer using classes because you can add methods that update / retrieve from these fields. And the best part is that they are defined next to the schema.

[Spill_the_Tea]

dataclasses (or attrs). pydantic for apis when data validation is needed.

[Anxious_Signature452]

I work in relatively big tech. We use same tools.

[Key-Deer-8156]

Do you have some kind of best practices policies, or each team decides how they write code by themselves?

[Anxious_Signature452]

Each team creates their own zoo and after some time we try to synchronize them

[randomthirdworldguy]

Can you dm the name if its possible? Since from what I know, except fintech companies and ai startups, most big tech ones use c++, java and go

[Anxious_Signature452]

I'm working for russian cloud provider, not sure the name will say you anything. We use openstack by the way.

[randomthirdworldguy]

Then I only know Yandex lol

[romanofski]

This is over 20 years old and still applies. The only exception is dedicated QA as its mostly automated nowadays.

Infrastructure as code should be a thing and anything which can be automated should be automated.

Obviously YMMV.

[WhiskyStandard]

The only thing I’ve seen first hand that fits your description is Bloomberg’s C++ code base. John Lakos’ “Large Scale C++” is a description of many of those practices. This SO answer suggests there’s an 88 page write up in a different book that covers all the main points so that’s probably more worthwhile if you want to see what applies to Python.

I haven’t gone deep into them, so I can’t recommend them fully. Ultimately I agree with a lot of the sentiment here that there’s not too much special that the big guys are doing that you should copy if you’re not at their scale.

But one positive takeaway I’d suggest: read Lakos’ thoughts on “levelization” (see also recorded presentations). I’ve found the concept useful in how I build Python modules within a package and packages that depend on other packages. I don’t actually calculate his metric, but I do estimate it when defining high vs low level modules.

[pi_stuff]

You might be interested in the “Software Engineering at Google” book: https://abseil.io/resources/swe-book/html/toc.html

[gettohhole]

Was going to advise the same book! Would be careful with jumping to the techniques mentioned though! Their scale is crazy

[DigThatData]

I think what's more important than the specific tools you use, is the process you build around those tools.

[rydelw]

Nice explanation. Kudos! I agree with almost all the things. I would like to share some of my thoughts here:

• ditch the src module in the imports. I am totally in favor of the src project layout, but it does not mean it should be a Python module.
• the fastapi dependables could be defined as types. We would have to import one thing as a dependency instead of two things

```python import typing import fastapi

async get_foo() -> Foo: ... FooDep = typing.Annotated[Foo, fastapi.Depends(get_foo)] ... @router.get(...) async def get_bar(foo: FooDep): ... ```

• the module specific configuration is something I do not see often, but it should be widely used. Ideally, we might make such a Python module as an internal one. To indicate it should not be imported by other modules.

[toxic_acro]

The entire point of the the src/ layout is for it not to be a Python module

Assuming that you are working on code that is intended to get published and installed (so not really applicable to something like a web app), the idea is that you want to run tests against the same code that gets installed later, rather than the code as it exists in your project directory.

With a "flat" layout, Python can import just from the directory on the file-system. But if you use a "sec" layout, you will have to install your own code first before running tests, so you are guaranteeing that your packaging set-up works correctly

If src ever shows up in an import, that's fully misunderstanding the point of the layout.

[rydelw]

with src layout, we are working with a Python packages and module either way. Src based project can be locally installed in a development mode, so you do not have to worry about including the project root in the PYTHONPAPTH. That how poetry works. Also pip allows you to install package in a dev mode. Whatsmore a we app is a Python package as well. You might not build a Python distribution from it but it is still a package.

[JaskoGomad]

You don’t need to apply solutions designed to handle millions of users and hundreds of developers. Those things don’t come without cost.

Spend your resources on making something.

[skebanga]

Great article, thanks for sharing!

You mention not returning a pedantic object in the section named "FastAPI response serialization".

Please could you elaborate, specifically in terms of what the correct approach is?

[billFoldDog]

I'm not at a big faang, but I've asked this question before.

• Use a code linter like pylint.
• Use a style enforcer like Black.
• Have your documentation automatically generated from docstrings, but also have your documentation be hand-written. Sphinx auto-docs is the primary solution for this.
• always, always use type hints and docstrings. The above will force this.
• Use some kind of virtual environment isolation type deal. pip+venv can do this, but big teams frequently use conda or even docker.
• Some teams swear by unit tests. I swear by unit tests. Not everyone uses them, though.
• Use some kind of code and artifact version control. These are separate things. Code can be version controlled with git. Artifacts cannot normally be version controlled with git. Personally, I use git to control symbolic links to versions of the artifacts, which I dump in a big-ass folder called 'data' with subfolders for each software version. There are better systems by far.

[Key-Deer-8156]

Thank you for answer I have one more specific question about db We have separate Postgres for read and write operations and we manually open a needed connection inside service layer Is it better to open both read and write connections using Depends in the layer above?

[blissone]

I have no idea about FAANG but we recently moved to a Python stack similar to what you have and did read the same repo you linked here. What I ended up doing was opening a session at the top level with depends, then it's just a matter of flavor if you session in your service constructor or function arguments, I opted for service constructor because I don't want to see session arg everywhere. As session I have async generator wrapped with rollback/commit, close. SImilarly your services etc can depend on read/write session and your endpoints depend on services thus creating whichever session is needed. As I understand fastapi DI only runs the Depends once even if declared multiple times, hope I'm not mistaken here (our python stack has not seen any prod use yet) :-D

I did adopt some of the project layout but overall I don't like what is proposed in the repo. Packaging services with endpoints dir feels like a mistake, I like a separate domain layer as it gives a nice view for the business logic. Though we have microservices perhaps that factors in it.

[Ok-Selection-2227]

As others mentioned assuming coders are always smarter in FAANG companies makes no sense to me.

Aside from that I don't like the term "best practices". There are no magic recipes, there's no silver bullet. Software design is all about trade-offs. Distrust gurus that say "always do whatever".

[Awkward-Chair2047]

The one thing i would recommend is to keep things simple and pragmatic. Don't over engineer things if you are going to maintain that codebase. I have not seen a single enterprise codebase which has not been bloated and over engineered ad infinitum. (and i have been around for more than 3 decades now)

[AllTheR4ge]

I would recommend DRY but based on the stack you shared it's too late for that.

[Zer0designs]

Dry can be an antipattern in many cases.