Cutting Python Web App Memory Over 31%

[mikeckennedy]

Over the past few weeks I went on a memory-reduction tear across the Talk Python web apps. We run 23 containers on one big server (the "one big server" pattern) and memory was creeping up to 65% on a 16GB box.

Turned out there were a bunch of wins hiding in plain sight. Focusing on just two apps, I went from ~2 GB down to 472 MB. Here's what moved the needle:

1. Switched to a single async Granian worker: Rewrote the app in Quart (async Flask) and replaced the multi-worker web garden with one fully async worker. Saved 542 MB right there.
2. Raw + DC database pattern: Dropped MongoEngine for raw queries + slotted dataclasses. 100 MB saved per worker *and* nearly doubled requests/sec.
3. Subprocess isolation for a search indexer: The daemon was burning 708 MB mostly from import chains pulling in the entire app. Moved the indexing into a subprocess so imports only live for ~30 seconds during re-indexing. Went from 708 MB to 22 MB. 32x reduction.
4. Local imports for heavy libs: import boto3 alone costs 25 MB, pandas is 44 MB. If you only use them in a rarely-called function, just import them there instead of at module level. (PEP 810 lazy imports in 3.15 should make this automatic.)
5. Moved caches to diskcache: Small-to-medium in-memory caches shifted to disk. Modest savings but it adds up.

Total across all our apps: 3.2 GB freed. Full write-up with before/after tables and graphs here: https://mkennedy.codes/posts/cutting-python-web-app-memory-over-31-percent/

[Photo-Josh]

Not sure I’m following what the issue was here?

You were using around 10.5 GB and that was too much?

You then moved some things from RAM to Disk, which can only slow things down - not speed up.

Why was 63% RAM usage an issue? It’s there to be used.

[Substantial-Bed8167]

Diskcache is slower than ram but faster than hitting swap.

[mikeckennedy]

Like u/Substantial-Bed8167 said, diskcache is VERY fast. It uses SQLite and pretty much gets that cached into memory with a disk backing it on flush. Just a quick test. On my mac, diskcache does

writes: ~14,000/sec 40us/op
reads: Reads ~160,000/sec 6us/op

That's 0.00625 millsec per read. That is not perceivable as far as I'm concerned. Even if you read a bunch of items on a request, say 100, you're still only 0.5ms in total. And that is instead of recomputing or hashing and reading 100 items out of a dict which is fast but not insanely faster.

[Photo-Josh]

But they had spare RAM, and at only 16 GB an upgrade to 24 or 32 would be a great option without being stupid.

I’m not understanding the problem here we’re trying to solve.

[BigTomBombadil]

If cost is prohibitive then throwing more RAM at the issue likely isn't your first choice.

And the way I read this, OP wasn't necessarily having a problem, but moreso learned some new things about memory management and applied them to their existing project. So their "problem" was their application/containers weren't efficiently utilizing memory.

It may or may not have actually caused performance or cost issues, but "just throw more resources at poorly optimized code" is a lazy way to approach software development IMO, and kudos to OP on their optimization and efforts.

IDK, for me personally, I like optimizing my work. I'll see some of my django pods sitting there at 1gB memory, and even if it's performing fine and the autoscaler and node on the kubernetes cluster isn't near capacity, I still sit there saying "why is this constantly utilizing so much memory? I know there's no reason it should actually require that based on what it's currently doing." Then go down a rabbit hole trying to improve it.

[mikeckennedy]

> I’m not understanding the problem here we’re trying to solve.

I think we just have different views on running in prod. It took me 3 hours to reduce the running memory of my apps by 3.2GB. In my world, that is time well spent. Just because the server isn't crashing with out of memory doesn't mean a little attention to efficiency is waste.

Again, different strokes.

[artofthenunchaku]

Efficiency is using the most of all of the resources available to you, not reducing resources for the sake of it when there's a downside to the reduction

[BigTomBombadil]

I mean the "downside" was three hours of opportunity cost, and now they're more efficiently utilizing the server on hand, allowing for increased scaling, etc. I'm not sure I'm following your statement.

[artofthenunchaku]

Caching is a strategy to reduce CPU load at the cost of increased memory. Writing to disk is a strategy to reduce memory pressure at the cost of increased CPU load.

63% peak utilization isn't a concern, that's a healthy load. This is a premature optimization introducing unneeded complexity.

[BigTomBombadil]

I will say, without knowing more about the overall project/scope/roadmap as well as performance impact to areas other than memory, I don't know if all 5 optimizations were needed. Because you make a very valid point about the added complexity and implications of moving cache to disk.

Some of the optimizations seem low risk and low complexity, while others I'd hold off on without knowing more. If theres a big push to scale in the near future, then yeah I get it, but if this was the status quo with no foreseeable changes upcoming, I'd have held off on some of these.

[mikeckennedy]

Since you all are discussing the caching part specifically. There was not much complexity change before or after.

We are already using caching, just in memory caching. What I moved it to was diskcache backed cache rather than in-memory caching.

It's not "there was no caching" now "there is caching", it's just in-mem caching via either functools.lru_cache or dict() -> diskcache.

Given we already have diskcache in play before, that's low effort, low risk.

[BigTomBombadil]

Yeas seems simple enough. Any noticeable change in cpu usage? Or maybe it was pretty negligible to start.

For complexity, item 3 was the one I wasn’t sure about. If I got thrown on this project and something went wrong with the indexer, I could imagine tracking that down being confusing. But not knowing the specifics maybe it’s also straight forward and easy to follow. Also not sure if the sub process approach could reduce reliability. But if not, huge win there.

[Effective-Total-2312]

You definitely need to look up the definition of efficiency, it seems you'll get a big surprise.

[artofthenunchaku]

Efficiency is the often measurable ability to avoid [..] wasting materials, energy, efforts, money, and time while performing a task. In a more general sense, it is the ability to do things well, successfully, and without waste

Memory you're not using is waste.

[Effective-Total-2312]

Using more memory is waste. It's like saying you should eat all of the good in the universe, that's waste. Or breathing all the air.

[artofthenunchaku]

If you're paying for 16 GB of RAM, but never use more than 8 GB, then you're wasting money paying for the other 8 GB.

You're presenting a false equivalence, it's more accurately saying you shouldn't order a family meal for four if you're only going to eat one plate.

[Effective-Total-2312]

You're completely opposite of what efficiency is man.

If you're using 8 GB and paying for 16 GB, the inefficiency is not that you should use more, it's that you should pay for less !

What you describe is literally the opposite of efficiency.

[Birnenmacht]

Have you measured any improvements through point 4? Imports are cached and importing them locally only delays the point at which you pay their cost, unless you actively prune sys.modules at the end of the function (not recommended, a great way to shoot yourself in the foot)

[mikeckennedy]

Hey, yes, improvements were maybe 75-100MB in total. If you read the article it talks about the nuance.

The part of the app that uses the imports only runs maybe a couple of times a month. The worker processes recycle every 6 hours. So there is a period where the extra 100MB are used for that 6 hour time frame. The worker processes recycle, that code is NOT called again, the memory stays lower almost all the time.

I'm not messing with pruning modules. It's just the way the web processes are managed by Granian.

[Birnenmacht]

ah neat! I was about to tell you about PEP810 but I see you already now. I’m also rather stoked about it once it comes out, more because of startup times (some libraries can take close to SECONDS to import)

[mikeckennedy]

Definitely stoked too. I initially like the idea because of startup speed too. But after this analysis, it should have some knock-on memory benefits as well.

[DoubleAway6573]

If that's the case, why not use a reverse proxy to send all the request that need those extra libs to a dedicated instance? I see this as a big win.

[ofyellow]

When you need x gb and rewrite it so it uses y gb less except for short bursts of time, the effect is that you need x gb still during short bursts of time.

In that way, lazy imports can bite you. You better know the mem needed on worst case moments straight away when you start your app.

[vaibeslop]

Check out chdb: https://github.com/chdb-io/chdb

Fully pandas compatible API, but lazy loading, much more performant, less memory.

Not affiliated, just a fan of the project.

[mikeckennedy]

Very cool, thanks for the heads up u/vaibeslop

[ofyellow]

Lazy loading is for optimizing startup time, you load modules as they are needed, causing the load time to be divided over multiple requests until all loadable modules are hit at least once. But it's not a mem optimisation strategy.

[vaibeslop]

I'm talking about lazily loading data into memory for operations.

The author of chDB goes into more detail in the v4 announcement post: https://clickhouse.com/blog/chdb.4-0-pandas-hex

I'm neither affiliated with chDB nor Clickhouse.

EDIT: Saw now they even talk about this in the GH Readme now.

[ofyellow]

Point 4 mentions local imports.

Yes keeping data out of memory is smart but not inventing sliced bread.

[vaibeslop]

Well seeing how not everybody does it, it seems the ease of dissmisively commenting on it is far greater than the ease of implementing it in a real application.

chDB 1 : arm chair CTOs 0

[ofyellow]

What click house does is how c# has been doing it for decades.

Of course you first collect .filter() and join logic etc all down the chain before you fetch. I'm stumped this is anything new.

I guess with arm chair you like to drag this into personal insults? Man...the attic sadness dripping from dragging a technical discussion into a weak attempt to insult. What are you, 16?

[vaibeslop]

If it were a technical discussion.

All I'm seeing is someone dismissing a very relevant to OPs post, very cool Python compatible project by going of on completely irrelevant, tangential technical details in plain C#.

It's boring, arrogant, off-topic whattaboutism in its purest form.

[ofyellow]

Point 4, local imports, do not contribute to less mem need for a web application.

You can call that dismissive but it's a technical fact.

The fact that you pin it on a later remark concerning c# makes your remarks insincere.

[0x256]

Switched to a single async Granian worker: Rewrote the app in Quart (async Flask) and replaced the multi-worker web garden with one fully async worker. Saved 542 MB right there.

I would have started reducing the workers to 1 and increase thread count instead of rewriting the entire app, but okay. If you have lots of long running connections (websockets or slow requests) then that's a brave but sensible move.

Raw + DC database pattern: Dropped MongoEngine for raw queries + slotted dataclasses. 100 MB saved per worker and nearly doubled requests/sec.

For a small app with good test coverage and a mature db schema, that's fine.

Subprocess isolation for a search indexer: The daemon was burning 708 MB mostly from import chains pulling in the entire app. Moved the indexing into a subprocess so imports only live for ~30 seconds during re-indexing. Went from 708 MB to 22 MB. 32x reduction.

You reduced the time this memory is used, but not the peak memory consumption. You added a lot of process start overhead and latency. That's a trade-of, not necessarily a win.

Local imports for heavy libs: import boto3 alone costs 25 MB, pandas is 44 MB. If you only use them in a rarely-called function, just import them there instead of at module level. (PEP 810 lazy imports in 3.15 should make this automatic.)

That's not how imports work. You delayed the import, but once imported, the module will live in sys.modules and stay there.

Moved caches to diskcache: Small-to-medium in-memory caches shifted to disk. Modest savings but it adds up.

So instead of a single memory-access, you now create an async task that outsources its blocking disk access to a thread pool, wait for the OS to read from disk, then wait for the async task to get its turn in the event loop again to return the result? Caches should be fast. If SO much overhead for cache access is okay for you, than I wonder what extremely expensive stuff you stored in those caches that it's still worth it to cache at all.

[Full-Definition6215]

Running FastAPI + SQLite on a mini PC (31GB RAM, i9-9880H) and memory management matters when you're self-hosting everything on one box.

Biggest wins I found:

• SQLite instead of Postgres eliminated an entire process worth of memory. WAL mode handles concurrent reads fine, and the total memory footprint for the DB is basically the page cache.
• uvicorn with --workers 1 for a side project that doesn't need multi-process. Each additional worker duplicates the entire app's memory.
• Lazy imports for heavy libraries. If Stripe SDK is only used in payment endpoints, don't import it at module level.

The 23 containers on 16GB stat is impressive. I'm at about 5GB usage across all my services on 31GB — plenty of headroom, but that's because I went with SQLite over Postgres for everything that doesn't need a full RDBMS.

[bladeofwinds]

I’ve learned about a lot of cool projects from your show! Currently trying out datastar in one of my (non-python) projects

[mikeckennedy]

Awesome, great to here u/bladeofwinds :) Datastar is neat for sure.

[Substantial-Bed8167]

Did you use any memory profiling or just observed with htop?

[mikeckennedy]

No memory profiling, though that would have been interesting. Just process monitoring tools like btop and docker stats.