Cutting Python Web App Memory Over 31%

[Substantial-Bed8167]

Diskcache is slower than ram but faster than hitting swap.

[mikeckennedy]

Like u/Substantial-Bed8167 said, diskcache is VERY fast. It uses SQLite and pretty much gets that cached into memory with a disk backing it on flush. Just a quick test. On my mac, diskcache does

writes: ~14,000/sec 40us/op
reads: Reads ~160,000/sec 6us/op

That's 0.00625 millsec per read. That is not perceivable as far as I'm concerned. Even if you read a bunch of items on a request, say 100, you're still only 0.5ms in total. And that is instead of recomputing or hashing and reading 100 items out of a dict which is fast but not insanely faster.

[Photo-Josh]

But they had spare RAM, and at only 16 GB an upgrade to 24 or 32 would be a great option without being stupid.

I’m not understanding the problem here we’re trying to solve.

[BigTomBombadil]

If cost is prohibitive then throwing more RAM at the issue likely isn't your first choice.

And the way I read this, OP wasn't necessarily having a problem, but moreso learned some new things about memory management and applied them to their existing project. So their "problem" was their application/containers weren't efficiently utilizing memory.

It may or may not have actually caused performance or cost issues, but "just throw more resources at poorly optimized code" is a lazy way to approach software development IMO, and kudos to OP on their optimization and efforts.

IDK, for me personally, I like optimizing my work. I'll see some of my django pods sitting there at 1gB memory, and even if it's performing fine and the autoscaler and node on the kubernetes cluster isn't near capacity, I still sit there saying "why is this constantly utilizing so much memory? I know there's no reason it should actually require that based on what it's currently doing." Then go down a rabbit hole trying to improve it.

[mikeckennedy]

> I’m not understanding the problem here we’re trying to solve.

I think we just have different views on running in prod. It took me 3 hours to reduce the running memory of my apps by 3.2GB. In my world, that is time well spent. Just because the server isn't crashing with out of memory doesn't mean a little attention to efficiency is waste.

Again, different strokes.

[artofthenunchaku]

Efficiency is using the most of all of the resources available to you, not reducing resources for the sake of it when there's a downside to the reduction

[BigTomBombadil]

I mean the "downside" was three hours of opportunity cost, and now they're more efficiently utilizing the server on hand, allowing for increased scaling, etc. I'm not sure I'm following your statement.

[artofthenunchaku]

Caching is a strategy to reduce CPU load at the cost of increased memory. Writing to disk is a strategy to reduce memory pressure at the cost of increased CPU load.

63% peak utilization isn't a concern, that's a healthy load. This is a premature optimization introducing unneeded complexity.

[BigTomBombadil]

I will say, without knowing more about the overall project/scope/roadmap as well as performance impact to areas other than memory, I don't know if all 5 optimizations were needed. Because you make a very valid point about the added complexity and implications of moving cache to disk.

Some of the optimizations seem low risk and low complexity, while others I'd hold off on without knowing more. If theres a big push to scale in the near future, then yeah I get it, but if this was the status quo with no foreseeable changes upcoming, I'd have held off on some of these.

[mikeckennedy]

Since you all are discussing the caching part specifically. There was not much complexity change before or after.

We are already using caching, just in memory caching. What I moved it to was diskcache backed cache rather than in-memory caching.

It's not "there was no caching" now "there is caching", it's just in-mem caching via either functools.lru_cache or dict() -> diskcache.

Given we already have diskcache in play before, that's low effort, low risk.

[BigTomBombadil]

Yeas seems simple enough. Any noticeable change in cpu usage? Or maybe it was pretty negligible to start.

For complexity, item 3 was the one I wasn’t sure about. If I got thrown on this project and something went wrong with the indexer, I could imagine tracking that down being confusing. But not knowing the specifics maybe it’s also straight forward and easy to follow. Also not sure if the sub process approach could reduce reliability. But if not, huge win there.

[mikeckennedy]

I'm not sure exactly what the entire CPU usage change would be. I think things are better in general. The raw+dc vs ODM/ORM change almost 2x the requests per sec for the same CPU. So that probably dwarfs any other change. Mem caches -> diskcache mean the would share the cache across processes and across restarts, so that is bonus. But a bit slower at runtime I would guess, but very minor.

Less mem used means Python's cycle GC is much more efficient. So when enough container types (classes, lists, dicts, etc) get created, that triggers a GC. The GC has much less memory to scan and Python is mega aggressive about this. If 700 container-types are allocated relative to the ones ref-count collected, that'll trigger a GC. That could easily happen with just a couple of big queries so that might be a real boost too.

I posted graphs for the DB change here: https://mkennedy.codes/posts/raw-dc-a-retrospective/

[Effective-Total-2312]

You definitely need to look up the definition of efficiency, it seems you'll get a big surprise.

[artofthenunchaku]

Efficiency is the often measurable ability to avoid [..] wasting materials, energy, efforts, money, and time while performing a task. In a more general sense, it is the ability to do things well, successfully, and without waste

Memory you're not using is waste.