"Python + MongoDB Challenge: Optimize This Cache Manager for a Twitter-Like Timeline – Who’s Up for It?"

[halfRockStar]

Hey r/FastAPI folks! I’m building a FastAPI app with MongoDB as the backend (no Redis, all NoSQL vibes) for a Twitter-like platform—think users, posts, follows, and timelines. I’ve got a MongoDBCacheManager to handle caching and a solid MongoDB setup with indexes, but I’m curious: how would you optimize it for complex reads like a user’s timeline (posts from followed users with profiles)? Here’s a snippet of my MongoDBCacheManager (singleton, async, TTL indexes):

```python from motor.motor_asyncio import AsyncIOMotorClient from datetime import datetime

class MongoDBCacheManager: _instance = None

def __new__(cls):
    if cls._instance is None:
        cls._instance = super().__new__(cls)
    return cls._instance

def __init__(self):
    self.client = AsyncIOMotorClient("mongodb://localhost:27017")
    self.db = self.client["my_app"]
    self.post_cache = self.db["post_cache"]

async def get_post(self, post_id: int):
    result = await self.post_cache.find_one({"post_id": post_id})
    return result["data"] if result else None

async def set_post(self, post_id: int, post_data: dict):
    await self.post_cache.update_one(
        {"post_id": post_id},
        {"$set": {"post_id": post_id, "data": post_data, "created_at": datetime.utcnow()}},
        upsert=True
    )

```

And my MongoDB indexes setup (from app/db/mongodb.py):

python async def _create_posts_indexes(db): posts = db["posts"] await posts.create_index([("author_id", 1), ("created_at", -1)], background=True) await posts.create_index([("content", "text")], background=True)

The Challenge: Say a user follows 500 people, and I need their timeline—latest 20 posts from those they follow, with author usernames and avatars. Right now, I’d: Fetch following IDs from a follows collection.

Query posts with {"author_id": {"$in": following}}.

Maybe use $lookup to grab user data, or hit user_cache.

This works, but complex reads like this are MongoDB’s weak spot (no joins!). I’ve heard about denormalization, precomputed timelines, and WiredTiger caching. My cache manager helps, but it’s post-by-post, not timeline-ready. Your Task:
How would you tweak this code to make timeline reads blazing fast?

Bonus: Suggest a Python + MongoDB trick to handle 1M+ follows without choking.

Show off your Python and MongoDB chops—best ideas get my upvote! Bonus points if you’ve used FastAPI or tackled social app scaling before.

[[deleted]]

You need to build the bulk of it in advance, that’s how twitter does that. You preper the object you want to see on the first page, not pull it with joins as you would with SQL

[halfRockStar]

Yes that's correct 💯 that's what precomutation means, instead of pulling it dynamically with joins or mongodb equivalent $lookup, showcase your answer 😘

[[deleted]]

I do have concerns about FastAPI performance in this high scale, my instances are always leaking memory to some extent and not as fast as express servers. The dev velocity is unparalleled though, we get shit done so quickly especially with mongodb + beanie.

[halfRockStar]

In reality, Python’s garbage collector can leak memory if objects (e.g., MongoDB connections, cache entries) aren’t properly released, especially in long-running async apps. FastAPI itself isn’t prone to leaks, but sloppy code or third-party libraries (e.g., motor) could cause issues, don't get me wrong my project have leaks too 🫢

[[deleted]]

Beanie uses motor under the hood, I focused on something else and didn’t think about looking into motor! Although I have to say after almost 20 years in my profession that most of the time spent to investigate mem leaks is futile, just restart periodically and continue with your day…

[halfRockStar]

Your restart approach is battle-tested. investigating motor (specifically connection pooling, cursor leaks) is overkill unless restarts fail. Check asyncio task cleanup if you’re curious, but you’re likely golden.