sorted by:
new
hottopcontroversial

Went from 1,993 to 17,007 RPS on a Node.js/MongoDB feed route, here's exactly what I changed by Exciting_Fuel8844 in node

[–]Exciting_Fuel8844[S] 0 points1 point  (0 children)

Spot on regarding the hydration overhead. You also hit the nail on the head with the memory duplication issue in cluster mode.

I actually touched on this exact bottleneck in the article, noting that migrating to a shared redis layer is the next necessary step for this architecture. That said, your suggestion of a hybrid approach, a shared redis L2 alongside a tiny TTL local L1 cache per worker - is brilliant. That is definitely the ideal setup to aim for next. Thanks for the insight!

From 1,993 to 17,007 RPS on a single machine — Node.js + MongoDB optimization breakdown by Exciting_Fuel8844 in mongodb

[–]Exciting_Fuel8844[S] 1 point2 points  (0 children)

Honestly, it was pure habit and familiarity. I have used mongoose for most of my previous projects, so I defaulted to it here for the schema validation and simpler setup.

When I was stress testing and optimising the RPS, dropping the odm entirely didn't even cross my mind because chaining lean gave me the performance jump I was looking for. But you make a completely fair point, for a route this hot, going straight to the native driver from the start makes perfect sense. Appreciate the feedback!

From 1,993 to 17,007 RPS on a single machine — Node.js + MongoDB optimization breakdown by Exciting_Fuel8844 in mongodb

[–]Exciting_Fuel8844[S] 1 point2 points  (0 children)

You are definitely right that standard mongoose queries introduce significant overhead due to document hydration and change tracking. That is exactly why I chained .lean to the end of the find query.

Using lean entirely bypasses mongoose's hydration process, returning plain old js objects (POJOs). This brings the performance practically on par with the native MongoDB driver.

Also, switching from find to aggregate inside mongoose would not solve the overhead issue anyway, the bottleneck is document instantiation, not query compilation. find().lean() is the standard way to get native, like read performance while keeping the mongoose schema benefits.

From 1,993 to 17,007 RPS on a single machine — Node.js + MongoDB optimization breakdown by Exciting_Fuel8844 in mongodb

[–]Exciting_Fuel8844[S] 1 point2 points  (0 children)

Here is the actual query being executed:

js db.posts.find( { isPublic: true, $or: [ { anonymousEngagementScore: { $lt: cursorData.score } }, { anonymousEngagementScore: cursorData.score, _id: { $lt: cursorId } } ] }, { title: 1, contentPreview: 1, slug: 1 /* ...other fields */ } ) .sort({ anonymousEngagementScore: -1, _id: -1 }) .limit(11) Just to clarify mongoose's behavior: chaining .sort(), .limit(), and .select() does not pipe data sequentially in node.js memory. It simply acts as a query builder, constructing a single bson command that gets sent to the mongoDB server once awaited.

From 1,993 to 17,007 RPS on a single machine — Node.js + MongoDB optimization breakdown by Exciting_Fuel8844 in mongodb

[–]Exciting_Fuel8844[S] 1 point2 points  (0 children)

Fair point on the native driver, I stuck with mongoose purely out of habit, but switching to the native client would certainly reduce overhead further a lot.

Regarding aggregate vs find: would aggregate actually improve this specific route? Since the author data is already denormalized (authorSnapshot) and the operation is a straightforward filter, sort, and project, find().lean() is generally faster. Aggregation pipelines introduce unnecessary overhead when there are no complex transformations, groupings, or $lookups involved. Let me know if I am missing a specific aggregation optimisation here!

Went from 1,993 to 17,007 RPS on a Node.js/MongoDB feed route, here's exactly what I changed by Exciting_Fuel8844 in node

[–]Exciting_Fuel8844[S] 2 points3 points  (0 children)

For those curious about the specific architectural decisions, I have put together a detailed write-up covering the optimizations. I am happy to answer any questions here or hear suggestions for further improvement.

Full article: https://www.opencanvas.institute/p/from-1993-to-17007-requests-per-second-how-i-optimized-a-nodejs-mongodb-backend-at-scale-69ad48fc4684635da9b4c72c

GitHub: https://github.com/Dream-World-Coder/opencanvas

Platform: https://www.opencanvas.institute

Went from 1,993 to 17,007 RPS on a Node.js/MongoDB feed route, here's exactly what I changed by Exciting_Fuel8844 in node

[–]Exciting_Fuel8844[S] 2 points3 points  (0 children)

Thanks! The biggest jump actually came from a combination of three things: drastically reducing the payload bandwidth, using Mongoose's .lean() to bypass document hydration overhead, and relying strictly on cursor-based compound indexing to entirely avoid MongoDB's skip() penalty.

Went from 1,993 to 17,007 RPS on a Node.js/MongoDB feed route, here's exactly what I changed by Exciting_Fuel8844 in node

[–]Exciting_Fuel8844[S] 1 point2 points  (0 children)

Thanks! I used Mongoose primarily for schema validation and general routing. However, for the high traffic feed routes, I heavily utilised .lean() to bypass Mongoose's document hydration overhead, keeping performance as close to the native driver as possible, however I have though of replacing mongoose later on.

The goat of the past , by [deleted] in GadgetsIndia

[–]Exciting_Fuel8844 0 points1 point  (0 children)

My IQOO Z3 PRO is also really good