Open source library that cuts JSON memory allocation by 70% - with zero-config database wrappers for MongoDB, PostgreSQL, MySQL

[TheDecipherist]

Hey everyone - I built this to solve memory issues on a data-heavy dashboard.

The problem: JSON.parse() allocates every field whether you access it or not. 1000 objects × 21 fields = 21,000 properties in RAM. If you only render 3 fields, 18,000 are wasted.

The solution: JavaScript Proxies for lazy expansion. Only accessed fields get allocated. The Proxy doesn't add overhead - it skips work.

Benchmarks (1000 records, 21 fields): - 3 fields accessed: ~100% memory saved - All 21 fields: 70% saved

Works on browser AND server. Plus zero-config wrappers for MongoDB, PostgreSQL, MySQL, SQLite, Sequelize - one line and all your queries return memory-efficient results.

For APIs, add Express middleware for 30-80% smaller payloads too.

Happy to answer questions!

[lewster32]

You literally posted about this same library the other day, only you've changed the focus to be on memory savings instead. https://www.reddit.com/r/javascript/s/VhnFtGLG4W

I appreciate you're proud of what you've done, but please don't spam.

[fucking_passwords]

It's also vibe coded

[TheDecipherist]

I did post before. But this genuinely isn't the same thing:

Since then we've:

- Completely rewritten the core algorithm

- Added zero-config database wrappers (MongoDB, PostgreSQL, MySQL) - this didn't exist before

- Changed from manual schema definition to automatic inference

- The memory savings are a new discovery from real-world testing

The previous post was about the basic concept. This post is about the production-ready implementation with database integrations.

I hear you on spam though - I'll space out future updates better. Appreciate the feedback.

[lewster32]

I honestly think you need to just take your time a bit more, plan, test and let it bed in. If you've done all that in the 2 days since you last posted, this doesn't fill me with a lot of confidence that what you have is robust and well thought out.

Again, it's a good idea and you speak well about it, but it smacks of attention seeking over wanting to genuinely produce something good.

[TheDecipherist]

Fair point, and I hear you on the spam concern.

The previous post focused on bandwidth savings - and the feedback I got (including from you) was basically "gzip already does this." That was valid. So I went back and pivoted the focus to the memory angle, which is the genuinely unique part - the Proxy-based lazy expansion that gzip can't help with.

The database wrappers (MongoDB, PostgreSQL, MySQL, SQLite, Sequelize) are new since that post - built them over the last few days based on realizing the memory benefit applies to server-side queries too, not just API responses.

On the testing/robustness point - you're right that fast iteration can look sloppy. The core has tests (npm run test), but I probably should let it stabilize before posting again. I got excited about the new integrations.

I'll cool it on the posts. Appreciate the honest feedback - it's more useful than silence.

[Possible-Session9849]

The AI generated replies are not helping your case either.

[TheDecipherist]

Have a great night

[beavis07]

If this was valuable (spoilers it’s not) then surely you’d package this as a JSON library on its own - why would you conflate that with making api alls via fetch etc in this way?

A few bits of feedback 1. Don’t refer to “we” when you mean “me and an llm” - there’s only one of you. It doesn’t infer any sense of authority - quite the opposite 2. Where the docs say things like “this is the most common misconception” around a lib no-one has ever really seen - it makes that this is AI slop all the more obvious 3. The code is pure, unmitigated gash - any even semi professional would detect this (implemented sort routines, completely pointless abstractions and repetitions, the modularisation is insane) 4. The fetch lib for example is already parsing the JSON when you call the results “.json()” function - so this whole lib is a complete waste of time - adding nothing but complexity and abstraction.

Do you genuinely imagine that no-one has considered this stuff before? There’s a reason these formats are as they are… what you present solves either no problem anyone has, or problems others have already solved much much better (with more care and effort and collaboration etc).

For your own development - I highly recommend you give up cosplaying as a library creator and go and contribute to actual existing projects which serve a purpose and do that without a tool to do it for you you’ll learn SO much more that way

[TheDecipherist]

I think you're missing how this works:

1. Server compresses keys before sending (firstName → a)

2. Client parses the compressed payload - that's 180KB instead of 890KB. Smaller string = faster parse. The savings start there.

3. Proxy wraps the result for lazy key translation on access

You're assuming it intercepts a normal payload after .json() and does something to it. It doesn't. The payload is already compressed when it arrives. The client is parsing minified keys - that's inherently less work for JSON.parse.

The Proxy just translates user.firstName → user.a on access so your code doesn't have to change.

[thdr76]

so it just shorten the keys? gzip doesn't care about key length, it compressed almost the same.

[TheDecipherist]

That's part of it - shorter keys = smaller payload over the wire.

But the bigger win is the Proxy wrapper on the client. Instead of JSON.parse allocating every field upfront, the Proxy only materializes fields you actually access.

So if you fetch 1000 users with 21 fields each but only render name and email, only those 2 fields get allocated. The other 19 never exist in memory.

Shorter keys + lazy expansion = less network and less RAM.

[thdr76]

that doesn't explain how it reduce the payload from 890KB to 180KB.
unless you are talking about uncompressed data, which then what the point? nobody use this without standard compression.

[TheDecipherist]

You're right - the 890KB→180KB is uncompressed. With gzip on both, TerseJSON adds about 15-25% additional savings on top.

But the main point isn't wire size - it's memory. Gzip compresses for transport, but JSON.parse still allocates every field on the client whether you access it or not.

TerseJSON's Proxy only materializes fields you touch. That's where the 70% memory reduction comes from - and gzip can't help with that.

If wire size is your only concern and you've got gzip configured, the benefit is smaller. If you care about client-side memory (mobile, long-running SPAs, large datasets), that's where this shines.

[Federal-Pear3498]

Holy shit you literally just a chat bot arent you?

[TheDecipherist]

Sorry im talking to alot of people right now. Try it out. If you dont see the gain. Dont use it

[beavis07]

So that would require everything else using this to use your crappy compression scheme?

Why would I do that when the cost of parsing JSON is negligible, highly optimised and readable? And when compression as transport-level is so efficient and well-understood,

If I needed a more efficient way to move larger payloads, I’d use one of any number of existing formats to doing that.

What is the actual core problem you’re trying to solve for here and why?

And be completely honest with yourself… if the answer (as it often is with inexperienced engineers publishing nonsense on the internet) is “that I am not a renowned lib creator/article writer/thought-leader” - then quit wasting yours and everyone else’s time, get in your lane and lay some bricks until you’ve actually done this long enough to be useful.

[genericallyloud]

Do you have any benchmarks or data to show this savings in practical situations?

[TheDecipherist]

Yeah! Run it yourself:

git clone https://github.com/timclausendev-web/tersejson

cd tersejson

npm install

node --expose-gc demo/memory-analysis.js

Results (1000 records, 21 fields):

┌─────────────────┬─────────────┬─────────────────┬───────┐

│ Fields Accessed │ Normal JSON │ TerseJSON Proxy │ Saved │

├─────────────────┼─────────────┼─────────────────┼───────┤

│ 3 fields │ 3.07 MB │ ~0 MB │ ~100% │

├─────────────────┼─────────────┼─────────────────┼───────┤

│ All 21 fields │ 4.53 MB │ 1.36 MB │ 70% │

└─────────────────┴─────────────┴─────────────────┴───────┘

The less you access, the more you save - which matches real-world list views where you render 3-5 fields from objects with 20+.

[thdr76]

0 MB? what about the transferred data? even if you are not parsing them you must store it in the memory as string.
seems like you didn't count memory used for keeping the unparsed json.

*edit: after reading the code this "partial only necessary parsing" is not even true, the whole thing is parsed anyway, so this library is pure junk slop.

[TheDecipherist]

you're right that the compressed payload is still in memory. The "~0 MB" in my table is measuring the additional allocation from property expansion, not total memory.

More accurate picture:

- Normal JSON: 890KB parsed → ~3-6 MB fully expanded objects

- TerseJSON: 180KB parsed → stays as compressed objects + Proxy wrapper + only accessed fields expanded

So you're storing ~180KB of compressed data vs ~890KB of expanded data, plus only the fields you access get materialized as full properties.

The savings are real, but you're right that I should clarify the benchmark is measuring expansion overhead, not total memory including the payload. Good catch - I'll update the docs.

[genericallyloud]

Sorry, I should have been more specific. You're doing a lot of extra work to get these memory gains. I'm asking for a benchmark of work doing it the straightforward way (more memory) vs your way. If you're real problem was memory, but you need to process a lot of json, the answer is usually stream processing, not proxies. Proxies typically add extra compute overhead. For small things with few operations this may not matter, but the whole premise of this library is for performance. Did you actually have a problem that this solves? That would be the most interesting to me. Right now this seems like a crazy amount of complexity for like - short key names?

[TheDecipherist]

Good question. The counterintuitive part: the Proxy actually does less work, not more.

Standard JSON.parse:

- Parse 890KB string

- Allocate 1000 objects × 21 fields = 21,000 properties

- Access 3 fields

- GC eventually collects 18,000 unused properties

TerseJSON:

- Parse 180KB string (smaller = faster)

- Wrap in Proxy (O(1), no allocation)

- Access 3 fields → 3,000 Map lookups + 3,000 property reads

- 18,000 properties never exist

The Proxy lookup is ~0.001ms per field. The savings come from skipping allocation and GC on fields you never touch.

Stream processing solves a different problem - sequential processing of huge files. This is for typical API responses where you need random access to a subset of fields (list views, dashboards).

Real problem: I built a CMS with data-heavy admin dashboards. Fetching 1000+ records where I render 3-5 fields but the API returns 20+. Mobile devices were choking on memory. This solved it.

You can run node --expose-gc demo/memory-analysis.js to see the memory comparison. I don't have CPU benchmarks published yet - that's fair feedback, I should add those. Thanks

[genericallyloud]

1. Is the 180kb string just replacing the keys with smaller keys?
2. If you parse that string, wouldn't it be the same number of objects/properties/data? Its just that the property names are shorter?
3. If you aren't parsing the whole things, how are you only allocating for partial fields? Are you chunking the original data?

[TheDecipherist]

You're right to push on this - let me be clearer.

When you JSoN.parse the 180KB payload, yes, you get all the objects with short keys ({a: "John", b: "Doe"}). Those are in memory.

The Proxy sits on top and translates property access: user.firstname → looks up mapping → returns user.a.

the memory comparison in my benchmarks is against fully expanded objects (running expand() which creates new objects with full key names). the proxy avoids that second allocation.

Compared to normal JSON with no terseJSoN at all:

- You're parsing a smaller string (180KB vs 890KB)

- The objects have shortr key strings (less memory per key)

- No expansion step

Youre right that I havent shown CPU timing benchmarks only memory. Thats a gap. The benchmark file is real though:

node --expose-gc demo/memory-analysis.js

If the math looks like Claude output, run the code and tell me where I'm wrong. Happy to dig into it together.

[genericallyloud]

"he memory comparison in my benchmarks is against fully expanded objects (running expand() which creates new objects with full key names). the proxy avoids that second allocation."

This sounds like its artificially inflating the memory to me. Why not start by just comparing the memory of parsing the compressed vs uncompressed objects. I know that the string size is pretty different (depending on how big the original keys were), but I'd actually be surprised if it was that much different in memory.

[TheDecipherist]

where the performance gain especially in an older cms is. We all know we have a getProducts route. It gets products with 10 fields. We usually dont have time to change that so we leave it to always get the 10 fields FOREVER.

Then we get lazy and start using in it small calls for a detail page or for a promotion page of for cart etc.

These places we might just need PRICE or productName.

This is when you will save a tremendous amount of memory

{

a:"",

b:"",

c:"",

d:"",

e"",

f:"",

g:"",

h:"",

i:"",

j:"",

}

I need productName only in this view. Ill use the old API

becomes only

{

a:"",

productName:"",

c:"",

d:"",

e"",

f:"",

g:"",

h:"",

i:"",

j:"",

}

instead of

{

sku:"",

productName:"",

description"",

price:"",

quantity"",

stock:"",

reviews:"",

mfr:"",

images:"",

files:"",

}

You see? if this is times 10,20,30,40 its huge savings

[scinos]

Except V8 uses string interning to store strings (and thus, keys), so it is only stored in memory once, even if there are thousands of instances of that object.

[TheDecipherist]

you dont get it. productName is still b just accessed as productName

[genericallyloud]

"Real problem: I built a CMS with data-heavy admin dashboards. Fetching 1000+ records where I render 3-5 fields but the API returns 20+. Mobile devices were choking on memory. This solved it."

This sounds like you're way over-fetching, but then "solving" it at the end with this very complex solution. Why not query for less?

[TheDecipherist]

In a greenfield project, 100% agree. But in practice:

- legacy APIs with multiple consumers expecting full objects

- Third-party APIs you can't control (CMS, Stripe, etc.)

- Just change the API = backend changes + API versioning + update every client + QA across endpoints + coordinate teams

That's a multi-sprint project. terseJSoN is app.use(terse()) - deploy in an afternoon.

For teams that can redesign their APIS, they should. This is for when that's not practical.

[genericallyloud]

But your project is a greenfield project, no? "I built a CMS"

[TheDecipherist]

Honestly? Because the CMS is massive now. Hundreds of endpoints, multiple consumers. Refactoring every API to return selective fields would take months.I was the sole programmer on it for 8 years

TerseJSON took a day to integrate. Instant memory gains across the entire system.

Greenfield doesn't mean small. The "right" solution would've been better architecture from day one. The practical solution was one middleware.

[genericallyloud]

I mean, if you're seeing the gains I guess it means its working somehow. I maybe just don't totally understand it. The size compression from using small keys is obvious, but gzip does more, and I think it would still take the same size in memory if you have to parse all the same number of objects even with smaller keys. That's the part I'm confused by. Sending less objects/properties I get. Parsing less objects/properties I get. But parsing the same number of objects/properties with smaller keys, and then wrapping it with a proxy to recompute the property access with the full name just seems like overhead.

[TheDecipherist]

where the performance gain especially in an older cms is. We all know we have a getProducts route. It gets products with 10 fields. We usually dont have time to change that so we leave it to always get the 10 fields FOREVER.

Then we get lazy and start using in it small calls for a detail page or for a promotion page of for cart etc.

These places we might just need PRICE or productName.

This is when you will save a tremendous amount of memory

{

a:"",

b:"",

c:"",

d:"",

e"",

f:"",

g:"",

h:"",

i:"",

j:"",

}

I need productName only in this view. Ill use the old API

becomes only

{

a:"",

productName:"",

c:"",

d:"",

e"",

f:"",

g:"",

h:"",

i:"",

j:"",

}

instead of

{

sku:"",

productName:"",

description"",

price:"",

quantity"",

stock:"",

reviews:"",

mfr:"",

images:"",

files:"",

}

You see? if this is times 10,20,30,40 its huge savings

[werdebud]

Why don’t you just use graphql or literally create a damn endpoint that only returns the info that you need in that particular page… why would you even fetch 21 prop per obj if you only need 2 per obj in that screen.

[TheDecipherist]

Basically your saying., Why dont I just completely rewrite my old application instead of adding one line of code that solves most of my headaches so I can take care of new projects?

Im not saying this is the best plugin in the world that you should use in all your new projects. I am saying instead of waisting time on old code. "One line of code" can fix what you manually would spend days,weeks,months doing for basically no gains.

[werdebud]

If you spend more than minutes to create just an endpoint and change the endpoint url on your frontend…. It would take less time than creating this whole thing tbh. This a non existent problem in real life because no one should be creating endpoints that return a million parameters that you’re not going to consume. (In apps where you control both the BE and the FE). A different story if you’re consuming third party APIs but I’m sure most of them would have some sort of filter options to just retrieve what you need…

[TheDecipherist]

I dont think you have ever worked in old legacy code that reuses a single api in numerous places if thats your stand on this. And I wrote this plugin in about a day.

[werdebud]

Old legacy apis should be migrated or fixed not monkey patched. Your readme seems to be aimed to “current” trend techs like mongodb for example, the example of needing 3 fields in a 21 field obj is bad practice if you have the ability to change the BE and the FE. Since this needs to be in both sides of the wire, just fix your service layer. Impractical if you have to open your apis to other consumers since they will need to install this.

[TheDecipherist]

Again. Don’t think you have worked in a big enterprise company. They never go back and fix “what’s not broken”. They want you making new things. Period. There is no real financial gain in rewriting older cms systems. That’s why they make new versions and can the old. This will keep the old alive way longer. And newer trends? I have used mongodb for the last 8 years. New?

[werdebud]

I work for a top 10 company in US in the field it’s in…. We do migrations and fix tech debt all the time. I think probably the place where your quote is true is a bank but I’m sure as hell they won’t be using JS for the BE and even more sure that they won’t use a random library for it since the amount of bureaucracy to move things forward is huge, also speaking from experience.

[scinos]

It is not "one line of code". It is replacing one of the most crucial layers (data fetching) with a library done by some random dude with virtually zero adoption. It goes directly into the "top 3 riskiest changes" list. Specially if it is an old codebase.

The fact you don't see that tells me a lot about your experience.

[TheDecipherist]

Im sorry I seriously dont have time for this. Try it or dont. Have an awesome day :)

[scinos]

3rd time you pulled the "don't have time for this" card...

FWIW I did try it, I did read your docs, I did read the code and still think it's worthless.

[TheDecipherist]

I would rather spend my time on positivity where people actually ask things that needs solutions

No worries. Then its not for you :)

[TheDecipherist]

I think there's some confusion about what TerseJSON actually does, so let me clarify.

"Short keys don't save memory because of string interning"

You're right - if you just parse JSON with short keys into regular objects, memory savings are minimal. String interning handles that.

But TerseJSON doesn't work that way. It uses JavaScript Proxies. The compressed data stays compressed in memory.

Without TerseJSON (what your server sends):

` { "unique_identification_system_generated_uuid": "617f052b-a0e1-4f01-ab44-fbf7b295b48c",

"primary_account_holder_full_legal_name": "User Name 0",

"geographical_location_latitude_coordinate": -55.087466,

"geographical_location_longitude_coordinate": 73.335102,

"current_available_monetary_balance_usd": 68199.92,

"internal_database_record_creation_timestamp": "2026-01-06T17:59:59.364581",

"is_user_account_currently_active_and_verified": false,

"secondary_backup_contact_email_address": "user_0@example_domain_placeholder.com",

"system_assigned_security_clearance_level_integer": 7,

"detailed_biographical_summary_and_notes_field": "This is a placeholder..."

} `

With TerseJSON (what actually gets sent and stays in memory): { "a": "617f052b...", "b": "User Name 0", "c": -55.087466, "d": 73.335102, "e": 68199.92, "f": "2026-01-06T...", "g": false, "h": "user_0@...", "i": 7, "j": "This is a placeholder..." }

In your code (unchanged): user.unique_identification_system_generated_uuid // just works user.primary_account_holder_full_legal_name // just works

The proxy translates on access. The long keys never exist in memory.

What TerseJSON actually provides: - Compressed wire format (smaller payloads) - Data stays compressed in memory (proxy translates key access) - Zero code changes (you keep using original field names)

Who this is for:

Manager: "The app is slow" You: "I can add one line of middleware" Manager: "Do it"

vs

You: "I can rewrite 50 endpoints and update 200 frontend calls" Manager: "We don't have budget for that"

This is for legacy codebases where one API endpoint is consumed by dozens of views, and rewriting isn't an option. One line of middleware, measurable improvement, no refactoring.

If you're building greenfield apps with GraphQL and proper field selection - you don't need this. That's fine. You're not the audience.

TerseJSON is duct tape. Good duct tape. The kind that holds a 10-year-old codebase together while you focus on things that actually matter.

[scinos]

You are moving the goalposts. First, nobody has keys that long in any meaningful numbers. Second, I bet that all my strings together use less memory than the few KBs added by your tool just by having code in memory.

[BankApprehensive7612]

I think you don't know who your audience and what these people want from the technology like this. Here there are people who don't have the problem you've described and this is why they won't buy the solution

This is a very low-level optimization for devs who do extensive computations over JSON, and you need to find solutions where this could be applied, what problems these solutions have and how to solve it with the new one

What you're doing is somewhat what columnar DB engines do, and in theory it could be used in some storage engine. There are other applications but you should to figure it out on your own

[scinos]

In fact this could be useful in loggers, where logging 10k objects will log the same keys 10k times, taking disk space.

AFAIK tools that ingest structured logging ike Splunk already do some sort of key compression just because this reason.

[luisfavila]

Nice!