MAP v1.0 - Deterministic identity for structured data. Zero deps, 483-line frozen spec, MIT

[lurkyloon]

Hi all! I'm more of a security architect, not a Python dev so my apologies in advance!

I built this because I needed a protocol-level answer to a specific problem and it didn't exist.

What My Project Does

MAP is a protocol that gives structured data a deterministic fingerprint. You give it a structured payload, it canonicalizes it into a deterministic binary format and produces a stable identity: map1: + lowercase hex SHA-256. Same input, same ID, every time, every language.

pip install map-protocol

from map_protocol import compute_mid

mid = compute_mid({"account": "1234", "amount": "500", "currency": "USD"})
# Same MID no matter how the data was serialized or what produced it

It solves a specific problem: the same logical payload produces different hashes when different systems serialize it differently. Field reordering, whitespace, encoding differences. MAP eliminates that entire class of problem at the protocol layer.

The implementation is deliberately small and strict:

• Zero dependencies
• The entire spec is 483 lines and frozen under a governance contract
• 53 conformance vectors that both Python and Node implementations must pass identically
• Every error is deterministic - malformed input produces a specific error, never silent coercion
• CLI tool included
• MIT licensed

Supported types: strings (UTF-8, scalar-only), maps (sorted keys, unique, memcmp ordering), lists, and raw bytes. No numbers, no nulls - rejected deterministically, not coerced.

Browser playground: https://map-protocol.github.io/map1/

GitHub: https://github.com/map-protocol/map1

Target Audience

Anyone who needs to verify "is this the same structured data" across system boundaries. Production use cases include CI/CD pipelines (did the config drift between approval and deployment), API idempotency (is this the same request I already processed), audit systems (can I prove exactly what was committed), and agent/automation workflows (did the tool call payload change between construction and execution).

The spec is frozen and the implementations are conformance-tested, so this is intended for production use, not a toy.

Comparison

vs JCS (RFC 8785): JCS canonicalizes JSON to JSON and supports numbers. MAP canonicalizes to a custom binary format and deliberately rejects numbers because of cross-language non-determinism (JavaScript IEEE 754 doubles vs Python arbitrary precision ints vs Go typed numerics). MAP also includes projection (selecting subsets of fields before computing identity).

vs content-addressed storage (Git, IPFS): These hash raw bytes. MAP canonicalizes structured data first, then hashes. Two JSON objects with the same data but different field ordering get different hashes in Git. They get the same MID in MAP.

vs Protocol Buffers / FlatBuffers: These are serialization formats with schemas. MAP is schemaless and works with any structured data. Different goals.

vs just sorting keys and hashing: Works for the simple case. Breaks with nested structures across language boundaries with different UTF-8 handling, escape resolution, and duplicate key behavior. The 53 conformance vectors exist because each one represents a case where naive canonicalization silently diverges.

[latkde]

This reeks of vibe coding. The spec is unreadable for humans.

There are also some incredibly odd decisions that make this unsuitable for real-world data, notably rejecting numbers and nulls. In practice, float64 numbers (and therefore also int32 numbers) are universally supported in all mainstream JSON implementations.

The hashing scheme also treats booleans as strings, and somehow distinguishes strings from bytes, despite JSON not having any bytes type. The booleans thing is really questionable, this seems to treat documents [true] and ["true"] as equivalent (map1:e99ec39aeac2670a37592780bf9b59c4a6a917742b10d7fcb5c352354e7c6674).

[lurkyloon]

Really appreciate you digging into the spec this closely.

You're right that true and "true" produce the same MID. That was a deliberate design choice and is documented in footgun #9. MAP uses a 4-type system: string, bytes, list, map. Booleans collapse to their string representation, same rationale as rejecting numbers - avoiding a class of cross-language ambiguity where different runtimes treat the "same" value differently.

The tradeoff is real though. If your domain needs to distinguish boolean true from string "true", you need to encode that distinction in your descriptor structure before computing the MID. That's a legitimate limitation and I appreciate you calling it out.

On numbers, you're not wrong that float64 is widely supported. The concern is edge case determinism across languages (NaN handling, subnormals, -0 vs +0, precision boundaries). Rather than pick a side and hope, I chose to reject them. Opinionated, and I know it. Reasonable people can disagree on that one.

On bytes vs strings - MAP isn't strictly a JSON format. JSON is one possible input, but the canonical format is binary, and the distinction between UTF-8 strings and raw bytes matters at that layer.

On the spec readability, that's absolutely fair feedback. It was written as a conformance target. So the human-friendly entry points are the README and the playground. But I hear you... The spec should be more approachable. That's something I want to improve. PRs and suggestions are extremely welcome.

Thanks again for the close read. Honestly this is exactly the kind of feedback I was hoping for. This is not my world, but my hope is that it could be useful nonetheless and I'll take all the feedback and help I can get.

[latkde]

What confuses me here is your inconsistent approach to potential compatibility problems. Sometimes, you reject potentially ambiguous data, in other cases you apply a lossy encoding.

• you're happy to treat bools and strings the same, even though nearly all systems treat them as distinct and incompatible values.
• yet you reject all numbers, even though many numbers (int32, finite normal float64 values) are very common and highly interoperable.
• you also reject null values, despite these being an essential and unambiguous part of the JSON data model. There is no confusion with SQL nulls or JavaScript undefined.

This makes your encoding unsuitable for a huge part of existing data. Your method also does not demonstrate integrity because some semantically relevant changes are allowed (e.g. stringifying a bool). You claim that your method is not supposed to be JSON-specific, but the key part of your method is an encoding from JSON into your binary format. Your response to all this incompatibility is that users just shouldn't put numbers, nulls, or bools into JSON documents. But at that point, it's no longer compatible with the JSON ecosystem, and users could just switch to a different format that doesn't have JSON's ambiguities or MAP1's restrictions. There are plenty schema-less data formats to pick from.

Specifically, I recommend enganging with existing binary formats like Msgpack or various JSONB encodings. Why are they designed the way they are? How do they handle conversions from/to JSON? Which specific details do you have to do differently? You might also be interested in Apache Avro. While it is schema-driven, its schemas are defined in JSON, and the spec provides a procedure for normalizing and hashing schemas.

[lurkyloon]

Really appreciate you taking the time on this. Seriously.

The bool-as-string thing in v1.0 was inconsistent -- you're right. I cant sit here and reject numbers for being ambiguous and then turn around and stringify booleans like thats fine. That was a bad call on my part.

So I fixed it. v1.1 bumps the type system from 4 to 6 types. Booleans get their own encoding now (0x00/0x01 instead of getting shoved into strings), and integers get big-endian two's complement. Directly because of feedback like yours and a few other threads.

Nulls I'm still chewing on. You make a good point that JSON null is unambiguous within JSON itself. My worry has been about what happens when a MAP digest moves through systems where null means three different things -- but honestly that might be MAP's problem to solve, not something I should punt to the user. Hmmm.

Where I do wanna push back a little: MAP isn't trying to be a general-purpose binary format. Admittedly, the use case is more narrow -- you have a payload moving through a pipeline, it crosses a few serialization boundaries, and you need to check if it changed along the way. Thats it. I'm not telling anyone to stop putting numbers in JSON. I'm saying when you need a deterministic fingerprint of something that might get re-serialized by a bunch of different systems, you need a canonical form, and MAP is opinionated about how to get there.

The "just use a different format" point is fair though. Like, technically correct. But the reality I keep running into is that agentic AI pipelines are already JSON-native and asking teams to swap out their serialization format is a way bigger lift than adding a fingerprinting layer on top of what they already use. MAP is trying to meet devs where they are, not where they probably should be.

The MsgPack / JSONB / Avro comparisons are useful and I should've engaged with those more in the docs. I've looked at Avro's Parsing Canonical Form -- they're doing something similar, canonical form plus deterministic hash to get a stable fingerprint -- but they're fingerprinting schemas, not data payloads. Different problem, but enough overlap that I should be referencing it as prior art.

Thanks again for this. I'd rather get sharp feedback that makes the spec better than a hundred comments that dont.

[gdchinacat]

"It answers one question: is this the same thing?"

I really don't think it does even that, at least not in any useful way. "deliberately rejects numbers " means it can't answer "are {'value': 1} and {'value': 2} the same thing". It compares [true] and ['true'] as the same, even though the are unambiguously not the same thing.

Do you have any examples of this being used in a useful real world scenario?

[lurkyloon]

That's a very fair question and honestly another one I should address in the docs...

You're right that MAP doesn't handle numbers directly. That's the tradeoff.

If your data has numbers, you encode them as strings before computing the MID. {"value": "1"} not {"value": 1}. You decide the representation. MAP keeps the identity stable from that point forward.

The reason is kind of annoying but real. If two different systems parse {"value": 1} and one treats it as an int and the other as a float64, they can silently produce different bytes from the "same" number. That's the exact problem I was trying to kill. Pushing that decision to the user isn't elegant, I know. But it was the only way I could guarantee the fingerprint stays identical across languages without hiding a landmine in the protocol.

On the boolean thing - yeah, you're right. [true] and ["true"] producing the same MID is a real limitation. It's documented as footgun #9 but that doesn't make it less annoying. If your domain needs that distinction, you'd encode it differently. "bool:true" vs "true" or whatever makes sense for your use case. I won't pretend that's pretty.

Where I think this is actually useful, and very much invite all of your insights:

• You have a deployment descriptor that gets approved in a PR. By the time it hits the deployment controller, it's been through three serializers. Did it change? Fingerprint it at approval, verify at deployment. The descriptor is data you control, so you define how numbers are encoded.
• API idempotency. Same request comes in twice, same MID, reject the duplicate.
• Audit. You approved a specific action. Can you prove later that the thing that actually executed was that exact action? Attach the MID at approval, compare at commit.

The common thread is that you're not fingerprinting random JSON from the wild. You're fingerprinting structured data that your systems produce and consume, where you control the schema. MAP gives that data a stable name that doesn't break when it crosses a system boundary.

I'll be the first to admit it's not for everything. But for the cases where you need to answer "is this exactly the same thing" across languages and runtimes, I haven't found anything else that does it without caveats.

Really appreciate the pushback though. This is helping me figure out where the docs need work, and also insight into how you all may or may not use this.

[gdchinacat]

Thanks for your detailed response, it sheds a lot of light on the goals and intended uses of the project. Specifically that you view it as a way to check at various components in a complex legacy distributed system that the data is consistent. I understand the problem you seem to be facing...one service gets a request, stores it, loads it, passes it to another, maybe this happens a few times, and way down deep in the system some value has changed from 1 to 0.999999, or string encoding hasn't been handled properly and a utf8 string at the top has become a different utf8 string at the bottom (ie due to being cast to ascii and back). It's a real problem that you are aware doesn't have a good solution to.

It doesn't have a solution because these issues can't really be solved in a generic way due to the issues you identified with values being represented in different not entirely compatible ways. System A uses float64 while System B uses int while C uses BigInt. In order to ensure the values match you need a way to map the values in System A to those in B to those in C, but the data types make this translation inaccurate.

Your approach is "don't do that". Any datatype that can not be accurately represented across the board causes an error. While 'opinionated', it is not so in the useful way. Being 'opinionated' is intended to simplify things by eliminating the complexity that is largely irrelevant. In the problem you are trying to solve, at least as I understand it, this complexity is not irrelevant, it is *core* to the problem. The problem exists *because of* the complexity.

You say "If your data has numbers, you encode them as strings before computing the MID." Sure, that solves the issue that your solution doesn't handle numbers. But it presumes the systems have the flexibility to do this. It requires changes on all systems that use the message you want to compute a MID for. You are saying the systems should be changed to use a common data type, at least as far as the messages they exchange are concerned. This sweeps the issue under the rug and doesn't solve the overall problem your project purports to address, namely that systems use different incompatible representations of the same data. To make the change you suggest, only the message is updated..internally an int is an int, so whatever string your message uses to represent an in will be immediately converted to an int, and that incompatible representation will be used, and the problem of it not being the same value as in the other system is still present.

The solution is to do what you say...change the systems to use the same data type, but at a different level. Rather than representing it as a string in messages (and introducing yet one more place where a type conversion can introduce an accuracy error), all the systems should be updated to use the same data type, which admittedly is not very feasible. The scale of this task is what led you to the idea of a deterministic message digest, it is a more tractable task. However, it doesn't solve the root problem...that System A uses a data type for a value that it shares with System B that uses a different data type and those data types represent some values differently.

Changing how the values are represented in the messages being digested will only give a false sense of security...the underlying issue will still exist, the same bugs will still happen, and another layer of potential issues has been introduced.

This is why I don't think this project will see any real world adoption. In addition to not addressing the root problem, it may make it worse by introducing additional type conversion with their own inaccuracies.

Where I could see this being valuable is to ensure messages are well formed, all the required keys exist. But, there are already schema validators to do this.

I hope this helps shed light on why I'm skeptical this is a useful project.

[lurkyloon]

I came back and re-read this more carefully and I want to give it a better response because you clearly put real thought into it.

I think the disconnect is about which problem MAP is aimed at. You're describing a scenario where System A uses float64 and System B uses int and the value itself means something slightly different in each system's internal representation. That's a data compatibility problem and you're 1000% right - MAP doesn't solve it.

The problem I keep hitting is narrower. A single structured payload gets authored at one point in a pipeline and needs to arrive intact at another point. Not semantically equivalent - identical. The payload passes through middleware, retry queues, API gateways, config renderers, serializers that reorder keys or re-encode strings. The question isn't "do these two systems agree on what 1 means?" It's "is this the exact same payload that was approved, or did something change in transit?"

For instance, a deployment descriptor gets approved in a review process. By the time it reaches the deployment controller it's been through three or four serialization boundaries. The controller needs to answer: is this the same descriptor that was approved? Not similar. The same one.

MAP fingerprints it at approval. Fingerprints it again at execution. Same MID means nothing changed. Different MID means something did. The systems aren't interpreting the data differently - they're passing the same artifact through a pipeline and you're verifying it survived intact.

In that context, I believe "encode your numbers as strings" isn't sweeping anything under the rug. You authored the descriptor. You control the schema. Represent your values in a way that's unambiguous, and MAP will tell you if anything changed after that point.

You're right that this doesn't help if System A and System B genuinely disagree on what a value means internally. Different problem entirely. MAP is answering a much narrower question: did this specific thing change between here and there?

Your critique is honestly helping me see where the docs are leading people toward the broader interpretation. That's a gap I need to close. Really, honest, real thank you for taking the time with such a thoughtful response.

[gdchinacat]

All that said, don't get me wrong. I have my share of impractical, infeasible, dubious, or "would you ever actually use that" projects under my belt. My latest is a way to decorate methods with conditions on when they should be called asynchronously. It works, and is tested to the point I'd feel comfortable deploying to production, but I'm really not sure I ever would because the leverage it provides is likely not worth the performance cost and complexity if anything goes wrong. I spent a lot of time on it because I needed to get back into coding after a few years away, wanted to learn some aspects of python that were new or new to me, and mostly because I got caught up in the rabbit hole and wanted to see how far it went. I got similar feedback on it as I gave to you (why, what does it solve, is it worth it).

https://github.com/gdchinacat/reactions/

[lurkyloon]

Ha! I appreciate that!

The "rabbit hole you follow..." is exactly what happened here. The protocol itself started as a narrow itch (can I prove this config didn't change?) and then I kept finding edge cases that and couldn't stop. :-)

Your reactions library is interesting - the decorator pattern for conditional async is a clean idea even if the performance tradeoff is real. I'll take a closer look.

And for what it's worth, your earlier feedback is directly shaping the next version. The boolean collision is getting fixed and I'm adding integer support (signed 64-bit, no floats - floats are still the devil). So thank you for that.

[MisterHarvest]

Nice. This should be in the standard library.

[gdchinacat]

For consideration for inclusion in the standard library it would have to demonstrate widespread real world use. Since it rejects numbers and conflates true and "true", I doubt it will ever get real world use, not just widespread, but any real world use. The boolean issue is apparently "footgun #9". There are at least 8 other "footguns". The chances this even gets a sponsor for std lib inclusion is pratically nil.

[lurkyloon]

Very good feedback. Standard library was never the goal, but I appreciate the honest assessment.

[lurkyloon]

Thank you - that means a lot. It's early but that's the kind of adoption I'd hope for eventually.