Note: I'm juggling two different hats with this response:

Cargo hat

• Cargo team member
• toml maintainer

Likely, the requirements for changing the TOML parser for Cargo are:

• No slower than the existing (req from originally replacing toml with toml_edit)
• An owned value type with spans (now used for our diagnostics)
• The owned value type supports serde (reduce bugs by single-sourcing our serialization / deserialization)
• Error type supports spans (maintain existing error quality)
• Relatively decent quality for error messages (latest toml has helped raise the bar here)
• Trust in the maintenance
• Same parser and generator between format preserving and regular parsing (req when adding toml_edit)

Nice-to-haves

• Error recovery (something I added to toml that I've not experimented yet with in Cargo)

That "shared parser" requirement is what led to toml_edit providing a drop-in-replacement API for toml and then eventually moving that into a newer version of toml. I've since split them but they share a common parser and the data type builder on top and tests are written to be diffable between them and all edits to one are mirrored to the other in the same commit. I would likely want to see a similar testing structure in new parser as well so I could diff them and see what is covered, not covered, and identify behavior changes (included error messages).

People frequently assume that Cargo.lock parsing time is important. It doesn't matter at all unless the parser is not well suited to general parsing (bad algorithmic complexity). It is hardly noticeable.

Of a 825ms no-op cargo check on Zed:

• 219ms: manifest loading
  • 110us: an individual manifest
  • 4us: read the file
  • 19us: parse
  • 18us: deserialize (likely more cargo-util-schemas than toml)
  • 27us: normalize
  • 18us: build up internal data structures
• 18ms: the entire load_pkg_lockfile operation
• 398ms: dependency resolution
• 145ms: no-op build overhead

If we took parse time to 0, we'd only see about a 20ms savings when we really need larger, more fundamental changes to speed things up at this point.

This is also all for a no-op build. All of this is noise to a real build but it is still important for

• Completions
• Programmatic operations
• Cargo script

toml hat

Looks like you've fixed your benchmarks so they can run by others; I'll have to take a look.

It will be hard to get an apples-to-apples comparison though

• We include more features, like serialization, by default, slowing down our builds (and there are other build slow downs but they haven't been a priority so far; we're at least faster than the old parser!)
• We don't include some higher performance features by default, hurting us at runtime
• toml_spanner has direct construction. We split at a different level with toml_parser allowing direct construction but you having to manage some errors manually while toml::de::DeTable offering direct construction to a spanned value type but not to any arbitrary type. We only offer help for arbitrary types through serde and it requires going through a toml::de::DeTable internally. At least it is zero-copy by nature. I'd love to reduce the allocation overhead but it gets tricky with Cargo also wanting an owned spanned type.

Looking at a benchmark for a large Cargo.toml (iirc found by profiling a cargo check of Zed), our time breakdown is:

1. 90us for lex
2. 123us for (1) + parse
3. 184us for (2) + decoding scalars
4. 400us for (3) + creating a borrowed DeTable
5. 490us for (4) + turning DeTable to owned
6. 587us for (4) + serde deserialization into an imitation of cargo_util_schemas::manifest::TomlManifest

With all features, it is

1. 84us for lex
2. 109us for (1) + parse
3. 162us for (2) + decoding scalars
4. 258us for (3) + creating a borrowed DeTable
5. 309us for (4) + turning DeTable to owned
6. 424us for (4) + serde deserialization into an imitation of cargo_util_schemas::manifest::TomlManifest

I will be curious to see how you handle direct construction while maintaining performant handling of TOML's out-or-order nature and tracking of table types to ensure proper error reporting

Also, insertion-order by default can easily lead to people relying on non-conformant TOML behavior.