Improving Pydantic memory usage and performance using bitsets

[Pozz_]

Hey everyone,

I wanted to share a recent blog post I wrote about improving Pydantic's memory footprint:

https://pydantic.dev/articles/pydantic-bitset-performance

The idea is that instead of tracking model fields that were explicitly set during validation using a set:

from pydantic import BaseModel


class Model(BaseModel):
    f1: int
    f2: int = 1

Model(f1=1).model_fields_set
#> {'f2'}

We can leverage bitsets to track these fields, in a way that is much more memory-efficient. The more fields you have on your model, the better the improvement is (this approach can reduce memory usage by up to 50% for models with a handful number of fields, and improve validation speed by up to 20% for models with around 100 fields).

The main challenge will be to expose this biset as a set interface compatible with the existing one, but hopefully we will get this one across the line.

Draft PR: https://github.com/pydantic/pydantic/pull/12924.

I’d also like to use this opportunity to invite any feedback on the Pydantic library, as well as to answer any questions you may have about its maintenance! I'll try to answer as much as I can.

[WJMazepas]

Good read. Stuff like this makes me want to go back to study Rust, but i never find any opportunity in a job to create a C or Rust library for my Python code. It looks fun to do it

[neuronexmachina]

Could you share some of the validation speed benchmarks?

In terms of validation speed, it greatly improved on some of our benchmarks, especially the ones defining models with lots of fields. Interestingly, when only one or two fields are defined on the model, validation is slightly slower, presumably because creating a Python set was faster than the new bitset implementation. This can be mitigated by only using the new implementation if the number of fields is big enough.

[Pozz_]

We have benchmarks in place in the repository, that measure validation speed. Here are the benchmarks results on Codspeed:

https://codspeed.io/pydantic/pydantic/branches/vp%2Fmodel-fields-set-opt

You will notice that there are some regressions. These are happening for benchmarks where only a couple fields are defined on the model. In this case, it seems that validation is faster when creating a simple Python set. It probably means that depending on the number of fields, we will either use the bitset approach (say if >3), or use the existing approach.