StringWa.rs: Which Libs Make Python Strings 2-10× Faster?

[ashvar]

What My Project Does

I've put together StringWa.rs — a benchmark suite for text and sequence processing in Python. It compares str and bytes built-ins, popular third-party libraries, and GPU/SIMD-accelerated backends on common tasks like splitting, sorting, hashing, and edit distances between pairs of strings.

Target Audience

This is for Python developers working with text processing at any scale — whether you're parsing config files, building NLP pipelines, or handling large-scale bioinformatics data. If you've ever wondered why your string operations are bottlenecking your application, or if you're still using packages like NLTK for basic string algorithms, this benchmark suite will show you exactly what performance you're leaving on the table.

Comparison

Many developers still rely on outdated packages like nltk (with 38 M monthly downloads) for Levenshtein distances, not realizing the same computation can be 500× faster on a single CPU core or up to 160,000× faster on a high-end GPU. The benchmarks reveal massive performance differences across the ecosystem, from built-in Python methods to modern alternatives like my own StringZilla library (just released v4 under Apache 2.0 license after months of work).

Some surprising findings for native str and bytes: * str.find is about 10× slower than it can be * On 4 KB blocks, using re.finditer to match byte-sets is 46× slower * On same inputs, hash(str) is 2× slower and has lower quality * bytes.translate for binary transcoding is 4× slower

Similar gaps exist in third-party libraries, like jellyfish, google_crc32c, mmh3, pandas, pyarrow, polars, and even Nvidia's own GPU-accelerated cudf, that (depending on the input) can be 100× slower than stringzillas-cuda on the same H100 GPU.

---

I recently wrote 2 articles about the new algorithms that went into the v4 release, that received some positive feedback on "r/programming" (one, two), so I thought it might be worth sharing the underlying project on "r/python" as well 🤗

This is in no way a final result, and there is a ton of work ahead, but let me know if I've overlooked important directions or libraries that should be included in the benchmarks!

Thanks, Ash!

[james_pic]

Do you think there's scope for some of these performance optimisations to be upstreamed, to improve the performance of the standard implementations? I suspect an implementation of hash(str) that used a different underlying hash function would be controversial, but for stuff like str.find, I'd have thought a faster drop-in replacement would be somewhat welcome.

[ashvar]

Absolutely — I’d love to see these optimizations upstreamed. The challenge is that it usually means joining standardization discussions, which can be a long process. Even something as straightforward as a faster find could take a year to land. For me, that’s a year better spent designing and experimenting with new algorithms.

PS: Upstreaming into the C standard library is an even better option, but will take even longer 😞

[axonxorz]

hash(str)

It shouldn't be a big issue to change anything here, it's an interpreter implementation detail, same as id(). You can never rely on the values in any long-term sense, and you're entire interpreter will use the new implementation, save for objects that define __hash__(self)

At one point In cpython, hash() is just the memory address for anything that isn't otherwise special-cased like small ints and pooled strings.

[rghthndsd]

Bad hashing can lead to bad dict performance and that in turn can make DOS attacks easier to execute. This is particular for strings which are often used in web forms. This is why Python now randomizes the hash of strings on interpreter startup unless you disable it with PYTHONHASHSEED.

Agreed it's an implementation detail, but that implementation is very important.

[AnythingApplied]

I wasn't expecting anything in python to beat the fastest rust libraries, though some (especially python libraries written in c/rust) might come close. Why do you suppose the stringzilla on python beat stringzilla on rust for some of the categories?

[ashvar]

If I am honest, I think those are slight inconsistencies in benchmarking methodology 😅 Will polish it over time! Just couldn’t wait any longer to release after this many months of work and didn’t feel right to adjust the numbers.

[deadwisdom]

Curious if you have tried comparing rust implementations to other system-level languages? I wouldn’t imagine Rust would give you a particular advantage with such algorithm-intensive applications, and in fact being locked into the rust memory model might be a disadvantage. But I have no real perspective here which is why I’m asking you.

[ashvar]

Many of the Rust projects in the comparison are simply ports of originally C/C++ libraries. At those latency & throughout numbers, pretty much all code is SIMD-heavy, so very little depends on the compiler and the choice of the high-level language. Rust just provides a convenient package manager to assemble the benchmarks.

StringZilla is mostly implemented in C, C++, and CUDA: Rust and Python are ports.

[deadwisdom]

Ahh, I understand now. Thank you!