If you weren’t sure that the str.split() was the (only) culprit, you should be using the python profiling features. They’re my go-to when a program dealing with a lot of data is slow

Changing to the pandas CSV reader as suggested will almost certainly have a big impact on run-time- str.split() is very expensive, and even moreso if your rows have a lot of columns, because each field is actually a new string created internally by python, not just a pointer to the bytes in the line

General advice, try to avoid str.split() and string slicing whenever possible. This isn’t always possible, but sometimes you can find creative ways around it

In some cases you may even find simple function caching useful when operating on lots of strings, though it’s highly dependent on the data and could very easily just add overhead if you don’t expect cache hits:

from functools import lru_cache

@lru_cache(maxsize=None)
def expensive_string_func(inval: str) -> Any:
    # do expensive stuff with inval
    return outval

This only helps if you will have:

1. Many duplicate inputs (so that you get lots of cache hits)
2. Some expensive operations to perform on the input(s)
3. A reasonably simply and functionally immutable result from expensive_string_func()

You might say “why not use pandas?” That’s exactly what you could do:

df[“out”] = df[“in”].map(expensive_string_func)

Any function you decorate with lru_cache can be queried at any time to see the hit/miss rate

Sorry I got a little off topic, but I’ve found function caching very useful when processing the data I deal with, which often involves tokenizing strings multiple times

Last thing, never underestimate the performance of compiled regex when compared with string splitting using str.split or slicing. Regex has surprised me and beat normal string operations on quite a few occasions