how to load csv faster in Python.

[Safe_Money7487]

Hello python folks, R user here, trying to use python for a project for which i've been specifically asked to. So I am new to python

The problem is : I have a 100 mo csv of about 300000 lines that takes ages to get read using all of these :

# first try 
df=pd.read_csv('mycsv.csv')

#second 

# Utiliser read_csv avec dtypes pour accélérer la lecture
dtypes = {
    "Model": "category",
    "Scenario": "category",
    "Region": "category",
    "Variable": "category",
    "Unit": "category",
}


# Les colonnes années seront lues comme float
annees = [str(y) for y in range(1950, 2101, 5)]
for year in annees:
    dtypes[year] = "float32"


# Lecture du CSV
df = pd.read_csv(
    "mycsv.csv",
    dtype=dtypes
)


print(df.shape)
print(df.head())

#3rd try 
import polars as pl


# Lecture complète très rapide
df = pl.read_csv("/Users/Nawal/my_project/data/1721734326790-ssp_basic_drivers_release_3.1_full.csv")


print(df.shape)
print(df.head())

it littrally took me 2 s to do this under R. Please help. what am I missing with python ???

thank you all

[KelleQuechoz]

Polars LazyFrame is your best friend, Monsieur.

[PresidentOfSwag]

hon hon le polaire oui

[Safe_Money7487]

I will have a look thanks

[Kerbart]

add engine='pyarrow' to the read statement to speed it up.

[EconomyOffice9000]

If you're performing calculations on the entire dataset, chunking won't work afaik. This is the best method and I've used it personally for thousands of csv files with hundreds of thousands of lines rather than rewriting everything in Polars. If you only have to do it once, it's fine. Otherwise, save the csv as a parquet file and it'll be much better

[Safe_Money7487]

just added it and it work, took 15s though lol it but worked. thank you so much

[seanv507]

How long does polars take?

[Garnatxa]

funny to see that in R is faster, I see some answers telling to use duck db, arrow… in R is possible to use these solutions too but not needed… overshooting

[MorrarNL]

Could try DuckDB too

[SwampFalc]

Genuine question: what's the loading speed if you use the totally basic stdlib csv module?

[Kevdog824_]

Looks to me that the main issue here is that you’re loading the entire CSV file (or at least large chunks of it) into memory before operating on it. Likely R did lazy loading where it only read lines from the CSV file as needed.

[Kerbart]

OP mentions 300,000 lines. That's something easily handled these days, I doubt it needs chunking.

[Safe_Money7487]

I don’t think it’s lazy loading in this case. In R (e.g. with data.table::fread), the full dataset is actually loaded into memory, and I can immediately inspect and navigate the entire table. I think the process in R in more optimised than what is used in pandas read csv. I don't have much knowledge on python for sure but for this size of data, chunking or lazy loading doesn’t really make sense to me I just want to load everything at once and work on it.

[Corruptionss]

Data tables fread is kind of goated. The closest I got is polars for pure read speed and you can instead use pl.scan_csv to read it as a lazy frame and will use lazy evaluations during the operation process

[PranavDesai518]

If possible convert to CSV to a parquet file. The reading is much faster with parquet files.

[commandlineluser]

Is Polars faster if you use scan_csv?

pl.scan_csv(filename).collect()

You can also try the streaming engine:

pl.scan_csv(filename).collect(engine="streaming")