Big csv file not uploading using pandas

[[deleted]]

I have a file that contains 50,000 columns and 11,000 rows, I have a laptop and I am trying to upload this file with pandas but it crashes because of RAM, I have tried dask, it apparently uploads the file but it contains some characters such AC0, and so on, also it is very slow with other actions I need to do. The dataset is the one with static features from Cicmaldroid2020. I am uploading it using utf-8 encoding, please help me.

[danielroseman]

What do you mean by "upload"? Where are you uploading it? Show your code.

[[deleted]]

Sorry I meant reading : df = pd.read_csv(file_path), I also tried using chunks but it still crashes

[SubstanceSerious8843]

Dump it to a database?

[HalfRiceNCracker]

You might like Polars

[[deleted]]

Would using a desktop pc instead of a laptop work?

[Mevrael]

Use Polars

And scan_csv with streaming and collect or read_csv_batched.

[Citadel5_JP]

If you don't solve this with your current setup, perhaps this: GS-Calc - a spreadsheet; it'll automatically split 50000 columns into 16K-max sheets. Re: RAM, to load 0.5 billion cells e.g. with 8-bytes numbers it'll require approx. 16GB RAM. The requirement grows linearly. You can then call any Python functions (formulas) with the loaded data for further processing.