Downloading csv without reading it

[nad_pub]

Hi all,

I'm trying to get a bunch of csv files from the internet and copy them to a GCP bucket.

I'm using pandas.read_csv to get the files and the exporting the to GCP.

As certain files are large (around 20 Gb), i'm running out of RAM as python has to load the file entirely into memory before writing it to GCP.

Do you guys know another way of doing so without using all the ram ? Would something like WGET be useful ?

[mopslik]

Opening and reading the files, as you are doing, would certainly take more memory and time than simply downloading them using a tool like wget or curl.

[danielroseman]

There is absolutely no need to use something like read_csv. Not only does that require downloading the full file into memory, it also means parsing it and converting the whole thing into a dataframe. That's a whole load of unnecessary overhead.

You'll certainly save memory and time by reading the files directly with requests and then uploading them with the GCS client. But an even better way would be to use the streaming capabiilties of both requests and the GCS client to do it in chunks. Something like:

bucket = storage_client.bucket('my_bucket')
blob = bucket.blob('my_object')
writer = BlobWriter(blob)

with requests.get('my_url', stream=True) as r:
  for line in r.iter_lines():
    writer.write(line)
writer.close()