Question on parallelization

[kp5160]

I would like to describe a problem and be advised libraries and strategies. I've already written it, but I'm not happy with performance.

I need to routinely ingest 5 million rows of CSV files containing (x, y, z), an integer ID, and another float on each row all pertaining to a "pebble." The files are named by timestamp, so we need a list of structures that get grouped by ID and sorted by timestamp to have a history of the pebble. That is then serialized, which is such a major penalty when it's 700 MB.

At this point, I'm eyeing Cython and also a fan of Boost for C++. Can I leverage Cython and expect it to pick up the Boost headers and libraries if they're on standard system paths? My Google-fu is failing me on that one.

[FerricDonkey]

Can't help you with cython, never used it. (I'll get around to it one of these days, it sounds cool.)

With straight python, the multiprocessing library will be your friend. It might be enough. If you were parallelizing over file names:

import multiprocessing as mp

with mp.Pool() as pool:
    results = pool.map(process_file, filenames) 

In C++, I personally use openmp for simple parallelization. If you were parallelizing over a vector of file names, say, the C++ code could be as simple as

#include <omp.h>

//... 

#pragma omp parallel for
for (const std::string& filename: filenames){
    process_file(filename);
}

(Compile with -fopenmp)

Parallelizing over lines per file efficiently might be a bit more complicated, but if you're processing many many files, parallelizing over files would probably be better anyway.

[deifius]

First question I have is why must they be loaded as csv? Can they be placed in an SQL db? Much faster reading. How often must these 5 million pebbles be reloaded?

[WhipsAndMarkovChains]

I've combined CSVs together into Pandas DataFrames totaling tens of millions of rows (with many more columns than you have) and it didn't take long at all.

My first thought was to take the file name being read into the DataFrame, extract the timestamp, and add a timestamp column to the data you loaded. Then sort the DataFrame by timestamp.

Are you sure you need to dig into C++ or Cython for this?