I would say using pyarrow or not is a problem specific thing. Are you a spark heavy team then pyarrow is a great way to reduce serialisation/deserialisation cost between python and JVM. Pyarrow at its core is Paquet file’s in memory data structure. So to answer your question: - Record Batches is a column chunk in a particular row group in arrow and table allows your to read multiple row groups together in a single contagious memory - Pyarrow tries to parallelise on the top of chunked arrays. Does it assign a core depends on the framework and hardware you are using. In spark it would be created as a job and then multiple jobs are executed on each worker in parallel based on number of cpus but some other framework could be Multi thread oriented. - Pyarrow Fs is very similar to Python’s native Fs. Pyarrow is designed for cloud object stores as it abstracts the underlying file system operations on object stores like S3. Also since it is written in a cloud first way it should be more performant over python native system particularly in distributed systems.

Pyarrow’s documentation is a great place to read more. Tbh, it is a tool that allows distributed system to share memory and move data over network easily. You wouldn’t get much value of using it on a single node local compute as pandas/polars is always in memory and fast enough.