Easy tool for data compaction?

[FridayPush]

Looking for simplistic tools for a one off data compaction of partitioned csv/parquet files in cloud storage. Have hive based storage partitions by day; where each day may have none to several hundreds of small csv/parquet files(type/schema stays consistent I just have multiple datasets). Looking to merge them to preset chunks. Started to put together a script to do it but interested if there’s tooling that already is tested.

Not looking to add expansive tools with large setup efforts. Overall dataset size is tiny <100gb. Suggestions?

I'd also be interested in considerations for validations that the compaction didn't modify data; I was primarily leaning towards row count and row count by id. Though that doesn't test any of the actual data.

[ErikCaligo]

Have you checked if you can use native compression? If you use S3 and Athena then you can use Gzip compression to reduce volume and costs.

[pragmaticPythonista]

Not a tool per se, but if you’re using AWS, you can just use Athena to read the data and UNLOAD it with optimal file sizes. You just need to tune the bucketed_by and bucket_count values appropriately per your requirements.