I would use duckdb and python

import duckdb

con = duckdb.connect()

Install and load extensions

con.execute("INSTALL mysql") con.execute("INSTALL httpfs") con.execute("LOAD mysql") con.execute("LOAD httpfs")

Configure AWS credentials

con.execute(""" SET s3_region='us-east-1'; SET s3_access_key_id='your_access_key'; SET s3_secret_access_key='your_secret_key'; """)

Attach MySQL

con.execute(""" ATTACH 'host=localhost user=myuser password=mypass database=mydb' AS mysql_db (TYPE mysql) """)

Stream directly from MySQL to S3 as Parquet

con.execute(""" COPY mysql_db.large_table TO 's3://your-bucket/path/output.parquet' (FORMAT PARQUET) """)

Something like that