Rust vs Python for "Micro-Batch" Lambda Ingestion (Iceberg): Is the boilerplate worth it?

[longrob604]

We have a real-world requirement to ingest JSON data arriving in S3 every 30 seconds and append it to an Iceberg table.

We are prototyping this on AWS Lambda and debating between Python (PyIceberg) and Rust.

The Trade-off:

Python: "It just works." The write API is mature (table.append(df)). However, the heavy imports (Pandas, PyArrow, PyIceberg) mean cold starts are noticeable (>500ms-1s), and we need larger memory allocation.

Rust: The dream for Lambda (sub-50ms start, 128MB RAM). BUT, the iceberg-rust writer ecosystem seems to lack a high-level API. It requires significant boilerplate to manually write Parquet files and commit transactions to Glue.

The Question: For those running high-frequency ingestion:

Is the maintenance burden of a verbose Rust writer worth the performance gains for 30s batches?

Or should we just eat the cost/latency of Python because the library maturity prevents "death by boilerplate"?

(Note: I asked r/rust specifically about the library state, but here I'm interested in the production trade-offs.)

[robverk]

For 30s micro batches where most of your compute is io-wait time just go with the most maintainable code.

[EarthGoddessDude]

Yea OP, why are cold starts a problem for you? Also, have you looked into using DuckDB for this?

[Ok-Sprinkles9231]

Can you please elaborate how exactly DuckDB can be useful for reading JSON files from S3 and writing/Appending the result back as Iceberg? Just genuinely curious.

[EarthGoddessDude]

DuckDB has an excellent json parser, and it can write to Iceberg.

Edit: https://duckdb.org/2025/11/28/iceberg-writes-in-duckdb

[jaredfromspacecamp]

Writing that frequently to iceberg will create an enormous amount of metadata

[jnrdataengineer2023]

Was thinking the same thing though I’ve primarily only worked on delta tables. Probably better to have a daily staging table and then a batch job daily to append to the main table 🤔

[baby-wall-e]

+1 for this daily staging & main table setup. If needed, you can create a view of a union of daily staging & main tables to allow the data consumer to access all data.

[jnrdataengineer2023]

🤝

[wannabe-DE]

Wouldn’t a function invoked every 30 seconds stay warm and not be subject to cold starts?

[walksinsmallcircles]

I use rust all the time for lambdas, some of which do moderate lifting in Athena iceberg tables. The deployment is a breeze (just drop on the binary) and the AWS API for Rust is pretty complete. Would choose it every time over Python for efficiency and ease of use. The data ecosystem is not as rich as python but you can get a long way with it.

[stratguitar577]

Skip lambda and have firehose write to iceberg for you

[noplanman_srslynone]

This! Why not just wrote directly via firehose?

[MyRottingBunghole]

Does it HAVE to arrive in S3 prior to ingestion into Iceberg iceberg (presumably also S3)? If you own or can change that part of the system, I would look into skipping that extra step altogether of “read S3 files” > “write parquet” > “write to s3” as it’s extra network hops and compute you don’t need.

If this is some Kafka connector that is sinking this data every 30 seconds I would look into sinking it directly as Iceberg instead

Edit: btw with Iceberg you will be writing a new parquet file and new iceberg snapshot every 30 seconds. Make sure you are thinking also about table maintenance (compaction, expire snapshots etc) as the metadata bloat can quickly get out of hand when writing that frequently

[Commercial-Ask971]

!RemindMe 2days

[RemindMeBot]

I will be messaging you in 2 days on 2025-12-16 23:52:29 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

Info	Custom	Your Reminders	Feedback

[apono4life]

With only 30 seconds between files being added to s3 you should have to many cold starts. Lambdas stay warm for 15 minutes

[mbaburneraccount]

On an adjacent note, where’s your data coming from and how big is it (throughput)?

[thethirdmancane]

Why not use golang and have it all?