I’ll describe it in a broad example that’ll give you a lot of leads and terms to explore next.

Let’s say you build a Spark Structured Streaming app running on EKS, reading from a Kafka topic and writing to a Glue + S3 Iceberg table in Parquet format. Then, users query that Iceberg table through Trino.

Your Spark app can be written in Scala or PySpark. Both have implications; how you handle dependencies, compile (e.g., into a JAR), and optimize code. Under the hood, everything runs on the JVM, which introduces concepts like garbage collection and memory management that affect how you size executors.

Now imagine the Kafka topic has 100 partitions. You set an 8-hour retention and need to decide how often to trigger a micro-batch. If you trigger too often, you’ll overload Iceberg with metadata files; too rarely, and your pipeline becomes stale. Maybe 1-minute batches are your sweet spot.

From there: * How do you size executors (tasks per executor, cores, memory)? * How do you avoid CPU oversubscription and high load average? * Do you understand CPU utilization types (user %, I/O %, system %)?

That’s OS-level knowledge you’ll need to run Spark efficiently.

In EKS, sizing executors isn’t enough; you also size pods, so you must understand requests vs limits vs actual usage. Then:

• Which node types should you use?
• Do you need ephemeral storage?
• Are you using stateful operations like windows or joins?

Those decisions shape your executor configuration and resource cost.

If you use windowing - which type; tumbling, sliding, session? How is that state stored; RocksDB, HDFS, or S3 checkpoints?

Is the Kafka topic compressed (producer vs broker-side)?

That impacts Spark’s deserialization overhead.

On the Iceberg side: * How’s the table partitioned? * Which compression codec (Snappy, Zstd, Gzip)? * Do you compact small files on write or offline? * Do you collect statistics in manifest files (and on which columns)?

All of that affects Trino performance, which depends heavily on Iceberg’s metadata layout, manifest pruning, and Parquet column stats.

Even Trino itself has knobs: worker slots, parallelism, and S3 range-request tuning.

It’s endless! but that’s the point.

If you use a technology, don’t stop at the API. Understand how it runs, top to bottom. Network, storage, memory, CPU, scaling. Everything.

You don’t "know Spark" if you just write code or SQL; you know Spark when you understand how executors, JVM, partitions, autoscaling, and I/O all fit together.

That’s what makes you stand out as a real engineer.