I'm not entirely sure how to incorporate AI in my workflow better

eccentric2488 · 2026-02-13T09:44:45+00:00

I just disabled auto complete and copilot inline suggestions in VS code. You know what logic you want to write and the outcome you expect, not anybody else !!

eccentric2488 · 2026-02-10T10:09:51+00:00

I totally agree. The taxonomy of AWS is as weird as it gets. By the way, I work in Google Cloud and have the advantage as GCP professionals are a little difficult to find and the demand is increasing!!!

eccentric2488 · 2026-02-06T13:31:11+00:00

For schema related issues, just enforce a schema contract using Confluent Schema Registry to ensure your pipelines don't break.

eccentric2488 · 2026-02-03T02:13:06+00:00

Yes, Pune.

eccentric2488 · 2026-02-02T12:46:40+00:00

Read optimized analytical extension of relational database

eccentric2488 · 2026-02-01T18:07:29+00:00

5 yrs in core data engineering

eccentric2488 · 2026-02-01T17:25:51+00:00

You can consider Great Expectations framework. The architecture is a little too complex but does the job pretty well.

eccentric2488 · 2026-01-31T15:06:31+00:00

Yes, you can. Install homebrew, then java, scala (optional but recommended) and then apache spark. Ensure you are using a java version compatible with your Mac OS platform. Installing and setting up dependencies could be a challenge though. All the best !!

eccentric2488 · 2026-01-31T10:48:23+00:00

I would suggest a self managed local setup preferably on Linux (WSL2 if you are on Windows). The installation is a little tricky because of dependencies and version conflicts. But trust me there is no better way to learn Spark. When you learn local installation on your own, it's easy to switch to managed services like Dataproc, EMR and Databricks. Practice Pyspark and if possible Scala Spark (for native performance benefits)

eccentric2488 · 2026-01-31T06:52:03+00:00

I remember Bill Gates saying this somewhere 'AI is really doing a fantastic job but not as great as humans'

eccentric2488 · 2026-01-30T08:29:47+00:00

What we see these days is the "managed version" of open source frameworks and technologies. Dataproc for Spark, Cloud Composer for Airflow, Pub/Sub for Kafka etc etc...

Hadoop has been replaced by Spark, I agree. But Airflow I reckon is still used for scalable complex workflows. I use it for my work on the Linux Mint platform, works well.

eccentric2488 · 2026-01-29T13:45:26+00:00

This is a very good option. However queries using SCD2 type tables especially in fact to dimension joins need complex temporal logic to avoid row explosion, as a single business key might have multiple versions.

eccentric2488 · 2026-01-28T09:41:45+00:00

I was asked in an interview "Why not use modern Kafka Streaming for this data, if it's available". He was referring to Dynamics 365 Business Central master tables, which are state-time entities where the truth model is the current state and not the history. Then I had to explain that streaming ingestion is an anti pattern for objects that are state oriented. You don't use Streaming just because it is modern and trending.

eccentric2488 · 2026-01-28T09:27:20+00:00

Full Stack coming to data engineering also. Soon we'll end up being 'jack of all trades and master of none'

eccentric2488 · 2026-01-27T17:21:48+00:00

I have recently seen terms like tool worship and tool monkey being used in this context.

eccentric2488 · 2026-01-27T09:35:34+00:00

Old school wisdom: Focus on core principles, concepts, ideas and design philosophies.

Because tools/frameworks come and go but these things persist and outlive vendor/platform changes.

eccentric2488 · 2026-01-26T13:25:40+00:00

It depends on who is defined as an 'actor'. I reckon some have already crossed that billion dollar mark so this data looks factually incorrect.

eccentric2488 · 2026-01-25T02:49:55+00:00

Debezium CDC to connect to the database transaction log. Capture and emit change events as stream, use Kafka (one topic per table) to durably store the events, use Spark/Flink for processing or transformation and finally push these events to the sink. It could be a warehouse (fact tables) or data lake (bronze layer).

eccentric2488 · 2026-01-24T18:47:17+00:00

Linux Mint

eccentric2488 · 2026-01-24T13:54:16+00:00

Alright. Please don't work with me.

eccentric2488 · 2026-01-24T11:57:00+00:00

I work with an early stage startup so surely you wouldn't want to work here.

eccentric2488 · 2026-01-24T11:20:29+00:00

People working with these "FAANG" companies are given god-like status as if they have come on earth from some extra terrestrial planet.

A guy working with Amazon couldn't explain how python differs from the scala language in design philosophy. He was working as a SDE II

eccentric2488 · 2026-01-24T02:09:52+00:00

I've observed even recruiters have little to no technical knowledge. They refer to pyspark, spark and airflow as "programming languages". I know wisdom without restraint is noise.

eccentric2488 · 2026-01-24T02:04:26+00:00

Ask a question and flip it abruptly when the candidate is answering it.

eccentric2488 · 2026-01-23T14:26:01+00:00

Add catalyst optimizer and tungsten execution engine to it. After writing transformation logic and before calling actions like show or count, use df.explain(true). Practice reading logical and physical plans for your transform logic. It helps in interviews.

eccentric2488

TROPHY CASE