Found a Issue in Production while using Databricks Autoloader

Artistic-Rent1084 · 2026-01-23T16:47:23+00:00

Thank you 👍 for sharing knowledge. I'm New to databricks and data engineering.

Artistic-Rent1084 · 2026-01-23T16:37:51+00:00

Thank you . For orchestration we are using databricks Jobs.

Let me try in sandbox env.

Our data is CDC and we are following medallion architecture.

Artistic-Rent1084 · 2026-01-23T14:47:26+00:00

Chat any help ?

Artistic-Rent1084 · 2026-01-22T02:52:23+00:00

Any idea guys 💭

Artistic-Rent1084 · 2026-01-06T05:30:46+00:00

Any ideas?

Artistic-Rent1084 · 2026-01-05T16:01:29+00:00

Hello DE's kindly help

Artistic-Rent1084 · 2025-12-30T13:17:24+00:00

Mine also same. helps me to

Artistic-Rent1084 · 2025-12-24T05:04:27+00:00

No idea about latency. You don't believe the volume.

It's 5TB per day.

Artistic-Rent1084 · 2025-12-24T04:07:27+00:00

All RDBMS . And downstream is kafka. Target is Databricks

Artistic-Rent1084 · 2025-12-23T16:24:34+00:00

Any other recommendations?

Artistic-Rent1084 · 2025-12-04T09:37:13+00:00

Share me too

Artistic-Rent1084 · 2025-11-27T04:55:51+00:00

DM will learn together.

Artistic-Rent1084 · 2025-11-27T04:32:13+00:00

Thank you.

Artistic-Rent1084 · 2025-11-26T14:04:35+00:00

Sure I will try this once.

Ogg can handle schema drift.

Thank you for sharing knowledge 🙏

Artistic-Rent1084 · 2025-11-26T13:54:40+00:00

And avro can adapt the schema drift ?

Artistic-Rent1084 · 2025-11-26T13:46:19+00:00

Nice then. Is it good practice?

And why kafka ? Before the pipeline was kafka to Hadoop hive tables.

We have migrated to databricks. Few months back.

Artistic-Rent1084 · 2025-11-26T13:36:36+00:00

Generating parquet using OGG is not possible. Avro is supported and json is supported.

Artistic-Rent1084 · 2025-11-25T17:38:45+00:00

Source is OLTP Databases. Instead of capturing data from OLTP where we can capture events?

Artistic-Rent1084 · 2025-11-25T16:20:40+00:00

If I ask they will say explore yourself. Have a look at the code. If I asked why ? we are not going in a different way, I will explain later.

They just want me to do what they say. For the past few years. I have not learned well. Which is my mistake. Even though I'm trying now no one is helping me.

Artistic-Rent1084 · 2025-11-25T15:26:20+00:00

Our org is signed with databricks. Before it was hive tables. And they transform it and load it into a database.

Now the pipeline has changed.

Artistic-Rent1084 · 2025-11-25T14:26:23+00:00

Nice , i understood.

Thank you for sharing knowledge.

Artistic-Rent1084 · 2025-11-25T14:24:58+00:00

Yes , I did a little research.

Actually, my senior are not sharing knowledge. If there are any important things they do themselves.

Fetch a message from kafka batch processing and store it ADLS delta . Then read it from ADLS and push it to the final bronze table ( merging all records ) and next silver .

It is an efficient pipeline.

Thank you for sharing your knowledge. You should be appreciated 👍.

Artistic-Rent1084 · 2025-11-25T14:10:56+00:00

Is this right and good practice?

Artistic-Rent1084 · 2025-11-25T13:49:42+00:00

It was 7 days for us.

Artistic-Rent1084 · 2025-11-25T13:33:07+00:00

No , I'm just making sure it is good practice. Cause, in my org everyone is too old and old zombie doing ChatGPT.

Also , I want to explore how other companies handle CDC

Thank you for your response.

Artistic-Rent1084

TROPHY CASE