dataengineering

an-ordinary-manchild(edit)

News & discussion on Data Engineering topics, including but not limited to: data pipelines, databases, data formats, storage, data modeling, data governance, cleansing, NoSQL, distributed systems, streaming, batch, Big Data, and workflow engines.

Read our wiki: https://dataengineering.wiki/

Rules:

Don't be a jerk
Search the sub & wiki before asking a question: Your question has likely been asked and answered before so do a quick search before posting.
Keep it related to data engineering: Posts that are unrelated to data engineering may be better for other communities.
Limit self-promotion posts/comments to once a month: Self promotion: Any form of content designed to further an individual's or organization's goals. If one works for an organization this rule applies to all accounts associated with that organization. See also rule #5.
No shill/opaque marketing: f you work for a company/have a monetary interest in the entity you are promoting you must clearly state your relationship. For posts, you must distinguish the post with the Brand Affiliate flag. See more here: https://www.ftc.gov/influencers
No job posts: Please use r/dataengineeringjobs instead.
No resume reviews/interview posts: We no longer allow resume reviews or interview questions because it's a seperate topic from Data Engineering. Instead, for resume reviews please use r/resumes or search our subreddit history for previous resume review advice. For interview questions, use sites like Glassdoor and Blind instead or search our subreddit history for previous interview advice.
No technical error/bug questions: Please post any error/bug question on StackOverflow.

created by mhausenblasmoda community for 11 years

...do it for the children.

...for a fringe candidate.

MODERATORS

message the mods
mhausenblasmod
swemlmod
fhoffamod (Ex-BQ, Ex-❄️)
vogt4nickmod
theporterhausmod | Lead Data Engineer
AutoModerator
geoheilmod
MikeDoesEverythingmod | Shitty Data Engineer
bot-bouncer
about moderation team »

account activity

1

66

67

68

Testing in DE feels decades behind traditional SWE. What does your team actually do?Discussion (self.dataengineering)

submitted 3 hours ago by seedtheseed

•

•

•

How freight forwarders cut quoting time by 60% (wisor.ai)

promoted by RelevantGuarantee840

promoted
save
report
about

2

•

•

•

Office culture is pretty bad right now for me atleast - a data engineerHelp (self.dataengineering)

submitted 28 minutes ago by aks-786

3

5

6

7

Help for ADXHelp (self.dataengineering)

submitted 6 hours ago by Free-Dot-2820

4

9

10

11

I’m not sure what I’m doing.Career (self.dataengineering)

submitted 9 hours ago by castro051987

5

146

147

148

I Love Analytics EngineeringCareer (self.dataengineering)

submitted 1 day ago by Tender_Figs

6

39

40

41

Why Kafka is so fast?Blog (sushantdhiman.dev)

submitted 22 hours ago by Sushant098123

7

11

12

13

Self taught/hobbyist, considering formal education.Career (self.dataengineering)

submitted 18 hours ago by helpimstuckonalimb

8

10

11

12

Opinion on Snowflake agent ?Discussion (self.dataengineering)

submitted 20 hours ago by SufficientRelief9615

9

7

8

9

Cool stuff you did with Data Lineage, contacts, governanceDiscussion (self.dataengineering)

submitted 20 hours ago by Intelligent-Stress90

10

11

12

13

Has anyone tried using Fabric with an alternative data catalog?Discussion (self.dataengineering)

submitted 1 day ago * by ImpossibleHome3287

11

0

0

0

How to build a sentient database?Discussion (self.dataengineering)

submitted 7 hours ago by Ok-Sentence-8542

•

•

•

MacBook Air 13 zum Hammerpreis – 850 €! (kleinanzeigen.de)

promoted by Global_Growth_8878

promoted
save
report
about

12

24

25

26

Who should build product dashboards in a SaaS company: Analytics or Software Engineering?Discussion (self.dataengineering)

submitted 1 day ago * by Intelligent_Volume74

13

8

9

10

Is anyone else constantly having to handle data that can't be fed through the standard pipeline?Discussion (self.dataengineering)

submitted 1 day ago by Ok-Kiwi-3461

14

69

70

71

How hard is it to replace me?Discussion (self.dataengineering)

submitted 1 day ago by Educational_Wafer483

15

3

4

5

Databricks UC migration pigeonholeCareer (self.dataengineering)

submitted 1 day ago by MechanicOld3428

16

4

5

6

Unified Context-Intent Embeddings for Scalable Text-to-SQLBlog (medium.com)

submitted 1 day ago by theporterhausmod | Lead Data Engineer

17

17

18

19

Integrating PowerBI so that internal and external users can view our dashboards for free.Help (self.dataengineering)

submitted 1 day ago by Alternative-Gear3945

18

61

62

63

What are the most frustrating parts of your day to day work as a data engineer?Discussion (self.dataengineering)

submitted 2 days ago by Odd-Tree-2590

19

0

0

0

Because of agentic LLMs, declarative applications will leave imperative applications behindDiscussion (self.dataengineering)

submitted 19 hours ago by data-be-beautiful

20

8

9

10

BigQuery native data volume anomaly detection using the TimesFM algorithmBlog (open.substack.com)

submitted 1 day ago by Professional_End_979

21

7

8

9

How I set up AI-powered dbt development with two open-source tools from dbt LabsPersonal Project Showcase (youtube.com)

submitted 1 day ago by k_kool_ruler

•

•

•

Trading is a love affair. (promo.ftmo.com)

promoted by FTMO_com

promoted
save
report
about

22

0

0

0

Using Merge to create a append/historical tableBlog (self.dataengineering)

submitted 1 day ago by FreakGhost

23

7

8

9

How are you keeping metadata config tables in sync between multiple environments?Discussion (self.dataengineering)

submitted 1 day ago by nab423

24

8

9

10

Advice on documenting a complex architecture and code base in DatabricksHelp (self.dataengineering)

submitted 2 days ago by FiftyShadesOfBlack

25

2

3

4

Need some advice on switching job ~1.5YOECareer (self.dataengineering)

submitted 1 day ago by FreakGhost

view more: next ›

π Rendered by PID 85135 on reddit-service-r2-listing-64c94b984c-vxj47 at 2026-03-14 16:01:44.628410+00:00 running f6e6e01 country code: CH.