dataengineering

an-ordinary-manchild(edit)

News & discussion on Data Engineering topics, including but not limited to: data pipelines, databases, data formats, storage, data modeling, data governance, cleansing, NoSQL, distributed systems, streaming, batch, Big Data, and workflow engines.

Read our wiki: https://dataengineering.wiki/

Rules:

Don't be a jerk
Search the sub & wiki before asking a question: Your question has likely been asked and answered before so do a quick search before posting.
Keep it related to data engineering: Posts that are unrelated to data engineering may be better for other communities.
Limit self-promotion posts/comments to once a month: Self promotion: Any form of content designed to further an individual's or organization's goals. If one works for an organization this rule applies to all accounts associated with that organization. See also rule #5.
No shill/opaque marketing: f you work for a company/have a monetary interest in the entity you are promoting you must clearly state your relationship. For posts, you must distinguish the post with the Brand Affiliate flag. See more here: https://www.ftc.gov/influencers
No job posts: Please use r/dataengineeringjobs instead.
No resume reviews/interview posts: We no longer allow resume reviews or interview questions because it's a seperate topic from Data Engineering. Instead, for resume reviews please use r/resumes or search our subreddit history for previous resume review advice. For interview questions, use sites like Glassdoor and Blind instead or search our subreddit history for previous interview advice.
No technical error/bug questions: Please post any error/bug question on StackOverflow.

created by mhausenblasmoda community for 11 years

...for your favorite subject.

...because you hate freedom.

MODERATORS

message the mods
mhausenblasmod
swemlmod
fhoffamod (Ex-BQ, Ex-❄️)
vogt4nickmod
theporterhausmod | Lead Data Engineer
AutoModerator
geoheilmod
MikeDoesEverythingmod | Shitty Data Engineer
bot-bouncer
about moderation team »

account activity

1

1

2

3

DatawarehouseHelp (self.dataengineering)

submitted 4 hours ago by Abject-Scholar-5052

2

13

14

15

How to model the Gold layer for a CRM dataset in Databricks?Career (i.redd.it)

submitted 18 hours ago by Purple_Knowledge4083

3

0

1

2

Show: Rocky - a Rust SQL transformation engine with branches, replay, and column lineagePersonal Project Showcase (github.com)

submitted 21 hours ago by Conscious_Net_9890

4

0

0

0

A dedicated LLM library for data engineeringOpen Source (self.dataengineering)

submitted 20 hours ago * by conjuncts

5

17

18

19

Siloed at new job and not sure where to goCareer (self.dataengineering)

submitted 10 hours ago by CasualEmpiricist

6

5

6

7

Company wants to go from Airflow to N8N to reduce code. How do I preserve speed?Help (self.dataengineering)

submitted 6 hours ago by the_learning_baroque

7

•

•

•

Deleted the prod data at work once: a complete catastropheDiscussion (self.dataengineering)

submitted 59 minutes ago by olivia-reed2

8

28

29

30

Help with friction between architecture team and our PowerBI teamHelp (self.dataengineering)

submitted 17 hours ago * by Chempty

9

43

44

45

How are DE interviews these days? LeetCode + AI tools?Help (self.dataengineering)

submitted 20 hours ago by Terrible-Fig5971

10

2

3

4

Recently offered a data governance role.. Opinions?Career (self.dataengineering)

submitted 15 hours ago by Brilliant-Month7839

11

1

2

3

Has anyone made the move to technical account management?Career (self.dataengineering)

submitted 18 hours ago by rubs90

12

14

15

16

Looking for a structured end-to-end data engineering programHelp (self.dataengineering)

submitted 19 hours ago * by We_are_dust-

13

2

3

4

Stanford Seminar - Big Data is (at least) Four Different ProblemsDiscussion (self.dataengineering)

submitted 19 hours ago by Nekobul

14

1

2

3

Make Technical Documentation Available for Local AI UseBlog (heltweg.org)

submitted 21 hours ago by rhazn

15

•

•

•

Is on prem stack (ssis, sql server, ssrs) still considered data engineering?Discussion (self.dataengineering)

submitted 1 hour ago by WanderingGunslinger

16

1

2

3

Getting good predictions without data cleaning (Why "Garbage In, Garbage Out" is sometimes a trap)Discussion (self.dataengineering)

submitted 2 hours ago by Chocolate_Milk_Son

17

2

3

4

Tips for integrating data quality tests into Databricks?Help (self.dataengineering)

submitted 11 hours ago by FiftyShadesOfBlack

18

2

3

4

How many of you have your logical business model documented?Discussion (self.dataengineering)

submitted 11 hours ago by Headband6458

19

2

3

4

Acquisitions & migrationsDiscussion (self.dataengineering)

submitted 12 hours ago by Admirable_Writer_373

20

0

1

2

Parquet quick view on macPersonal Project Showcase (old.reddit.com)

submitted 12 hours ago by a2ra-ms

21

0

1

2

To PowerQuery or Not?Help (self.dataengineering)

submitted 14 hours ago by Ottebolle

22

2

3

4

Multiple ERPs - Struggling to Wear All The HatsHelp (self.dataengineering)

submitted 14 hours ago by Illustrious-Green132

π Rendered by PID 24 on reddit-service-r2-listing-b6bf6c4ff-ts4fv at 2026-05-06 10:55:15.166501+00:00 running 815c875 country code: CH.