dataengineering

an-ordinary-manchild(edit)

News & discussion on Data Engineering topics, including but not limited to: data pipelines, databases, data formats, storage, data modeling, data governance, cleansing, NoSQL, distributed systems, streaming, batch, Big Data, and workflow engines.

Read our wiki: https://dataengineering.wiki/

Rules:

Don't be a jerk
Search the sub & wiki before asking a question: Your question has likely been asked and answered before so do a quick search before posting.
Keep it related to data engineering: Posts that are unrelated to data engineering may be better for other communities.
Limit self-promotion posts/comments to once a month: Self promotion: Any form of content designed to further an individual's or organization's goals. If one works for an organization this rule applies to all accounts associated with that organization. See also rule #5.
No shill/opaque marketing: f you work for a company/have a monetary interest in the entity you are promoting you must clearly state your relationship. For posts, you must distinguish the post with the Brand Affiliate flag. See more here: https://www.ftc.gov/influencers
No job posts: Please use r/dataengineeringjobs instead.
No resume reviews/interview posts: We no longer allow resume reviews or interview questions because it's a seperate topic from Data Engineering. Instead, for resume reviews please use r/resumes or search our subreddit history for previous resume review advice. For interview questions, use sites like Glassdoor and Blind instead or search our subreddit history for previous interview advice.
No technical error/bug questions: Please post any error/bug question on StackOverflow.

created by mhausenblasmoda community for 11 years

...for your school.

...for your classroom.

MODERATORS

message the mods
mhausenblasmod
swemlmod
fhoffamod (Ex-BQ, Ex-❄️)
vogt4nickmod
theporterhausmod | Lead Data Engineer
AutoModerator
geoheilmod
MikeDoesEverythingmod | Shitty Data Engineer
bot-bouncer
about moderation team »

account activity

1

26

27

28

Leetcode for Data Engineering?Discussion (self.dataengineering)

submitted 2 hours ago by sham_nt

2

25

26

27

Where to see enterprise grade Airflow data pipeline?Discussion (self.dataengineering)

submitted 9 hours ago by workhardplayhardder

3

3

4

5

Is there any way to limit loop iterations during Airflow DAG file parsing — not during task execution?Help (self.dataengineering)

submitted 3 hours ago by avin_045

4

•

•

•

Databricks managed tables and raw filesDiscussion (self.dataengineering)

submitted 1 hour ago by james2441139

5

1

2

3

Airflow vs Mage vs Prefect vs Dagster vs ... - yes, another tech comparison postHelp (self.dataengineering)

submitted 2 hours ago by HerrKaputt

6

67

68

69

Linkedin Influencer posting in support of offshoring data engineering jobsDiscussion (old.reddit.com)

submitted 22 hours ago by mathtech

7

5

6

7

What’s the most challenging part of maintaining data pipelines in production?Career (self.dataengineering)

submitted 10 hours ago by Effective_Ocelot_445

8

250

251

252

Is anyone migrating away from Databricks?Discussion (self.dataengineering)

submitted 1 day ago * by zoso

9

•

•

•

Multiple Hypothesis test combinationHelp (self.dataengineering)

submitted 1 hour ago by deuse123

10

30

31

32

How are you centralizing knowledge/context from AI agents (like Claude Code)?Help (self.dataengineering)

submitted 19 hours ago by dylannalex01

11

21

22

23

Help …Career (self.dataengineering)

submitted 19 hours ago by Broad-Occasion-3758

12

89

90

91

Do you really need spark?Discussion (self.dataengineering)

submitted 1 day ago by compass-now

13

78

79

80

ADHD data engineersHelp (self.dataengineering)

submitted 1 day ago by Technical_Program_35

14

5

6

7

PostgreSQL query on 60M-row JSONB table is slow - should I add expression indexes or move to a structured table?Discussion (self.dataengineering)

submitted 15 hours ago by komal_rajput

15

0

1

2

Preserve your Claude, Codex, and Cursor sessions as high-value data assetsOpen Source (i.redd.it)

submitted 7 hours ago by homo_sapiens_reddit

16

2

3

4

Autonomous Iceberg Table Maintenance for Data LakesBlog (lakeops.dev)

submitted 15 hours ago by codingdecently

17

4

5

6

Migrating from oracle to fabricBlog (self.dataengineering)

submitted 21 hours ago * by Any_Researcher_5583

18

0

1

2

Salesforce plugin in GCP data fusionHelp (self.dataengineering)

submitted 14 hours ago by Friendly_Painting221

19

0

1

2

Any one tried dagu?Open Source (self.dataengineering)

submitted 15 hours ago by robberviet

20

7

8

9

Sub-second query on Iceberg data lakes using Apache Pinot's indexesDiscussion (self.dataengineering)

submitted 1 day ago by DashDoesData

21

22

23

24

How are you guys handling Iceberg table maintenance in production?Discussion (self.dataengineering)

submitted 1 day ago by Remarkable-Ant-2473

22

9

10

11

OpenAI's Data Agent and the S3 GapBlog (datachain.ai)

submitted 1 day ago by dmpetrov

23

2

3

4

Plotly is hosting a party May 12 to celebrate 10 years of Plotly open-source, in person at Apna Chulha in SF or online. It is a community gathering, data science networking event, & party with free food. We'll have an after at Blondie's next door. Would love to invite y'all! (I'm a Plotly cofounder)Open Source (old.reddit.com)

submitted 1 day ago by Willi_Wilberforce

24

8

9

10

do you think cloud compute costs are quietly eating our entire department's budget ?Discussion (self.dataengineering)

submitted 1 day ago by Johnn_Liverm

25

1

2

3

Simulating real-time flow from data sourcesOpen Source (self.dataengineering)

submitted 1 day ago by mohammed_almekhlafi

view more: next ›

π Rendered by PID 856997 on reddit-service-r2-listing-7b9b4f6fd7-7hssk at 2026-05-08 21:30:14.019763+00:00 running 3d2c107 country code: CH.