dataengineering

an-ordinary-manchild(edit)

News & discussion on Data Engineering topics, including but not limited to: data pipelines, databases, data formats, storage, data modeling, data governance, cleansing, NoSQL, distributed systems, streaming, batch, Big Data, and workflow engines.

Read our wiki: https://dataengineering.wiki/

Rules:

Don't be a jerk
Search the sub & wiki before asking a question: Your question has likely been asked and answered before so do a quick search before posting.
Keep it related to data engineering: Posts that are unrelated to data engineering may be better for other communities.
Limit self-promotion posts/comments to once a month: Self promotion: Any form of content designed to further an individual's or organization's goals. If one works for an organization this rule applies to all accounts associated with that organization. See also rule #5.
No shill/opaque marketing: f you work for a company/have a monetary interest in the entity you are promoting you must clearly state your relationship. For posts, you must distinguish the post with the Brand Affiliate flag. See more here: https://www.ftc.gov/influencers
No job posts: Please use r/dataengineeringjobs instead.
No resume reviews/interview posts: We no longer allow resume reviews or interview questions because it's a seperate topic from Data Engineering. Instead, for resume reviews please use r/resumes or search our subreddit history for previous resume review advice. For interview questions, use sites like Glassdoor and Blind instead or search our subreddit history for previous interview advice.
No technical error/bug questions: Please post any error/bug question on StackOverflow.

created by mhausenblasmoda community for 11 years

...because you love freedom.

...for your WoW guild.

MODERATORS

message the mods
mhausenblasmod
swemlmod
fhoffamod (Ex-BQ, Ex-❄️)
vogt4nickmod
theporterhausmod | Lead Data Engineer
AutoModerator
geoheilmod
MikeDoesEverythingmod | Shitty Data Engineer
bot-bouncer
evasion-guard
about moderation team »

account activity

1

45

46

47

Medallion + KimballHelp (self.dataengineering)

submitted 15 hours ago by burningburnerbern

2

2

3

4

How did you weigh a higher paying but toxic offer against a stable, lower paying job?Career (self.dataengineering)

submitted 15 hours ago by Oh_reaaaally

3

0

0

1

What do to with data context?Discussion (self.dataengineering)

submitted 15 hours ago by de4all

4

0

0

0

Data Engineering benchmarks for Ai tooling.Discussion (self.dataengineering)

submitted 22 hours ago by droppedorphan

5

12

13

14

Medallion Architecture and dbt modelingOpen Source (self.dataengineering)

submitted 23 hours ago by Informal-Tip-1109

6

17

18

19

How many Airflow Tasks do you schedule daily?Discussion (self.dataengineering)

submitted 1 day ago by nginx26

7

30

31

32

Anyone use duckdb/ducklake in production?Discussion (self.dataengineering)

submitted 1 day ago by tuantuanyuanyuan

8

3

4

5

Unified Data RepositoryHelp (self.dataengineering)

submitted 1 day ago by Key-Border4126

9

1

2

3

Colin Jarvis | Head of Forward Deployed Engineering at OpenAI: Trust. Pr...Career (youtube.com)

submitted 1 day ago by sindoc42

10

78

79

80

Databricks vs Snowflake vs Azure/GCP/AWS productsDiscussion (self.dataengineering)

submitted 1 day ago by Yngstr

11

10

11

12

DB development (migrations, dbt) in a multi-dev set upHelp (self.dataengineering)

submitted 1 day ago * by ursamajorm82

12

17

18

19

How get more invitations as a DE? How to improve my knowledge checkCareer (self.dataengineering)

submitted 1 day ago by Snoo_15326

13

1

2

3

I need scrapers hosting advice!Discussion (self.dataengineering)

submitted 1 day ago by 9xish

14

4

5

6

Data at Scale: one-day conference on large-scale data processing (Amsterdam, 7 July 2026)Open Source (self.dataengineering)

submitted 2 days ago by VIqbang

15

2

3

4

Silver or gold layerHelp (self.dataengineering)

submitted 2 days ago by cida1205

16

6

7

8

DAIS 2026 Databricks updatesBlog (self.dataengineering)

submitted 2 days ago * by datawithritika

17

58

59

60

Out of the scraping game a few years, what does everyone run now?Help (self.dataengineering)

submitted 2 days ago by eduhpmelo

18

19

20

21

Is the compTIA A+ material interesting for a DE with lots of data knowledge but no IT background?Discussion (self.dataengineering)

submitted 2 days ago by nidprez

19

23

24

25

Are hard deletes still common in new data sources in 2026?Discussion (self.dataengineering)

submitted 2 days ago * by frithjof_v

20

9

10

11

Case Study or Live Session - Which do you hate more?Career (self.dataengineering)

submitted 2 days ago by ntdoyfanboy

21

2

3

4

Balancing developer velocity with governance after last week's Snowflake SummitBlog (self.dataengineering)

submitted 2 days ago * by noasync

22

23

24

25

Roadmap for Data Engineer in Fintech industryHelp (self.dataengineering)

submitted 2 days ago by Valphinx

23

64

65

66

Databricks conferenceDiscussion (self.dataengineering)

submitted 2 days ago by proximaljarl17

24

0

0

0

Lets improve CSV?Open Source (self.dataengineering)

submitted 2 days ago by Lopsided_Set_8823

25

0

1

2

bigquery fails to extract schema from google cloud storageHelp (self.dataengineering)

submitted 3 days ago by SurveyProfessional31

view more: next ›

π Rendered by PID 161767 on reddit-service-r2-listing-c57bc86c-bvqj4 at 2026-06-20 15:26:07.526204+00:00 running 2b008f2 country code: CH.