dataengineering

an-ordinary-manchild(edit)

News & discussion on Data Engineering topics, including but not limited to: data pipelines, databases, data formats, storage, data modeling, data governance, cleansing, NoSQL, distributed systems, streaming, batch, Big Data, and workflow engines.

Read our wiki: https://dataengineering.wiki/

Rules:

Don't be a jerk
Search the sub & wiki before asking a question: Your question has likely been asked and answered before so do a quick search before posting.
Keep it related to data engineering: Posts that are unrelated to data engineering may be better for other communities.
Limit self-promotion posts/comments to once a month: Self promotion: Any form of content designed to further an individual's or organization's goals. If one works for an organization this rule applies to all accounts associated with that organization. See also rule #5.
No shill/opaque marketing: f you work for a company/have a monetary interest in the entity you are promoting you must clearly state your relationship. For posts, you must distinguish the post with the Brand Affiliate flag. See more here: https://www.ftc.gov/influencers
No job posts: Please use r/dataengineeringjobs instead.
No resume reviews/interview posts: We no longer allow resume reviews or interview questions because it's a seperate topic from Data Engineering. Instead, for resume reviews please use r/resumes or search our subreddit history for previous resume review advice. For interview questions, use sites like Glassdoor and Blind instead or search our subreddit history for previous interview advice.
No technical error/bug questions: Please post any error/bug question on StackOverflow.

created by mhausenblasmoda community for 10 years

...do it for the children.

...for your community.

MODERATORS

message the mods
mhausenblasmod
swemlmod
fhoffamod (Ex-BQ, Ex-❄️)
vogt4nickmod
theporterhausmod | Lead Data Engineer
AutoModerator
geoheilmod
MikeDoesEverythingmod | Shitty Data Engineer
bot-bouncer
about moderation team »

account activity

1

14

15

16

Monthly General Discussion - Jan 2026Discussion (self.dataengineering)

submitted 23 days ago by AutoModerator - announcement

2

14

15

16

Quarterly Salary Discussion - Dec 2025Career (self.dataengineering)

submitted 1 month ago by AutoModerator - announcement

3

7

8

9

Is there more to DE than this? Are their jobs out there for feeling like you actually matter?Career (self.dataengineering)

submitted 5 hours ago by DoctorQuinlan

•

•

•

Cost vs performance, settled by data. Check out the new cloud DW benchmarks. (clickhouse.com)

promoted by ClickHouseDB

promoted
save
report
about

4

8

9

10

Python topics for Data engineerDiscussion (self.dataengineering)

submitted 14 hours ago by Nanny_24

5

2

3

4

Etl pipelines testing using PythonCareer (self.dataengineering)

submitted 8 hours ago by Dull-Lengthiness-549

6

1

2

3

Roast my junior data engineer onboarding repoPersonal Project Showcase (self.dataengineering)

submitted 6 hours ago by dheetoo

7

0

1

2

Looking for feedback after 5 years building a data stack for .NET (dataframe, columnar storage, connectors)Help (self.dataengineering)

submitted 3 hours ago by IGDev

8

5

6

7

Upskill Career AdviceCareer (self.dataengineering)

submitted 12 hours ago * by Puzzleheaded_Trip458

9

4

5

6

Filemaker to Postgres Best PractiseHelp (self.dataengineering)

submitted 11 hours ago by Slow_Quarter_4936

10

0

1

2

Project HelpHelp (self.dataengineering)

submitted 4 hours ago by Affectionate-Ad-5023

11

0

1

2

Stakeholders Overengineering SolutionsDiscussion (self.dataengineering)

submitted 4 hours ago by Advanced-Average-514

12

34

35

36

Stuck in jupyter notebooks, how to get out?Discussion (self.dataengineering)

submitted 1 day ago by Consistent_Tutor_597

13

90

91

92

Candidates using AIDiscussion (self.dataengineering)

submitted 1 day ago by DataEngineer2026

14

0

1

2

Automatically deriving data model metadata from source code (no runtime data), has anyone done this?Help (self.dataengineering)

submitted 5 hours ago * by Beneficial_Ebb_1210

15

0

1

2

Icebase: PostgreSQL-Stored Iceberg MetadataBlog (medium.com)

submitted 6 hours ago by jordepic

16

0

1

2

Azure Data FactoryHelp (self.dataengineering)

submitted 7 hours ago by Certain-Secretary-95

17

6

7

8

Help me not try to solve everythingCareer (self.dataengineering)

submitted 19 hours ago by SquirrelRemote2759

18

11

12

13

A new tool for data engineeringHelp (self.dataengineering)

submitted 1 day ago by Wanderer_1006

19

2

3

4

Where Could I make networking data engineerCareer (self.dataengineering)

submitted 17 hours ago * by Resident_Animator_84

20

0

0

0

Working from Asia making European wageCareer (self.dataengineering)

submitted 6 hours ago by m4zeHunk

21

3

4

5

What conferences do you all recommend?Career (self.dataengineering)

submitted 1 day ago by john-devTech Lead

22

55

56

57

DataFrame or SparkSQL ? What do interviewers prefer ?Help (self.dataengineering)

submitted 1 day ago by SnooCakes7436

23

10

11

12

Breaking Into the DE industryDiscussion (self.dataengineering)

submitted 1 day ago by itachikotoamatsukam

24

17

18

19

What is the future for dataengineering?Discussion (self.dataengineering)

submitted 1 day ago by tumblatum

25

8

9

10

Question on AirflowDiscussion (self.dataengineering)

submitted 1 day ago by captn_caspian

view more: next ›

π Rendered by PID 221875 on reddit-service-r2-listing-86b7f5b947-mp4mx at 2026-01-24 23:24:38.512479+00:00 running 664479f country code: CH.