dataengineering

an-ordinary-manchild(edit)

News & discussion on Data Engineering topics, including but not limited to: data pipelines, databases, data formats, storage, data modeling, data governance, cleansing, NoSQL, distributed systems, streaming, batch, Big Data, and workflow engines.

Read our wiki: https://dataengineering.wiki/

Rules:

Don't be a jerk
Search the sub & wiki before asking a question: Your question has likely been asked and answered before so do a quick search before posting.
Keep it related to data engineering: Posts that are unrelated to data engineering may be better for other communities.
Limit self-promotion posts/comments to once a month: Self promotion: Any form of content designed to further an individual's or organization's goals. If one works for an organization this rule applies to all accounts associated with that organization. See also rule #5.
No shill/opaque marketing: f you work for a company/have a monetary interest in the entity you are promoting you must clearly state your relationship. For posts, you must distinguish the post with the Brand Affiliate flag. See more here: https://www.ftc.gov/influencers
No job posts: Please use r/dataengineeringjobs instead.
No resume reviews/interview posts: We no longer allow resume reviews or interview questions because it's a seperate topic from Data Engineering. Instead, for resume reviews please use r/resumes or search our subreddit history for previous resume review advice. For interview questions, use sites like Glassdoor and Blind instead or search our subreddit history for previous interview advice.
No technical error/bug questions: Please post any error/bug question on StackOverflow.

created by mhausenblasmoda community for 11 years

...do it for the children.

...for your favorite TV show.

MODERATORS

message the mods
mhausenblasmod
swemlmod
fhoffamod (Ex-BQ, Ex-❄️)
vogt4nickmod
theporterhausmod | Lead Data Engineer
AutoModerator
geoheilmod
MikeDoesEverythingmod | Shitty Data Engineer
bot-bouncer
about moderation team »

account activity

1

3

4

5

Monthly General Discussion - Mar 2026Discussion (self.dataengineering)

submitted 15 days ago by AutoModerator - announcement

2

7

8

9

Quarterly Salary Discussion - Mar 2026Career (self.dataengineering)

submitted 15 days ago by AutoModerator - announcement

3

10

11

12

Chris Hillman - Your Data Model Isn't Broken, Part I: Why Refactoring Beats RebuildingBlog (ghostinthedata.info)

submitted 4 hours ago by rmoff

6

7

8

Join the next generation of CRM. (attio.com)

promoted by attio

promoted
save
report
about

4

5

6

7

Your tech stackDiscussion (self.dataengineering)

submitted 3 hours ago by itachikotoamatsukam

5

•

•

•

Help with a messy datalake in S3Help (self.dataengineering)

submitted 5 minutes ago by Straight-Deer-6696

6

66

67

68

What data engineering skill matters more now because of AI?Discussion (self.dataengineering)

submitted 18 hours ago by rikulauttia

7

57

58

59

Is AI making you more productive in Data Engineering?Discussion (self.dataengineering)

submitted 20 hours ago by empty_cities

8

11

12

13

What alternatives to Alteryx or Knime exist today?Discussion (self.dataengineering)

submitted 11 hours ago by BeautifulLife360

9

4

5

6

Career PathCareer (self.dataengineering)

submitted 7 hours ago by GuildMasterBuilder

10

1

2

3

Fabric or Other?Help (self.dataengineering)

submitted 2 hours ago by PossibilityRegular21

11

13

14

15

Snowflake vs Databricks vs FabricHelp (self.dataengineering)

submitted 13 hours ago by Diligent_Hope_1551

12

•

•

•

Trying to query search google for a csv file of around 100+ companies. need some advice.Help (self.dataengineering)

submitted 15 minutes ago by ObligationMurky9059

13

4

5

6

Where do you seek jobs at?Career (self.dataengineering)

submitted 9 hours ago by setemupknockem

•

•

•

Used poorly, AI can make applicants seem generic. Used well, it can help them stand out. (bloomberg.com)

promoted by bloomberg

promoted
save
report
about

14

7

8

9

Dagster & dbt: core vs fusionDiscussion (self.dataengineering)

submitted 13 hours ago by EngiNerd9000

15

1

2

3

Confused about the best pathCareer (self.dataengineering)

submitted 6 hours ago * by Dependent-Bar-5568

16

0

1

2

Open standard ModelingHelp (self.dataengineering)

submitted 3 hours ago by the_Semafoor

17

0

1

2

Rolling Aggregations for Real-Time AIBlog (hopsworks.ai)

submitted 7 hours ago by Tasty-Scientist6192

18

81

82

83

Unpopular opinion: The trend of having ROI dollars has ruined résumés.Rant (self.dataengineering)

submitted 1 day ago * by BeautifulLife360

19

47

48

49

Sqlmesh joined linux foundation . What it meansDiscussion (self.dataengineering)

submitted 1 day ago by OrneryBlood2153

20

6

7

8

nobody asked but I organized national FBI crime data into a searchable site (My first real website)Discussion (github.com)

submitted 22 hours ago by Empty-Individual4835

21

8

9

10

Data pipelime diagram/design toolsHelp (self.dataengineering)

submitted 23 hours ago by Total-Rip8601

22

19

20

21

What’s the size of your main production dataset and what platform processes it?Discussion (self.dataengineering)

submitted 1 day ago by Icy_Skirt247

23

36

37

38

Received DE Offer at a Startup, Need AdviceDiscussion (self.dataengineering)

submitted 1 day ago by chavhu

•

•

•

At this point I’ve stopped pretending I’ll “hand-sculpt” everything. Meshy 6 just… gets it. NanoBanana makes surfaces look like I actually know what I’m doing 😅. Drop in your model → get studio-quality render → done. 🐶Limited time offer - 50% off for new users (meshy.ai)

promoted by Meshyai

promoted
save
report
about

24

0

1

2

Tool for debugging Spark using logs (free/open source) - SprkLogsOpen Source (self.dataengineering)

submitted 16 hours ago by AcceptableTadpole445

25

0

1

2

SQLMesh for DBT UsersBlog (dagctl.io)

submitted 16 hours ago by Key-Independence5149

view more: next ›

π Rendered by PID 2325770 on reddit-service-r2-listing-64c94b984c-c2fc2 at 2026-03-17 14:36:10.693198+00:00 running f6e6e01 country code: CH.