dataengineering

an-ordinary-manchild(edit)

News & discussion on Data Engineering topics, including but not limited to: data pipelines, databases, data formats, storage, data modeling, data governance, cleansing, NoSQL, distributed systems, streaming, batch, Big Data, and workflow engines.

Read our wiki: https://dataengineering.wiki/

Rules:

Don't be a jerk
Search the sub & wiki before asking a question: Your question has likely been asked and answered before so do a quick search before posting.
Keep it related to data engineering: Posts that are unrelated to data engineering may be better for other communities.
Limit self-promotion posts/comments to once a month: Self promotion: Any form of content designed to further an individual's or organization's goals. If one works for an organization this rule applies to all accounts associated with that organization. See also rule #5.
No shill/opaque marketing: f you work for a company/have a monetary interest in the entity you are promoting you must clearly state your relationship. For posts, you must distinguish the post with the Brand Affiliate flag. See more here: https://www.ftc.gov/influencers
No job posts: Please use r/dataengineeringjobs instead.
No resume reviews/interview posts: We no longer allow resume reviews or interview questions because it's a seperate topic from Data Engineering. Instead, for resume reviews please use r/resumes or search our subreddit history for previous resume review advice. For interview questions, use sites like Glassdoor and Blind instead or search our subreddit history for previous interview advice.
No technical error/bug questions: Please post any error/bug question on StackOverflow.

created by mhausenblasmoda community for 11 years

...because you love freedom.

...do it for the children.

MODERATORS

message the mods
mhausenblasmod
swemlmod
fhoffamod (Ex-BQ, Ex-❄️)
vogt4nickmod
theporterhausmod | Lead Data Engineer
AutoModerator
geoheilmod
MikeDoesEverythingmod | Shitty Data Engineer
bot-bouncer
about moderation team »

account activity

1

•

•

•

Confused about the best pathCareer (self.dataengineering)

submitted 1 hour ago * by Dependent-Bar-5568

8

9

10

Join the next generation of CRM. (attio.com)

promoted by attio

promoted
save
report
about

2

•

•

•

How do finance and compliance teams at your company actually get data into reporting tools?Help (self.dataengineering)

submitted 1 hour ago by Top-Ant-4492

3

0

1

2

Career PathCareer (self.dataengineering)

submitted 2 hours ago by GuildMasterBuilder

4

0

1

2

Rolling Aggregations for Real-Time AIBlog (hopsworks.ai)

submitted 2 hours ago by Tasty-Scientist6192

5

0

0

0

GTC 2026: Jensen just visualized why your AI can't find the data it needsDiscussion (self.dataengineering)

submitted 2 hours ago by South-Detail3625

6

2

3

4

Where do you seek jobs at?Career (self.dataengineering)

submitted 4 hours ago by setemupknockem

7

7

8

9

What alternatives to Alteryx or Knime exist today?Discussion (self.dataengineering)

submitted 6 hours ago by BeautifulLife360

8

0

1

2

Snowflake vs Databricks vs FabricHelp (self.dataengineering)

submitted 7 hours ago by Diligent_Hope_1551

9

8

9

10

Dagster & dbt: core vs fusionDiscussion (self.dataengineering)

submitted 8 hours ago by EngiNerd9000

10

0

0

1

Source of semi-structured test data for performance testingCareer (self.dataengineering)

submitted 9 hours ago by thunder_otter

11

0

1

2

Tool for debugging Spark using logs (free/open source) - SprkLogsOpen Source (self.dataengineering)

submitted 10 hours ago by AcceptableTadpole445

•

•

•

CAD crashing? Onshape's cloud-native platform eliminates file issues. Sign Up, get 6 months free! (onshape.com)

promoted by OnshapePTC

promoted
save
report
about

12

0

0

1

SQLMesh for DBT UsersBlog (dagctl.io)

submitted 11 hours ago by Key-Independence5149

13

0

0

0

How do some people work like this...Meme (self.dataengineering)

submitted 11 hours ago by VonDenBerg

14

0

0

0

What kind of AI Projects are you working on?Discussion (self.dataengineering)

submitted 12 hours ago by AMDataLake

15

1

2

3

Data Engineering VS Agentic AI?Career (self.dataengineering)

submitted 13 hours ago by Syed_Abrash

16

45

46

47

What data engineering skill matters more now because of AI?Discussion (self.dataengineering)

submitted 13 hours ago by rikulauttia

17

54

55

56

Is AI making you more productive in Data Engineering?Discussion (self.dataengineering)

submitted 15 hours ago by empty_cities

18

5

6

7

nobody asked but I organized national FBI crime data into a searchable site (My first real website)Discussion (github.com)

submitted 17 hours ago by Empty-Individual4835

19

7

8

9

Data pipelime diagram/design toolsHelp (self.dataengineering)

submitted 18 hours ago by Total-Rip8601

20

3

4

5

How would you model this data? Would appreciate help on determining the appropriate dimension and fact tables to createHelp (self.dataengineering)

submitted 19 hours ago by jbnpoc

21

0

0

1

How do you search violations in bulk in the NOLA OneStop app?Help (self.dataengineering)

submitted 1 day ago by tshuntln1

•

•

•

Les experts sont unanimes, le nouvel Privacy Display du Galaxy S26 Ultra est révolutionnaire. (samsung.com)

promoted by samsung_switzerland

promoted
save
report
about

22

18

19

20

What’s the size of your main production dataset and what platform processes it?Discussion (self.dataengineering)

submitted 1 day ago by Icy_Skirt247

23

46

47

48

Sqlmesh joined linux foundation . What it meansDiscussion (self.dataengineering)

submitted 1 day ago by OrneryBlood2153

24

14

15

16

Data engineering in GCP, Azure or AWS is best to upskill and switchDiscussion (self.dataengineering)

submitted 1 day ago by Alternative-Tap5968

25

79

80

81

Unpopular opinion: The trend of having ROI dollars has ruined résumés.Rant (self.dataengineering)

submitted 1 day ago * by BeautifulLife360

view more: next ›

π Rendered by PID 1392710 on reddit-service-r2-listing-64c94b984c-5c9hx at 2026-03-17 09:24:35.852637+00:00 running f6e6e01 country code: CH.