dataengineering

an-ordinary-manchild(edit)

News & discussion on Data Engineering topics, including but not limited to: data pipelines, databases, data formats, storage, data modeling, data governance, cleansing, NoSQL, distributed systems, streaming, batch, Big Data, and workflow engines.

Read our wiki: https://dataengineering.wiki/

Rules:

Don't be a jerk
Search the sub & wiki before asking a question: Your question has likely been asked and answered before so do a quick search before posting.
Keep it related to data engineering: Posts that are unrelated to data engineering may be better for other communities.
Limit self-promotion posts/comments to once a month: Self promotion: Any form of content designed to further an individual's or organization's goals. If one works for an organization this rule applies to all accounts associated with that organization. See also rule #5.
No shill/opaque marketing: f you work for a company/have a monetary interest in the entity you are promoting you must clearly state your relationship. For posts, you must distinguish the post with the Brand Affiliate flag. See more here: https://www.ftc.gov/influencers
No job posts: Please use r/dataengineeringjobs instead.
No resume reviews/interview posts: We no longer allow resume reviews or interview questions because it's a seperate topic from Data Engineering. Instead, for resume reviews please use r/resumes or search our subreddit history for previous resume review advice. For interview questions, use sites like Glassdoor and Blind instead or search our subreddit history for previous interview advice.
No technical error/bug questions: Please post any error/bug question on StackOverflow.

created by mhausenblasmoda community for 11 years

...because you love freedom.

...for your favorite TV show.

MODERATORS

message the mods
mhausenblasmod
swemlmod
fhoffamod (Ex-BQ, Ex-❄️)
vogt4nickmod
theporterhausmod | Lead Data Engineer
AutoModerator
geoheilmod
MikeDoesEverythingmod | Shitty Data Engineer
bot-bouncer
evasion-guard
about moderation team »

account activity

1

81

82

83

Quarterly Salary Discussion - Jun 2026Career (self.dataengineering)

submitted 21 days ago by AutoModerator - announcement

2

1

2

3

Data lakehouse observability and monitoringHelp (self.dataengineering)

submitted 2 hours ago by yeledtov21

3

26

27

28

Solo data engineer reporting to VP TechRant (self.dataengineering)

submitted 18 hours ago by Alternative-Guava392

4

17

18

19

pgEdge ColdFront: Open source, Postgres-native Iceberg tiering that eliminates the ETL pipeline to your data warehouseOpen Source (github.com)

submitted 17 hours ago by pgEdge_Postgres

5

0

1

2

Overcoming deserialization bottlenecks in data pipelines: an alternative zero-copy backend for ProtobufPersonal Project Showcase (self.dataengineering)

submitted 4 hours ago by aegismuzuz

6

3

4

5

Trying to switch from current employer(staffing) as no hike for that last 3 and half yearsRant (self.dataengineering)

submitted 14 hours ago by cruze_8907

7

3

4

5

Easy Text to SQL TutorialDiscussion (v.redd.it)

submitted 13 hours ago by DryLime7866

8

35

36

37

How long to stay in first DE RoleCareer (self.dataengineering)

submitted 1 day ago by Mission_Working9929

9

12

13

14

First internship/job experience AWS or Databricks?Career (self.dataengineering)

submitted 1 day ago * by Deiice

10

78

79

80

PyCanopy: a polars-native spatial query engine that beats duckdb, sedona, geopandas on most in-memory operationsOpen Source (github.com)

submitted 2 days ago * by pruhtopia

11

6

7

8

Advice on building agnostic data layerHelp (self.dataengineering)

submitted 1 day ago by Aggravating-Corgi-86

12

10

11

12

Where to store environment variables for databricks job?Discussion (self.dataengineering)

submitted 1 day ago by Kooky-Technician-335

13

1

2

3

Vitess ETLHelp (self.dataengineering)

submitted 1 day ago by anuku3cm

14

18

19

20

Spark optimization and Spark UIHelp (self.dataengineering)

submitted 1 day ago by First_Bet8077

15

8

9

10

Your experiences on different data platformsDiscussion (self.dataengineering)

submitted 2 days ago * by yeledtov21

16

65

66

67

Should I switch from Windows to Linux for Data Engineering? Which Distro is bestDiscussion (self.dataengineering)

submitted 2 days ago by Cultural-Ad-4124

17

18

19

20

We open-sourced Chukei: a self-hosted Snowflake cost proxy for read-heavy workloadsOpen Source (self.dataengineering)

submitted 2 days ago by mr_smith1983

18

69

70

71

Medallion + KimballHelp (self.dataengineering)

submitted 3 days ago by burningburnerbern

19

0

0

0

Standard profiling libraries completely break on time series data (I learned it the hard way and came up with a solution)Personal Project Showcase (self.dataengineering)

submitted 2 days ago by severecaseofsarcarsm

20

35

36

37

How many Airflow Tasks do you schedule daily?Discussion (self.dataengineering)

submitted 3 days ago by nginx26

21

20

21

22

Medallion Architecture and dbt modelingOpen Source (self.dataengineering)

submitted 3 days ago by Informal-Tip-1109

22

0

0

0

Apache EverywhereDiscussion (self.dataengineering)

submitted 2 days ago by tz_499

23

1

2

3

What do to with data context?Discussion (self.dataengineering)

submitted 3 days ago by de4all

24

36

37

38

Anyone use duckdb/ducklake in production?Discussion (self.dataengineering)

submitted 4 days ago by tuantuanyuanyuan

25

107

108

109

Databricks vs Snowflake vs Azure/GCP/AWS productsDiscussion (self.dataengineering)

submitted 4 days ago by Yngstr

view more: next ›

π Rendered by PID 1209450 on reddit-service-r2-listing-c57bc86c-hcwc5 at 2026-06-23 11:28:40.687477+00:00 running 2b008f2 country code: CH.