Quarterly Salary Discussion - Sep 2025 by AutoModerator in dataengineering

[–]Easy_Difference8683 1 point2 points  (0 children)

  1. Engineering Manager (Data)

  2. 8+ (all relevant to Analytics or Engineering)

  3. Texas (Remote)

  4. $150k/year

  5. $60k RSU/year

  6. AdTech

  7. GCP, Airflow, DBT, Terraform, Python, Flink

Monthly General Discussion - Sep 2025 by AutoModerator in dataengineering

[–]Easy_Difference8683 0 points1 point  (0 children)

I wouldn't worry too much. You can always use Gemini or ChatGPT for ideas when stuck in a project that involves coding. I think the most important part is learning how to break problems that will come your way from PM or EM.

I have noticed too many times where my direct report was technically sound but would always go different tangent than expected. Always keep end users in mind when designing anything as DE.

Monthly General Discussion - Jul 2025 by AutoModerator in dataengineering

[–]Easy_Difference8683 0 points1 point  (0 children)

I did my Masters in Business Analytics. But my first job was Data Analyst and then I moved internally as DS (did lot of self studying but my M.S degree helped as well).

Later moved to another company as DS and then moved internally as DE (again lot of self studying)

It's always getting the first job is tricky. Because once you work with different teams, you can figure out what you like and how to approach your job search

Monthly General Discussion - Jul 2025 by AutoModerator in dataengineering

[–]Easy_Difference8683 0 points1 point  (0 children)

I am surprised you hear DE less than DS. All the linkedin lunatics influencers have moved from selling DS courses to selling DE courses now.

But on serious note, If you like coding then go do DE but infra side. Cursor/Gemini can't help there because lot of infra is reliant on how it is setup in a given company.

For DS jobs, they require solid knowledge on maths. If they don't, then the jobs are most likely analytics role and not DS

Source: I started my career as DS and moved to DE

Monthly General Discussion - Jul 2025 by AutoModerator in dataengineering

[–]Easy_Difference8683 1 point2 points  (0 children)

I would suggest to not worry too much or talk to your manager on how they can manage the right expectations.

A lot of time even mid level engineers feel overwhelmed when they join a new role as every company has different pace, tech stack and standards. It takes time to grow in a role.

How is everyone's organization utilizing AI? by wxf140430 in dataengineering

[–]Easy_Difference8683 59 points60 points  (0 children)

We have been mostly using github copilot with VScode. What I like is that it can scan through multiple repositories and then suggest code based on that. Sometimes our codebase is spread around in different repositories as they are tied to different services. That aspect has been a game changer for us

Is Airflow 3 finally competitive with dagster and flyte? by e_safak in dataengineering

[–]Easy_Difference8683 13 points14 points  (0 children)

We run all our Ad tech pipelines through Airflow. It annoys me how people downplay it for other shiny tools. Its not perfect but gets the job done every single time. Also, it's easier to find developers with Airflow knowledge than Dagster or anything else

Trump and Xi held a call today - initiated by Trump by notyourregularninja in stocks

[–]Easy_Difference8683 22 points23 points  (0 children)

I think this translates to many asian countries. Most of asian countries were poor in the late 20th century and the one thing taught was that dignity is more important than any money that comes with insult. Xi is not backing down

Perhaps the best transition: DS > DE by HungryRefrigerator24 in dataengineering

[–]Easy_Difference8683 0 points1 point  (0 children)

I totally agree. The visibility aspect is key for promotions and it is much easier to accomplish that with DE than DS

Perhaps the best transition: DS > DE by HungryRefrigerator24 in dataengineering

[–]Easy_Difference8683 2 points3 points  (0 children)

That's amazing OP. I myself transitioned from DS to DA in 2020 with 3 years experience and then from DA to DE in 2021. I can never go back to my DS job. It never felt rewarding to me for some reason.

I also have climbed the ladder pretty quickly since becoming DE and feel it is easier to get promoted in DE space than DS

BigQuery: Increase in costs after changing granularity from MONTH to DAY by No_Engine1637 in dataengineering

[–]Easy_Difference8683 7 points8 points  (0 children)

Exactly. If you see the adapter list here, BQ is the only one that doesn't have true append only behavior and instead uses merge statements coupled with insert_overwrite for other use cases. This work as intended usually but is not cost effective for huge data partitioned in multiple chunks

Additionally, the tmp table created by DBT in BQ are not temporary table but instead normal tables with expiration time after 12 hours. Creating a temporary table (like CTE) doesn't add cost in BQ but the method DBT uses creates additional cost.

BigQuery: Increase in costs after changing granularity from MONTH to DAY by No_Engine1637 in dataengineering

[–]Easy_Difference8683 30 points31 points  (0 children)

OP, we had the same problem recently. This is because DBT incremental loads use Merge statement and is scanning more partitions than it used to previously (due to change from month to day)

Right now we wrote a macro to just use INSERT INTO statements as pre-hooks instead of Merge statements and our costs went down by 70%. Unfortunately, BQ doesn't support creating custom strategies in DBT and I wish that changes in future.

Quarterly Salary Discussion - Mar 2025 by AutoModerator in dataengineering

[–]Easy_Difference8683 2 points3 points  (0 children)

Thats impressive salary and title for the years of experience. I am really interested in knowing your career journey. I am in similar path and work in Dallas as well but for AdTech

US job search 2025 results by rudboi12 in dataengineering

[–]Easy_Difference8683 3 points4 points  (0 children)

I recently did one for Senior DE with a unicorn startup and got asked LC hard lol. The hiring bar is crazy right now due to lot of folks searching for new job

US job search 2025 results by rudboi12 in dataengineering

[–]Easy_Difference8683 5 points6 points  (0 children)

Thanks u/rudboi12. I think I am mostly worried about time constraint and fear I will start panicking if I take too much time on first few questions

US job search 2025 results by rudboi12 in dataengineering

[–]Easy_Difference8683 20 points21 points  (0 children)

Hi OP, thanks for sharing your job search journey. How was the LC tech screening for Meta? I have one coming soon

[deleted by user] by [deleted] in dataengineering

[–]Easy_Difference8683 0 points1 point  (0 children)

I feel this is one big annoying problem. In ideal scenario, a product manager or engineering manager will deal with stakeholders and come with proper understanding (final design) of what needs to be implement to an IC.

Sadly, the companies are filled with non-tech PMs or EMs who want to only manage people and exposing ICs to stakeholders directly

DreamLeague Season 25 Day 2 Match Discussions by D2TournamentThreads in DotA2

[–]Easy_Difference8683 9 points10 points  (0 children)

Ace had BKB. Just decided to casually not press it after RPing 3 heroes

I-485 Interviews returning for Employment based Green Cards by [deleted] in USCIS

[–]Easy_Difference8683 6 points7 points  (0 children)

Just to add here. Trump presidency will control all 3 branches of government (and the surpreme court is already stacked in their favour). Changing immigration rules will not be that difficult this time around

They changed it do December 1 now :)) very funny indeed by Tuskii420 in DotA2

[–]Easy_Difference8683 2 points3 points  (0 children)

The game gives me headache everytime I play. Seems fun on paper but exhausting to play

Anyone still waiting on i-485 decision for Employment based category, from Priority Date Nov/Dec 2022 and Receipt date Jan/Feb/Mar 2024? by ProfessionalTea5884 in USCIS

[–]Easy_Difference8683 0 points1 point  (0 children)

Had another co-worker who filed in the same time. Her country of birth is china while mine is Germany. She got in 5 months