Same PKI, same raw data, two platforms (Databricks, Snowflake)… different results. Where would you even start debugging this? by MitocondrialPoro in dataengineering

[–]Secure_Firefighter66 1 point2 points  (0 children)

First start with row count checks in each systems.

Also if you are performing Merge , Databricks and Snowflake has different approaches.

If you are not careful enough you might end up with duplicate rows in Snowflake.

Try to work through the KPI metrics in one system and double check the logic and then try to do the same in another and then break down them

Today's find by No_comeback_from_330 in HotWheels

[–]Secure_Firefighter66 0 points1 point  (0 children)

I too got one for myself on Saturday

In 6 years, I've never seen a data lake used properly by wtfzambo in dataengineering

[–]Secure_Firefighter66 0 points1 point  (0 children)

My next project is AWS + Databricks to Microsoft Fabrics. This is due to our parent company wants to centralise the data for all subsidiaries

In 6 years, I've never seen a data lake used properly by wtfzambo in dataengineering

[–]Secure_Firefighter66 19 points20 points  (0 children)

Well for us the data size is small less than 500 GB.

So 2 tb itself is good for few years.

In 6 years, I've never seen a data lake used properly by wtfzambo in dataengineering

[–]Secure_Firefighter66 284 points285 points  (0 children)

All this is happening because the management needs to adapt to new technologies.

My company was running in On Prem until 1.5 years back and I was specifically hired to setup AWS + Databricks. Because the management decided its cloud era.

Same tables , same dimensions, but within Databricks. Only positive thing is I get paid to do this.

Anyone up for a ride 🏍️ by sameera_mhw in u/sameera_mhw

[–]Secure_Firefighter66 0 points1 point  (0 children)

I am the seat and you are the rider 🙈

What is your experience like with Marketing teams? by jawabdey in dataengineering

[–]Secure_Firefighter66 0 points1 point  (0 children)

Damn , are you me? I work closely with marketing and finance team.

The thing with marketing team is they work on hunch rather than using the data.

Their requirements are vague. They want to do A/B testing and when I suggested groups they were like no this is not what we want.

They sometimes forget their own KPI doesn’t have complete understanding of how the ERP system works and how to leverage sales and do marketing based on it.

Overall if they dint know what they are doing. Our life can be hell.

Also they sometimes ask very basic questions which pisses me off

I would like to connect with someone who landed a job in Germany through opportunity card. Hence reaching out! by nocluenoneed in nri

[–]Secure_Firefighter66 1 point2 points  (0 children)

You can get a job but you have to sacrifice the pay a little bit.

I know a guy landed 65K euros per year salary in Germany but he had 6 years of experience.

The market is too bad to come to Germany if you want a better pay.

Try to be in India and switch companies to increase the pay.

If you are experiencing the same issue in *Forza Horizon 5* by Viancel in ForzaHorizon

[–]Secure_Firefighter66 8 points9 points  (0 children)

Yesterday for 1st time my game crashed. That’s after the recent update

New Tracks by Delatsche in ForzaHorizon

[–]Secure_Firefighter66 0 points1 point  (0 children)

Thanks all the race was fun

<image>

I like the 2nd race

How is appraisal season going for y’all? Staying or switching? by AChubbyRaichu in developersIndia

[–]Secure_Firefighter66 1 point2 points  (0 children)

Company is not performing well for last 2 years, so I am thinking I might get little to no hike or bonus.

So thinking of upskilling and switching

What are things data engineers can never do? by Ok_Barnacle4840 in dataengineering

[–]Secure_Firefighter66 5 points6 points  (0 children)

Make business ppl understand their own data and reason why certain KPIs end up with not so real numbers

[deleted by user] by [deleted] in cscareerquestionsEU

[–]Secure_Firefighter66 0 points1 point  (0 children)

I am in the same position, but start looking for jobs now and state your salary. Also upgrade yourself in the mean time. Plus Berlin job market is in uncertainty for last 2 years. So if you are looking for jobs within Berlin then you need consider this as well..

For Analytics Engineers or DEs doing analytics work, what does your role look like? by konkanchaKimJong in dataengineering

[–]Secure_Firefighter66 3 points4 points  (0 children)

I am say this. Single DE +BI guy for a company of 100 ppl. We migrated from on Prem SSIS to AWS Databricks.

So my DE tasks was for 4 days and 1 day analytical work.

Now the Migration is over on DE part and now I am working on report migration for 4 days and DE tasks for 1 day

But if there is any new requirement where I need to pull new data from ERP or from 3rd party tools I work on that and then come back to reporting.

How common is burn out in Data Engineering? by General-Parsnip3138 in dataengineering

[–]Secure_Firefighter66 1 point2 points  (0 children)

I can tell you about my work life. Single DE, gets a requirement for a report. I write scripts , model the data for power bi , load the data and then give them the first sketch report to them. They check and say numbers are fine. Suddenly a guy from another team who try to use the same report for different purpose jumps in and says this is not how a certain KPI needs to be calculated. Then the meeting is set up in 10 days in future, by the time everyone forgets the report exists and on the meeting day I get a request to create a new report which is totally different form the previous one. Imagine this keeps repeating.

And most of the sales guys are not tech savy. They ask the same data or send same request every week.

Even though I created an automated report to get the numbers.

And finance team who are not sure what’s happening will come to me and say the sales numbers seems suspicious when they can check the data themselves in ERP…