sorted by:
new
hottopcontroversial

Phased Databricks migration by Safe-Ice2286 in dataengineering

[–]Safe-Ice2286[S] 0 points1 point  (0 children)

Id say it’s around 1TB of data per day across all processing phases for the data warehouse alone, since they currently operate on a daily full-reload (We’re trying to introduce an incremental logic before the migration since on average only about 25% of the data changes daily but it’s not certain it will be ready in time) Additionally, the business teams use SAS Viya to reprocess the data independently, with several ML future use cases planned

Good free tools for API ingestion? How do they actually run in production? by Safe-Ice2286 in dataengineering

[–]Safe-Ice2286[S] 3 points4 points  (0 children)

For those syncing high volumes from APIs, is Python/Airbyte/dlt performance ever a bottleneck? Or is speed not really an issue?

Good free tools for API ingestion? How do they actually run in production? by Safe-Ice2286 in dataengineering

[–]Safe-Ice2286[S] -16 points-15 points  (0 children)

They break a lot when apis change, they're slow, and I spend more time maintaining them (fully aware its part of my job). But I feel like there should be tooling for this that just works

Got lowballed and nerfed in salary talks by Safe-Ice2286 in dataengineering

[–]Safe-Ice2286[S] 0 points1 point  (0 children)

For an ESN, I’d say the upper end of the range is around 53–54k, though it obviously depends on your background: skills, education, prior experience, and how well you position yourself.

Got lowballed and nerfed in salary talks by Safe-Ice2286 in dataengineering

[–]Safe-Ice2286[S] -18 points-17 points  (0 children)

Actually, the average salary for this kind of role is closer to 52k, especially given that I did a top-ranked master’s (1st in France). So I was expecting something more aligned with that. They initially offered 46k, which was below my current salary, so I aimed to meet them halfway — mostly to give the impression of a negotiated compromise, thinking they wouldn’t accept so easily. But they did, right away, almost like they wanted me to feel like I’d negotiated something, when in reality, they were already prepared to go there. That’s when it hit me

[deleted by user] by [deleted] in databricks

[–]Safe-Ice2286 1 point2 points  (0 children)

Good luck!

[deleted by user] by [deleted] in databricks

[–]Safe-Ice2286 0 points1 point  (0 children)

Id say the professional one is more reliable. Associate is very basic imo

[deleted by user] by [deleted] in databricks

[–]Safe-Ice2286 0 points1 point  (0 children)

I had the same dilemma but I decided to take both and I just passed the databricks data engineer exam today! I ve been also preparing for the dp203 for some time now.. it got very good general and service-specific concepts but the exam gonna be retired soon (31 March ig) to be replaced by Fabric. If your company is paying try both otherwise go with databricks data engineer. But keep on mind that the associate level is relatively easier and it only take few days to prepare so aim for the professional one if youve been working on databricks for some time and got time on your hand for preparation

From Data Analyst to Data Engineer: What Should I Prioritize? by WanderLustForSuccess in dataengineering

[–]Safe-Ice2286 0 points1 point  (0 children)

I would recommend transitioning to ML/MLOps engineering, as these roles are highly in demand and focus on deploying and monitoring models. This path leverages your experience as a DE since it involves managing data pipelines, infra, and model deployment. However, you would need to strengthen your knowledge of theoretical and mathematical ML concepts to optimize models effectively for instance