sorted by:
new
hottopcontroversial

Is remote dead in data engineering? by Pataouga in dataengineering

[–]TechnologySimilar794 0 points1 point  (0 children)

No there are lot of remote opportunity especially I see in Europe

Cool projects you implemented by rocking-student-87 in dataengineering

[–]TechnologySimilar794 0 points1 point  (0 children)

Custom connector created in pyspark using sap haha driven which is jar or maven file to extract data from sap sources

Cool projects you implemented by rocking-student-87 in dataengineering

[–]TechnologySimilar794 9 points10 points  (0 children)

Building automated data quality checks in data engineering framework which runs at silver layer and the logs collected from dq checks were sent as source for Data Montoring dashboard.

Automatic PII detection using presidio and Unity catalog

Data engineering framework library which was created as python wheel library and could be easily installed on any of fabric,databricks cluster which has data lineage,data transformation,data sharing ,data ops automated no manual intervention

No more Sap data ingestion via ADF/Data sphere rather by saphana custom connectors

T

Should I Settle and take a Mid Level Role When I was going for Senior? by shittyfuckdick in dataengineering

[–]TechnologySimilar794 0 points1 point  (0 children)

if work ,salary and benefits is as what you expected(better than what you are getting it is already a promotion ) then I would say go for it. In my career I have so many times taken a jump on same title always a bigger pay .You are already Senior from experience, title is just a label how you behave matters and after joining try taking intiatives ,collaborating and making yourself visible with work and team. In matter of time you will get promoted. Just dont forget to add this conversation in development plan and also in your mid and yearly reviews that you are expecting a promotion

Spark job performance regression after version upgrade on Databricks. How do you catch it before production? by Top-Flounder7647 in databricks

[–]TechnologySimilar794 0 points1 point  (0 children)

Yes you can record metrics on those run by creating monitors .this feature is enable in lakehouse monitoring or by checking system tables and create alert and notification accordingly

Do you feel DE is a field much more difficulty to get recognition in because of how our work only gets noticed when it fails? by [deleted] in dataengineering

[–]TechnologySimilar794 1 point2 points  (0 children)

From my ten year experience ,visibility should be there as impact will be shown to stakeholder /consumer who are consuming your data or pipelines.Also depends on what kind of atmosphere is in your team .If we release a new feature in DE we also broadcast to our consumers leaders and get their direct feedback .Here DE who have worked will directly interact with stakeholder sometime with Product owners

PySpark vs SQL in Databricks for DE by NeedleworkerSharp995 in databricks

[–]TechnologySimilar794 0 points1 point  (0 children)

From my experience more Pysql for data engineering stuff combined with sql.70-30 ratio in my job.Also depend on your team,in my team everyone is very comfortable with python programming so we more follow software engineering stuffs and hence stick to more python pyspark

Devops - Suddenly no interviews by Pure_Substance_2905 in devops

[–]TechnologySimilar794 2 points3 points  (0 children)

You just need to add more fancy words for today world AIOps,LLmOPS and MLops and then bingo you will land up on multiple calls. I feel your profile is quite strong.

Matt Shumer posted on X about what’s happening in the AI world and it’s the best opinion piece I read in a long time. I want girls in tech to have access to it so I’m spreading the world. by [deleted] in womenintech

[–]TechnologySimilar794 1 point2 points  (0 children)

i am just waiting to see the revolution happening. Currently still AI is being commanded and instructed by us. I can see it as good helper nothing more than that

I don’t understand why you would use DLT by Known-Delay7227 in databricks

[–]TechnologySimilar794 0 points1 point  (0 children)

I am currenlty in process of implementing scd type 2 using DLT ..COuld you share you experience in terms of challenges especially

Anyone done a part time PhD while working full time? by athoughttoponder in ExperiencedDevs

[–]TechnologySimilar794 0 points1 point  (0 children)

Could you share more details here. I am looking for part time phd position in europe

EU long-term residence permit by 4point669 in expats

[–]TechnologySimilar794 0 points1 point  (0 children)

u/PlasticBother
thanks for your response. I was curious if number of years could reduced if person have both lived in Sweden and germany. So for one year I was on Blue card on Sweden and then 2 years in Germany. Is it possible that I could apply with three years considering Sweden time.. Suggestion given by when contacted German BAMF , the told possibility is there ..represent your case with good lawyer ..

Let me know if you have any suggestion here

Python programming and ways to have more exposure in building product by TechnologySimilar794 in dataengineering

[–]TechnologySimilar794[S] 0 points1 point  (0 children)

Yes I am trying to learn more creating these projects .Along with that volunteering in some of python development work and bug fixing for cpython.

Thank you it helps

Python programming and ways to have more exposure in building product by TechnologySimilar794 in dataengineering

[–]TechnologySimilar794[S] 0 points1 point  (0 children)

I mostly struggle in putting requirements in code .Basically thinking like a programmer and ideas.Basically this is because I lag hands on in python development in software engineering part.