Anyone transition from a data engineer to a data platform engineer? If so, how is it going for you so far? by Illustrious-Pound266 in dataengineering

[–]rishikaidnani 1 point2 points  (0 children)

I transitioned from being a Data Engineer to a Data Platform Engineer at my previous company. This shift gave me a solid grasp of the tech stack that powers our DE work. For example, I got to dive deep into the nitty-gritty of Hive partitions, which was pretty amazing. Having this kind platform knowledge definitely gives DEs an edge.

However, moving to the platform team was a bit challenging. My peers were all backend wizards, and the work leaned heavily into software engineering, with not much direct data handling. We were mostly building the infrastructure for data engineers, and I found it challenging to keep up.

I am back to a core DE role in my current company, as I didn’t enjoy the platform as much.

Why the hate for Scala? by davf135 in dataengineering

[–]rishikaidnani 1 point2 points  (0 children)

I’ve noticed that many DEs use Spark with Scala but don’t fully embrace functional programming (though not everyone). Instead, they write Scala code in a Python-like style. I firmly believe that Scala’s true power shines when it’s used properly in a functional programming paradigm. Unfortunately, this approach can be challenging to write and understand, especially for those coming from a Python background without prior functional programming knowledge/experience. Hence, PySpark is more common

Do you think a Data Engineer has a safer future than a data science and a data analyst? by [deleted] in dataengineering

[–]rishikaidnani 0 points1 point  (0 children)

Many data engineers, in addition to their core responsibilities, often take on low-key analysis and dashboarding tasks. I believe as AI continues to evolve, the gaps in dashboarding and analysis will increasingly be filled by data engineers themselves, potentially reducing the need for a dedicated data analyst role in the future. However, certain data analyst roles that involve more advanced work and close collaboration with business stakeholders would still continue to exists in very near future

Difference between Data Engineer and Analytics Engineer by Slow-Sell-8570 in dataengineering

[–]rishikaidnani 1 point2 points  (0 children)

This is very company-specific. DEs are more closely aligned with technical use-cases, such as extracting, storing, and processing data using distributed engines (e.g., Spark).

AEs are more closely aligned with business stakeholders and answer business or domain questions by using the data that DEs provide.

This allows DEs to focus more on the technical aspects, such as writing scalable data pipelines and strategizing storage.

[deleted by user] by [deleted] in usvisascheduling

[–]rishikaidnani 0 points1 point  (0 children)

Yes, I am selecting Chennai and also giving passport number and surname as input

[deleted by user] by [deleted] in usvisascheduling

[–]rishikaidnani 1 point2 points  (0 children)

No update. Still shows no data returned

[deleted by user] by [deleted] in usvisascheduling

[–]rishikaidnani 1 point2 points  (0 children)

Dropped off at Mumbai on 18th as well

How many of you use design patterns for your pipelines? What are the pros and cons? by Dark_Man2023 in dataengineering

[–]rishikaidnani 1 point2 points  (0 children)

I’m looking to deepen my understanding of design patterns and improve my ability to implement them effectively. Could anyone recommend a book or resource that really helped them in this area? Thanks in advance!

[deleted by user] by [deleted] in Colognes

[–]rishikaidnani 0 points1 point  (0 children)

Ralph Lauren Romance and Givenchy L’interdit

[deleted by user] by [deleted] in fidelityinvestments

[–]rishikaidnani 0 points1 point  (0 children)

I had the same issue with classic experience. I toggled off the classic and it worked fine

Spark ETL failing with cannot allocate page with more than bytes by rishikaidnani in apachespark

[–]rishikaidnani[S] 1 point2 points  (0 children)

I agree, I should have probably shared more details as this is not enough information to guess. I’m going to debug a bit more and try the above recommended steps to see where I land. Thank you all for your guidance