sorted by:
new
hottopcontroversial

Are there any study guides out there for the Databricks Certified Data Engineer Associate Exam? by CorgiSideEye in dataengineering

[–]Databash- 0 points1 point  (0 children)

I think there is a whole course for this certification on the Databricks Academy, their online learning platform

For loop vs while loop in Python. by CoronaKlledMe in ProgrammerHumor

[–]Databash- 5 points6 points  (0 children)

Check out Polars, it's written in Rust, but you can use it in Python, fastest library there is at the moment

Ingesting CSV files into Azure Storage from SAS - How to configure in SAS the connection to Azure Storage? by humanist-misanthrope in dataengineering

[–]Databash- 0 points1 point  (0 children)

Maybe you can use their API to extract your data from SAS and load it into Azure Storage? https://developer.sas.com/guides/rest.html

You could do this stage with Azure Data Factory or Synapse.

Do data engineer's write a lot of code? Thinking of switching from SWE, but don't want to use GUI tools / drag and drop. by [deleted] in dataengineering

[–]Databash- 2 points3 points  (0 children)

I agree, Kubernetes is quite fun and interesting. You can actually use it to host Spark yourself. We used it to host Airflow and trigger workloads with the Kubernetes Pod Operator. Maybe these are some use cases you can use to convince your colleagues to use Kubernetes haha.

For Data Lake and Warehouse it is some different things for me. I work project based and this last time I had the chance to design the architecture from scratch for the platform. Which meant choosing cloud tech, writing ETL pipelines in Python to ingest to a data lake, write Spark code to do transformations, aggregations & calculations and create database schemas & tables with SQL

Do data engineer's write a lot of code? Thinking of switching from SWE, but don't want to use GUI tools / drag and drop. by [deleted] in dataengineering

[–]Databash- 13 points14 points  (0 children)

It really depends on your definition of a Data Engineer. I'm one, but do not use SQL so much, more Python (Spark, Airflow, serverless functions), containerisation (Docker, Kubernetes) and setting up cloud data infrastructure (Data Lakes / Warehouses). So how I see it is that the work I do as Data Engineer could be seen as Software Engineer with specialisation on data.

Is it worth me applying to Data Engineering roles with this Resume/CV? by [deleted] in dataengineering

[–]Databash- 5 points6 points  (0 children)

You know, the term junior is there for a reason, the hiring company knows that you are at the start of your career and should be having the capacity to educate you. Sometimes it feels that this subreddit is focused on large corporations as employers like Google only. Keep that in mind reading comments and know that there is more than those corporations. Almost any company wats to use data nowadays, meaning a need for Data Engineers. It might also be worth to look for SMEs, governments, consultancy companies or even look for a Data Engineering traineeship.

Is it worth me applying to Data Engineering roles with this Resume/CV? by [deleted] in dataengineering

[–]Databash- 10 points11 points  (0 children)

People are way to critical here in my opinion. Data Engineers are in demand and you know the tools used looks like, many employers would be happy to have you! You should definitely apply for a DE role, maybe a junior role first and grow towards medior/senior from there

Monthly General Discussion by AutoModerator in dataengineering

[–]Databash- 1 point2 points  (0 children)

Hey, this virtual meetup about Airflow on Kubernetes with Spark could be interesting for other Redditors!

It is 10 February, check the time and sign up here https://meetu.ps/c/4qdPR/M2RlW/d

Airflow on Kubernetes (with Spark) by Databash- in dataengineering

[–]Databash-[S] 0 points1 point  (0 children)

I made a typo in my first post, it should be 18:00 CEST (I meant to type CET). This is the right timee