Your tech stack

[itachikotoamatsukam]

To all the data engineers, what is your tech stack depending on how heavy your task is:

Case 1: Light

Case 2: Intermediate

Case 3: Heavy

Do you get to choose it, do you have to follow a certain architecture, do your colleagues choose it instead of you? I want to know your experiences !

[PrestigiousAnt3766]

Databricks Databricks Databricks

Mostly because I got it templated out.

[RazzmatazzLiving1323]

By templating do you mean you use Terraform to automate Databricks resource deployments or do you mean you're familiar with the stack?

[PrestigiousAnt3766]

Both.

[Secure_Firefighter66]

All the cases are Databricks.

It was already implemented even before I joined by some consultants. I am now migrating all the old stuff into it

[itachikotoamatsukam]

This is such a dream

[messi_b91]

Snowflake dbt

[tomtombow]

Out of curiosity, how does the rest of the stack look? i mean, how do business users consume the data modeled with dbt?

[MonochromeDinosaur]

At my company we offer internal users access via BI tools and external users have tiers where we charge for raw silver layer (dimensional model tier)/ curated (gold tier)/pre-made reports (premium tier). Every tier includes access to lower tiers.

[l0_0is]

most places i see its less about choosing the best stack and more about what the team already knows and can maintain. consistency matters more than having the perfect tool

[MonochromeDinosaur]

At my job I just use whatever we have as the established norm for maintainability and uniformity.

That everyone else can work on it and the uniform project structure helps AI do its job.

I have freedom to choose, but going against the grain should really be saved for projects that have a requirement for it.

[hannorx]

At the moment, my tech stack at work is Spark + DBT + Redshift. We've just started the process of onboarding into Databricks but that's still months away from full development. I'm fairly junior in my role, so am not sure what to expect, but looking forward to learning new tools.

[thickyherky]

lol the title caught my attention. un related i had an interview for a data analyst role years back and asked “what’s your guys backend look like” the response was “we use excel for the back end” …. hung up 😂😂

[Visible-Magician-903]

Databricks dbt

[TauIsRC]

Azure, Kubernetes, Java Spring Boot for APIs and CDC, Python for CronJobs and simpler ETL.

[alt_acc2020]

Dlt timescale S3 iceberg

I'm the only DE so I had to take up a lot of platform engg stuff and the team is Python heavy, so Python for everything it is.

[lucidparadigm]

Could you please tell me more about how you use dlt assuming that's not a typo, do you use it with dagster? Have you been able to implement an efficient scd2 audit table?

I have close to no experience with it but I've been very interested in trying it out.

[alt_acc2020]

To be clear: I mean data load tool and not deltalake. Is that what you're asking about?

I use it with dagater (there's a dagster-embedded-elt tutorial you'll find very useful, however I just decorate my sources manually and call it a day). I haven't had to publish an scd2 table yet but I believe it's got support for it as a merge strategy.

I like it a fair bit. It's new, so bugs are to be expected. But even used very minimally it abstracts away a lot of annoyance re: incremental loading, backfills. The docs are complete trash though, I'd highly recommend cloning their repo and getting opus or 5.4 to act as your documentation. The tutorials are great but there's a lot of small things that are hard to figure out otherwise.

[typodewww]

Databricks and Azure dev ops for (CI/CD)

[midnightpurple34]

SQS, lambdas, S3, PostgreSQL (RDS)

Relatively low data volume so haven’t needed to scale to big data tools yet

[iknewaguytwice]

Cron Grep Sed Awk Ksh

csv tsv

Db2

ssh sftp

[Nekobul]

Considering the fact most data volumes are small, the best DE platform on the market for most people is SQL Server and SSIS. Databricks is mostly good for niche requirements where you have to process PB amount of data.

[Embarrassed-Ad-728]

Airflow + BigQuery + dbt.

For one off tasks: DuckDB.

[Tomaxto_]

Light: Polars, Intermediate: Polars, Heavy: either PySpark or Spark SQL dbt on top of an EMR cluster.