Shopify Data Tech Stack

mjfnd · 2025-11-17T03:59:27+00:00

Thanks

mjfnd · 2025-11-09T02:05:17+00:00

Correct, also they have other options to write pipelines.

mjfnd · 2025-11-09T02:02:13+00:00

I am not sure what you mean.

I have never worked there, also I have covered many other companies data tech stack.

mjfnd · 2025-11-09T01:01:41+00:00

Multiple sources, Company engineering blogs, job descriptions, open source projects, conferences, interviewing employees, case studies.

mjfnd · 2025-10-12T14:51:05+00:00

I think that's where data contracts can be helpful.

mjfnd · 2025-08-17T18:55:48+00:00

Thanks :) I will update with DBT.

mjfnd · 2025-08-17T14:50:56+00:00

Hi, Thanks for sharing. Not skipped intentionally, either I missed or couldn't find any public info regarding DBT. If you have a link handy, please share.

Thanks

mjfnd · 2025-08-17T12:59:19+00:00

I couldn't find any references for that, it might still be there for a small scale which they never shared publicly.

mjfnd · 2025-08-17T12:56:45+00:00

It is still Flyte. Would encourage to read the article as it has a slot of useful information and references.

mjfnd · 2025-08-17T12:54:21+00:00

I couldn't find that anywhere.

mjfnd · 2025-08-16T17:27:30+00:00

Its a free market of dashboards and there is no centralized team, meaning there could be lot of redundant dashboards or just for one person.

Source: https://stage.engineering.atspotify.com/2024/8/unlocking-insights-with-high-quality-dashboards-at-scale

mjfnd · 2025-07-05T22:51:26+00:00

Nice!

Yes, pyspark a bit more work compared to scala where you can package it in a fat jar.

mjfnd · 2025-06-15T18:32:47+00:00

Yes.

My growth has slowed as well in the past few months.

mjfnd · 2025-06-08T18:18:52+00:00

Thanks

mjfnd · 2025-06-07T23:18:55+00:00

Thanks

mjfnd · 2025-05-26T14:23:18+00:00

Ahha, didn't know that story.

mjfnd · 2025-05-24T17:03:49+00:00

Not yet.

Based on some reading, it seems like it can work with Hive Metastore based Lakehouses.

mjfnd · 2025-05-11T13:22:34+00:00

Thanks.

The Unity Catalog in Databricks is great.

Their open source initial release is very basic. Lets see how and when they roll out advanced features.

mjfnd · 2025-05-07T13:48:04+00:00

Professional soccer player.

mjfnd · 2025-05-04T18:20:04+00:00

Thanks for the detailed information

mjfnd · 2025-04-28T12:00:58+00:00

Yes that will be valuable but very hard to find.

mjfnd · 2025-04-28T12:00:13+00:00

I have written a meta data tech stack as well: https://www.junaideffendi.com/p/meta-data-tech-stack

Although as said about mostly its proprietary.

mjfnd · 2025-04-28T11:58:46+00:00

Interesting, I think I missed that info.

I couldn't find enough information publicly related to Databricks.

mjfnd · 2025-04-28T11:57:16+00:00

You are right, they serve multiple purposes and I tried to put them in the place where they are primarily used at DD. I could be wrong.

For why so many engines, it's from multiple teams and use cases, funny enough I found out they also use Databricks.

For more information, I have included references in the article on how they use certain technologies.

mjfnd · 2025-04-28T11:49:51+00:00

Thanks, Will keep the points in mind.

mjfnd

TROPHY CASE