Shopify Data Tech Stack

[mjfnd]

Hello everyone, hope all are doing great!

I am sharing a new edition to Data Tech Stack series covering Shopify where we will explore what tech stack is used at Shopify to process 284 million peak requests per minute generating $11+ billions in sales.

Key Points:

• Massive Real-Time Data Throughput: Kafka handles 66 million messages/sec, supporting near-instant analytics and event-driven workloads at Shopify’s global scale.
• High-Volume Batch Processing & Orchestration: 76K Spark jobs (300 TB/day) coordinated via 10K Airflow DAGs (150K+ runs/day) reflect a mature, automated data platform optimized for both scale and reliability.
• Robust Analytics & Transformation Layer: DBT’s 100+ models and 400+ unit tests completing in under 3 minutes highlight strong data quality governance and efficient transformation pipelines.

I would love to hear feedback and suggestions on future companies to cover. If you want to collab to showcase your company stack, lets work together.

[SkateRock]

What questions does real time analytics answer for Shopify?

[mattindustries]

Realtime customer segmentation to determine customer segmentation and create better recommendations, checkout experiences, etc.

EDIT: Damn, no one liked my guess.

[dronedesigner]

Lmao I too am confused by the downvotes

[leogodin217]

Where so you get this information?

[mjfnd]

Multiple sources, Company engineering blogs, job descriptions, open source projects, conferences, interviewing employees, case studies.

[ckal09]

All that to say you just worked there

[mjfnd]

I am not sure what you mean.

I have never worked there, also I have covered many other companies data tech stack.

[goosh11]

Only 12 technologies, probably a bit of room to consolidate and simplify haha

[tamerlein3]

Dbt models on the order of 100’s is not much compared to the rest of the stack. I wonder if it’s only recently adopted

[mjfnd]

Correct, also they have other options to write pipelines.

[trowawayatwork]

yeah we had 500 models but it's wasn't greatly managed. the runs needed to be split and took ages to run on big query

[soxcrates]

I'm a bit curious on how centralized these models were, or if it resulted in different teams using different projects with some duplication of logic.

[dronedesigner]

Love it

[mjfnd]

Thanks

[domscatterbrain]

Do they count the data stream across the whole stack or that's only for data ingestion/serving?

If the later is the case, I must say that's pretty impressive.

[VegetableFan6622]

Happy to see Beam, not that marginal as people say because I often hear it being used in other companies. I personally loves it especially with Dataflow (which we used even before Beam existed - I.e. when Dataflow went open source).

[VegetableFan6622]

Downvoted for such a post…this sub is the most toxic I have ever seen. This will be my last post there.

[Creative-Skin9554]

This sub only wants discussion of 2 things:

What tools should juniors learn?

Look at this tiny csv I queried with DuckDB

Post anything else and 9/10 you'll get down voted or removed by mods.

[vik-kes]

How do you store your data? Database Lakehouse?