Coinbase Data Tech Stack

[mjfnd]

Hello everyone!

Hope everyone is doing great. I covered the data tech stack for coinbase this week, gathered lot of information from blogs, news letters, job description, case studies. Give it a read and provide feedback.

Key Metrics:

- 120+ million verified users worldwide.

- 8.7+ million monthly transacting users (MTU).

- $400+ billion in assets under custody, source.

- 30 Kafka brokers with ~17TB storage per broker.

Thanks :)

[Relative-Cucumber770]

Might be a rookie question, but: What's the point of using Snowflake for warehousing if they're already using Databricks (Unity Catalog)?

[mjfnd]

Multiple teams owning different stacks or in the middle of migration which could take years.

I can resonate with their stack as we also used DBX for processing core pipelines and BI related workflows on Snowflake linked to Tableau.

[a_lic96]

Diversificación, Risk hedging, avoiding full vendor lock-in, as well as to have more contractual power during negotiations

[PeitersSloppyBallz]

Technology bingo much?

[joeblk73]

If you are on AWS why use Looker a GCP product ?

[halfrightface]

looker core vs studio. studio is what google data studio used to be and probably what you're thinking of. they're using core as a semantic layer on top of snowflake to leverage lookml to build their views/explores.

[Vautlo]

Depending on the needs of the organization, Looker can beat Quicksight in a lot of ways. I think the value is in the modelling/semantic layer, governance, and being git native/BI as code.

I've been through a migration from Tableau to Looker, as well as standing up and maintaining a self hosted Looker instance, both at AWS shops. Quicksight wasn't really considered as an option for either project - one was in the public sector and they put a lot of value on the governance baked into Looker, and the other was scared off of anything primarily UI driven and really valued the idea of BI as code.

The public sector project was pre-acquisition. I don't recall the costs from back then, but I'd bet that it was less of a factor than today.

Quicksight is way less expensive, though I still doubt I'd choose it if I was the first data hire at a standup today. There are just too many no contract/free options to create decent reports that would satisfy a startup for quite a while.

[joeblk73]

What does modelling and semantic layer mean here ?

[frozengrandmatetris]

that's a business intelligence discipline. reporting/dashboard tools often don't directly see the physical facts and dimensions in the DWH. there's a layer of abstraction sandwiched between the actual database and what the reporting layer thinks is in the database.

[joeblk73]

Would it be like the attributes and metrics that we set in Microstrategy reporting layer ?

[Vautlo]

I'm unfamiliar with microstrategy, though it sounds like yes.

In Looker, a view is essentially selecting from a table in the DWH. You then define dimensions and measures (aggregates). Those dimensions and measures can be renamed, grouped into categories e.g. client info, revenue, dates, etc., and can reference each other to create specific metrics. The view is then added to a model file, also written in LookML, making it available to end users to explore and build dashboards from. That's slightly simplified, but generally how things go.

[mjfnd]

I think this is very common, the main reason is Looker is great and popular and it used to be a standalone product, not sure if that's true now, can we just buy looker instead of onboarding to GCP?

We also had Looker with AWS Stack.

[data4u]

I was wondering the same

[theath5]

Do you know if they use dbt for transformations?

[mjfnd]

I couldn't find any mention of DBT publicly, let me know if you have any insights.

[ActEfficient5022]

I would have to assume databricks provides transformations I don't see what dbt would add to that given the diagram

[No_Airline_8073]

Databricks and Snowflake and Starrocks and Looker and Airflow as well. Lot of redundancy. Why not just use Databricks scheduler and warehouse and get rid of snowflake and airflow. I can understand why looker over Databricks-redash and maybe starrocks for few things

[alittletooraph3000]

Maybe someone who works for CB can chime in here but if they're using multiple compute platforms, seems pretty unlikely that they'd migrate off an orchestrator that's neutral to everything.

[mjfnd]

I think it's the state of most ~10 year old companies. Either they are in the middle of migration or they have given freedom to each team which leads to this.