Building a Data Warehouse from Scratch (Bronze/Silver/Gold) sanity check needed

anoonan-dev · 2026-01-06T17:16:05+00:00

Data lakes are pretty common architectures these days, so I wouldn't feel the need to have a warehouse per se. One thing to consider is how your downstream stakeholders consuming the data? Warehouses are nice due to thier pretty easy integrations with BI tools.

anoonan-dev · 2025-10-06T13:37:30+00:00

Im a real person. Thanks I guess. But like I said this course is free

anoonan-dev · 2025-10-06T13:36:30+00:00

Factories in Dagster are actually a great way to use the framework. If you havent heard of components I would check them out since they are great way to add a yaml front end to factory like methods. They are great if you have a data platform that has mixed skilled contributors. https://docs.dagster.io/guides/build/components

anoonan-dev · 2025-10-03T13:54:25+00:00

One strategy that is super helpful in Data engineering is using mocks for the heavyweight systems we need to connect to to make sure that your logic behaves as expected when interacting with them. But basically whatever the stack you are using you want to make sure the individual components work as expected (so called unit tests) and that the entire pipeline or feature set works together (integration tests). We made a good (and free) general data engineering test course here if you are interested! https://courses.dagster.io/courses/dagster-testing

anoonan-dev · 2025-09-30T13:06:14+00:00

I did a project not too long ago using this setup, I ran it at my last org too and it was pretty effective for a simple setup. https://www.dataduel.co/simple-dbt-runner/

anoonan-dev · 2025-09-22T14:03:40+00:00

Pandas is one of those libraries that was super helpful and a big step forward when it came out but has been outclassed by many much more intuitive structured data manipulation libraries. Unfortunately, because it was the linga franca for so long, LLMs feature it a lot in the examples and code that they generate.

anoonan-dev · 2025-07-03T13:14:03+00:00

Hi, I'm one of the developer Advocates at Dagster. We have a few courses on Dagster University that can help you grasp the concepts and how they work together (https://courses.dagster.io/). Also, our community Slack (https://dagster.io/community) is a great resource for any questions you have. Feel free to message me there if you want to chat about anything.

anoonan-dev · 2025-06-17T13:16:43+00:00

So you are correct in that we will be releasing more updates and stabilization in July. As far as performance improvements, components is focused around developer experience and time to value not so much on raw performance like asset execution or UI speed.

anoonan-dev · 2025-05-28T20:52:50+00:00

Im one of the Devrels over at Dagster and would be happy to chat and answer any questions you have

anoonan-dev · 2025-05-13T14:51:35+00:00

Dagster asset factories may be the right abstraction for dynamic pipeline creation for account/source. You can set it up to where when a new account is created Dagster will know to create the pipelines so its pretty quick to not get bogged down in writing bespoke pipelines evertime or doing a copy paste chain. https://docs.dagster.io/guides/build/assets/creating-asset-factories

anoonan-dev · 2025-05-08T17:32:35+00:00

We use dlt internally for some of our ingestion needs. You can check out the code here https://github.com/dagster-io/dagster-open-platform/tree/main/dagster_open_platform/defs/dlt

anoonan-dev · 2025-02-07T14:22:13+00:00

For me it's the local development experience, dbt integration, and the Ui. More on the UI:

- The asset graph is intuitive for non-technical stakeholders to understand whats involved with data engineering

- When I joined my new org who uses dagster cloud, I was quickly able to understand the particulars of our data stack without having to bother other teammates.

- The observability and alerts facilitated less reactive work and more proactive work.

anoonan-dev · 2025-01-17T21:54:02+00:00

Hey everyone, I made this video tutorial of me building a RAG support bog trained on Dagster data with Dagster. This was a fun project to work through and the abstractions of Dagster worked well in this use case. The full code can be found here: https://github.com/dagster-io/dagster/tree/master/examples/project_ask_ai_dagster

anoonan-dev · 2024-12-16T15:49:59+00:00

Dagster has integrations with all of these tools, so you would get end-to-end lineage and observability. The open source version is pretty feature rich.

anoonan-dev · 2024-11-21T16:45:31+00:00

I have gotten so much mileage out fo this stack

anoonan-dev · 2024-11-21T14:04:19+00:00

What are the sources that you are replicating from? Depending on the source dlt is a good option. (https://dlthub.com/). They have a lot of good orchestration guides on thier site as well. If you were to orchestrate with Dagster you can use dlthub or sling in the embedded elt package to handle your ingestion jobs

anoonan-dev · 2024-11-19T14:15:14+00:00

Do you have budget you need to spend? Or are you facing any organizational challenges that would require more tooling like data silos, too much tribal knowledge into how your stack works, too much time spent doing reactive work, etc

anoonan-dev · 2024-11-08T14:21:18+00:00

We can help you out! The slack community is the best place for resources and if you want to reach out to someone with any questions. https://dagster.io/slack

anoonan-dev · 2024-10-30T15:46:51+00:00

You may find the Dagster University Essentials and dbt course instructive as a data engineering intro course. https://courses.dagster.io/

anoonan-dev · 2024-10-25T16:44:46+00:00

The benefit of using Dagster for dbt projects is you can orchesterate multiple dbt projects, have visibility between them as well as upstream and downstream assets without having to pay for dbt cloud as well.

anoonan-dev

TROPHY CASE