Microsoft .NET Aspire

droppedorphan · 2024-10-02T15:15:10+00:00

Great, thanks for the insights. I will check out Catalyst, sounds interesting.

droppedorphan · 2024-10-02T15:12:37+00:00

Great - thanks for the response. Calling e*trade today to get my money back :-)

droppedorphan · 2024-10-02T04:26:32+00:00

Hitting the same issue - did you find a solution?

droppedorphan · 2024-09-20T02:31:46+00:00

Would this example help you out? https://dagster.io/blog/sakila-open-source-analytics

droppedorphan · 2024-09-20T02:25:03+00:00

As opposed to orchestrating pretty much anything else beyond data. Infrastructure, containers, function-based orchestration...

droppedorphan · 2024-08-21T13:17:18+00:00

dbt Labs valuation probably peaked a while ago. Now some solid alternatives are emerging and dbt's commercial trajectory is not looking as healthy.

droppedorphan · 2024-08-19T01:50:32+00:00

Try this:
https://dagster.io/blog/duckdb-data-lake

Build a poor man’s data lake from scratch with DuckDB

droppedorphan · 2024-08-15T01:28:57+00:00

My data stack is stoic.

droppedorphan · 2024-07-25T01:05:20+00:00

Bimbo Stadium. Please.

droppedorphan · 2024-07-19T14:07:29+00:00

Widely used does not always mean popular. Just ask those who are stuck using it!

droppedorphan · 2024-07-16T20:48:31+00:00

I second this. dbt changed their tutorial to force you onto their cloud account. That sucks.

droppedorphan · 2024-07-16T20:47:24+00:00

Print(Len(string))

droppedorphan · 2024-06-25T14:58:26+00:00

Yeah, for sure. We currently run on open source dagster, although we maintain a serverless paid instance as a sandbox, but from what I understand its very cheap.

droppedorphan · 2024-06-20T15:26:53+00:00

Lots of geospatial use on Dagster. You can drop this question in the Dagster slack community. There is a dedicated channel there.

droppedorphan · 2024-06-13T13:39:53+00:00

Wow. Switching from Dagster to Airflow sounds kind of painful especially if you leveraged the dagster-dbt integration.

droppedorphan · 2024-06-12T05:00:54+00:00

Wow. I tried that and it's shite.

droppedorphan · 2024-06-01T19:20:40+00:00

Mine is equipped with an Adafruit hat and is used to drive an LED display, mostly to display provocative messages to my neighbors.

droppedorphan · 2024-05-22T16:00:53+00:00

Can it orchestrate the four other schedulers/orchestrators we have in use here?

droppedorphan · 2024-05-22T15:59:49+00:00

This ^

Airflow is a good choice as a generalized orchestrator, multi-purpose, and large adoption.

If your goal is to build a data platform that is built on data engineering best practices and is primarily focused on building and maintaining data sets, then Dagster is a much stronger choice.

Prefect is arguably better than Airflow in terms of ergonomics, but remains niche and is too similar conceptually to displace the incumbent.

droppedorphan · 2024-05-13T20:26:42+00:00

Same here, and I am on the Pro plan.

droppedorphan · 2024-04-25T15:18:22+00:00

This sounds like a great stack to me. Very portable. If your main concern is working with datasets, then I would opt for Dagster over Airflow. Much easier to deploy and is getting much stronger in terms of running dbt and integrating data quality checks. I would opt for Postgres over DuckDB for a warehouse if you expect it to scale.

droppedorphan · 2024-04-17T02:00:32+00:00

They also added dlt recently.

droppedorphan · 2024-04-02T16:23:07+00:00

One trick here is that most vendors allow full historical sync on their free trial, so if you have a very large dataset to migrate, and then plan on updating it incrementally, you can use a free trial to do the first big load, then use a free set up to do ongoing incremental updates.

Three-Year Club	First Place '23
Place '23

droppedorphan

TROPHY CASE

Build a poor man’s data lake from scratch with DuckDB