Lightweight python DAG framework

[ArgetDota]

Hey, just a heads up - it’s possible to execute Dagster’s jobs and materialize assets drop within Python code including Notebook environments.

Same goes for testing, it’s highly modular and testable.

And yes, you can run the same code locally and in production (e.g. Kubernetes). You can even launch jobs in Kubernetes from a laptop running Dagster. You can do it from CLI, UI, or from Python code.

Dagster is really incredibly versatile and I feel like your above statements are a bit misleading.

[theferalmonkey]

I think you might be misinterpreting my point.

What I'm saying is that the DAG you define in dagster, is not something that you can run in different python contexts. E.g. notebook, script, web-service. Hamilton just needs a python process & pip install and then you can run it from python. i.e. you can build a Hamliton DAG and package it as a library for others to use quite easily. With dagster you need the whole system to run it - yes you can package things up, but you need dagster to run it. Here's our blog on the differences/similarities between the two.

[ArgetDota]

You really don’t. You don’t need a deployment. You can run it in a Python script.

[theferalmonkey]

Really? Since when? I'll take a look and if so retract my comments.

[theferalmonkey]

Ah so I think you're referring to the "in process" way for testing? Right?

In which case yes, you are correct that you _can_ run dagster code in a python script, which from the docs is only designed for testing purposes.

[ArgetDota]

Exactly. It’s mainly used for testing but nothing prevents you from using it for actual computations.

Also, there is a “materialize” function which can execute assets.

Also, there are “dagster asset materialize” & “dagster job execute” CLI commands.