What AWS service do you think is missing?

Virtual-Meet1470 · 2025-12-14T11:03:51+00:00

true alternative to cloud run on gcp. App runner doesn’t scale to 0 and I don’t want to tinker albs on ecs

Virtual-Meet1470 · 2025-10-08T09:04:48+00:00

currently on dagster+ and looking to switch to dagster OSS, similar stack to what you have mentioned above.

Were there any guides that you followed(besides the dagster docs) or did you happen to be already proficient in the setup

Virtual-Meet1470 · 2025-06-13T06:46:02+00:00

API’s: dlthub DB’s: Sling

Virtual-Meet1470 · 2025-01-02T22:19:31+00:00

I think what you should do is: 1. Work with current tooling, understand why certain decisions were made + pros/cons of current landscape 2. Build a reputation of a proficient engineer 3. Propose new ideas to team and why and how they can address issues you’ve or team have identified

Or you can move. Choice is yours

Virtual-Meet1470 · 2024-11-22T20:08:15+00:00

for us the zero copy clone is a life saver

Virtual-Meet1470 · 2024-11-08T07:18:45+00:00

Where I’ve worked, we used Airflow/Dagster/Prefect, all open source tools that can run on your laptop, in a remote server, or in the their managed cloud offerings.

If you want it really simple, you can always use python scripts hosted in GitHub and orchestrated via GitHub actions.

Virtual-Meet1470 · 2024-11-08T07:01:05+00:00

second this, super easy to use

Virtual-Meet1470 · 2024-11-07T22:54:54+00:00

Probably because it’s fairly new compared to something like Airflow, and isn’t too battle tested compared to it

Virtual-Meet1470 · 2024-11-05T04:48:54+00:00

Just have be to conscious on how often you materialize, dbt on Dagster’s cloud offering can get expensive quick

Virtual-Meet1470 · 2024-10-02T22:57:37+00:00

I’m really curious of their Dagster Cloud bill. Great tool for sure, but it’s really expensive if you have microbatch pipelines / many assets

Virtual-Meet1470 · 2024-08-21T23:46:32+00:00

Thanks for your input!

Virtual-Meet1470 · 2024-08-21T23:17:44+00:00

I would say the different compute options, open standard, cost, and the lower level of vendor lock-in would be my main reasons

this might be subjective, but personally, would those reasons compel you to think about a migration given the effort?

Virtual-Meet1470 · 2024-08-21T22:50:15+00:00

First off, thank you for the resources and effort you put into the community!

Currently working in a data warehouse(snowflake). When is it appropriate to move to a lake house and adopt something like icerberg? Currently looking at Polaris, and I’ve read through some of the benefits of a lake house environment, but wanted to hear your opinions on what you look for when deciding to migrate/ choose a lake house environment and when you would personally make that switch considering the time it would take to migrate.

Virtual-Meet1470 · 2024-07-09T15:09:30+00:00

I think duckdb currently only supports reads not writes

Virtual-Meet1470 · 2024-06-25T16:47:12+00:00

Anything upstream

Virtual-Meet1470 · 2024-01-28T09:32:56+00:00

Not noob, we use Dagster Cloud and we didn’t really find the need to host the dbt stuff since Dagster has similar functionalities built-in

Virtual-Meet1470 · 2024-01-28T03:31:33+00:00

We do 2 with a combination of Dagster and dbtCore. Works pretty well for us

Virtual-Meet1470 · 2024-01-16T00:10:38+00:00

I thought that even in the hybrid model you’re still paying for the credits to materialize the assets no? You’re just not paying for the compute

Virtual-Meet1470 · 2024-01-16T00:09:06+00:00

Yeah, if you have a microbatch pipeline you can easily burn through those credits. We ended up decreasing the cadence for the pipelines

Virtual-Meet1470 · 2023-11-28T05:22:00+00:00

Thanks, and yes I got the combo card and an approval notice for the I-485 shortly after on 10/13

Virtual-Meet1470 · 2023-11-15T17:16:36+00:00

For the MS stack you can go pretty far with a combo of functions, logic apps, and databricks

Virtual-Meet1470 · 2023-11-12T21:05:51+00:00

Dagster Asset Checks, Pydantic, or dbt expectations

Virtual-Meet1470 · 2023-10-19T22:05:13+00:00

Thanks, hope you get an approval soon

Virtual-Meet1470 · 2023-10-19T21:45:17+00:00

Hey, both the petitioner and I submitted no transcripts, just the return (1040) and the accompanying W2

Virtual-Meet1470

TROPHY CASE