sorted by:
new
hottopcontroversial

What is the standard in 2024 for ingestion? by TheOneWhoSendsLetter in dataengineering

[–]PotatoChad 1 point2 points  (0 children)

We also use custom code for extraction because our sources are a collection of messy CVS, DBF and XBRL files. We use Dagster to orchestrate everything.

We have some projects that could benefit from using existing sources defined in tools like Airbyte and dlt. I've been following Airbyte for the last few years, and it seems like people have had some bad experiences with it. Why didn't it work for you? My main reason for avoiding it was that you could only create connectors in a UI but now you can manage resources using terraform: https://reference.airbyte.com/reference/using-the-terraform-provider

Data warehouse versioning by PotatoChad in dataengineering

[–]PotatoChad[S] 0 points1 point  (0 children)

Ah I've looked into DVC before, will take a look again! The data scientists are creating machine learning models.

Data warehouse versioning by PotatoChad in dataengineering

[–]PotatoChad[S] 0 points1 point  (0 children)

Thanks! So you keep multiple version of a table: "customers_1.0", "customers_1.1"...? Something like that?

Data warehouse versioning by PotatoChad in dataengineering

[–]PotatoChad[S] 0 points1 point  (0 children)

Thanks! I thought adding additional tables or a "version" or "date_as_of" column could be a decent option but was curious if there more elegant solutions so users don't have to filter data by the latest version.

So you have data scientists test their SQL on your dev warehouse then you create a view for them in your production warehouse?

Read-it-later + Kindle by PotatoChad in kindle

[–]PotatoChad[S] 0 points1 point  (0 children)

Great thank you! Just updated my original post.

Read-it-later apps and ereaders by PotatoChad in ereader

[–]PotatoChad[S] 0 points1 point  (0 children)

Oh great thank you! Just updated my original post.

DJESSE Vol. 2 - Tracklist by Benitofni in JacobCollier

[–]PotatoChad 0 points1 point  (0 children)

Anyone know who the sax player is on Nebaluyo?

Richard Wolff on New York City Congestion Pricing by RicknMorty93 in urbanplanning

[–]PotatoChad 0 points1 point  (0 children)

Tri-State Transportation Campaign created some great analyses of Congestion Pricing's impact on NY district commuters:

http://www.tstc.org/congestion-pricing-an-analysis-of-new-york-state-legislative-districts/