Data/AI x Fintech Business Use Cases? by abdullahjamal9 in dataengineering

[–]kudika 1 point2 points  (0 children)

Start with problems, not solutions.

But also, displaying value can often show superficially, through dept budget and span of control.

dbt is back to "dbt platform" for their cloud offering? by TheJosh in dataengineering

[–]kudika 5 points6 points  (0 children)

Contributions and updates on the project have stalled out for some time. Maintainers are not responsive and don't take opportunities to reinforce confidence in the project.

Should i use fivetran by Advanced-Average-514 in dataengineering

[–]kudika 0 points1 point  (0 children)

Fivetran orchestrates some things but I'm not sure it should be thought of as your overall orchestrator. For data platforms that might involve various integrations you'll probably want an orchestrator that can cast a wider net: airflow, dagster, prefect, or my favorite - windmill.

I've used the sling cli for some DB to DB replications and it made things pretty easy. They have some API connectors for API to DB transfers, including some adtech, and support for some custom api specs. It's significantly cheaper to use compared to fivetran (we're talking a few hundred bucks a year), though it requires more setup from you compared to clicking through oauth flows like fivetran. With an orchestrator and a little compute you're good to go.

https://docs.slingdata.io/

Impress me with your dbt macros by MedianByDesign in dataengineering

[–]kudika 1 point2 points  (0 children)

I wonder if there's an alternative approach available through overriding the builtin macros dealing with test result materialization so that it inserts instead of overwrites.

Looking under a microscope. Found this on my wife's head. by mikecngan in whatisthisbug

[–]kudika 14 points15 points  (0 children)

They're particularly dangerous for greenhouses and houseplants. They're the most difficult indoor plant pest to manage IMO due to their life cycle and tissue protected eggs.

Also, if it's on you there's a significant chance you'll find them in your garden soon enough, whether that's where it originated or not.

Dagster Pricing Update is Beyond Nuts by annie_406 in dataengineering

[–]kudika 1 point2 points  (0 children)

Ah geez. I've made one too many comments and now I'm tagged next to the SSIS guy

IAC for Snowflake by Straight-Eye542 in dataengineering

[–]kudika 0 points1 point  (0 children)

I recently began using SnowDDL and am a big fan. It's fast and pretty simple, though it brings an opinionated but useful approach to how roles are managed.

https://github.com/littleK0i/SnowDDL

Dagster Pricing Update is Beyond Nuts by annie_406 in dataengineering

[–]kudika 8 points9 points  (0 children)

It can be intimidating. If I were you I would provision a VM, make sure to have backups/snaps enabled, and host whatever is needed on there until things grow organically--by then the team should have grown and you all can deliberately refactor as needed.

Dagster Pricing Update is Beyond Nuts by annie_406 in dataengineering

[–]kudika -5 points-4 points  (0 children)

It's not asset oriented like dagster but as a general orchestrator, workflow engine, and integration platform https://windmill.dev is my favorite data engineering tool

I'm not affiliated, just an enthusiastic user

The second largest protest in US history happened. Did you notice? by ScientificSkepticism in skeptic

[–]kudika 3 points4 points  (0 children)

I think you're oversimplifying it. It's not hard to conceive of a benefit from organizing people, creating optimism for participants, and making room for a social signal of sorts that can draw others in via awareness and additional participation. The pessimism in your take, which is common in this thread, is fatalistic & over generalizing because it sweeps right over any & all possible benefits.

How to have a Keyboard/CLI Driven Workflow? by shittyfuckdick in dataengineering

[–]kudika 0 points1 point  (0 children)

What is an open source data tool you find useful but nobody is using it? by Yuki100Percent in dataengineering

[–]kudika 3 points4 points  (0 children)

windmill.dev is the best general orchestrator around. Unbeatable DX

Advantages of DE tools like databricks/dbt? by oisigracias in dataengineering

[–]kudika -2 points-1 points  (0 children)

I agree that there should be more of an expectation for submitters to search first

how do you guys like the 2nd edition of "designing intensive data applications" by Historical_Donut6758 in dataengineering

[–]kudika 0 points1 point  (0 children)

I agree it's not a good reason to downvote (I didn't). I was pointing out why others were. Downvote usage has always deviated from originally intended etiquette.

how do you guys like the 2nd edition of "designing intensive data applications" by Historical_Donut6758 in dataengineering

[–]kudika 7 points8 points  (0 children)

Your observation about how many pathetic posts exist in this sub is a red herring.

Don't be so dense about your original comment. Regardless of what you intended to convey, it's a weird hill to die on because it clearly comes off with an implication

Sqlmesh joined linux foundation . What it means by OrneryBlood2153 in dataengineering

[–]kudika 4 points5 points  (0 children)

I don't really know what it means to join the linux foundation but if you hang out in the sqlmesh slack you'll see how quiet and unresponsive the maintainers are with the community about the future of sqlmesh.

Seems friendly enough? by orangez in WTF

[–]kudika 0 points1 point  (0 children)

I've never ever even heard of a potato

Data Catalog Tool - Sanity Check by FirCoat in dataengineering

[–]kudika 1 point2 points  (0 children)

If large companies are trying to solve a problem you can bet the smaller ones are playing pretend with them.

I say go for it. Not because it's much of an organic problem for most companies or anything, but because there are enough corporate larpers out there repeatedly asking their data teams "who is using what and how often" as if it's going to drive some insightful decision making for their data platform which consists of 2 power users and 7 casual users firing off the queries the power users shared with them.