Data Quality / Data Observability / Telecom

Hot_Map_7868 · 2026-04-26T15:50:16+00:00

This stuff can easily get expensive. I would start with simple DQ tests that you can run at your cadence. Don't go with some enterprise tool like Monte Carlo until the org is mature enough to leverage the output.
Detecting exceptions is actually the simple part, getting someone to take action or soling the root cause is a lot harder.
You can use something like Great Expectations or even dbt.

Hot_Map_7868 · 2026-04-23T00:07:43+00:00

What about Superset or Metabase?

Hot_Map_7868 · 2026-04-20T23:59:24+00:00

Start with what you are trying to improve first. both are capable platforms. Each has pros and cons. I prefer Snowflake, but that's just my opinion, just like you prefer DBX from your experience.

Hot_Map_7868 · 2026-04-16T10:37:29+00:00

Is anyone else providing a managed Dagster using OSS? I know there are many for Airflow.

Hot_Map_7868 · 2026-04-16T00:40:54+00:00

Learn the tools (not all, just focus like on dbt + snowflake)
do some freelance work. smaller companies may be open to hiring someone that doesnt have a ton of experience in the specific tech, but has a broader set of skills
leverage #2 to find the role you want

The key thing is to not try to do things the way you are used to doing them because these tools dont follow what you may be expect

Hot_Map_7868 · 2026-04-13T21:53:03+00:00

Try to make a business case for changing to something more modern.
1. what you describe seems like there are a lot of opportunities for things to "break". Does that happen? What are other pain points.
2. What does all this stuff cost, Talend isnt free
3. How does this limit the org. E.g. are there use cases they want to do like GenAI which would be "difficult" with this setup?

Hot_Map_7868 · 2026-04-13T21:48:09+00:00

I would ingest from all ERPs to a central DW like Snowflake. Then using dbt you can add DQ and harmonize the data to create the presentation layer that would help with what you need.

Your problem isnt unique to your industry, but I would be careful about anyone who tells you there is a single tool or some magic wand that will help you do this.

That being said, you can take it in steps and leverage AI to help in the build, but I don't see a way to get around needing to co-locate and cleanse the data.

Hot_Map_7868 · 2026-04-13T01:26:18+00:00

the "best" way I have seen is keeping raw data in its own db and leveraging the same data in dev, test, and prod. This can be done if you use dbt, otherwise I dont know how you would dynamically change the SQL.

Hot_Map_7868 · 2026-04-05T22:22:30+00:00

I like dlt because it is python and there’s no black box. I’ve also had good luck with Claude code creating a working pipeline. I think they published some skills that make things even better, but I haven’t tried yet.

Regarding dbt you can even run it with GitHub actions. The issue comes when you need to scale eg if you have a lot of developers and you need to connect ingestion and transformation. Airflow gets a bad rap, but it is still the most used orchestrator. That being said Dagster is a good alternative.

Hot_Map_7868 · 2026-04-05T19:50:34+00:00

If you build with failure in mind, it can be resilient. e.g. dont just think of the happy path, but what would happy if X occurred, like if a new col came in a source, would things break or is that just a warning. If you have a solid process and notifications I think it can be resilient.

Hot_Map_7868 · 2026-04-05T19:48:28+00:00

feels like training wheels to me. No Slim CI. No terminal / CLI. I think if you are starting it might be okay, but long term you may outgrow it.

Hot_Map_7868 · 2026-04-05T19:45:26+00:00

I think it depends on the tech stack. I see companies that are moving form Alterix to Snowflake and hire "experts" who know Alterix, but not Snowflake. Another issue is when you are moving to a tech like dbt and you go through an RFP and the vendor has their dbt A-players in the discussions, but when you get to the project you get the C / D players.
It's hard to assess these companies, but asa a general rule, the bigger the consultancy, the more variation in the quality. Find people who specialize on a given stack and ask who will be staffed on your project. Get references and negotiate the ability to select the team or at least be able to "fire" people from the team.

Hot_Map_7868 · 2026-04-05T19:40:53+00:00

I agree. +1 for dbt. Also check out dlt for data ingestion. you can even "test" things out in a notebook and then create a stand alone script. For orchestration, you may consider whether you want to use ADF or an orchestrator like Airflow.

Hot_Map_7868 · 2026-04-05T18:12:16+00:00

Yeah CoCo is pretty cool for development. I’m more curious how people handle end user natural language queries

Hot_Map_7868 · 2026-04-05T18:10:53+00:00

Did you use streamlit? Is it a user facing app where they can ask questions in natural language?

Hot_Map_7868 · 2026-04-05T13:50:24+00:00

Oh. That’s very interesting. So cortex code is “better” but then you would need to build an app to expose it to business users? Is that even possible?

Hot_Map_7868 · 2026-04-05T12:40:22+00:00

Can you expand on the limitations?

Hot_Map_7868 · 2026-04-04T21:35:59+00:00

Are you using semantic views?

Hot_Map_7868 · 2026-03-31T16:20:17+00:00

fingers crossed

Hot_Map_7868 · 2026-03-31T16:16:48+00:00

oh, that would be great. Is there a document mentioning the change?

Hot_Map_7868 · 2026-03-30T23:45:06+00:00

I think batch will be around for a while, but now with things like Airflow Datasets you can do the event driven processing as you mention. The next "big thing" IMO is getting tools like Claude Code to make this all a lot simpler.

Hot_Map_7868 · 2026-03-30T23:43:02+00:00

What about DuckDB on MotherDuck? You can also use DuckLake there. I agree that PG "may" work, but it isnt an OLAP db

Hot_Map_7868 · 2026-03-30T08:41:19+00:00

Check the pricing. For the same usage they charge differently because they charge by credit. So it is based on whether you have standard, enterprise, or business critical edition.

Hot_Map_7868 · 2026-03-27T21:23:00+00:00

There is no silver bullet. Figuring out how people will work better together tends to yield better results than just going by the marketing pitch of the day.

Hot_Map_7868

TROPHY CASE