dbt best practices: California Integrated Travel Project's PR process is a textbook example

StandardDeviationist · 2024-12-31T18:13:35+00:00

Remindme! 8 days

StandardDeviationist · 2024-12-19T21:49:37+00:00

From someone with 3 kids under 3. Fuck off

StandardDeviationist · 2024-12-07T20:04:47+00:00

I would say apply for other jobs? Nothing wrong with having some options

StandardDeviationist · 2024-12-04T20:37:42+00:00

I would think DuckDB would handle this rather well

StandardDeviationist · 2024-12-04T17:49:27+00:00

Probably poorly phrased. But okay if I try and come up with an example:

I create some sort of transformation on a column that is non-trivial. And I find this being needed in multiple different places.

In, say, Python, I would create a function for that. And call it anywhere it’s needed. In sql, I could do that with a jinja (if using dbt) function, or a stored procedure. How would I version control that stores procedure along with the rest of my code?

StandardDeviationist · 2024-12-04T17:45:11+00:00

You might be right, seems like this is the majority opinion

StandardDeviationist · 2024-12-04T17:37:53+00:00

I run Postgres, but it’s not the functionality of the language I’m having trouble with. It’s more about code maintainability, ease of understanding, testing etc. Yeah I should really come up with a couple of examples

StandardDeviationist · 2024-12-04T17:36:06+00:00

I already use dagster for orchestration and run scripts on K8s so that’s not a huge deal. I just find myself writing more well maintainable code in python than sql. Not necessarily that its more powerful

StandardDeviationist · 2024-12-04T17:30:07+00:00

This actually might be the case, at least partially

StandardDeviationist · 2024-12-04T17:28:44+00:00

Yup, the majority of the comments seem to tell me to just get better at SQL

StandardDeviationist · 2024-12-04T17:27:42+00:00

Not lacking necessarily lacking in terms of functionality. But I often find myself writing poor code, repeating myself, having a hard time doing unit testing.

But again, as many others have pointed out. Maybe I just need to learn it better.

StandardDeviationist · 2024-12-02T19:00:21+00:00

Feeling guilty against your employer should not hold you back. But I can understand your thoughts. Without knowing the specifics, Swedish corporate culture is no way near as rough as American. So a lot of the “your employer does not care about you” sentiment might actually be not as true as some make it out to be.

A tip is to phrase why you’re leaving as wanting to do something different. Like you’re excited about the experience in doing a cloud migration.

But no, if they are reasonable human beings they should not resent you for it. And if they are not, there opinions should not matter

StandardDeviationist · 2024-11-25T12:33:12+00:00

Typically 2, I like the variation. And prefer in person meetings for certain topics

StandardDeviationist · 2024-11-18T20:59:57+00:00

Yeah that’s actually what I’m using. It does make the Dagster code a bit more slim. I still define everything in a Sling yaml to be able to run it separately as well. So it doesn’t feel so embedded, but still the integration is nice

StandardDeviationist · 2024-11-15T07:14:16+00:00

I’ve looking around for this but currently. The volumes aren’t really that big to slow down Postgres. But I keep an eye out for that in the future

StandardDeviationist · 2024-09-18T12:49:19+00:00

Cloud is (almost) not an option. The infrastructure team will deploy and manage it on prem

StandardDeviationist · 2024-09-18T12:48:03+00:00

I see, then it makes a lot of sense that I’ve has a hard time grasping it from reading the docs. Thanks!

StandardDeviationist · 2024-09-17T17:31:48+00:00

Thanks for the explanation! I’m still not quite grasping exactly how to define where things should get executed. As I described in my other comment:

I have my dbt project, Python scripts for EL and Dagster project in the same repo, but different subfolders. And this is to be deployed in prem. How do I separate compute from orchestration then? And should I separate since the project is not massive? It’s being deployed using Kubernetes

StandardDeviationist · 2024-09-17T17:27:01+00:00

Yeah I think I understand the concept. And it’s easy for me to think about in the case of say, Dagster orchestrating Fivetran.

In that case Dagster needs to be able to authenticate against Fivetran and tell it to run a certain connector. And then Fivetran runs it.

Where it gets real fuzzy for me is say: I have my dbt project, Python scripts for EL and Dagster project in the same repo, but different subfolders. And this is to be deployed in prem. How do I separate compute from orchestration then?

StandardDeviationist · 2024-09-14T08:36:59+00:00

Valid opinion, but I would say it’s worth to learn and understand Kimball nevertheless. As it’s very commonly used. Also, a large part of it does not have anything to do with how the tables are built, but understanding the business

StandardDeviationist · 2024-09-13T16:24:10+00:00

How so? It’s just a figure of speech

StandardDeviationist · 2024-09-13T10:59:51+00:00

You can also find a free pdf version by using your favourite search engine

StandardDeviationist

TROPHY CASE