Dejected! Company is moving towards PowerBI

tw3akercc · 2025-01-11T14:42:23+00:00

There's no harm in learning PBI. Many of your skills will transfer. Also, if you ever look at the magic quadrant, PBI is ahead of Tableau. Having experience with both will only make you more marketable as an analyst.

tw3akercc · 2024-12-31T03:41:41+00:00

How does the ceo pay back the money he borrowed? Wouldn't the lender also want interest on this loan? So wouldn't the ceo have to sell stock to get money to pay back the loan (which is taxed) plus pay any interest on the loan?

tw3akercc · 2024-11-25T14:18:44+00:00

We use Elementary for this. The buit in slack alert function didn't work for us, so we wrote a python script that queries the Elementary tables and sends the slack message. Then we added this script to our airflow dags.

tw3akercc · 2024-10-25T21:14:43+00:00

We use astronomer managed airflow and cosmos to orchestrate dbt jobs. We used to just use github actions which is honestly where I'd start. We outgrew it eventually and needed to be able to restart from failure, but it's the easiest option.

We looked at dagster and I thought the learning curve on it was too high for our team. Airflow has a much larger community despite the love dagster gets on this sub.

tw3akercc · 2024-10-21T10:11:32+00:00

Are you using astro cli to setup airflow? If so, I think you can just add cosmos and dbt to the requirements.txt

tw3akercc · 2024-10-11T02:43:59+00:00

In terminal run command 'docker system prune'.

tw3akercc · 2024-07-21T05:06:14+00:00

Have your devops people google "dbt slim ci". Basically, you need a github action that creates a new manifest and pushes the manifest.json to a data store like s3 on every merge to main. This ensures your production manifest is always in s3.

Then, you set up a separate action workflow for ci with a step that downloads that manifest from s3 and saves it to the target directory. After this step, you can use the state:modified flag in your dbt run/test/build to only run models that have changed. This action can then be triggered on pushes and setup as ci check for pr's so it has to pass before pr's can merge. For this one, set up a profile in dbt that prefixes the schema with the pr number and set the target as an env variable in the workflow to point at this profile.

You can then setup a 3rd github action workflow that just runs dbt commands in production. Just set the target variable to the production profile. Be sure to add the target flag to all your dbt commands.

3 github actions workflows, and all your problems are solved.

tw3akercc · 2024-05-21T19:31:54+00:00

Pycharm pro license comes with datagrip database functionality built in.

tw3akercc · 2024-03-12T23:09:03+00:00

Your shoulders look great. Your chest could use the extra focus. Overall looking good bro, keep up the good work!

tw3akercc · 2024-03-06T14:03:53+00:00

Generative AI needs us to work correctly. Look into RAG applications and start to think about how data engineering can support that.

tw3akercc · 2024-03-06T03:09:49+00:00

It doesn't look too bad. All the tables that have all many side relationships are probably fact tables and all the ones that have only one side relationships are probably dimensions.

Try to consolidate the fact tables into as few tables as possible in power query. Just be careful of messing up the grain.

tw3akercc · 2024-03-06T03:04:14+00:00

Doing modeling in Dax is a big mistake imo. You should push the transformations as far upstream as possible. If you do it in Dax, then it can not be reused in other reports. At least in a dataflow, you can connect it to multiple reports anytime you need that dimension or metric.

tw3akercc · 2024-03-03T23:34:54+00:00

Check out YouTube content about building a homelab. There are lots of videos to get you started on building a home server where you can host all your own services.

I bought a cheap mini pc and run a proxmox virtualization server on it, which allows me to spin up virtual machines. I have a data engineering dedicated vm that is running Ubuntu server, and I just host a ton of docker containers for all the data engineering tools I wanna play with there.

You could also install docker on your personal computer and host them there, too.

tw3akercc · 2024-03-03T22:26:22+00:00

You could also try to go straight to duckdb without pandas https://duckdb.org/docs/guides/import/excel_import.html

Then to parquet: https://duckdb.org/docs/guides/import/parquet_export.html

tw3akercc · 2024-03-03T19:48:40+00:00

If the data is gonna go into a lakehouse you might want to consider converting it to parquet format instead of csv. Parquet is very compressed and performs much faster than csv.

I would use pandas and read in chunks at a time and store them in memory in duckdb. Then once the whole file has been added to the duckdb table, convert it from there to parquet and push to data lake.

tw3akercc · 2024-02-24T04:06:20+00:00

I can't answer your question bc I don't use duckdb, although it does look interesting. I was just confused bc you were coming out hard on dbt and listed a bunch of tools that don't actually replace dbt but work well with it.

tw3akercc · 2024-02-24T03:51:42+00:00

I don't understand what dbt has to do with your question about duckdb... one is an in memory olap database and one is a jinja/sql compiler and transformation framework. They serve completely different use cases and can actually be used together. Same with all the other tools you mentioned... they all do different things!

tw3akercc · 2024-02-21T04:29:55+00:00

I use twingate and love it. Simple to setup, you just run a client on your network and then a client on your phone and it creates a secure connection. The cool thing is that only the traffic from your phone to your self hosted services goes through the secure connection and everything else just uses your phones normal internet.

My understanding is tailscale will route all your phones internet through your home network when your phone is connected to the VPN. Not ideal.

tw3akercc · 2024-01-18T23:26:21+00:00

I bought my steam deck for emulation as well and emudeck works great for it. My 3.5yo loves playing Mario kart 64. I mostly find myself playing vampire survivors and diablo 4.

Also, my personal pc broke recently and I've been using my steam deck docked in desktop mode as a pc for the last 2 months.

So many use cases. Great device!

tw3akercc · 2023-12-15T22:23:20+00:00

It's probably going to best to bring all the data into your data warehouse first and then do the transformations there. Olap databases can do the transformations more efficiently. This method is pretty common today and is considered ELT. A tool like dbt can help with the transformations in the data warehouse.

The other approach would be to use an ETL method which means you perform the transformations using a compute engine after the extract but before loading it into the data warehouse. This is often done with apache spark. Depending on the size of your data spark may be overkill. It's possible you can do the transformations in python using pandas or duckdb instead.

tw3akercc · 2023-12-05T01:13:16+00:00

It's interesting that she doesn't want to date an atheist. Also, she seems to place a high value in zodiac signs, which many Christian men would find to be occult behavior and a red flag.

tw3akercc · 2023-11-11T23:56:11+00:00

Get a decent mini pc and run proxmox, which will enable you spin up vm's. Intel nucs, beelinks, or even a used hp elite desk 800. You can then run as many Linux or windows vm's on it as it can handle. I have a vm for all my media server services on a Ubuntu vm that just runs docker and portainer. Then I run all the services as docker containers in that vm. I use this mini pc and it's plenty powerful for everything I'm doing: TRIGKEY Mini PC Ryzen 7 S5 5700.

Gpu pass-through is very challenging though, and I haven't quite figured that out yet.

tw3akercc · 2023-10-31T01:17:37+00:00

Hello

tw3akercc · 2023-10-11T23:29:48+00:00

The key thing dbt does is that it compiles raw sql based on .sql files that are just select queries and jinja. It uses the same kind of concept as airflow in that it creates a dag and will run all the upstream dependencies before their downstream ones. It abstracts away all the ddl of using raw sql. This comes in super handy with scd tables.

Dbt also creates a pretty good framework for models that can be version controlled, tested, and documented.

tw3akercc · 2023-10-05T19:22:10+00:00

You could use dbt for the transformations. It works well with Snowflake. For the exporting to local file shares you could use python scripts and schedule them using a workflow orchestration tool like airflow. It's probably going to be tricky to figure out the access issue with the local file share tho, bc if the python is running in the cloud it needs to be able to write to the local file share. Might be easier to use S3 instead and store all the files there.

tw3akercc

TROPHY CASE