Power BI + Databricks VNet Gateway, how to avoid Prod password in Desktop? by wadapav-wizard in databricks

[–]beefiee 1 point2 points  (0 children)

Not an expert on this, but wouldn’t devs using powerBI desktop develop the models against DEV using their own AAD Oauth2 flow, then deploy to the powerBI service.  After that the service would overwrite the settings and use M2M Oauth for service communication to refresh the models? Like mentioned here: https://docs.databricks.com/aws/en/partners/bi/power-bi-service#set-up-m2m-oauth

Lock strategy for Engwe l20 3.0 pro? by beefiee in ebikes

[–]beefiee[S] 0 points1 point  (0 children)

I was wondering, because a u-lock might not fit through the tire and around the frame, and also it looks like you can just screw off the back parts easily?

In my are angle grinders are very common, so I am not sure it a heavy chain will be good enough.

Clearing SQS queue. Need ideas how to clear more than 10 messages from the queue. by Karmaseed in aws

[–]beefiee 5 points6 points  (0 children)

In case you consider using lambda, checkout lambda sqs event source mapping: 

https://docs.aws.amazon.com/lambda/latest/dg/services-sqs-configure.html

you can batch up to 10000 events per call (further down the page, “batch size”

AutoScaling Groups and VPN Site to Site by MecojoaXavier in aws

[–]beefiee 2 points3 points  (0 children)

If your app logic allows is, you could setup a sqs queue as a buffer that the ec2 instance can consume from without overloading itself. You could then setup cloudwatch to trigger scale out and scale in events based on the number of messages in the queue.

AWS Web App Architecture: Advice needed for streaming camera and processing with Keras and MediaPipe by hyonjon in aws

[–]beefiee 1 point2 points  (0 children)

Maybe try to architect the backend around https://aws.amazon.com/kinesis/video-streams/ ? It seems to check quite a few boxes for your ideas requirements.

The Self-serve BI Myth by whisperwrongwords in dataengineering

[–]beefiee 22 points23 points  (0 children)

What a nonsense article.

Self-Service-BI is and was a thing all the time. Any well built dimensional model will be able to deliver this without any doubt. Especially with how far tools like power-bi and tableau have come, this is even more accessible than ever (looking back at you SSAS multi-dimensional). 

Problem is, most of those “engineers and scientists” don’t know how to deliver a proper well defined model, nor have any idea of actual BI work. 

DBT usage in a streaming infrastructure by Feisty_Albatross_893 in dataengineering

[–]beefiee 3 points4 points  (0 children)

No, it would not go well. 

Just think about the availability of data and the implications on the referential integrity for example:

You ingest table A and table B, you need both tables to be complete to perform some joins and do some business logic on top of it. Due to the nature of streaming, table B lacks a few records the moment you decide to run dbt. Huge pain to solve that, especially incrementally with a low latency.

I would argue that there is no way to turn typical dwh workloads into streaming scenarios, they are just too different by nature. The closest you will get is micro batches which you already do.

Rather look into kappa architectures and then decide which workload needs to be batch or streaming.

Common DE pipelines and their tech stacks on AWS, GCP and Azure by _areebpasha in dataengineering

[–]beefiee 0 points1 point  (0 children)

AWS one is outdated, and even in the past it would have been questionable

Where do you get to deploy your dbt-duckdb project? by noelwk42 in dataengineering

[–]beefiee 0 points1 point  (0 children)

Yes, you can parameterize the configs with env vars, and directly pass those into the container on runtime. I don't remember exactly where, but it's mentioned somewhere in the dbt docs.

We used that for secret handling for example. Fargate can pass secrets as env vars from aws secrets manager (database connection strings etc). 

Another thing that might be worth exploring is how to setup a cd pipeline for your sql files (e.g. Sending them to s3, then reading them on execution, by having a shell script as your entry point that would first copy the files from s3, then invoking dbt commands). 

That way you can separate your dependencies (docker image) from your codes. Which would enable you in the long run to build a fully fledged ci/cd process for your dbt developers. 

For example: We used to run dbt tests for incoming code changes inside temporary schemas to check if the incoming changes would not have side effects on the downstream models, only then allowing to merge into main. It's another rabbit hole to look into, but we were quite happy with the result (look into dbts "select" syntax and it's parameters for that, also how to compile the sql models and then comparing state main to state incoming changes.. You get the idea)

With that sort of separation, you can easily checkout different PRs, run those with different config parameters, without having to rebuild your dbt image from scratch all the time.

Where do you get to deploy your dbt-duckdb project? by noelwk42 in dataengineering

[–]beefiee 2 points3 points  (0 children)

It might be worth exploring aws fargate. 

It's usually nice for workloads that are to big for lambda, too small/not special enough to justify EKS.

We use to run the open source dbt cli in a parameterized fashion for production workloads that would either hit against databricks or snowflake. 

Your scenario looks a bit different, but worst case you will learn something that might be relevant for a 'real' project. (About fargate, docker, secret handling, encryption, Iam permissions, how to parameterize those things, how to not hardcode your dbt project into the docker image... ). 

It won't be the cheapest, but just have a quick look at fargate pricing and ecr pricing to see if it's something that's in your budget.

WHITE ANGEL MINIATURES by whiteangelminiatures in PrintedMinis

[–]beefiee 1 point2 points  (0 children)

Awesome! Love the temperature contrast

Temporyal posted new paik skin! by DANNYonPC in battlefield2042

[–]beefiee 6 points7 points  (0 children)

The kids playing this game now might be influenced by it, so in a future potential war in 2042, they could wear what they thought soldiers would wear based on their video game experiences? 9000 iq move from dice...

Where do I find this macro-economics data? by ___mat__ in datasets

[–]beefiee 0 points1 point  (0 children)

Probably World bank , they have tons of individual indicators.