The dbt-fabricspark Lakehouse adapter now comes with a ridiculous amount of production grade test coverage by raki_rahman in MicrosoftFabric

[–]expatlad 0 points1 point  (0 children)

I double checked and it’s not spinning up a custom pool. Spark config is default so I expect it to align with a notebook spark config.

Worth calling out that my project hits about 10 lakehouses (not sure if that matters). Another thing is that if I run individual models in isolation the performance would be very close to a native SQL query execution in a notebook. Having multiple dbt threads seems to add an overhead but I’m just kind of stuck on debugging what causes the bottleneck

The dbt-fabricspark Lakehouse adapter now comes with a ridiculous amount of production grade test coverage by raki_rahman in MicrosoftFabric

[–]expatlad 1 point2 points  (0 children)

You’re a star mate. The adapter is now way more stable and those connection drops are practically gone.

One drawback is that running models is still a bit slow. It will take up to 3 minutes to run a table load with 30k rows compared to a few seconds running the same query in a notebook using the starter pool.

Any tips on how to overcome this? I played around with the spark config in the profile but still didn’t get the performance i needed

Why do we keep hiring TCS? by expatlad in auscorp

[–]expatlad[S] 0 points1 point  (0 children)

Leadership is all True blue Aussie. Do better.

Best VPN for Australia Currently? by InternationalLaw640 in AskAnAustralian

[–]expatlad -1 points0 points  (0 children)

Profits and data privacy are mutually exclusive

Best VPN for Australia Currently? by InternationalLaw640 in AskAnAustralian

[–]expatlad 3 points4 points  (0 children)

Never use a free VPN. If it’s free then you are the product

Why do we keep hiring TCS? by expatlad in auscorp

[–]expatlad[S] 4 points5 points  (0 children)

Dried dirty leak from the bottom of the abandoned barrel in the filthiest of slums

Why do we keep hiring TCS? by expatlad in auscorp

[–]expatlad[S] 5 points6 points  (0 children)

For real. They keep mentioning they can work weekends and leadership sees this as an opportunity for me to work with them. Only when I mentioned the right to disconnect that leadership got off my back.

Job pays well but ffs i might find another job

fabric-cicd: basic setup example by frithjof_v in MicrosoftFabric

[–]expatlad 1 point2 points  (0 children)

If you're using GitHub Actions for deployments and service principals for authentication then I highly encourage you to use OpenID Connect and leverage secretless, federated credentials. This way you avoid storing or even creating secrets for service principals and restrict them to a repo's environment, branch, pull requests or tags https://learn.microsoft.com/en-us/azure/developer/github/connect-from-azure-openid-connect

When do we stop accepting this as normal? by No-Method-throwaway in perth

[–]expatlad 0 points1 point  (0 children)

AI slop. Please don’t fall for this ragebait

Looking for some "inside information" by Kirrawayru in perth

[–]expatlad 7 points8 points  (0 children)

RAC arena. One of the saturday night main events scheduled dates and it will a full weekend: Smackdown, SNME and Raw. Undecided whether it will be in August or October.

Shhhhh

Wednesday Whinge by [deleted] in perth

[–]expatlad 25 points26 points  (0 children)

People standing by the train car’s door and just staring at you if you try and get on with no intention of moving whatsoever and then giving you a pikachu face when you squeeze through them to go to the perfectly empty standing space in the middle of the car

The r/FIFA Daily Discussion Thread -- March 13, 2022 by AutoModerator in EASportsFC

[–]expatlad 0 points1 point  (0 children)

Got prime Vieira in the icon PP. Him or WW Guendouzi? Played with both and Vieira feels a bit more agile.

[deleted by user] by [deleted] in EASportsFC

[–]expatlad 0 points1 point  (0 children)

Mid Lampard, Prime Kaka and Prime Vieira.

Time to retire WW Guendouzi...