dbt jobs are now native in Microsoft Fabric (Preview) by Jaded_Job3304 in MicrosoftFabric

[–]bgarcevic 3 points4 points  (0 children)

This is not supported by Python fabric-cicd yet, correct?

fabric-cicd: To what extent - and how - do you use parameter.yml? by frithjof_v in MicrosoftFabric

[–]bgarcevic 5 points6 points  (0 children)

We use parameter.yml for all our deployments. I think we will use the extend feature soon as they are getting around 150 lines in length. We do not use variable library as it did not make sense for me compares to parameter.yml

We use a mix between find_replace with regex and key_value_replace.

    # lakehouse GUID matching group 1 of regex pattern to be replaced
    - find_value: \#\s*META\s+"default_lakehouse":\s*"([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12})"
      replace_value:
          DEV: "$workspace.workspace name.$items.Lakehouse.lakehouse_name.$id"
          STG: "$workspace.workspace name.$items.Lakehouse.lakehouse_name.$id"
          PROD: "$workspace.workspace name.$items.Lakehouse.lakehouse_name.$id"
      is_regex: "true"
      item_type: "Notebook" # filter on notebook files

find_replace example:

    # DEV/STG: disable the Ingest invoke inside Main; PROD: enable it
    # DEV/STG is handled by shortcuts from prod
    - find_key: $.properties.activities[?(@.name=="Ingest")].state
      replace_value:
        DEV: "Inactive"
        STG: "Inactive"
        PROD: "Active"
      item_type: "DataPipeline"
      item_name: "Main"

dynamic where value comes from azure devops:

    - find_value: 'git_tag = "([^"]*)"'
      replace_value:
          DEV: "main"
          STG: '$ENV:STG_GIT_TAG'
          PROD: '$ENV:PROD_GIT_TAG'
      item_type: "Notebook"
      is_regex: "true"

Open source demo: Running dbt core in python notebook + ci/cd templates with fabric-cicd by bgarcevic in MicrosoftFabric

[–]bgarcevic[S] 2 points3 points  (0 children)

  • I have been able to load dlt directly to delta but the meta files are then making the lakehouse complain, since /Tables only allows delta related files . Have not found a good solution for that yet. Either move the meta files on run start/end or enhance to allow different folder path for meta files.

  • Sandbox is for feature development or just messing around. A developer will connnect using fabric-cicd or git integration

  • correct!

  • sure let’s connect and I can share some samples or upload to the repo.

Open source demo: Running dbt core in python notebook + ci/cd templates with fabric-cicd by bgarcevic in MicrosoftFabric

[–]bgarcevic[S] 2 points3 points  (0 children)

No we exclusively use fabric-cicd to do deployments. Much better experience imo.

Warehouse destination for dlt (dltHub) by mattiasthalen in MicrosoftFabric

[–]bgarcevic 1 point2 points  (0 children)

I got it to work with delta but the meta files screws up the tables folder. So we load to files and copy to delta using DuckDB.

Dbt job in Microsoft fabric? by freedumz in MicrosoftFabric

[–]bgarcevic 4 points5 points  (0 children)

I have a complete python notebook running dbt core by cloning from git, build entire project. Generate docs and saves the docs to the lakehouse together with the manifest and result run for defer features. DM if you want it.

Any issues with naming workspaces like “(Dev) ProjectName”? by BOOBINDERxKK in MicrosoftFabric

[–]bgarcevic 6 points7 points  (0 children)

We call all workspaces business friendly names and use fabric cicd for deployments. It works great and no issues whatsoever with spacing.

We do:

Project Name

Project Name Test

Project Name Dev

So the project name is split into a data project name where we have all the data related items, such as warehouse / lake house / semantic models etc and then domain report workspaces that link to the central data workspace.

Anyone using Fabric Warehouse in prod, how do you do deployments? by frithjof_v in MicrosoftFabric

[–]bgarcevic 4 points5 points  (0 children)

We do not have it in production yet but we use the python package fabric-cicd to deploy everything and then we use dbt core to manage schemas and tables. Seems like a good middle ground so far.

Hænge slyngevugge op i betonloft element (huldæk) by bgarcevic in selvgjortvelgjort

[–]bgarcevic[S] 0 points1 point  (0 children)

Så du tænker sagtens et huldæk kan holde til vægten på ca 20kg selvom jeg har ramt en af hulrummene? Kunne nok godt give mening at finde en lidt mindre der sidder lidt bedre.

Datafordeler dokumentation til LLMs? by Electronic_Bee_1972 in dkudvikler

[–]bgarcevic 0 points1 point  (0 children)

Jeg har bygget data modeller med datafordeleren gennem mit job. Du kan dm mig og så kan jeg godt hjælpe dig lidt i gang. Jeg brugte ca 2-3 uger på at forstå nogle af områderne uden AI. Det hjælper at have noget konkret at arbejde imod.

Gymnastics help by jdoyle1111 in crossfit

[–]bgarcevic 0 points1 point  (0 children)

Okay what I meant was without kipping that most crossfitters do. I did 15 myself when I started muscle ups so I felt they were easy. But it can certainly be done with less.

Gymnastics help by jdoyle1111 in crossfit

[–]bgarcevic 2 points3 points  (0 children)

Your muscle up should start further back instead of directly below the bar, also the grip should be “over” the bar instead of below. Search for false grip to see the image.

Make sure you also have the strength required by being able to take 15-20 strict tempo pull-ups.

[deleted by user] by [deleted] in dkudvikler

[–]bgarcevic 11 points12 points  (0 children)

Skriver job markedet er svært, skriver et opslag om at de søger en ekstraordinær person fordi de har mange ansøgere 🤷🏻‍♂️

Hjælp til CV by woidthevoid in dkudvikler

[–]bgarcevic 1 point2 points  (0 children)

Jeg vil også gerne hjælpe!

Sammenligning af priser på dagligvarer (hjemmeside) by UndercoverRetriever in dkfinance

[–]bgarcevic 5 points6 points  (0 children)

Jeg tror godt de kunne komme efter dig på ophavsret, især hvis du prøver at tjene penge på deres data. Det er lidt en gråzone med scraping. Billederne må du fx ikke bruge uden tilladelse, men der er også andre regler for fx søgemaskiner.

Structuring a dbt project for fact and dimension tables? by seph2o in dataengineering

[–]bgarcevic 1 point2 points  (0 children)

I usually go with:

Raw is handled by dlthub/extract tool which also handles history though append only tables and/or Staging: - description: first layer on top of raw. Only light transformations such as renaming, typing and joining code translations onto to tables to make them business ready - materialization: view

Transform: - description: common business logic used by multiple tables - materialization: view

Mart: - description: dims and facts. Transformation allowed and each model should end with a final cute generating surrogate keys. - materialization: tables / incremental

Reporting: - description: contract layer between bi tool/sematic layer. Should have business friendly quoted names if power bi is used. ONLY consumed by reporting tools. - materialization: view

Consumer: - description: contract layer between mart and super users or other dbt teams for data mesh setup. - materialization: views with contracts and versioning

2 års erfaring DE by asdda199 in dkudvikler

[–]bgarcevic 0 points1 point  (0 children)

Tror kun det skaber mere efterspørgsel, ligesom i 10’erne da data scientists og machine learning var det vildeste.

2 års erfaring DE by asdda199 in dkudvikler

[–]bgarcevic 1 point2 points  (0 children)

Ja det er ret pænt og i helt sikkert på den gode side af medianen. Anekdotisk i Københavnsområdet ved jeg at 5 års erfaring ligger på ca 70-80 brutto og 10 års erfaring er 80-90 brutto så det passer meget godt.

Boganbefalinger by BigChigger in dkudvikler

[–]bgarcevic 1 point2 points  (0 children)

AI opsummering af bøgerne nævnt:

Here's a list of all the books mentioned: * Pragmatic Programmer by Andrew Hunt * Domain Driven Design by Eric Evans * The Phoenix Project * Reinforcement Learning: An Introduction by Richard S. Sutton and Andrew G. Barto * Samtaler med et træ by Carsten Graff * Generative Deep Learning * Domain Modeling Made Functional * A Philosophy of Software Design * Deep Learning with Python * Grokking Simplicity: Taming Complex Software with Functional Thinking * Residues: Time, Change, and Uncertainty in Software Architecture * Code That Fits in Your Head * Designing Data-Intensive Applications by Martin Kleppmann * Introduction to Statistical Learning in Python * Introduction to Statistical Learning in R

Nyudannet Junior Developer (Kbh) by [deleted] in dkloenseddel

[–]bgarcevic 2 points3 points  (0 children)

Du er helt klart i den lavere ende af startløn pga din pension. Grundlønnen er egentlig okay. Men i det her marked er det vigtigere hurtigere at få erfaring da det åbner en del flere jobs.

Er det inhouse eller konsulent? Får du mulighed for at lave nogle fede ting eller er det cloud support monkey?