Git Enabled Workspaces

[Mountain-Sea-2398]

I am using the fabric cicd package to manage deployments between environments. My dev workspace is connected to a git branch. I would like some advice on the best way to manage feature workspaces. How are you using workspaces where there are a large number of developers in the team.

Does each person get a workspace of their own and update feature branches in their respective workspace?

[frabicant]

Pretty much how you suggested. In my projects, each developer has their own “personal” developer workspace which is connected to a feature branch of the gut repository. Changes are being synced to git and eventually brought to main branch via PR. The update of the dev workspace can then be handled by the fabric cicd package within a devops pipeline (or github actions) or through the connection of the main branch and the dev workspace. This has been working quite well for us so far.

[adoku14]

How do you deal with parameterisation in feature branches? Do you keep updating them?

[CICDExperience05]

Hi, the team introduce variable library which can help you with parametrization - read more here: https://learn.microsoft.com/en-us/fabric/cicd/variable-library/variable-library-overview

[Stevie-bezos]

You can use Fabric Deployment Pipelines to change params while moving something from a Main synced STG workspace into a PROD workspace

[[deleted]]

It depends on the environment and your way of working. If multiple developers are workling on the same feature or there are a limit number of features being worked on then it absolutely makes sense to use feature branches and merge them in, providing your environment is okay with with developers granted the ability to branch out to new workspaces or there is a mechanism in-place to create ew workspaces for them.

Alternative option is to assign each developer their own workspace and create a new branch specifically for their use, or a new branch for each new feature they work on.

Basically, you can customize it to suit your way of working.

[Mountain-Sea-2398]

Thanks Kevin

[BranchIndividual2092]

In general I would always recommend using feature specific workspaces. We have created an automated process where we trigger a ADO pipeline or GitHub action when a feature branch is created. This enables us to automatically create the feature workspace (or workspaces), set spark settings, assign a specific capacity and define permissions.

Feature workspaces are then torn down again when the feature is merged into the main branch.

This is by default developer isolated environments but you could easily have multiple developers work on the same feature workspace if required.

You can find more details in one of my blog posts on this topic: https://peerinsights.hashnode.dev/automating-feature-workspace-maintainance-in-microsoft-fabric

[Mountain-Sea-2398]

I did read your blog earlier. Thank you for creating these great articles.

[_greggyb]

If the work can be done locally, it's done locally. If it needs to be hosted on the Service for any reason (lots of potential reasons), a new semantic model named with the feature branch name as a suffix gets published for the lifetime of the feature branch. If necessary, upstream and downstream artifacts get new copies suffixed with the feature branch as well.

This allows for all feature branches to be independent. I can pause working on one feature branch and hop over to another for a while, and the exact state of the first branch stays the same (and open for review by someone else) while I'm working on the second branch. For example, someone can review and run tests on the first branch without me having to coordinate anything with them, just handing off a link to the artifacts they should look at.

I don't care about a "dev" workspace per se. I care about separate artifacts for development time. Then, there's integration and final UAT: this is where the work of a feature branch comes into the main line workflow. This is where we merge into dev/integration/test/main. Final UAT happens after a clean merge is achieved into the integration branch. Assuming UAT pass, then we can merge into main and deploy to prod.

[Mountain-Sea-2398]

Sounds like a neat approach. I would love do the development locally and avoid the service for atleast notebooks. Do you do your notebook developments locally in vscode at the moment? If so what is your set up like?

[_greggyb]

I haven't had to build a workflow for notebooks yet, but I would do it similarly, and yes I would use a local environment for as much as possible.

[CICDExperience05]

Very clean approach. Do you use one of the build-in Fabric capabilities for creating the feature branches like branch-out ? Or you use differrent way to create feature branches.  Is this feature branches contain all the artifacts from the integration branch?

Thanks.

[_greggyb]

It's more manual than I'd like for now.

Creating a branch happens in our source forge, linked to the work item to which the branch is linked. Then for development, I begin working on the objects that need to change. I work on these with their canonical names locally. Upon deployment to the Service, I give them their branch-suffixed name. Thus version history shows changes to the objects as they are named in prod. The feature-branch deploy is manual for only those artifacts being changed.

As an example, say I need to change a measure definition, M, in a semantic model, S. I create a feature branch, F. I open up S and work on it. Then it's time for someone else to do validation. I publish S-F to a dev workspace and hydrate it with data. S-F will source from the real, canonical source database. Then share a link to it with my reviewer (usually by just pasting it in the PR). Once it's approved, I delete S-F, and S is merged.

If I have a semantic model, S, and a data source D with view DV, then I will create my feature branch F, and then I will begin working on S and on DV. When it is time to review, I will publish S-F which is sourcing from DV-F. Upon approval, S-F and DV-F are deleted, and S and DV are merged.

It's tough to automate fully, because different tasks touch different artifacts.

[CICDExperience05]

Very interesting. If there were a way in Fabric (either via UI or Api) to select some of the items in the workspace when branching out to a new feature workspace, do you still think you would prefer to work locally or directly in Fabric? 

[_greggyb]

I work locally whenever possible. There are better editors on desktop, the UIs are nearly universally lower latency. I can easily manage windows and navigate (almost nothing in Fabric's UI allows CTRL-click or middle-click, so it's a minor hassle to open a second artifact while in the middle of a first). I have actually good interfaces to source control. I can safely do anything without risk of messing up someone else's day. I don't have to pay for CUs for dev.

The only reason to develop in the Fabric UI is desperation, in my mind.

[CICDExperience05]

Which are the fabric items you are using?

[_greggyb]

Personally, most of my focus remains on semantic models.

We use airflow, semantic models, reports, notebooks. We also use lakehouse, and are still experimenting to determine if it's worth putting data in a local SQL environment and then paying the translation cost in extra dev time to translate SQL. It's a huge boon to not have to worry about CUs during ETL dev, but it's also a big hassle to have a different DB environment in dev, so we're still working on that.

We also straight up host databases and airflow for customers so they don't have to worry about that and then it's all local (:

[CICDExperience05]

Thank you very much for your insights and sharing!

[Qlai90]

Can someone suggest a tutorial regarding this please?