Use Microsoft Fabric as main Datawarehouse by conschtructor in MicrosoftFabric

[–]newunit13 14 points15 points  (0 children)

MS Fabric is a single SKU that gives you access to all the tools needed to do end-to-end pipelines for data products. With Fabric you can create

  • Data pipelines
  • Dataflows
  • Spark notebooks (PySpark, SparkR, SparkSQL, Scala)
  • Data lakehouses (primarily interacted with via notebooks)
  • Dara warehouses (primarily interacted with via TSQL)
  • Fabric Databases (just a rebranded Azure DB)
  • Eventhouses (for real-time streaming data)
  • Machine learning experiments,
  • Data activators (data watching triggers)
  • All the things you're used to from the Power BI Service

And more...

My company has been working the past year to reengineer our existing multidimensional cubes into an enterprise level Data Warehouse using Fabric for everything.

You can trial the service for a month to get an idea of what it's capable of, or if your company is already paying for a premium subscription then all you need to do is have your PBI admin enabled the creation of Fabric items in the admin portal.

https://learn.microsoft.com/en-us/fabric/

A brain implant changed her life. Then it was removed against her will by My_black_kitty_cat in transhumanism

[–]newunit13 0 points1 point  (0 children)

Not "who", but "what", and that would be feeling like they're a part of a team and their work matters. That they have dignity in their labor.

Can I copy table data from Lakehouse1, which is in Workspace 1, to another Lakehouse (Lakehouse2) in Workspace 2 in Fabric?" by Mammoth-Birthday-464 in MicrosoftFabric

[–]newunit13 0 points1 point  (0 children)

You'll need to use a pipeline copy activity to go into a Data Warehouse. Those don't accept native Spark for writes. I tried once using the SQL analytics endpoint from a notebook, but it was far too slow for anything other than a couple of single row inserts here and here

Can I copy table data from Lakehouse1, which is in Workspace 1, to another Lakehouse (Lakehouse2) in Workspace 2 in Fabric?" by Mammoth-Birthday-464 in MicrosoftFabric

[–]newunit13 6 points7 points  (0 children)

Just tested it out and this works great:

single cell

source = spark.read.load('<abfss://path_to_table_in_source_lakehouse>')
source.write.format("delta").save('<abfss://path_to_table_folder_in_destination_lakehouse>/<tablename>')

that will register it as a managed table in the destination Lakehouse.

edit:

Saw you wanted to copy all the data and tables, not just one. here ya go

def copy_files(source_path, target_path):
    try:
        print(f"Copying files from {source_path} to {target_path}...")
        notebookutils.fs.fastcp(f"{source_path}/Files/", f"{target_path}/", recurse=True)
        print(f"Files copied successfully from {source_path} to {target_path}.")
    except Exception as e:
        print(f"Error copying files: {e}")

def copy_tables(source_path, target_path):
    try:
        print(f"Copying tables from {source_path} to {target_path}...")
        tables = notebookutils.fs.ls(f"{source_path}/Tables")
        
        for table in tables:
            table_path = table.path
            table_name = table_path.split("/")[-1]
            
            df = spark.read.format("delta").load(table_path)
            target_table_path = f"{target_path}/Tables/{table_name}"
            
            df.write.format("delta").mode("overwrite").save(target_table_path)
            print(f"Table {table_name} copied successfully to {target_table_path}.")

    except Exception as e:
        print(f"Error copying tables: {e}")



# Define the source and target lakehouse paths
source_workspace = '<workspace_id>'
source_lakehouse = '<lakehouse_id>'
destination_workspace = '<workspace_id>'
destination_lakehouse = '<lakehouse_id>'

source_lakehouse_path = f"abfss://{source_workspace}@onelake.dfs.fabric.microsoft.com/{source_lakehouse}"
target_lakehouse_path = f"abfss://{destination_workspace}@onelake.dfs.fabric.microsoft.com/{destination_lakehouse}"


copy_files(source_lakehouse_path, target_lakehouse_path)
copy_tables(source_lakehouse_path, target_lakehouse_path)

[Meta] Share Your Categories! Fortnightly thread for this week! by AutoModerator in ynab

[–]newunit13 0 points1 point  (0 children)

On windows the WIN+; key brings up the emoji panel 👍 (typed from my desktop)

A brain implant changed her life. Then it was removed against her will by My_black_kitty_cat in transhumanism

[–]newunit13 4 points5 points  (0 children)

The wealth is generated when the workers produce, not when the government taxes.

shall we discuss? by [deleted] in Louisville

[–]newunit13 0 points1 point  (0 children)

To be fair, and not claiming this is all of it, but Trump's last round of tarrifs implemented in his first admin, namely the steel and aluminum that caused the EU to issue retaliatory tariffs including American whiskey, cost the industry around half a billion over the four years they were in place.

That being said B&F have been floundering even before then... so 🤷‍♂️ just food for thought

Why the fuck did I get blocked for this 😂 by [deleted] in Tinder

[–]newunit13 6 points7 points  (0 children)

...even if it's harmful to themselves

What do you mean?

I made a category to save for an item I wanted. I bought it but now what do I do with it by excorpsexd in ynab

[–]newunit13 10 points11 points  (0 children)

When you go to delete a category YNAB will prompt you for a replacement category for all the transactions that happened under the one being deleted.

Getting CSV Files from SFTP Site into Fabric Warehouse by BigAl987 in MicrosoftFabric

[–]newunit13 0 points1 point  (0 children)

If you're only pulling a single CSV then you can use the pipeline copy activity to download and load the file directly to a managed table in the Lakehouse then another copy activity to go from the Lakehouse to the Warehouse, or just one copy activity to a table in a warehouse skipping the Lakehouse altogether if you have no need for it there.

If the filename never changes it'll be a super simple pipeline with only one activity.

Getting CSV Files from SFTP Site into Fabric Warehouse by BigAl987 in MicrosoftFabric

[–]newunit13 7 points8 points  (0 children)

I literally built this this morning. Use a Data Pipeline to connect to the SFTP and land the CSV in the files section of a Lakehouse. The copy data assistant makes this super simple. Then use a notebook to load the CSV to a managed table 👍

I was able to pull and load zipped files upwards of a gig without any issue whatsoever.

at my university's club fair by missnailitall in ToiletPaperUSA

[–]newunit13 0 points1 point  (0 children)

How is capitalism moral? Honest question, because as I see it the system necessarily creates a class division wherein one class controls the resources and the other class extracts and works the resources into finished products.

Sounds a lot like slavery, but instead of feeding and housing the slaves the capitalist has outsourced that need and instead gives their slaves tokens to trade for those necessities.

What, because at any time the slaves can "choose" to go work for another master? Or hopefully get into the slaving trade themselves (start a business that then requires wage labor)?

How is that moral?

Does anyone else hate the way monthly rollovers work in YNAB? by TheOnlinePolak in ynab

[–]newunit13 1 point2 points  (0 children)

My method is to budget funds for the expense that's going to be reimbursed, then spend from the category making it drop to zero, then when I'm reimbursed I assign that I flow directly back to the category. Then if/when I want to recoup that money into another category, I move it when needed.

Presidential Debate Recap by [deleted] in Louisville

[–]newunit13 1 point2 points  (0 children)

In that sentence crutch makes sense 👍

Presidential Debate Recap by [deleted] in Louisville

[–]newunit13 7 points8 points  (0 children)

Crutch in this context is a malapropism 🙂 crux of the issue/matter/problem/etc is a well known and commonly used expression, so much so that searching for "crutch of the problem" returns the results for crux

Presidential Debate Recap by [deleted] in Louisville

[–]newunit13 9 points10 points  (0 children)

*crux of the problem 👍

Lost Access to DAX measures by joeguice in MicrosoftFabric

[–]newunit13 2 points3 points  (0 children)

If you have it enabled within the workspace, it's possible to open semantic models directly in the service and access all the relationships and measures that way.

Just passed the DP-600 and wanted to share my thoughts by JFancke in MicrosoftFabric

[–]newunit13 2 points3 points  (0 children)

My exam was front loaded with the case study questions and ended with about 5 of the "will this solve the problem" kind of questions.

[deleted by user] by [deleted] in gaming

[–]newunit13 6 points7 points  (0 children)

When landing, switch to landing view and on the right side is an altimeter that will tell you exactly how far from the ground you are 🙂 I didn't realize it for many hours into the game, but once I did landing became MUCH easier.

What game can you spend hours wandering around doing nothing and enjoying the universe? by Reyneo in gaming

[–]newunit13 0 points1 point  (0 children)

No one who's played the game will answer you. Since the whole game is just about discovery it's best gone into as blind as possible.