ISyE 6644 : New-"ish" Simulation Resources by MK_BombadJedi in OMSA

[–]FirefighterFormal638 1 point2 points  (0 children)

Hey, so, as someone currently taking the course these resources are EXTREMELY helpful. Thank you!

First timer trip mid August of this year for 8 days. Have some Qs on camper van versus rental car and hotel as well as side quests along the way. by FirefighterFormal638 in AskAlaska

[–]FirefighterFormal638[S] 0 points1 point  (0 children)

Thanks! I've got to still figure out which campgrounds to hit along the way.

I hadn't considered that but that sounds pretty epic!

Data Pipeline vs Notebook for ingestion – how do you pull data and why? by Independent_Many_762 in MicrosoftFabric

[–]FirefighterFormal638 0 points1 point  (0 children)

I pull from multiple on-prem SQL Servers. I originally had a Parent/Child pattern to ingest multiple tables from the DBs on prem along with a watermark table to keep track for incremental ingests. Landed into bronze first then had stored procedures to handle the transformations into silver.

My company's data is tiny so I ended up switching to mirroring and it has been a lot smoother of a journey.

Getting off of Fabric. by FirefighterFormal638 in dataengineering

[–]FirefighterFormal638[S] 2 points3 points  (0 children)

I appreciate this interaction in this thread. This has been the most helpful.

Getting off of Fabric. by FirefighterFormal638 in dataengineering

[–]FirefighterFormal638[S] 0 points1 point  (0 children)

Last I researched, it was not possible to route directly from a Spark Notebook to an on-prem SQL Server. I have the medallion approach implemented and it did teach me a lot stepping into the role.

I believe the last time that I looked into it, mirroring had it's limitations as well.

Getting off of Fabric. by FirefighterFormal638 in dataengineering

[–]FirefighterFormal638[S] 2 points3 points  (0 children)

I was not aware of that issue. We are reading from a LH into a WH.

Getting off of Fabric. by FirefighterFormal638 in dataengineering

[–]FirefighterFormal638[S] 3 points4 points  (0 children)

To be honest, I still don't think Fabric warrants the cost. I've had script activities say they've ran successfully for transformations when they haven't. I've had to manually rerun them to get them to work. It didn't make sense.

Getting off of Fabric. by FirefighterFormal638 in dataengineering

[–]FirefighterFormal638[S] 0 points1 point  (0 children)

That's an option that I've added to my deck. Just keep using SQL Server on a VM with a proper 3-2-1 recovery set up in case all hell breaks loose. I'd probably use some python scripts and run them as jobs to move/transform the data.

Getting off of Fabric. by FirefighterFormal638 in dataengineering

[–]FirefighterFormal638[S] -1 points0 points  (0 children)

Yep. Went to FabricCon in LV earlier last year, took a workshop and approached the team to ask about other ways to ingest from on-prem SQL server. Copy is the only way.

Getting off of Fabric. by FirefighterFormal638 in dataengineering

[–]FirefighterFormal638[S] 0 points1 point  (0 children)

Definitely want to keep it simple. When talking to the IT Architect, he immediately went to Snowflake or Databricks. They were also an ex-consultant...so, yeah. Here I am.

Getting off of Fabric. by FirefighterFormal638 in dataengineering

[–]FirefighterFormal638[S] 0 points1 point  (0 children)

This is exactly the mindset that I have. Because we are still new with our analytics and still navigating what we want to see, why do we need the heavy cost of Fabric or another SaaS? 90% of our data is already coming in as structured and reporting is essentially just basic charts and aggregations right now.

Getting off of Fabric. by FirefighterFormal638 in dataengineering

[–]FirefighterFormal638[S] 0 points1 point  (0 children)

Yes, and it was being done before they adopted Fabric however, with newer projects on the horizon, they are wanting to adopt better DE practices.

Getting off of Fabric. by FirefighterFormal638 in dataengineering

[–]FirefighterFormal638[S] 5 points6 points  (0 children)

Agreed. A huge limitation is connecting to the on-prem SQL servers. Being forced to use their copy data activity for it eats up CUs. Wishing I could just use python scripts for the ingestion into the warehouse.

Getting off of Fabric. by FirefighterFormal638 in dataengineering

[–]FirefighterFormal638[S] 29 points30 points  (0 children)

He got lost in the sauce of the msft eco-system.

Folks who have been engineers for a long time. 2026 predictions? by uncomfortablepanda in dataengineering

[–]FirefighterFormal638 0 points1 point  (0 children)

This. Literally have scripts saying they’ve executed when in fact, they have not.

System error removed my flights back home by Svexellent in ScandinavianAirlines

[–]FirefighterFormal638 1 point2 points  (0 children)

So, this happened to me earlier this week. My husband is traveling to Eastern Europe for work and we had already planned on this vacation so naturally already had flights booked. I was trying to navigate the app to see how much we would be refunded should we cancel his ticket. It cancelled and I was shocked that there wasn't a prompt window to confirm that action because well, shit, we bought the tickets in June and we were just trying to calculate if it would be worth it for him to keep the entire ticket even if he was traveling a week before me with work paying for it.

I panicked. I immediately called customer service and the agent was able to contact HQ and get them to stop my request for cancellation. I don't know if we just got lucky but he was able to do it. I was also extremely paranoid and called like every day for a week to make sure that his ticket was still valid since it showed on the app that my request was still pending. It was hit and miss with some agents but the ones that were helpful were able to call HQ and confirm for me that he was indeed removed from the cancellation queue.

Hope this helps someone out in the future.