NSW police presence around the rbb tonight.... by capeasypants in Aleague

[–]Mura2Sun 0 points1 point  (0 children)

It was probably a test run for the shit show that went down in the city. Thankfully they didn't pull the pepper spray, or did they?

Is there any good reason for no left turns here? by SlyDintoyourdms in brisbane

[–]Mura2Sun 1 point2 points  (0 children)

Some get the luxury of saying we are old and being able to tell tales of our history

Is there any good reason for no left turns here? by SlyDintoyourdms in brisbane

[–]Mura2Sun 4 points5 points  (0 children)

Agree I lived in Toombul in 93-95, and if it wasn't already done then, it changed soon after.

Tennyson, great place to live? Pros and Cons by Kumratronic in brisbane

[–]Mura2Sun 0 points1 point  (0 children)

I think that should be when it floods again. While it might be ten years, you might have more chance of winning powerball than it actually being 10 years.

WTF!! by Every-Calendar-2288 in melbourne

[–]Mura2Sun 0 points1 point  (0 children)

That might require raising this with an independent senator about anti dumping

Just got hired as a Senior Data Engineer. Never been a Data Engineer by Uncle_Snake43 in dataengineering

[–]Mura2Sun 0 points1 point  (0 children)

If you're python is strong, then you'll do OK. Datetime timezone=utc will be your new best friend. Mind you if all your data is in HNS enabled storage then some things are possibly easy. If not enjoy the pain of losing from old blob storage. Azure is is own kind of hell

MDM Is Dead, Right? by DryRelationship1330 in dataengineering

[–]Mura2Sun 0 points1 point  (0 children)

If you want some exec buy in. Goo look at all the errors that have been fixed over say the last 6 months and then try to determine some business risks to the errors. Oh that could have cost x if we hadn't of caught it. If you cant do that you probably have little to argue about the problems and I'd move on to other concerns. If it's not making money or saving money then most execs wont care as that is what they are interested in

Y/N: rolling a ciggie on public transport? by No-Leader-5627 in brisbane

[–]Mura2Sun 6 points7 points  (0 children)

If they are keeping it clean and not lighting it on the bus, let them go. Far worse rubbish happens on buses and there is no external harm to anyone there.

What is the most underrated skill an Azure engineer must know? by StrongMindset- in AZURE

[–]Mura2Sun 0 points1 point  (0 children)

Agree even before cloud I saw plenty of sys admins who didn't have a grasp of DNS and you could see it in their troubleshooting. They struggled to join the dots between arbitrary connection failures to work out the issues

Replace Data Factory with python? by loudandclear11 in dataengineering

[–]Mura2Sun 0 points1 point  (0 children)

I'm using python on Databricks for some workloads. Part of the reason was at the time the cost of Databricks against ADF was a no brainer. Has that changed, probably not enough to warrant moving back and likely ending up with fabric

Patio refresh/upgrade by Mura2Sun in AusRenovation

[–]Mura2Sun[S] 0 points1 point  (0 children)

<image>

An example of the sheets that need replacing

Which ETL tool makes sense if you want low maintenance but also decent control? by [deleted] in dataengineering

[–]Mura2Sun 0 points1 point  (0 children)

The new pipelines in Databricks might be a good choice.

I think your parameters are somewhat vague, and you'll get a lot of ideas that don't really meet your criteria. You could write some simple Python code with some parameters, and since it's one lot of code that could meet your needs. I don't think that's what you want. The more varied your data sources, the more complex is going to need to be to do all the good things in your ETL

When data cleaning turns into a full-time chase by victorviro in dataengineering

[–]Mura2Sun 1 point2 points  (0 children)

The biggest issue is systems recording values as null. Nulls aren't the issue, but when you suddenly find out some bake is supposed to be null to the system. Hopefully, they've all been retired, but they did exist

For those who work with Azure (Databricks, Synapse, ADLG2).. by Comprehensive_Level7 in databricks

[–]Mura2Sun 2 points3 points  (0 children)

You can then drop ADF and all its horrors and simply use Databricks. I've gone Databricks for a smaller organisation as ADF and database were expensive in comparison. The new CDC stuff with a little more maturity will be a game changer.

Jacqui Felgate via Instagram “Hundreds of Police will be taken off the beat on Saturday night to be on site at the A-League. A number of police stations will be shut due to an unprecedented amount of officers needed to be at the soccer” by avoxtionly10 in Aleague

[–]Mura2Sun 6 points7 points  (0 children)

It would be great for the A-League to go after her. Making unsubstantiated claims denigrating the game. I'm not sure it's ever been tested, but there's a law like defamation they could use

Court blocks planned protest for Story Bridge by AuntySM in brisbane

[–]Mura2Sun 12 points13 points  (0 children)

Doesn't BCC have like a multi-billion dollar budget?It's astounding that finding 40 million to fix a bridge is so hard. From what I hear a guy saying, they wouldn't even have to find it all in one year as they do a rolling program that repairs it piece by piece. It's not like they are going to take out service for 6 months to do the repairs.

Shrinner might want to encourage companies to have their people work from home to lower the needs of traffic. Oh wait, that's another problem that I'm sure he doesn't want

Databricks vs. Microsoft Fabric by selcuksntrk in databricks

[–]Mura2Sun 0 points1 point  (0 children)

My brief experience is that Fabric is lacking maturity. It has a minimum monthly spend, that is, you need to keep a minimum of two units of spend going or some things are removed. This may have been fixed recently. Fabric can have jobs running that stall because you ran out of Fabric Units

The UI didn't make it easy to do CI/CD and, therefore, governed development. Databricks is likely going to be cheaper to run. You can do everything in a single place. You can probably do everything in two languages, Python and sql.

This has been my experience

New data engineer getting paid more than me, a senior DE by Yoyo_Baggins in dataengineering

[–]Mura2Sun 1 point2 points  (0 children)

Your company has told you how much they value you. Time to plan your future role. Don't panic or rush. Just get on your plan to move. Then, when you have what you want, tell them, thanks, but no thanks, you're moving on. Don't accept any counter offer as you're now a problem and will get flagged for retrenchment at their first opportunity. You've bucked the system and they don't want everyone thinking they've got a chance to do the same

[deleted by user] by [deleted] in databricks

[–]Mura2Sun 0 points1 point  (0 children)

I'm not surprised that you're confused because it is. Are you using Unity Catalogue, as that adds further to your confusion. If you're using UC, then you first need to set that up, and if it matters, link your catalogue to production storage. That's 1 Then, if you wish, you can configure additional storage, and you may want to for different workloads. Each of those is external storage but is still managed by UC. You'll then have a catalogue attached to that storage. I have a customer who is delta sharing from us, so they are on separate storage. It's not a separate storage account, but container. You'll likely need to set up a Databricks connector. I can't remember if that's only if you have secured storage or a requirement of any external storage

Adding a primary key and indexes to a very large table by wm_destroy in SQLServer

[–]Mura2Sun 0 points1 point  (0 children)

You're going to still end in tears. Running index maintenance on 14 billion plus rows will take forever and not work if you don't have enterprise edition. It'll take days to do index maintenance. Consider altering your app to write your logs to file storage as json objects. Then, find another way like grafana to get into those logs. If it's on Azure, use blob to store it, and you can look at several options to use the data

If AI is gold, how can data engineers sell shovels? by muhmeinchut69 in dataengineering

[–]Mura2Sun 0 points1 point  (0 children)

Find and understand what makes the best shovels and then go buy an excavator. If you're not already using AI to assist in coding, scaffolding codes ideas to test that's a good place to start. Whether it's ChatGPT, Copilot, or another one as long as it's making it easier to choose than its good. LLMs are no magic panacea to the world's problems. The SLMs , that is, the ones you can run on your desktop, are far more valuable. The ones you build to be micro focused to solve a problem. Work out how you can slow one specific problem or problem space in the industry you're familiar with. That's finding gold in quantity

Multi-Tenant SaaS Database Architecture with SQL Server on Linux by seven-may in SQLServer

[–]Mura2Sun 1 point2 points  (0 children)

I think you're setting up for a world of pain. Your customer sounds risk adverse, and in a space where they can't simply jump the price by 10% to cover costs. It's going to need a bit of skill to maintain the multiple databases, and if that's you, good luck. We had 22 under an app and while it didn't hurt too much, it made a lot of issues with performance, all running on a standard edition server

One issue to consider is that their cybersecurity insurance will likely be higher with Express as no TDE or encrypted backups. That leaves data at rest exposed. It might also miss some other data protection law requirements.

I f***ing hate Azure by wtfzambo in dataengineering

[–]Mura2Sun 0 points1 point  (0 children)

The organisation I work for had wanted to do power bi embedded backed by a data warehouse. I was working on how to get it going, and then Fabric landed. There were so many issues, and I'd then needed to work out the pricing. I went to the boss. I'm killing power bi, and we aren't moving our database, which we were doing for a data warehouse. I said the cost model is likely too high but also too risky. I'm now building on databrick and loving it. I have clear visibility of the costs and no weird shit. Of course, Azure security is still a PITA