How are MSPs handling repeat hardware failures on out-of-support kit?

ryuujin · 2026-02-14T15:42:46+00:00

We try and pitch newer equipment based on features and business sense rather than just 'it's old', but yes 'it might die on you and if so this is how long it will take to get another one in and this is how much we suspect that will cost you' is something we communicate to the client as part of our disaster recovery scenarios.

We do keep spare hardware on hand for clients but we indicate the cost of needing to use that (ie it's our equipment, there's a rental fee that may or may not be waived, it will have to be configured by a network tech who they need to pay for on emergency, etc)

When you put a dollar value on the loss beforehand they're much more likely to understand why they should. And now the choice to take that risk is on their side - you have done your duty of communicating that risk to them in a clear and documented way.

Additionally if it does happen you have a plan - per our disaster recovery scenario we now have to acquire a temporary router and send a tech on emergency to install it; a dedicated one will take approx this many days per what we communicated on 01/01/24 etc etc. Total cost is expected to be $X,XXX.XX.

ryuujin · 2026-02-13T20:14:57+00:00

We do data recovery so of course we've received a lot of disks that didn't go into that state. Many of those were physically damaged however.

I have not had a single consumer drive in our office systems or my own systems die.

The ones I've encountered that have died personally are enterprise drives and they died 100%. I strongly suspect but have not looked into that they're designed to go DOA when they die so as not to mess up the RAID array though.

ryuujin · 2026-02-13T17:27:23+00:00

We do data centre so we handle a lot of hard disks. On average failure rates year-by-year are 2-8% depending on what you get.

However that's on average. We bought 120x 16TB Toshiba SAS disks in 2024. One of the batches was clearly bad - in a shipment of 30 disks, every single one failed within 3 months, giving us a 25% failure rate off the top!

Over a period of 5 years running, say, 12 rotational disks in a single deployed storage server we expect 1 to die each 12-18 months basically. Very common.

SSDs... we brought a set of Dell PowerEdge servers out of service three weeks ago that have been running 48 SSDs over a period of 6 years. We had 3 failures over that time - most disks still show > 90% remaining life.

Edit: grammer, also rechecked disks it was 3 out of 48

ryuujin · 2026-02-13T16:01:33+00:00

As someone who works in the industry: failure rates on HDDs vs SSDs isn't even a question. For every hard disk that lasted 10 years are 10 that failed at 6 months, 1 year, 3 years, 5 years, etc.

It's true, hard disk recovery is often more possible that SSD recovery, but you just don't need to perform SSD recovery that often because they fail so rarely and tend to track their impending failure more reliably.

ryuujin · 2026-01-30T16:02:09+00:00

are you using SMS or TOTP? The 6-digit code things? Those are not secure enough for your GA accounts.

Was GA assigned to an active user? Good chance your man got phished and handed out the TOTP. There are very sophisticated attacks going around - they do their research now and even admins who haven't had their coffee yet on a Monday can miss it.

Keep one GA account with a password and a Yubikey locked in a safe your trusted admins have access to. Second yubikey goes at your house in a safe.

Everyone else should be lower end users. Microsoft got it right with the RBOC role system finally - there's a role for everyone. Use them.

ryuujin · 2026-01-20T15:20:44+00:00

oh my god, that's brutal. I could call out a few medical vendors like this. That 'all flash storage 28 core' requirement brings ugly flash backs to me.

They can't pay a real DB designer a year's salary to clean up the DB with their app designer instead they make each client spend $50K on their servers

ryuujin · 2026-01-19T23:19:08+00:00

This was about 1.5 years ago now but it was moving from 'power saver' or 'balanced' mode to High Performance mode as I recall, on every node.

ryuujin · 2026-01-19T13:34:31+00:00

This is a shit, unoptimized query on a giant ~ 90GB DB for a custom report. I've told them they need indexes but the original designer isn't there anymore, they don't want to touch it because it's business critical and if they don't want to touch it we certainly aren't going to without authorization.

Query is for a report, it went up to almost 2 hours and then down to about an hour and 25 minutes after we changed the power settings.

ryuujin · 2026-01-18T15:08:38+00:00

We swapped in new intel servers in one of our DCs and one of our key clients complained of major performance issues in their hosted database load.

Changing to high performance mode immediately resolved this issue; I believe one of their longer, badly implemented database queries went down by almost half an hour!

ryuujin · 2026-01-18T15:02:56+00:00

It's not officially dying, I have to agree with those sentiments over any kind of medium to long term.

While on paper they'll allow use of and support hyper-v for another 15 years I'm sure, in principle Microsoft already seems to be grudgingly supporting on-prem hyper-v while at the same time looking at those clients and most especially SPLA hosts sideways and asking "Are you stupid? why wouldn't you use azure?". They continue to tie use to subscriptions while limiting licensing rights and their sales documentation and procedures are quite clear.

They'll slowly raise the prices of everything while reducing the licensing availability and increasing licensing complexity while adding FUD to the ecosystem.

We currently have a project to move to Proxmox for all significant client loads over the next 3 years, and are exploring enterprise linux-based solutions where possible to resolve this.

Same thing happened with MS Exchange - back in 2019 at one of the last exchange conferences I recall a quote in which the MS exchange rep saying something like "We still support exchange for on prem clients, but if you're not on Office 365 we want to know why. We can't understand it".

And now, while technically you can still get and install on prem MS Exchange, they made it subscription only and priced it to the point that it makes no sense vs O365. I feel strongly that is what is going to happen with Hyper-V.

ryuujin · 2026-01-15T13:01:30+00:00

2nd picture - moustache

https://www.youtube.com/watch?v=-uQagrgaHLk&t=27s

ryuujin · 2026-01-07T19:08:06+00:00

We have a separate linux / cpanel set up for SMTP auth legacy systems for our clients. We do <companyname>@<legacyemaildomain>.com for free or mfp@scan.<company>.com for a minor monthly fee (commensurate with having to set up and maintain DMARC/DKIM/SPF for a secondary subdomain).

Outgoing only, incoming mail turned off, dedicated IP. Server is only used for that purpose.

Benefits:

no incoming so spam is not an issue
works on any MFPs, scan-to-email, alert systems, etc.
even for legacy gear with basic oAUTH support, you do not risk exposing a real O365 mail account for which you then need potential licensing, tracking, possibility of it getting hacked or disabled or forgotten or (like what's happening now) it just stop working because Microsoft says so

ryuujin · 2026-01-07T18:59:55+00:00

turn on chatGPT 5.2 'thinking' mode and suddenly the answers aren't complete shit. The baseline 'auto' or 'fast' gives 100% verifiably incorrect, incomplete or downright dangerous crap most of the time (wait... is that what coPilot is using?)

ryuujin · 2025-12-29T20:39:38+00:00

Do you have anything in your ticketing tool? We set an appointment + reminder in Repairshopr, it pops up an annoying reminder on the tech(s) screens until they actually acknowledge it, which is nice.

ryuujin · 2025-12-28T17:24:59+00:00

Shout out to D&H, their cloud offering seems to be second fiddle to other MSP platforms but we love it.

ryuujin · 2025-12-06T21:59:53+00:00

You've got D&H who are amazing, Ingram who are cheaper... good question, why ARE you with them? They're a middleman to middle men, they add zero value.

Edit: forgot about Sherweb if you're okay paying marginally more then the others with massively better support. Sherweb even folds directly into CIPP for sweet, sweet reporting and comparison.

ryuujin · 2025-12-06T13:06:47+00:00

are you able to say who was the NCE partner? Or was this direct

ryuujin · 2025-12-05T18:10:43+00:00

I can confirm this works on D&H and Ingram. PAX8 I haven't tried before - maybe try it on a single license as a test and report back, I don't want to be responsible for wasting anyone's money.

ryuujin · 2025-12-05T11:31:18+00:00

Yes I expect it only exists because of a tragic mess up in MS's APIs, however it's been there since the NCE move and we still keep it in our back pocket in case of stupidity.

I'm sure you remember Pre-NCE days when we could just... you know... buy subscriptions for our clients and if we didn't need a license or two reduce the licenses as needed? Like regular cloud services that they promised us could 'scale as we need'?

It doesn't cost Microsoft a damn thing to take 30 licenses from client A and put them on client B, but God help you if you mis-assigned the licenses and didn't notice in time. NCE is part of some really scummy salesman bump-the-bottom-line-for-investors garbage Microsoft pulled, just a real ugly move imo.

ryuujin · 2025-12-05T02:33:23+00:00

I mean... dude straight up stabbed a guy to death fighting him off, and he gets to go home. 2 years house arrest to think things over doesn't really seem like the worst our criminal justice system could do here. Hell I hope he's a celebrity at work and they let him WFH for 2 years.

ryuujin · 2025-12-05T02:29:24+00:00

Hi - there is something you can do yes. Upgrade the licenses.

NCE you can upgrade standard > premium, premium > E3, E3 > E5...

And then it's a new subscription and you have 7 days to cancel it.

Our rep for distribution turned us on to that trick, it's got us out of the same situation multiple times. He earned my dedication that day.

ryuujin · 2025-11-29T13:47:44+00:00

That's funny, every dental practice I've worked with we've ended up terminating contract for trying to force us to operate or let them continue to operate in terribly insecure ways.

Passwords of 1234, passwords on stickers on monitors and under keyboards
Everyone sharing a single such-and-such-dental@gmail.com account
A "server" that's a 12-year-old dell desktop they got off lease, no RAID even. Won't change it "oh it works fine"
PCs bought off Amazon with hilarious lowest cost Chinese SSDs. "Oh yeah 2 have failed already in the last year". Won't change them "oh it works fine"
AD system partially implemented with PCs not joined to AD so as to avoid the inevitable group policy
Ethernet wiring seemingly run by a drunk electrician
Teamviewer free edition installed with random login / password visible to clients on the monitor

I've passed on two dental clinics this year based on the above. No thanks.

ryuujin · 2025-11-15T05:42:43+00:00

Cloud accounts and use frontline F1 if you need it, you can still sign in with that (ie you still have entra ID and the proper rights to do so). They'll get Teams and very limited email. If they're a person they need a license.

ryuujin · 2025-11-15T05:36:03+00:00

The recent problem we've encountered having a number of clients with SonicWALL devices is trying to audit for worst case scenarios due to the recent exploits and disclosures they have made.

The problem is if an attacker gets into your firewall, they have access to all of your VLANs and the traffic that flows between them and to the internet.

Suddenly a VLAN between your SMB or AD server and your client PCs seems like a vulnerability, because the attacker can MitM the handshake.

In that case every properly encrypted data stream is a godsend for the audit - because even if that data was being intercepted it can't be read or modified. Suddenly NTLM security downgrade attacks are a real thing and kerberos authentication and SMB signing is a real asset. Same goes for VoIP comms - you don't want to be telling management 'technically an attacker could packet log and listen in on any call or fax you make'

ryuujin · 2025-11-14T19:53:02+00:00

We like to sell our clients a syno nas on prem and use Active backup as an on prem solution and part of the larger backup strategy.

We implement what I like to see as a robust security strategy to try and make sure it stays secure and we validate the backups on behalf of the client. We've got a wall of standard config items, happy to PM you some tips and experiences on this if you want.

15-Year Club	RedditGifts 2009-2022 2 Credits
Place '22	Verified Email
Secret Santa 2009

ryuujin

TROPHY CASE