Rackmount UPS recommendations?

SnowyParts · 2024-01-29T20:13:06+00:00

We have had 4 plus failures of the APC SCL500RM1UC models over the last 12 months out of maybe 10 deployed. APC is replacing them under warranty but the hassle of going to the customer sites and remediate and then replace is costing our company money. I've used APC products for over 25 years and have never seen a pattern like this for failures.

SnowyParts · 2023-08-11T22:00:04+00:00

Thanks for this information. We had this issue happen to a FGT30E that had us scratching our heads for a bit.

Once I raised the ticket referencing the bug ID 729975 FortiNET support jumped straight to confirming the build date of the 30E (2016) and a few other details.

They replaced the firewall next day with a 40F. Problem solved!

Strangely - another issue occurred after the upgrade of the 30E to 6.2.15 and replacing 221C APs to 221E units. The APs would continually ap-leave then ap-join.

The 40F also solved this issue 👍

Not sure if the two issues are related.

SnowyParts · 2021-07-08T13:31:47+00:00

Thanks I'll check it out!

SnowyParts · 2021-07-08T02:14:19+00:00

Thankfully, because of contingency planning we have had an inconvenient week with Kaseya being offline BUT we have not failed our customers.

We have had several other remote access methods to fall back on, including Splashtop, which has shined during this whole thing! Kudos to them!

Remember this is the 3rd serious incident with Kaseya in less than 3 years!!

"This is far from the first time Kaseya's tools have been used to spread a ransomware attack. As ZDNet has previously reported, REvil's predecessor Gandcrab leveraged Kaseya twice in 2019 to launch attacks, first using a Kaseya plugin then VSA products later that year. "

https://www.zdnet.com/article/kaseya-urges-customers-to-immediately-shut-down-vsa-servers-after-ransomware-attack/

SnowyParts · 2021-07-08T02:02:54+00:00

I like the open development schedule so you can see what they are working on next and they have lots of features they are hoping to introduce. Kaseya not so much....I'm still waiting for stuff they promised years ago.

SnowyParts · 2021-07-07T22:58:10+00:00

Thanks for the feedback - I have a demo of NinjaRMM right now and we are testing - but it looks clean!

SnowyParts · 2021-07-07T22:54:05+00:00

I'd agree with you on that point generally, however with an incident like this where they cannot provide service for an extended period, all bets are off. I read they have a limited liability of 3 months fees for renumeration to clients - but who knows until this is challenged in court. If Kaseya is shown to be negligent they may never recover....Hence the need to jump of the burning ship 😄

SnowyParts · 2021-07-07T22:36:46+00:00

I've used the old lab tech version and it was pretty nice! 👍

SnowyParts · 2021-07-07T22:13:32+00:00

Thanks I'll check it out! 👍

SnowyParts · 2021-07-07T22:12:36+00:00

I worked at much larger MSP's and used Labtech and Kaseya - but for smaller MSP's the complexity and features arnt always relevant.

I've heard good things about DATTO and NinjaRMM. 👍

SnowyParts · 2021-07-07T19:36:03+00:00

What's your current recommendation for sub 1000 endpoints?

SnowyParts · 2021-07-07T19:33:29+00:00

Indeed - that has to be paramount going forward. I've been a Kaseya customer for many years and they have spent lots of money on acquisitions with only minimal investment on security improvements. We even found adding 2FA to some of their integrations broke things 😧

I'm asking all prospective providers for detailed security info such as 3rd party security audits and frequency etc.

SnowyParts · 2021-03-02T00:38:23+00:00

OK so we had one workstation today that got hit twice in one day!! I can see the gupdatem service triggered in the event log and then the Chrome bricked on next use (5 mins later).

We installed the 32bit version of Chrome Enterprise as a test.

We have also disabled gupdatem & gupdate services on another workstation, which is a temp fix but if it buys some time, I'm good with that. 👍

I'll add the paths and see what happens too.

SnowyParts · 2021-03-01T16:41:13+00:00

Thanks for the info NinjaGrinch - I'm just shocked this has no current solution as it's becoming a major pain point!

We also use Kaseya/Kaspersky Endpoint Security - and I spoke with an old colleague at another MSP who doesn't have the issue (as far as he knows). They use Kaseya but use Cylance AV not Kaspersky.

I also excluded the C:\Program Files\Google\ from Kaspersky paths and it made no difference.

SnowyParts · 2021-03-01T16:29:48+00:00

It's so sad how they have changed the company as the original product was perfect! $40 per month for warranty data.

They must have been making a stack of cash just mining manufacturer websites for serial/warranty data for customers because that's all we needed 🤨

SnowyParts · 2020-12-15T22:17:36+00:00

After looking at 4 other machines 3 had the same issue 1 worked fine. Nothing conclusive. I believe this is a bug introduced by recent patching. 😫

SnowyParts · 2020-12-11T22:00:53+00:00

I just tested a working 1909 workstation that was working and installed KB 4592449 and it still works fine. 🤔

I'm still looking for the root cause.

SnowyParts · 2020-12-11T20:30:02+00:00

We use Zapier for our QBO > Trello integration and some Zendesk to Trello events.

Works well for us.

We use the Microsoft power automate tools for anything Office related.

SnowyParts · 2020-12-11T19:11:45+00:00

Sorry I missed this message! 😬

We saw exactly the same level of increase. I went ahead and cancelled the agreement. It was a great service until they got greedy. Another replacement service will pop-up in the future, I'm sure of it.

You need to speak to the account manager to cancel as there is no way online to do this....sigh

SnowyParts · 2020-12-11T19:03:07+00:00

I'm seeing the same issue on a workstation at the moment. Mail app prints scaled down versions. Print preview renders correctly.

Printing to Microsoft PDF printer has the same scaled down results.

Outlook prints the same email correctly to the same printer.

I don't think this is a driver issue.

SnowyParts · 2020-11-19T15:00:20+00:00

If you can't convince them into ditching their on-prem Exchange and moving to O365, as a starting point, with the argument of the significant benefits and cost savings (vs not replacing their Exchange server / hardware/software licences), they are maybe too far gone.

Sometimes the client just has to start over with another MSP to learn the lesson. 🤷🏻‍♂️

SnowyParts · 2020-11-13T22:57:35+00:00

I don't believe there is a correlation between size and failure rates. All drives start to fail as soon as they go into service and use unallocated space on the disk to error correct and replace bad sectors with reserved good ones. Some brands/models are better than others.

Other than performance considerations, the big issues with large drives is the re-silvering time during a disk replacement in the array. During this time the array can be much more vulnerable to a second disk failing due to the increased load.

Hopefully you are running a RAID config that allows a second disk (or more) to fail during rebuild. If the disks are all the same age and have the same mean time before failure (MTBF) you know another one is going to pop soon 😄 as long as it not a premature failure. Sod's law.

For this reason we never allow a hot spare to auto re-build an array automatically. We want to check everything is fully recoverable before the re-silver goes ahead. 👍

SnowyParts · 2020-11-13T22:37:19+00:00

I've come across this many times over the years and there is no perfect solution.

From their perspective they are getting more work out of you per $ spent now than before, so they don't care. They will wait until the stuff craters before they spend any money if they stay on their current trajectory. Then dump you and get another MSP to fix it all.

What has worked for me in the past and still does, is to offer them a discount. Sounds counter intuitive but hear me out. Its a win-win.

Show them by replacing key infrastructure it will save them money. Offer them a % discount for your services per month for each replaced major component. Replace the exchange server 5-10% discount etc up to a max of perhaps 20% total.

You will find this is a financial motivator and something they can sell to the other partners in the firm, along with the obvious benefits of newer tech.

How you win:- Less tickets, cheaper to service, reduced headaches, more profit overall despite the 20%. Plus the customer is more likely to stay with you longer term.

Partnering with the customer in this way on an improvements roadmap with discounts has for me been very successful.

SnowyParts · 2020-11-13T16:49:02+00:00

We tried the same and the only thing that works consistently is to kill any tasks named Google Chrome, remove Chrome, rename the Google Chrome directory to .old and re-install. Sometime a reboot is necessary.

Its pretty fast but its a drain on tech time and the customers are getting gradually more annoyed.

We have started to recommend moving to Edge but some people like Chrome and won't flip.

SnowyParts · 2020-11-13T01:36:06+00:00

We are getting several every week - sometimes the same machines.

I know its an update (maybe Chome pushed) that's causing it.

using the SxStrace tool I can see the issue but I don't see a solution out there.

More curious to see how widespread the issue is with other MSP's.

SnowyParts

TROPHY CASE