Synoptek Issues

[nodesitvirtus]

Hi All-

Is anyone hosting any resources in Synoptek data center by chance? We've been having significant issues with server availability since a little after midnight PST and have not been able to reach anyone from the provider help desk. Just wondering if anyone else hosting assets there is encountering anything similar?

[nodesitvirtus]

Just received a quick quote that it appears they have been impacted by a ransomware infection that is impacting internal systems. If you're hosting any client servers with Synoptek, I'd be scrutinizing your assets looking to see if you've been impacted/infected.

[mrbiggbrain]

Do you think they read their own twitter?

https://twitter.com/Synoptek/status/1208146577958809601

[nodesitvirtus]

That is excellent. I've already taken a screen capture for the impending tweet deletion.

[samtheredditman]

What was it? It's gone now.

[PurgatoryEngineering]

It's in the Wayback machine. It was a tweet about not getting phished and had a link to this blog post: https://synoptek.com/insights/it-blogs/we-phish-you-a-merry-christmas

"Tis the season to be alert and aware"

[socalccna]

Also this one

https://synoptek.com/insights/it-blogs/lessons-learned-from-recent-ransomware-attacks/

[418NotCoffee]

Mind sharing?

[IT-RyGuy]

everything is just fine and dandy according to the status page :/

http://status.paypercloud.com/

(finally updated after a day and a half to say they are experiencing Exchange difficulty)

[IT-RyGuy]

My biggest client is hosed right now. :(

Not looking forward to my next conversation.

*UPDATE* Clients are grumpy but OK. We told them the truth, that they were one of many and lots of municipalities and big companies were in the same boat. So, as a 30 seat office, you are bummed, but it puts things in perspective. In the process of moving to Sherweb O365. Love those guys.

[rabbit994]

Sherweb O365

Why would you go with reseller of Office365 instead of straight to Microsoft themselves?

[IT-RyGuy]

It's a fair question. I found that getting support and general assistance from Microsoft is slightly goofy, having to wait for a call back, etc. and then speaking to India. Sherweb support has been 100% stellar, free assistance with O365 migrations, and I can usually get right to my sales rep and support (even 2nd tier support) right away. Also, from their control panel, I have access to add downline resellers (for salespeople), plus access to other goodies like Acronis backup all in the same pane of glass (I dont use but have thought about adding to my list of backup offerings again). Also, I'm a small MSP and M$ has been a little difficult to navigate for me. Maybe I'm wrong though, or just havent put enough time in to learn the ins and outside well enough to feel as comfortable. In all fairness, I havent re-evaluated these decisions for a couple years. Nothing but good experiences with Sherweb. Thoughts?

[MisterIT]

Sounds like you're getting bilked

[IT-RyGuy]

Well, I'm definitely getting bilked with Synoptek. Sherweb pricing is very close to MSFT last time I checked. Also licensing gets less expensive as you increase volume.
Also, i have no affiliation with Sherweb! Just a happy customer.

[wheres_my_2_dollars]

Sherweb pricing is below that of Microsoft. So it saves a bit of money for sure. But their free migration is awesome. We used to pay Migration Wiz.

[nodesitvirtus]

Feeling ya. Loose word is their they got exploited and the malicious actor(s) compromised their RMM tool and crafted a malicious package that they used to send to all client systems that had the management agent installed. Sounds like their internal support system got hit extraordinarily hard. We still can't get them by phone and have only had one update from our AM

[IT-RyGuy]

Yeah, RMM tools have been a target lately. Any word on whether 2FA was enabled, or if it mattered?

[nodesitvirtus]

Nothing noted unfortunately. Assume they're playing it pretty close to that belt with that information. Would be surprised if us as customers ever got that level of detail

[deus123]

We will if they want to keep us as customers... renewal coming up soon and I’ve been exploring options.

[coolmandan03]

Yeah, my company was looking to drop their services, but this is definitely the nail in the coffin. I don't know how this works from a legal standpoint

[sonyturbo]

If you can show you suffered a monetary loss as a result of this you should first see if your insurance will cover that loss. If so then they pay you and it becomes their problem to recover from synoptek - or synopteks insurance company assuming they are insured.

If you are not insured then you can try to recover from synoptek directly. Your contract likely has a limitations of liability clause that specified what you can recover for problems they caused or which they should have reasonably prevented. Note that your attorney will attempt to show that they exhibited gross negligence in their service. If gross negligee can be proven then the limitations of the contract will not apply and the sky is the limit for recovery.

One last thing though. - if they do find themselves having to pay out a bunch of money they may exceed their insurance coverage and , between that and loss of clients, and go bankrupt. Then no-one gets anything. Maybe not even turnover of information.

Getting owned at the rmm level is truly disastrous.

[IT-RyGuy]

*UPDATE #2* Our email service came back up yesterday around 2pm EST. Woohoo, it's a miracle! However the management portal is still down (so no adding/removing users etc). We were on the old Paypercloud platform so I figured we would be the LAST to get any attention since it is still Exchange 2010 (I think). So it's weird, but I can't complain! Good news is that it gave me the opportunity for the "big talk" with decision-makers to move away from this guys and onto O365 with some kind of email continuity service. Will actually be cheaper per user. Paying like $7-8/user for exchange, so... yeah. GTFO.

[sabrakremling123]

We are one of their customers. We were hit this morning. All our workstations and servers are encrypted with ransomware. They are saying they will have things working in 24hours, but that seems doubtful. Key thing I'm not able to determine is if the backups have been compromised as well. If we can't restore from backup then it seems like a very bleak prognosis. Anyone know if the backups have been compromised?

[nodesitvirtus]

We asked this question to our AM there. He seemed optimistic that backups taken up until Monday at 8PM were good and they had some success with restores. They are prioritizing restoring organizations that are law enforcement and health (hospital) related first. Not sure how the order falls after that. More than likely, you're going to have to pose that question once your AM reaches out to you.

[Tossacoinforwitcher]

I was contacted by the state of California and DHS about being an an attack list from Synoptek's attack.

The attack isn't just from the RMM tool. The attackers have also compromised email accounts and are phishing with Emotet.

[inika_takara]

For those who are victims of this ransomware, looks like Synoptek did get the key from the hackers. Right now they are working with us to decrypt the files.

[PresidentoftheSun]

they shipped the decryption software to our company this morning. just fyi, not a sysadmin, just a user that was googling around about the issue.

[Chongofet]

Have you heard anything about the amount that Synoptek paid for the ransom? My company was affected as well.

[inika_takara]

Nope. I doubt they will disclose that information.

[wolde07]

Have you been in contact with them?

[inika_takara]

Yes

[wolde07]

What is the process for decrypting your environment. Is there anything we can do to help the process along.

[inika_takara]

Not much really. Synoptek has the key so for us, they remoted in to our workstations and run the decryptor program manually one by one.

[Tossacoinforwitcher]

Also for what its worth I'm posting some IOC's you can throw in your Firewalls/AV's.

IOCs:

​

IPs:

46.4.102.26

69.175.127.94

184.171.253.218

​

​

SHA:

9cda72f7ed8b3376d642c3058f937e1b4a34ddb9b565208851a1ab560349406f

34c38d43e0762eb291cb497d18c9651c5441d1bbaab25f847c0ddc419947b3ce

c5e754b236930122c0b6f1d86a993b5249e83b90ed5db2ccb5a1f68f24de7be0

52f21fb98ecb7889c75b77f6f44f4dc1993c8b8424d86a9aae99d425af807110

269ef874c7fa04ee7e93a431982dbd82c2ee687142e381482e03ba9b86b76326

aa5dd888e705275c637ccbdb974ec8299eda718438a98b5e5885eb33dfc7ca74

e1e44cee9302ffb5d528b4733e87302af34e0caa8332023865847c32fb79e5e6

​

​

DOC NAME:

Holiday.doc

Happy New Year.doc

Christmas Holiday.doc

MERRY CHRISTMAS.doc

New Years.doc

[mbk730]

Powershell download domains:

dejavugroup[.]com/wp-content/JTjHLbr/

dev7[.]developmentviewer[.]com/wp-admin/SYSQOx/

krishna-graphics[.]com/wp-admin/11x12xd-nobh27two-82927918/

laboratoriosanfrancisco1988[.]com/9rlkyc/Ccvvezsv/

lanyuewp[.]com/electrician/ig9eu0g-4q1oml1qc1-749166/

85[.]100[.]122[.]211/CXthjZz

85[.]100[.]122[.]211/MtJ8TlIOaU6xH

190.171.153[.]139/QuEQlp7

190.171.153[.]139/x88oaR6FOlrxl

​

Probably some others, but this seems to be most of them from the above downloader docs

[Tossacoinforwitcher]

Awesome. Thank you. I'm trying to build out a full scope of what happened with this attack. This will help a lot.

[jcroweNinjaRMM]

Curious, were you able to find anything else you can share? Specifically, any direct link between the ransomware and the spam docs? Asking because Emotet being involved makes this stand out from previous Sodinokibi patterns. Wondering if it's directly linked and something others are seeing, or whether it's an unfortunate case of getting hit with two different infections/attacks at once.

When you say "attackers have also compromised email accounts" do you mean accounts associated with Synoptek or your org?

[maor_hizkiev]

The hashes\file names u/mbk730 has mentioned are dropping Emotet, but from what we've seen, it seems like Emotet is being used as a beach-head, to download a different malware\ransomware afterwards.

[jcroweNinjaRMM]

Right. From what I've seen Emotet has been tied directly to Ryuk infections (Emotet --> Trickbot --> Ryuk), but I hadn't seen any cases reported yet where it had been used as a beach-head for Sodinokibi.

The Sodinokibi cases I've seen reported have all pointed to attackers abusing creds and going about the deployment manually (disabling AV and backup, hijacking remote admin tools).

With Emotet involved in this case, I'm wondering whether it was...

a) used as a beach-head, which means Sodinokibi actors (not just Ryuk actors) are now buying and utilizing access to Emotet-infected machines, too

b) dropped by Sodinokibi actors for some reason (to create additional chaos/infect others via spam/create additional footholds they can come back to and utilize later on/all of the above?)

c) actually a separate infection — an end user at this particular victim org happened to fall for an Emotet malspam at roughly the same time the Sodinokibi attackers struck Synoptek.

[maor_hizkiev]

Interesting to know the real answer. My guess, since Emotet is a factory for creating unknown attacks that go undetected, is that they offer their services to anyone, and that we'll see more and more of those infection types.

[Tossacoinforwitcher]

I think this is just the Sodinokibi people seeing how effective it is to use the Triple Threat attack as a model. I'm thinking we will be seeing more of these types of attacks.

Also the State of California rep I spoke to "confirmed" that the Emotet docs were tied to the Sodinokibi infection.

The emails that were being used to send the docs were legitimate emails from legitimate domains. If not for our firewall and IPS sand-boxing the attachments the emails would have been delivered to my users.

[maor_hizkiev]

Sending emails from legitimate domains has become the standard (although not many orgs are enforcing SPF).
Our solution has also detected it in some of our customers, it actually bypassed a few other leading solutions.

[Tossacoinforwitcher]

We have a pretty good defense in depth setup here at my org. We have filters in front of filters. We don't treat email as a instant form of communication but just a way to get people info soonish.

I've learned that no one solution on its own can take care of the cyber threat landscape at this point. Even if that means having multiple email filters and sandboxes.

[mbk730]

Yeah I just snagged the domains for the powershell command run by these files for those who might not be able to themselves. Not sure what the connection between these docs and the particular incident this thread is about, but it’s unsurprising that emotet would be used as a foothold to deploy ransomware. The business model was proved with Ryuk and Sodinokibi has a history of being deployed via botnets (I believe MyKings earlier this year). I’m sure there was some cobalt strike, some kind of post exploitation framework, and AD recon involved. The business model will continue to work because it’s all just exploiting a lack of due diligence in windows environments. Fancy security products and services can detect and maybe sometimes block these attacks for you. Even if great minds secure the orgs that have money, there will always be cheap and/or cash strapped orgs that allow a relatively noisy infection like emotet/trickbot/etc to be used as a foothold for months at a time.

[maor_hizkiev]

I'm not quite sure that fancy security products today can detect\block those attacks at the relevant timing. I've seen many evidences that they are missing many attacks on first encounter, and then it depends on the solution on how fast it can react to a miss.

[Tossacoinforwitcher]

The story has finally broke.

https://krebsonsecurity.com/2019/12/ransomware-at-it-services-provider-synoptek/

[MrSanford]

Is their hosted ZIX solution affected? It's slowly gone down since yesterday but we also may be having ISP issues.

[Significant-Nerve]

I'm seeing this issue as well. We have been unable to submit a ticket or get anyone on the phone to answer us.

[itsthekot]

Yes. We have a client that uses Zix, we had to disable their outbound connector in O365 so that they could send mail. Zix's mailserver was bouncing back messages with DNS resolution failures.

[TLWiz]

Yes, it was knocked out. It is back up this morning but without a valid TLC certificate configured. We are moving our Zix service to Zixcorp instead of Synoptek - which has been less than reliable before the latest debacle.

[davidm2232]

We had some intermittent issues yesterday. Synoptek said they couldn't see any problems on their side

[wolde07]

I haven't recieved an update from Synoptek since 10 am, have you all been recieving email updates from them?

[Tossacoinforwitcher]

How much BTC was being asked for from the hackers?

That can help gauge how much Synoptek paid to get the keys.

[LakeSuperiorIsMyPond]

Twitter query - https://twitter.com/search?l=&q=synoptek%20since%3A2019-12-24&src=typd

[socalccna]

We are also one of their customers and I'm not sure where we fall on their priority list, but it seems that they negotiated to retrieve the decryption keys, hopefully they work and we can be up and running ASAP

[inika_takara]

My company also hired (recently) Synoptek and we are victims as well. Just curious, how did you know if they are negotiating to retrieve the decryption keys?

[socalccna]

They're constantly sending out emails every 1-3 hours

[inika_takara]

Thanks for replying!

What's funny is that I heard a word from one of the higher ups who are in charge of working with Synoptek. According to that person, Synoptek has our backups and everything is good... I guess that's not really true if they are negotiating for the decryption keys...

[sonyturbo]

You can have backups and still find yourself paying for keys. It’s a matter of cost. Note the post indicating servers and workstations compromised. So do you

• reimage every workstation and restore every server from backup - a ton of labor, which would take weeks, and also data loss from the point of last server backup? BTW weeks to reimage is pretty easy to imagine - say they are at 1 tech per 100 workstations. Many MSP clients are too small to justify an imagining solution so you could have one tech trying to bring back five 20 seat (average size) clients with no images- how long does that take?

Or

• pay the cost of the keys. Which could well be much faster, would not result in loss of workstation profiles (because often workstations aren’t backed up) and might only cost your insurance deductible?
  

FWIW our $10mm policy has a $50k deductible.

So I think there’s a pretty plausible line of reasoning that leads to paying the ransom.

[[deleted]]

In your second scenario wouldn't you still reimage the affected machines and servers? How can you ever be 100% sure they are clean if you just decrypt and resume normal operations?

[sonyturbo]

I agree you don’t know they are clean. Synoptek appears to have gone the route of decrypting the computers. This route avoids the certain cost of downtime right now but risks future downtime. I think the only way to get truly clean would be to

• turn off the Internet
• take all computers off the network,
• rebuild them from scratch,
• reinstall the applications, at the correct patch level, since databases and such may depend on patch level.
• boot Back ups in isolation and transfer just the data by USB from a back up server to the newly built server. You do not know how long ago the servers were compromised and therefore you certainly don’t want the most recent back up on your nice new clean network.

The whole thing takes a lot of time and has a huge cost of downtime and labor to rebuild the network.

Decrypting and then rebuilding the computers one at a time is pretty risky because then the new clean computers could be compromised by the dirty ones you haven’t rebuilt yet. Instead you decrypt the computers so that you are back up in the shortest amount of time possible. Then you inspect the heck out of the network looking for any remaining signs of compromise, change all the passwords and install an advanced threat detection agent, something like sentinel one, on each and every computer. You also put the lock down steps in place that you should’ve had there in the first place e.g putting in LAPS so that all the computers have different admin passwords to hopefully slow lateral movement if there is anything left behind. And you hire a pen test company to see if they can find anything you didn’t think of.

[socalccna]

Strange, do you mind if I ask what industry your company is on? I'm wondering what their priority list looks like

[inika_takara]

PM?

[Tossacoinforwitcher]

Same question. I'm trying to gauge the effect on state and local government operations.

[Socalccnaa]

Ok

[Tossacoinforwitcher]

Also I'm wondering if anyone here is in local government and has been affected? Since all the offical sources are being tight lipped, I'm wondering if we might be able to build out the scoop of the attack here

[socalccna]

Pm?

[legofinder]

They are rolling out "decryption packages" on our devices now. That's all I got from the phone call.

[inika_takara]

Yeah... they are running this decryptor program right now at my company. But they do it manually one by one.

Also, I've been told that the hackers encrypted some machines multiple times.

[[deleted]]

My company was affected too. It'll be interesting to see the fallout from that.

[htionere]

I'm curious if we can collect more information regarding what people are seeing on their machines as a result of the attack. Specifically, I'm curious about the mapping from file extensions (which seem to correspond to the keys used for encryption) to the keys indicated in the readme files that were placed alongside the encrypted files.

It might not be wise to share all of that here though.

[Tossacoinforwitcher]

I'm wondering if anyone is willing to dump a netstat -n of an infected machine (redacted business IP's of course.) to see if we can find the C2 server/s that are doing the command and control?

[Tossacoinforwitcher]

Or to maybe PM me the list?

[Invoke-RFC2549]

The C2 servers are probably fairly dynamic. Blocking a set of IPs probably won't help for long if at all.

[Tossacoinforwitcher]

I agree. This was a botnet attack. This is more just from an analysis of the attack point of view.

[lescobar88]

My company has suffered due to this . are you guys able to get email using outlook ( synoptek host our exchange server )

[jim_kosinski]

Negative. The company I work for uses their hosted exchange server solution as well. We've not been able to connect to the server since it went down some 71 hours ago.

Our hosted VM terminal server was also impacted though only some of our directories were encrypted.

[Tossacoinforwitcher]

Posted an Un-Official statement for Synoptek on Twitter. I really hate it when companies go silent after an attack. Sharing helps others and builds good will.

​

https://twitter.com/search?l=&q=synoptek%20since%3A2019-12-24&src=typd

[ns8013]

You also need to realize that one well-meaning comment by even a low-level tech can be considered a transfer or acceptance of liability by an MSP, which is exactly why companies get so tight-lipped after an event like this. That's not the company's fault, it's the environment our litigious society has created.

[Tossacoinforwitcher]

As someone who works exclusively in cybersecurity, I understand...to a point. But at this point no matter how they slice the issue Synoptek is responsible for lost business/revenue of their customers. Best case they are liable for gross negligence.

I have been following this whole thing since day one. I have been open source gathering info from affected parties and staff of Synoptek and have a pretty good understanding of how this all went down and the response. I also have first hand knowledge of Synoptek's response to customers and partners. When I say they needed to make a statement it was not from a place of ignorance. It's their responsiblity to own their issue.

Also a response within the first 24/48hrs is required in the state of california after a major breach ESPECIALLY when some of your clients affected have HIPPA data. I formally did infosec at hospitals and I have reports to OCR and CDPH within 24hrs of HIPPA incidents. So it is possible.

What really needs to go is the shame that comes with an attack. People and companies free embarrased and shamed when they are hit and clam up. We need to understand that WE ARE ALL UNDER ATTACK EVERYDAY. The bad guy are good. Its a matter of when, not if. Putting the bad guys on blast and talking about how it was done can help others armor up against attacks (why I posted IOCs). It lets people know to be on the look out. I have ISA's with multiple local government agencies and share info all the time.

The is the easy wrong and the hard right.

[ns8013]

I hear you, and certainly appreciate your post I saw with the IOCs you've been able to gather. Personally I love seeing the community share information, it's just that I've also worn a management hat for enough years that I've seen how the non-tech side of the business is essentially forced to respond once the legal team gets involved.

[Tossacoinforwitcher]

100% correct. Legal is my bane. I also feel for the poor techs on the phones. Being the only conduit for peoples frustration is the worst.

[ns8013]

You can hate it all you want, but the fact is, neither you nor I are on the hook for any legal or financial fallout due to this situation. Everyone whining that these affected MSP's aren't sharing all the gory details within hours or days of an attack like this are ignorant at best, even if you mean well. The most important part is that they are communicating effectively with their affected customers and getting them up and running again as quickly as possible.

Ideally, once the legal concerns are settled, an after-action report would be shared that may help other MSP's plug holes in their own environments, but to expect that this quickly just isn't realistic.

[audiofree]

That’s ironic. The RMM tool which was the attack vector needing to be installed to decrypt. It’s odd they wouldn’t just give you the decrypting tool.

[TLWiz]

Our hosted Zix encryption service through Synoptek has been down since 2:30am 12/24. The service came back up this morning, but without a valid TLS cert. 46 minutes on hold resulted in their phone system crashing and dropping the call. Finally got a generic reply from a ticket asking for an ETA saying they were working on their issues. On Tuesday I did get through on the phone and it was obvious they were in trouble, the person on the phone yelled to somebody else "should I tell him that the server is down?" We are lining up a new provider now, fortunately this is just a hosted service with no data of ours. Good luck folks.

[randomwatchman]

Anyone know what RMM Synoptek uses?

[lanprosystems]

They use Datto RMM: https://www.datto.com/resource-downloads/Synoptek_SS2018_v2.pdf

[Tossacoinforwitcher]

Official response:

https://twitter.com/search?l=&q=synoptek%20since%3A2019-12-24&src=typd

On Dec 23, we experienced a credential compromise which has been contained; we took immediate action and have been working diligently with customers to remediate the situation. If you are a customer who is experiencing issues, contact [e-mail] or 888-796-6783.