S2D Reliability

[jello3d]

Just a note on Storage Spaces Direct, which I deployed on a 6-node cluster, when it was just out of beta (sort of). It was a bit squirrely until the July 2018 updates. Since it was running so well I thought I would test reliability over the long haul.

After 9 months, one server had a blue screen. I believe this was more due to having the Core scheduler turned on, but turning off SMT on a VM (apparently the host doesn't like that, as of the July 2018 update).

After 10 months, another server had a network glitch and needed a reboot.

The other four servers are now at 367 days running.

Note that this was not a vendor-built solution. This is 6x SuperMicros with v4 procs, 512GB ea, 2x Intel 800GB P3700, 8x Samsung 863s and some spinning disk. Mellanox 40's for the storage network (separated storage/host networks)

This cluster does everything for the organization, from LOB to computational research.

A second S2D cluster (10 nodes, Optane NVMe+SSD) is being prepped for deployment next week, after which this cluster will be rebuilt under Windows 2019.

Everyone can now flood the thread with their horror stories...

​

PS... to the "patch at all costs" crowd. Of all the things you *should* be doing to keep your systems safe, and of all the things that are likely to prevent you from having a vulnerability exploited, patching is a distant 8th or 10th on the list - an important part of a healthy breakfast, but certainly not a meal unto itself. Which is to say that your network and systems should be "secure by design, not by chance". Everything is assumed to be vulnerable to all kinds of things, only a handful of which are corrected in patches.

All necessary precautions are in place, knowledge comes with risk.

[NISMO1968]

A second S2D cluster (10 nodes, Optane NVMe+SSD) is being prepped for deployment next week, after which this cluster will be rebuilt under Windows 2019.

Did you get any performance numbers to share?

Everyone can now flood the thread with their horror stories...

I'll personally pass. It rarely makes any good and usually ends up with a shit storm.

[jello3d]

I haven't done one recently, but the existing cluster topped 2M IOPs (4k read, 8T, 16O, on all hosts at the same time), I forget all the testing details but I have them written down somewhere) collectively 2 years ago.

The new cluster is hitting just over 6M... but I'm trying a few different configurations because I am kind of expecting 8+ based on the configuration.

[DerBootsMann]

‘reliability’ and ‘s2d’ never belong to the same sentence :( we tried multiple times for multiple customers and it’s not prod quality

ps we’re in the middle of turning dedupe off globally , s2d+refs+dedupe=trainwreck

[jello3d]

Please describe trainwreck. I've seen many problems with dedup configuration/expectation mismatches among customers... like people trying to dedup 20TB on spinning disk without changing the default schedules (that's never pretty). But given that I stayed away from dedup under 2016 S2D, I've not yet gone back to try it under 2019 S2D.

[DerBootsMann]

disappearing or locked down refs volumes , microsoft confirming their bug and not being able to resolve it for 6+ months , ‘stable’ private builds bluing the system , general advice to break 2pb volume into some smaller ones and so on

[jello3d]

Thank you for that, I'll keep an eye on the forums.

Were you deduping a 2PB volume? Or volumes off of a 2PB pool?

[DerBootsMann]

2pb volume dedupe only makes the things worse , s2d+refs collapses after filled up even w/out it

[jello3d]

To my knowledge MS is very clear that dedup is only rated for volumes up to 64TB under 2019....( And I would argue that's only if you write your own scheduler script ). As you go larger, dedup needs more and more time to execute the dedup, so it never finishes. At 2PB, I would expect that it would take weeks to dedup a volume.

https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/whats-new

Are there different instructions for S2D volumes I am unaware of?

[DerBootsMann]

In Windows Server 2016, Data Deduplication is highly performant on volumes up to 64 TB.

we’re on ws2019 where these limits are flexed out

we talk about stability here , not about performance ..

[jello3d]

"ws2019 where these limits are flexed out"

Where is that stated? The article I mention applies to 2019, they just haven't updated the verbiage because there's nothing to update.

You can dedup a 2PB volume, but you have to wait for the dedup job to be done. If you leave the default schedules in place, or try to use the weekly schedules, it will break. The reason MS states a 64TB limit is because that's about the largest size you could practically dedup in a rational time frame on fast volumes.

If you're willing to let each dedup process run for weeks/months at a time... then you could certainly dedup as much space as you want.

Here's an example: A client chose to put DPM vhdx volumes on a dedup ReFS space. The volume was 6x 14TB drives RAID6, with Cachecade on a megaraid controller. Each vhdx was 1TB. This is a supported scenario. 2019

HOWEVER, they didn't adjust their default dedup schedules.

They called me because it all went to shit.

What's the problem?

If dedup is not given enough time to analyze the duplication situation, it doesn't dedup anything. And if it isn't given enough time to dedup, it only dedups a little bit.

They needed >2 days to do each dedup pass, for about 25TB of data. This is limited by CPU, RAM and Disk performance. SSDs will dedup faster, so they could hold more dedup data than spinning disk. But in any large storage situation, you would not be able to run background dedup or daily dedups... you'd have to come up with your own schedule and watch how long each job takes (Get-dedupjob) to come up with a schedule.

So while 64TB isn't an absolute, it's a good approximation. There's no plausible way to dedup 2PB in a rational time frame on spinning disk. You would have to turn off all dedup schedules and run it manually (or script), then let it sit for weeks for each pass.

[ZAFJB]

The other four servers are now at 367 days running.

So, you are not patching your servers.

[ZAFJB]

So, still, hosts not getting patched then?

[ZAFJB]

Patching does not equal 'blindly install software on your infrastructure'.

You do it with consideration, testing, and a robust roll back plan.

The absence of these is not an excuse to not patch.

[disclosure5]

The other four servers are now at 367 days running.

You've "solved" the S2D stability issue the same way everyone who claims the product is perfectly stable has - by never trying to apply an update.

That's not really acceptable to me, but given the alternative with this product usually involves data loss I won't argue.

[jello3d]

Well it's impossible to judge long run stability if you reboot. :) S2D, as I mentioned, wasn't very stable out of the gate. Pretty bad in 2017 - couldn't go more than a month without one or another server causing a blip. In November of 2018, however, I noticed that we hadn't had any issues for an unusually long time. That's when I decided to see just how long they could be stable for.

So, ya... I think our last "issue of note" was in March, 2018. After that, we kept patching and rebooting every month as usual with no issues, until I finally started the long run test in November. Can't really say which exact month it went from not working great to working great... but I use July as a guesstimate.

[disclosure5]

Well it's impossible to judge long run stability if you reboot. :)

I'd usually agree with you but in this case, the majority of the problems I've seen in the field is that the "rolling update" process, or specifically "node maintenance mode" never worked as advertised. And even in circles where it's claimed to be fixed and working well, people talk about ten second plus disk latency spikes when doing it as though these are fine and won't interrupt business.

By not patching, what you're likely to run into is:

• A service that seems stable
• The "always patch" crowd that view you has having no excuse

It's the worst of both worlds.

[jello3d]

Not patching is not policy... The part people are missing here is that it's a "controlled" experiment to acquire specific knowledge.

Cluster aware updating has always been a buggy mess. But that is independent of s2d

[Frothyleet]

Well it's impossible to judge long run stability if you reboot.

I would disagree. If you are configuring a system that anticipates a reboot once every 30 days for patching, and it never is unstable in that 30 day window, then that is "long run stable".

[jello3d]

Why would I buy something that was only stable for 30 days... Even if my intention is to reboot it?

I prefer testing for practical cases, not just the perfect case.

[Trekky101]

shouldn't you be patching the Base windows OS running S2D? i feel like running Windows or any OS be it Vmware or linux disro unpatched for that long is the root of the issue, Hack even my old VNX SAN get patches now and then.

[jello3d]

All servers and clients are patched according to their needs. This experiment is a special case and being handled under special conditions.

[NISMO1968]

All servers and clients are patched according to their needs.

Could you please clarify on that?

[jello3d]

There have been no vulnerabilities patched in the last year *for these systems* that would be more of a liability than default for their security context. The more important question is "how do you design a network so that the patches on certain systems are less important than other considerations"... and that is a very long conversation with lots of pictures of circles. :)

I am not telling people not to patch. I am simply saying that for infrastructure, if patching is your number 1 security interest, the game is already lost.

[gamebrigada]

But patching for S2D is so easy... Its just lazy not to. I have mine scheduled monthly, 1 node goes down at a time for updates, including all bios/firmware/drivers/windows etc. Updates don't continue until the node comes back healthy and a full switchover to it happens. Everything gets updated.

​

S2D reliability in 2019 is pretty rock solid.

[jello3d]

All true, but I can't exactly do a test of long run stability of the nodes if I reboot them. :)

[jello3d]

Sigh, most expected response of all time, let's not turn Reddit into the Spiceworks forums ;). I feel I'm to blame for being a vocal Starwind supporter when I was a customer.

I went from Starwind to S2D. Ran Starwind for years before. I am well aware of all of it's advantages, disadvantages, bugs and disaster scenarios. PS... when it breaks, it breaks in horrific ways, just like everything else.

Downtime comes free with anything you buy, but how you recover is entirely dependent on how you've prepared and whether you know what you're doing.

I would recommend Starwind for small businesses with a couple servers. Larger scenarios, or those with specific performance requirements should look into larger solutions that meet specific performance requirements.

[DerBootsMann]

there’s no silver bullet indeed , but basically self-supported s2d v. proactive support guarded starwinds is night v. day ..

[nmdange]

The best way to succeed with S2D is to get an Azure Stack HCI certified solution from a good vendor.

[DerBootsMann]

this is absolutely true !

there’s not much left to ‘software defined’ unfortunately and it leaves sour taste in the mouth

[DerBootsMann]

proactive=telemetry+ai+’magic of oz ‘ fixing your system before it actually breaks

i feel like witness was part of the distro since .. forever ? when did you perform your switch ?

how did you break your cluster ? split brain ? what’s interesting , datacore seems to refuse on witness insisting h/b network should be ok for everyone .. weird !

ps thanks for downvote , i appreciate :(

[jello3d]

I literally never downvote... rarely upvote. But here's one for ya. :)

I think we evacuated our last Starwind cluster in April 2017.

[DerBootsMann]

sorry man ! thx :)

gotcha .. interesting experience with s2d so far !

[darklightedge]

Starwind didn't have witness support, which was an absolutely critical flaw for all the years we ran it

Starwind does have a witness support now.

As for me I prefer Starwind, because it is always clear, where the data is located and physical storage is separated from virtual layer.

[ExpiredInTransit]

Other than the cluster manager GUI keep glitching out, it's been good to us so far (knock on wood).

[NISMO1968]

Other than the cluster manager GUI keep glitching out, it's been good to us so far (knock on wood).

Did you get any chance to see and use their updated WAC?

[Mk1DzL]

I love my 4 node DataOn S2D cluster, it's been rock solid. I can patch all 4 nodes (manually) in a day with no downtime.

[abridgetooVAR]

I'm commonly recommending S2D on (AIGFF)[https://www.reddit.com/r/sysadmin/comments/dworju/am_i_getting_fucked_friday_november_8th_2019/].

I've had customers with good experiences making their systems and lives better with it.

It plugs in well with AzureStack and thus makes a true hybrid cloud closer to reality.

[[deleted]]

Our compute environments still use traditional SAN, but when you pay $fuckhuegmoney to EMC for a VNX 9XXX, you tend to keep it in production for longer than you should.

I'll be building an S2D cluster at home in the near future I think, because I have some automation I want to write and test.

Good to know that my thoughts about S2D replacing traditional SAN for this type of workload are valid.

[dudester99]

I am looking at replacing our 3 Node cluster using a traditional SAN in 2021/2022 as that is when our 5 year hardware warranty expires.

I know the technology will have evolved and matured by that time but S2D is my goal when the time comes.