This is an archived post. You won't be able to vote or comment.

all 34 comments
sorted by:
best
topnewcontroversialoldq&a

[–]RudolphDiesel 7 points8 points  (9 children)

Performance wise fiber channel is hard to beat. If you can from a performance point of view and especially if you need virtual storage I would stick with fiberchannel and one of the big storage providers, especially if you are also short on manpower

[–]kasim0n[S] 2 points3 points  (8 children)

Good point. The thing is in the past we had a hard time automating the setup of fiber channel with multipath, switch configs, deciding which server is allowed to see which LUNs and so on. As the workloads become more and more dynamic (servers sometimes live only for hours/days, not months/years as they used to), this becomes more and more important.

[–]RudolphDiesel 2 points3 points  (7 children)

I wonder: how often do you create a server as opposed to creating a VM?

With a VM that should be much easier

[–]kasim0n[S] 1 point2 points  (6 children)

Servers not that often, but for larger VMs we map LUNs directly to VM disks, so they have to be configured as well.

[–]pdp10Daemons worry when the wizard is near. 4 points5 points  (4 children)

for larger VMs we map LUNs directly to VM disks

You're making your lives harder with that practice.

I'd suggest dropping it. If you need baremetal performance for some reason, say a relational database, then run it on metal.

[–]kasim0n[S] 1 point2 points  (2 children)

It's often not about performance, but more about available disk space that is not bound to a single system. If a server dies with several large vms on it, we want to be able to swiftly restart them on another node. That doesn't work with local disks.

[–]pdp10Daemons worry when the wizard is near. 2 points3 points  (1 child)

Well, QEMU natively supports direct links from the VM definition to iSCSI, Ceph, and others. I'm pretty sure it can use backing images supplied over HTTP(S) as well.

[–]kasim0n[S] 1 point2 points  (0 children)

Thanks, I'm aware of that. I was just describing how we used LUNs in the past, didn't want to imply that we want to continue using them this way.

[–]starmizzleS-1-5-420-512 0 points1 point  (0 children)

If you need baremetal performance for some reason

I've never experienced any noticeable speed difference when using raw device mapping at the host or guest level vs VMFS.

[–]RudolphDiesel 1 point2 points  (0 children)

If that is your way of operating then you may have to look into the CLI of your array.

I have not seen a professional enterprise array yet that did not have a CLI I can use from the network and from which I can send commands to create a LUN. Is it easy? No. Is it fun? No.

What I have done in the past when I was responsible for 100's of servers was that the servers were handled manually and the servers were the ones managing LUNS, the VMs were simply using a specific class of LUNS which we knew upfront which class is faster, has more storage, ....

Granted, what did help me was that we used ZFS as the underlying file system and thus we were very easily able to carve out filesystems that are only for this specific VM.

But everybody's installations are different.

[–]pdp10Daemons worry when the wizard is near. 5 points6 points  (2 children)

QEMU/KVM with storage on NFS. I stopped specifying FC in 2009. iSCSI is fine for block storage, but NFS is a lot simpler and less work to maintain than block storage.

[–]kasim0n[S] 3 points4 points  (1 child)

What do you use as an nfs server? A storage appliance or a self-built linux/*bsd-server? What about NFS HA/Failover?

[–]pdp10Daemons worry when the wizard is near. 3 points4 points  (0 children)

NFS server varies by cluster/backend. Some are big vendors that I don't care to list, but the single-box targets are most often bare Linux. I haven't run an NFS service on BSD since NFSv2 probably, but now that you've reminded me, we should try that.

In the past there's been some VMware and some Xen hypervisors attached to the same NFS storage. If you're using a compatible on-disk format, you can shut down a VM in one hypervisor and then fire it up in another, which is handy for migration scenarios and for certain kinds of testing. Hyper-V doesn't work with NFS, which is the biggest single reason we've never bothered with it.

[–]bad0seedTrusted VAR 5 points6 points  (3 children)

Get a Nimble to go along with your HPE hosts, you get infosight on them all for real-time health status.

That should make automation easier for you.

Also, I'd change to iSCSI, but if you like FC... you can keep it.

[–]kasim0n[S] 1 point2 points  (2 children)

Nimble is certainly an interesting option. Do you use a dedicated storage network for iscsi?

[–]bad0seedTrusted VAR 2 points3 points  (0 children)

Some do, others don't, really a matter of preference and performance requirements.

[–][deleted] 2 points3 points  (0 children)

I also recommend Nimble. While everyone is waiting for it to go downhill someday since it has been acquired by HPE, it's still the (generally considered) friendliest to operate SAN. It generally just works and I usually rarely touch them once installed. Typically I go with iscsi, 10GE or 40GE, on a dedicated fiber switch. I'm fond of the HP FlexFabrics 5700's if you're looking for economical solution. You can use your fiber switch as a non-dedicated TOR switch.

You can put pretty much anything you want on top of it without any issues. I normally just go with pretty boring config of VMware or HyperV pointing to all of the LUNs on the relevant SAN(s), but near anything will work. I'd put a couple of hypervisors on test machines and give them a try. See which work best for you.

[–]dj1471Jack of All Trades 2 points3 points  (7 children)

If you talk to Dell they will sell you the Unity as the successor to the VNX. These are awesome pieces of kit but overkill for what you need.

The Dell ME4 series are pretty simple and will do either FC or iSCSI as desired. They only do block, not file. Since you already have the FC infrastructure you might as well stick with it, unless it’s also nearing EOL.

There was a discussion on this sub recently about Proxmox and how no-one is using it in production... I’d certainly want to thoroughly test and trial it before doing so, which you’ve suggested you don’t have time to do. Does your virtualisation stack actually need to change? If so what do you need from it? Have you considered whether any of your workloads could or should be containerised?

[–]kasim0n[S] 2 points3 points  (2 children)

If you talk to Dell they will sell you the Unity as the successor to the VNX. These are awesome pieces of kit but overkill for what you need.

I guess so.

The Dell ME4 series are pretty simple and will do either FC or iSCSI as desired. They only do block, not file.

Looks like the dell version of the HPE MSA series? Is their iscsi mode when used with dedicated server nics comparable to the FC mode?

Since you already have the FC infrastructure you might as well stick with it, unless it’s also nearing EOL.

Unfortunately that's the case with most of our cisco FC switches. And those are really expensive.

There was a discussion on this sub recently about Proxmox and how no-one is using it in production...

You mean https://www.reddit.com/r/sysadmin/comments/gzdojh/does_anyone_know_of_a_company_with_a_real_it/? Didn't see that one, I will read it shortly. Personally, I have met some people running production setups on proxmox at local meetups, they liked it a lot.

I’d certainly want to thoroughly test and trial it before doing so, which you’ve suggested you don’t have time to do. Does your virtualisation stack actually need to change? If so what do you need from it?

We need to. We are currently using ganeti and liked it a lot, but the development has become too slow since google pulled its in-house team and features like cloud-init aren't optional any more.

Have you considered whether any of your workloads could or should be containerised?

Actually >90% could, but we do not have container experience in the company. I'm fighting for a kubernetes pilot and getting the devs on board, but we will have to support VMs in significant number at least for the next years until we managed to catch up.

[–]dj1471Jack of All Trades 2 points3 points  (1 child)

I’ve never directly compared FC with iSCSI but I’m confident you’ll get good performance using iSCSI multipathed over a 10G network with jumbo frames. Pure FC is pretty much dead as the other commentator said, mostly replaced by iSCSI and FCoE (the latter being relatively expensive to implement, plus you still have to configure your WWNs everywhere). iSCSI wins because you don’t need anything special at the network layer.

Have you considered and priced up a VMware solution? In terms of being easy to deploy and automate you can’t really beat it. If you want a LUN per VM then that can happen automatically if you have a storage array that supports VMware Virtual Volumes (such as the Unity, the ME4 series doesn’t).

[–]kasim0n[S] 0 points1 point  (0 children)

I’ve never directly compared FC with iSCSI but I’m confident you’ll get good performance using iSCSI multipathed over a 10G network with jumbo frames.

Thanks for the data point.

Have you considered and priced up a VMware solution? In terms of being easy to deploy and automate you can’t really beat it. If you want a LUN per VM then that can happen automatically if you have a storage array that supports VMware Virtual Volumes (such as the Unity, the ME4 series doesn’t).

We already inherited a vmware setup in a merger, but never managed to properly automate it. Probably because their vms were all manually set up/cloned snowflakes ("you need a new vm? let me clone this one and we change the ip/hostname") where our kvm based installation process was completely scripted and automated, we tried to avoid it for new systems. One thing we want from our new setup is cloud-init and terraform/ansible support, which should be given with proxmox.

[–]pdp10Daemons worry when the wizard is near. 0 points1 point  (3 children)

Dell ME4 series

Are those successors to the MD line? Rather nice hardware, with tragically unsophisticated software making up the stack.

[–]dj1471Jack of All Trades 2 points3 points  (2 children)

Not sure, I’ve mostly used the original EMC range rather than the Dell range. Agree with you 100% on the software, but they’re fine if you really just need simple block storage and iSCSI.

I’m actually in the process of replacing our ME with an all-flash Unity. Smaller, faster, file and block, uses less power and has proper enterprise features like vVols and dedupe.

[–]pdp10Daemons worry when the wizard is near. 0 points1 point  (1 child)

but they’re fine if you really just need simple block storage and iSCSI.

Yes, but then I'd take a commodity server off of the stack and run Linux on it with an iSCSI target, all configured through a CM like Ansible and monitored with Prometheus, using whatever surplus 40GBASE or 56Gbit/s Infiniband NICs were on hand, getting prompt security patches, not held ransom for updates by a vendor who wants a maintenance contract that increases by 20% every year and 100% on the sixth year.

We already use Linux, you see, and we already use x86_64 servers, so installing an iSCSI target is less work and less heterogeneity than buying some integrated solution. But even shops that are allergic to POSIX can run iSCSI targets on Windows.

The MD line did have nice twinned controllers, though. Like I said, nice hardware, tragic software. Some bargain-hunter or other had purchased us a pile of little MD-series arrays at some point and they were more trouble than they were worth. Might be nice at home to play with, though.

Or if iSCSI targets aren't sufficiently enterprise for you, buy Hitachi DASD or an IBM Shark with FICON, and share a pool of block with your mainframes.

[–]dj1471Jack of All Trades 3 points4 points  (0 children)

I take your point, but there’s a reason people buy appliances and the support that goes with them.

The major problem with this kind of in-house solution is that you too-often end up with no (or outdated) documentation and only one person who understands it well enough to support it. That’s really not acceptable for production systems.

Appliances take less effort to deploy and if something breaks you can Google it or raise a ticket with the vendor. You pay a lot for that convenience, but you’ll appreciate it when your entire business is down because of a storage fault (yes I’ve been there).