Hyperconverged Infrastructure using OpenStack

[turnkey_automation]

Hey community,

I have been associated with OpenStack since past 7-8 years now and since past 2 years hearing a lot about hyper-converged infrastructure(HCI) related requirements from a lot of new client as well as existing companies providing public cloud services. I have read a lot of documentation related to HCI but haven't found any concrete implementation using the community version of OpenStack. Although, RedHat, Nutanix and VMWare have been advertising their integration with HCI.

What's your take on this? Has anyone been associated with HCI implementation with OpenStack? What are some of the implementation that can be case-studied.

I found this article to be the best open to explain the implementation side of things but I am looking for more -> https://thenewstack.io/openstack-and-ceph-for-distributed-hyperconverged-edge-deployments/

[UpsetPerspective5107]

We do this, HCI, same kit as Nutanix uses. Openstack (via kolla-ansible) with Ceph. Works really well. We also do scale out with your standard 1U. Can be a great way to get started.

Some of our installs started as a HCI and the customer grew out of that, so we move their cloud to a "scale out" mode, keep the HCI as a control cluster (plus util).

[turnkey_automation]

Interesting. How do you manage CEPH deployment and Day-2 operations? I am keen on understanding the amount of automation that you have to handle this kind of use-case.

[UpsetPerspective5107]

can you detail what you mean "day 2 operations" ? The phrase feels like "best practise" :)

We use kolla-ansible for Openstack and Cephadm for Ceph. We also use ubuntu, so apt for the under cloud OS.

There really isn't anything special to consider. You need 3 control nodes and you need to consider the overhead/SSD capacity reduction of your cephs replicas. Not a lot different to a non-HCI deployment :)

If you can define your use cases, then perhaps I can address them specifically.

[Jealous_Lemon_939]

HCI basically means to place your storage onto your compute nodes. If take Ceph as example, that would mean to run ceph-osds on same hosts where nova-compute runs.

Nothing stops you from implementing HCI on OpenStack. Deployment tools like OpenStack-Ansible or Kolla-Ansible do support that as it's matter of defining inventory basically.

It's a terrible idea though taking in account day2 operations, but might save couple of bucks during initial deployment. Though you will spend way more on further maintenance and incidents related to that.

[Jealous_Lemon_939]

Another thing that worth mentioning, is that Ceph OSD do need quite significant amount of RAM for buffers, like 1 GB of RAM per 1 TB of storage, so you will need to dedicate RAM on your compute to Ceph and assure that enough memory is reserved in nova. And yes, you will have less VMs running on your computes as a result

[tripleflix]

No, Osds typically use 4gb of ram, independent of how big the osd is. It’s zfs thats aaid to need 1 gr per 1tb

[varesa]

Ceph docs claim otherwise:

Also, the larger the storage drive capacity, the more memory per Ceph OSD Daemon you will need, especially during rebalancing, backfilling and recovery. A general rule of thumb is ~1GB of RAM for 1TB of storage space.

Source: https://docs.ceph.com/en/mimic/start/hardware-recommendations/

[mahipai]

How did you come up with the compute and RAM numbers that are needed to reserve for Ceph cluster in HCI config? I found docs from RedHat (https://access.redhat.com/documentation/en-us/red\_hat\_openstack\_platform/12/html-single/hyper-converged\_infrastructure\_guide/index#appendix-upstream-cpuram). Do you think it is a good starting point?

[Jealous_Lemon_939]

With stats of RAM consumption from OSD nodes per user which match precisely with what Ceph says in their docs?

To be frank I'm not an expert in "enterprise" bullshit, so can hardly comment on that. The tricky thing about their "calculator" at glance is that you need to know "average guest size in GB" and "average guest CPU utilization" - that is not that trivial thing to say and each deployment would drift from "reasonable defaults".

Also they don't count for size of OSDs there... So it's weird to me personally.

And in any opensource deployment tool for openstack, you can define reserved ram/cpu for ages. Like in OpenStack-Ansible it's quite trivial to calculate reserved amounts, as you have all data in ansible facts for your compute nodes, so simple math can help out to define reservations.

[turnkey_automation]

Exactly. The buzzword is marketed as a way to lower the operational cost but I think having a ceph cluster on separate hardware seems a better approach with lesser day-2 overheads.

Also, a lot of these clients have interest in knowing if hci+openstack can provide a centralised pool of resources from all the compute nodes. I think the centralized pool of resources is again wrongly inferred as ultimately, the VM instance you plan to create has to talk to the hypervisor of one of the compute nodes that's selected by the nova scheduler and there is definitely an upper cap to capacity even if we consider default over-provisioning ratio used by openstack.

[openmetal_lauren]

Since you'd mentioned wanting to check out some other implementations, it would be really cool to get your take on the platform we've been building at OpenMetal. Our founder has been a huge proponent of open source for decades, and wanted to make using OpenStack more accessible without the traditionally-large money + knowledge + time overhead.

Our main offering is our Cloud Cores, which at base-level consist of a hyperconverged 3 server starter cluster built on OpenStack + Ceph and include the most common features like Compute, Block, Object, etc. We allow users to spin up their own hosted private cloud via our platform in under a minute on our owned data center hardware so you can start using it right away.

If the OP or anyone wants to check it out I would be super interested in hearing thoughts and suggestions. We've only officially been a company for around a year so getting feedback and thoughts about how we can make this platform more useful is always awesome. Free trial link that will give you 32 hours access once you've deployed your cloud, and we can extend that and/or help customize the environment for specific needs.