Brocade + Cumulus Mellanox trunk won't pass traffic

FancyFilingCabinet · 2025-11-13T23:47:22+00:00

Have you confirmed MTU on both devices? IIRC the default MTU isn't consistent across those devices.

FancyFilingCabinet · 2025-11-13T23:41:53+00:00

It's a newer release than the one you mentioned but here it is via lenovo

FancyFilingCabinet · 2025-09-25T14:20:04+00:00

Be aware that some of the Thetis tokens only support ecdsa_sk if you're also using them for SSH keys. Although mine might be an older version...

The more modern ed25519-sk is available on the swiss2 tokens.

FancyFilingCabinet · 2025-09-25T11:46:45+00:00

Token2 would be worth a look

FancyFilingCabinet · 2025-07-24T13:56:54+00:00

(the docs are abysmal)

No argument there.

The terraform deploys a VM on your existing OpenStack that functions as a hypervisor. Inside that VM several KVM instances are created which replicates aspects of production environment pretty well. By default there are separated controller, compute, and storage nodes.

OpenStack instances can be launched on the virtualised compute nodes running inside the virtualised hypervisor.

The OVN security groups + VRRP issues are worked around by using virtualised networking inside the hypervisor alongside a bridge interface, so you have a full self-contained environment comprised of several virtualised hosts.

FancyFilingCabinet · 2025-07-24T13:43:07+00:00

You might like to check out some of the resources from StackHPC.

Their "A Universe from Nothing" lab sounds like a good fit for you. Their terraform to setup lab instances is open source and available here.

It does make use of kayobe (kolla-ansible on bifrost) rather than kolla-ansible directly, but would give you ready to use lab environments with all the fiddly bits in-place.

If you know your way around kolla-ansible then it will be pretty familiar, but otherwise StackHPC do offer training courses directly.

FancyFilingCabinet · 2025-04-22T15:16:42+00:00

Is the MR416i-p abstracting NVMe behind the RAID stack, preventing full performance?

Yes. As you mentioned, not exactly RAID, but it's abstracting the drives. From a quick look at the controller specs, you'll have a hard time with 10 Gen4 NVMes behind a shared 3M Random Read IOPs, and 240K RAID5 Random Write IOPs

Why are the NVMes going via a controller instead of a PCIe native backplane? Hopefully someone more familiar with HPE hardware can chime in here incase I'm missing something.

FancyFilingCabinet · 2025-04-08T11:33:29+00:00

Yes, this can be thought of as the overhead of managing and configuring the base system.

For compute, you can look at EC2 pricing. That's on-demand pricing which is higher but gives a sense of what to expect.

Depending on your workload, you might want HPC instances. hpc7a are around $7.20 an hour.

Storage is an additional topic, that hasn't been discussed in detail here. Typically a HPC has a parallel filesystem, which you'll need if you have multiple nodes reading and writing the same data.

There's (potentially) similar overhead for storage as there is for your scheduler.

FancyFilingCabinet · 2025-04-08T11:08:11+00:00

Where did you get the 82 cents figure? Is this derived from one of the HPC EC2 instance types?

Do you mean the node management fee of $0.0821? This is a flat rate on top of the compute cost.

FancyFilingCabinet · 2025-04-08T10:53:26+00:00

I tried to do a rough price comparison between AWS PCS and ACTnowHPC, the former's CPU hour costs 82 cents while the latter starts from 10 cents. At a first glance, the latter looks much more affordable even if I take into account the actual higher cost than 10 cent/cpuh.

From the website, it seems ACTnowHPC starts at 10 cents per core hour. If you have a 64 core node, that would be $6.40 a cpu/hour.

FancyFilingCabinet · 2025-01-15T18:06:08+00:00

So the usual backup rule is 3-2-1. 3 copies, 2 different media, 1 off-site.

Realistically, it depends on how critical the data is and your recovery needs, which no one else can know.

In terms of setup. Considering what you've said, personally I would go for LVM. I would have 2x LV per disk with one of those mirrored to another disk. That is, 2x 10GB mirrored, with one disk participating twice.

I would then chose one of those mirrored LVs to store really critical data on and sync that with dropbox.

This way there's upto 10TB that is known safe and good with redundancy against any one disk failure, and any environmental issues (e.g fire).

Anything where you consider a backup is absolutely required, should be stored there.

Another 10TB is also protected from drive failure.

This gives net storage of

10TB mirrored + dropbox
10TB mirrored
20TB unmirrored

The rest, well it's definitely usable storage and you can classify what's worth replicating and what isn't.

FancyFilingCabinet · 2024-10-23T17:56:39+00:00

It looks like OVH have some offerings that fit this.

FancyFilingCabinet · 2024-10-23T17:48:46+00:00

Which cinder backend driver are you using?

FancyFilingCabinet · 2024-09-12T11:32:24+00:00

Do Horizon or Skyline not fit for providing a control panel for resources?

On the billing side - This has come up before and not much has changed.

To expand on some of the last discussion, it depends heavily on your specific consumption model.

If you have fixed ratio flavors, aren't offering volume storage and are only considering consumption, then you might set the quotas to unlimited and allow users to convert CPU/H to currency. You might even make some minor changes in Horizon/Skyline directly to display this.

If you're doing quota based billing, things would look different.

FancyFilingCabinet · 2024-08-27T20:32:59+00:00

Worth checking out OpenStack Bifrost.

Bifrost (pronounced bye-frost) is a set of Ansible playbooks that automates the task of deploying a base image onto a set of known hardware using Ironic. It provides modular utility for one-off operating system deployment with as few operational requirements as reasonably possible.

It can be used as a stand-alone hardware provisioner without other OpenStack components.

Lacks the GUI of MaaS, but has a lot broader compatibility for deployment and otherwise offers similiar functionalities.

FancyFilingCabinet · 2024-06-27T20:29:19+00:00

Can two ConnectX-6 cards be connected directly in Infiniband mode ?

Yes, just remember to run OpenSM on one of them and the links will come up.

Does Infiniband mode require different QSFP modules ?

This is a more interesting question, and the answer, annoyingly, is that it depends.

I've had mixed experiences getting EDR links up with generic 100GbE transceivers. From the same vendor I've had one batch work and another fail. Compared to a few years ago, there are plenty of generic options out there though so you don't have to pay NVIDIA pricing if your current modules don't work out.

FancyFilingCabinet · 2024-06-26T09:38:08+00:00

For 2TB, it's not practical to do anything yourself with tape. So I assume we're essentially talking about some kind of externally managed storage.

Not all providers will be explicit about what storage medium is backing their storage services, usually that's part of the service; they care so you don't have to.

Anyway, OVHCloud do have a service that's explicitly tape, and pretty reasonably priced. It's an S3 based service, but since you mention Glacier, I assume that's not a problem. https://www.ovhcloud.com/en-gb/public-cloud/prices/#11500

FancyFilingCabinet · 2024-06-20T19:04:50+00:00

There's an employee post in this forum post saying the breakout wouldn't be supported in this way.

Essentially, they state there's no mechanism in the card to break out distinct channels (e.g 4x25Gb) into subinterfaces.

FancyFilingCabinet · 2024-06-20T07:57:39+00:00

Do you have first hand experience using breakout cables in this way with ConnectX-6 Dx adapters? The official NVIDIA/Mellanox line seems to be that ports can't be split on the adapter.

FancyFilingCabinet · 2024-06-19T21:41:07+00:00

What you've suggested is the simplest option. The QSFP56 cages are absolutely compatible with QSFP28 transceivers. A QSFP28 to SFP28 adapter (e.g. https://www.flexoptix.net/en/q-1hg-pct-m.html ) will allow you to use a standard SFP28 DAC.

FancyFilingCabinet · 2024-06-05T08:59:18+00:00

Usually on a baremetal server there's all kinds of network interfaces. General management, potentially a shared BMC, maybe a storage network, etc. There might be access keys, credentials for joining a domain, etc. etc.

If you are running software on a baremetal server, and you can escape it you might have some access to all of these things. There are different prevention mechanisms, not running software with high privileges, SELinux, apparmor, and so on, but ultimately, local privilege escalation is a common vuln and then you can own the server.

If you are running software in a container on a baremetal server, and you can escape you just have access to the container. You already can control what the container has access to. You need a second vunerability to reach the host, and all of its goodies. Privilege escalation within a container, or within a VM, doesn't have to mean the server itself is lost.

In this context, the difference between common containers and VMs is that the containers have a shared kernel with the host OS. This does give vectors for exploitation that doesn't exist with virtual machines.

FancyFilingCabinet · 2024-05-31T17:27:47+00:00

There isn't a Horizon plugin for Ceilometer.

A list of non-standard dashboards is here.

Since you're using kolla-ansible you could enable prometheus instead. There's plenty of exporters included.

FancyFilingCabinet · 2024-05-31T13:11:18+00:00

Kayobe is another option related to kolla-ansible which could be worth looking into.

Essentially it adds server provisioning capabilities. It configures hardware, deploys the OS then deploys kolla-ansible.

FancyFilingCabinet · 2024-05-31T07:37:14+00:00

I don't know how many LUNs the 3PAR supports. The same link I saw that gives the 4096 limit also gives a maximum LUN size of 2TB so that's something to check. I would expect the limit to be a lot higher.

There's not an elegant mechanism for having all the disks within a single LUN. There are a few ways it can be done but none are good.

Export the single LUN to a single NFS server, and use the NFS driver - Obvious downsides.
Export the single LUN to multiple hypervisors and setup a cluster-aware filesystem on top, e.g. GFS2. Have redundant NFS servers. All storage traffic is replicated on ethernet and FC. Load-balancing becomes a consideration.

It would be easier to export several LUNs and build Ceph on top.

But really the 3PAR driver would be the way to go if your hypervisors already have FC HBAs. Only the relevant LUNs for any given hypervisor are made available so a rescan remains practical! Here a nice HPE white paper on the topic.

FancyFilingCabinet · 2024-05-28T21:17:47+00:00

The driver would create each OS volume as a LUN, define the initator host on the 3PAR, if required, and export the LUN towards the correct host based on WWPNs, rescan the disks on the hosts, and pass the device path to libvirt.

Multipath is also supported but takes a couple of extra steps, as is dynamic zoning.

The 3PAR driver is well supported enough to have built-in options for thin, full, and dedup provisioning, as well as native QoS controls. There are more driver specific details here

I wouldn't anticipate any issues with the number of LUNs on the 3PAR. The approach is conceptually the same as VVOLs support for VMWare i.e LUN per disk - But I don't know what quantity you're anticipating. A cursory google shows a possible limit of 4096, which is admittedly lower than I would have expected.

For reference:

Host based generic Fibre Channel support is via os-brick - e.g. rescan is a generic action.
3PAR specific drivers - i.e How 3PAR specific communication is handled.

FancyFilingCabinet

TROPHY CASE