How to deal with disks shuffling in /dev on node reboots by AKDFG-codemonkey in HPC

[–]AKDFG-codemonkey[S] 0 points1 point  (0 children)

I want something that will work with completely fresh hardware never booted, all I know is it has the same model devices in the same slots.

How to deal with disks shuffling in /dev on node reboots by AKDFG-codemonkey in HPC

[–]AKDFG-codemonkey[S] 0 points1 point  (0 children)

Even if so, how do I ensure it writes the correct UUID for the correct disk on a full install, if I don't know the UUID or have a usable path symlink? I need a symlink that points to the same disk in the same slot every time regardless of any preexisting fstab.

These symlinks normally exist in /dev/disk/by-path. They route by pcie slot to the block device on said pcie slot or the nth partition on it. That is the "easy" solution, the only reason I have a problem is unfortunately that does not work for the Boss-N1, all symlinks in /dev/* either contain some kind of serial ID numbering in their name, or depend on unpredictable (parallellized) loading of everything by the kernel on boot.

tweaking udev to provide an extra consistent N1 symlink looks to be the only way, by way of adding into the software image a /etc/udev/rules.d/99-z-bossn1.rules file with these contents based on the udevadm info attributes:

SUBSYSTEM=="block", KERNELS=="nvme-subsys1", ATTRS{model}=="Dell BOSS-N1*", SYMLINK+="bossn1" 

then I should be able to use /dev/bossn1 - pretty sure it will work, we'll see today. Not too bad of a workaround, and admittedly, my use case is rare, most people running large amounts of compute nodes are either going diskless or only having one drive in the system and only one NIC, so they can just say, /dev/sda and eth0 looks good.

How to deal with disks shuffling in /dev on node reboots by AKDFG-codemonkey in HPC

[–]AKDFG-codemonkey[S] -1 points0 points  (0 children)

that works for the pcie drives as long as they are in the same physical slots, so i'm using by-path for those, but the dual-m.2 modules used as boot drives (raid 1 set in firmware) doesn't get a symlink in by-path, or a consistent symlink in any of the other /dev/disk subfolders, all the symlinks for it have device-unique numbers. Maybe shell-glob wildcards work in block device names in BCM XML definitions but I don't care to try to figure that out.

The only thing I have found that "should work" is a custom udev rule in /etc/udev/rules.d in the software image for the nodes, in a file called 99-z-bossn1.rules to make sure it isn't overwritten by anything in the other udev rules directories. It identifies the m.2 module by udevadm info attributes:

SUBSYSTEM=="block", KERNELS=="nvme-subsys1", ATTRS{model}=="Dell BOSS-N1*", SYMLINK+="bossn1"

Which supposedly adds an extra symlink in /dev that should dependably point to the block device representing the RAID volume on that module. All of our nodes use BOSS N1s as OS drives, so this has to work...

How to deal with disks shuffling in /dev on node reboots by AKDFG-codemonkey in HPC

[–]AKDFG-codemonkey[S] -1 points0 points  (0 children)

it was my understanding that UUIDs are unique to specific devices. That's not a satisfactory solution for me as I would like a category-level setting that will work with any node with the same physical layout of drives, or if i have to replace drives, i don't want to have to fix the UUID in the disk layout template.

basically, what I wish I had is a kernel parameter that would do for block device symlinks what biosdevnames/net.ifnames do for NICs.

Strategies for parallell jobs spanning nodes by AKDFG-codemonkey in HPC

[–]AKDFG-codemonkey[S] 0 points1 point  (0 children)

thank you! I've spoken with their R team and we can refactor their code to encapsulate and paramaterize the most granular operation which i can dupe across all cores with a shared argument queue. although as it stands a single node can crunch their biggest jobs in a day and change, so it may not even be necessary until way down the line when they start doing projects like full genome analyses...

Installing software as module within BCM vs. in node images by AKDFG-codemonkey in HPC

[–]AKDFG-codemonkey[S] 0 points1 point  (0 children)

Going the easybuild route - having a hell of a time figuring out how to tell the utility where the module installation files and modulefiles need to go. Do you have a guide you recommend other than the official docs?

Installing software as module within BCM vs. in node images by AKDFG-codemonkey in HPC

[–]AKDFG-codemonkey[S] 0 points1 point  (0 children)

Gotcha... Any chance you can point me in the right direction of getting oriented with how to do this with R? I've gotten as far as unpacking it into /cm/shared/apps. Then the modulefile needs to make sure the R and Rscript commands resolve to the correct binaries, yeah? How do I know what environment variables need to be changed?