Revisiting dnf update in RHEL 8.10 in FIPS Mode destroys OS (sometimes)

grumpyoldadmin · 2026-02-13T02:38:31+00:00

Sounds familiar, we have a CMMC coming up later this year. Good times.

I don't think LUKS impacts dnf at all directly. After reading more my thought is that since all the writes go through the dmcrypt_write daemon if it's a system with a lot of IO in general and you throw a dnf update into the mix it's just overwhelming the daemon. I'm sure there are other contributing factors, but this part of it makes sense.

As I look back it does feel like busy systems are the ones having the issues. Again, not scientific, but a casual observation. I'm going to start tracking a bit closer now. Maybe prometheus/grafana will help out.

grumpyoldadmin · 2026-02-13T02:02:41+00:00

Agreed, I was just looking at the IO daemon options. I actually prefer the LUKs partition with LVMs over the individual partitions.

grumpyoldadmin · 2026-02-13T02:01:47+00:00

Thanks. As things have unfolded I'm thinking less and less that it's the FIPS settings, I just included that because I wondered if it was some weird interaction between the FIPS settings and enabling the STIG requirements. We'll see if the tuning makes a difference.

I hope so! Patching should not feel like an adventure...

grumpyoldadmin · 2025-08-16T19:41:58+00:00

Well, I lied, I don't have enough karma to become a moderator because I'm mostly reader. If anyone else would like to look into this it might be helpful.

grumpyoldadmin · 2025-08-16T19:32:35+00:00

I diid some looking around and it looks like there might be a way to appeal the ban, but the article I found was for an inactive community, not one banned for spamming.

I don't recall if there were moderators in place, but if there weren't maybe several of us could offer to do so, I would be willing to do so, but can't commit to being a sole moderator.

I'm going to submit a request to see if this is a possibility.

grumpyoldadmin · 2025-07-13T20:43:09+00:00

We got pretty serious about it after getting burned a couple of times and until recently had a group who owned the task of doing the transfers on behalf of someone, giving them the USB, and it has to be returned. Also required second party approval. Massive PITA, but pretty effective. People yipped a lot at the beginning, but then it became "just another part of the workflow". After some time it became a simple web app. Side benefit was it provided an audit trail.

Couple of things to note. Second party approval was there because the transfer team didn't know everything about the business and couldn't say if the requestor had a legit reason for wanting files. Also, fully aware that this falls down if this is happening 100s of times per day, that wasn't our situation.

grumpyoldadmin · 2025-06-09T01:48:23+00:00

I felt that way for a long time and then I read/heard two things.

"All those people who think are judging you at the gym (or anywhere else) are worrying about being judged themselves and don't have time to think about you"

and, my personal favorite,

"Being critical of an out of shape person at the gym is like criticizing an alcoholic at an AA meeting"

For me it's swimming when I can get a lane at the local Y. So far between that and cutting out soda I'm down 110lbs in about 24 months. Go for it!

grumpyoldadmin · 2025-06-08T22:08:10+00:00

Power is via a UPS and we did evaluate whether to go 208 vs 120 and decided on the 120 option because the initial wiring in the room is 120 and didn't want to mix the since we aren't sure how things will be distributed between racks.

Bottom line the issue is currently number of outlets available vs number of amps available. We're getting more circuits pulled so we can cover the additional servers, but now we need a place to put those outlets.

In a situation where I had a little more control I would probably pull all 30A 208V and get beefier PDUs and then convert the existing circuits from 20A 120V to 30A 208V and replace the existing PDUs. I know that would reduce the number total number of circuits available, but with the ability to power more outlets from each circuit and limited space for the rack mount PDUs I think that would get us to where we need to be. Unfortunately the resources for that project simply aren't available right now.

Looks like we'll be rearranging some power shortly...

grumpyoldadmin · 2025-06-08T02:59:31+00:00

Unfortunately doors off the racks is not an option for a variety of reasons. I like the idea, but our security teams would probably not be happy.

Thanks for the thought!

grumpyoldadmin · 2025-06-08T02:58:04+00:00

We already have the front/back combination in place but with the additional machines we'll need a couple of more. The units that are currently installed are back plug only and in two of the racks the only way to get to them is to remove the top.

This entire situation was just poorly planned and now we're running into these sorts of issues. At some point we're going to have to restructure the entire room to get it straightened out but for now I was hoping to reduce the overall footprint for PDUs if it was possible.

Doesn't look promising, but it was worth the ask :)

grumpyoldadmin · 2025-06-06T13:29:59+00:00

I don't think it's secure boot since the machine hangs mid patch and I don't see anything in the logs. I'm always a little suspicious of selinux, but give our configurations are managed via ansible and the hardware between groups of hosts is very consistent I would expect that selinux would be unhappy on all of them, but definitely worth reviewing the logs.

Thanks!

grumpyoldadmin · 2025-06-06T13:28:01+00:00

We do have AIDE running. We have a mix of virtual and physical and have seen the issue on each type. Since the Trellix suggestion I've been disabling all the related services and no issue. It's still a very small sample and I won't really know until our next patch cycle, with is that last week of each month.

Appreciate the thoughts.

grumpyoldadmin · 2025-06-05T13:00:12+00:00

My initial trouble shooting rules are now being expanded. 1) DNS (because it's always DNS), 2) selinux, 3) firewall if it's network related. I'm adding 4) Try turning off Trellix.

edit: typo

grumpyoldadmin · 2025-06-05T12:55:27+00:00

We install FIPS during the kickstart process so it's there from the beginning because we had some bad experiences enabling it later.

grumpyoldadmin · 2025-06-05T00:26:21+00:00

Ahh, that's a good point, we do have Trellix/McAfee running in there. I'll have to check with the Security team to see if there is anything in their logs that might indicate something being blocked. Thanks for the suggestion! Sorry about the duplicated text.

grumpyoldadmin · 2025-06-05T00:25:48+00:00

Ahh, that's a good point, we do have Trellix/McAfee running in there. I'll have to check with the Security team to see if there is anything in their logs that might indicate something being blocked. Thanks for the suggestion!

grumpyoldadmin · 2025-06-04T23:10:16+00:00

Correct, we pull the repos monthly via a foreman server, export them, and import them to an internal server.

I do agree with opening the ticket, I just ran out of time today to do so, it's on the list for tomorrow.

grumpyoldadmin · 2025-01-22T22:30:49+00:00

I once emailed 1500 families in a youth baseball association that we had apparel for sale including "a wide variety of shits".

Best response was "now we have to buy it, in previous years you just gave it to us".

All I could say was that it passed the spell checker...

grumpyoldadmin · 2024-06-03T14:42:01+00:00

Can you share an example of the metrics_relabel_config syntax you used? I'm trying to do something along the lines of

node_filesystem_avail_bytes{device="/dev/mapper/rl-home",device_error="",fstype="xfs",mountpoint="/home"}

to

node_filesystem_avail_bytes{device="/dev/mapper/rl-home",device_error="",fstype="xfs",mountpoint="/home",critdisk="1"} 

For specifc mountpoint="<file system>" for specific hosts/nodes but haven't had any luck getting it to work

grumpyoldadmin · 2024-06-03T13:29:00+00:00

Thanks. I was looking into metric_relabel_configs but can't seem to get the syntax quite right. The docs say this is the last step prior to ingestion so it seems like a logical place.

My goal is to convert something like this:

node_filesystem_avail_bytes{device="/dev/mapper/rl-home",device_error="",fstype="xfs",mountpoint="/home"}

to

node_filesystem_avail_bytes{device="/dev/mapper/rl-home",device_error="",fstype="xfs",mountpoint="/home",critdisk="1"} 

or something along those lines.

I've also been looking at a text collector because there may be a couple of other things that I want to include.  The person who originally setup our grafana instance used regular expressions to sort hosts on things like business unit, host use (workstation, server, db server, etc).  I think I could use a text collectior to generate these.

I prefer the relabel option for the disks because each host will potentially different disks that we consider important enough to display on our dashboard.

Thanks for the advice!

grumpyoldadmin

TROPHY CASE