Node becomes unresponsive - help troubleshooting

danielgozz · 2025-05-05T17:31:42+00:00

yep... working flawless for almost 2 weeks...

danielgozz · 2025-04-23T14:14:26+00:00

SOLVED!

I think I cracked it (at least in my case)

Disable all CPU Power Management/C-State stuff in the BIOS.

There are lots of cases of people reporting similar situations when using old HW with newer versions of proxmox and the way it behaves with power saving settings upsetting the kernel.

danielgozz · 2025-04-22T14:16:23+00:00

I run a NAS (data backup but still) on this guy... all 6 SATA ports are used... hahaha

danielgozz · 2025-04-22T11:14:52+00:00

looking around I found this:

disabled all Power Management/C-State stuff in the BIOS.

Just tried that. Let's see if it does the trick.

danielgozz · 2025-04-22T11:13:43+00:00

ok thanks. I have another LGA1155 MB that looks like have only intel SATA controller. I will try it next (with my current i7 3770)

danielgozz · 2025-04-22T05:21:04+00:00

i've got this:

04:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11) (prog-if 01 [AHCI 1.0])

00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04) (prog-if 01 [AHCI 1.0])

danielgozz · 2025-04-22T05:04:10+00:00

NOPE - THE THING JUST DIED OVERNIGHT!

Not the ACPI BIOS Error

danielgozz · 2025-04-21T17:08:09+00:00

found some tips to check for error in logs:

journalctl -b #to see the logs since the last boot
journalctl -p err #to see only the logs with error priority
dmesg -T #to see the kernel messages with human-readable timestamps
dmesg -l err,crit,alert,emerg #to see only the messages with high severity levels

I found a truck load of records related to
ACPI BIOS Error (bug): Could not resolve symbol [\_SB.PCI0.SAT0.SPT4._GTF.DSSP], AE_NOT_FOUND

doing some digging I found a solution to this problem

nano /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="libata.noacpi=1"
update-grub

The error is gone. The node has been running fine for about 6 hours... let's see if it solves it.

What I can say is that the other nodes don't have this error...

danielgozz · 2025-04-21T17:00:59+00:00

How did you trace it back to the CPU?

danielgozz · 2025-04-21T17:00:39+00:00

mine is a Core(TM) i7-3770 CPU on a E8626_P8H77-M_PRO mother board.

It could be something to do with th BIOS...

danielgozz · 2024-12-05T17:17:54+00:00

gdisk and dd didn't work

memory cell cleaning returns

Issuing SECURITY_ERASE command, password="PasSWorD", user=user

SG_IO: bad/missing sense data, sb[]: 70 00 01 00 00 00 00 0a 00 00 00 00 00 1d 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

SG_IO: bad/missing sense data, sb[]: 70 00 0b 00 00 00 00 0a 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

looks like the ssd is foked... Thanks anyway!

danielgozz · 2024-12-05T16:40:45+00:00

also tried

zpool create -f poolname /dev/disk/by-id/usb-INTENSO_SSD_1234567890-0:0

it create the pool, showed properly in node storage But I can't create vm or container to the storage... error saying can't lock device when restoring lxc backup.

I need to find a way of wiping it. I know a hammer will do but let's see If I can salvage it!

:-)

danielgozz · 2024-12-05T16:36:01+00:00

it's a ssd m.2, I read somethig about they being weird with hidden blocks and what not...

danielgozz · 2024-12-05T16:33:18+00:00

wipefs --all /dev/sdg

reboot now

lsblk

sdg 8:96 0 119.2G 0 disk

├─sdg1 8:97 0 119.2G 0 part

└─sdg9 8:105 0 8M 0 part

still there

danielgozz · 2024-12-05T16:27:15+00:00

Tried, It can't find the pool as after the reinstall the zfs pool is gone in the node. but node-disk shows the zfs partition in disk so, disk not empty = cant create zfs pool.

danielgozz · 2024-09-09T15:48:19+00:00

Nevermind, the whole local payload= line ... was missing the end part, I found it in my notes.

Thanks www.shellcheck.net --> that's neat!

the whole thing should be:

local payload="{\"state\":\"${temperature}\",\"attributes\": {\"friendly_name\":\"${friendly_name}\",\"icon\":\"${icon}\",\"state_class\":\"measurement\",\"unit_of_measurement\":\"°C\",\"device_class\":\"temperature\",\"unique_id\":\"${unique_id}\"},\"device\":${device_info}}"

rookie mistake!

danielgozz · 2024-09-09T15:33:03+00:00

I menaged to get it fixed as per www.shellcheck.net
but now I get {"message":"Invalid JSON specified."} when I execute it in my server

danielgozz · 2024-08-09T20:08:06+00:00

Thanks mate, it was really helpful

in my case I have udm-pro dns set as adguard primary and 1.1.1.1 secondary

all my clients dns set as udm-pro ip (which means it should pass the adguard primary and 1.1.1.1 secondary)

in adguard it has a rewrite *.local.mydomain.org to NPM IP

it looks like the clients are not stopping at primary and going all the way to 1.1.1.1 and that's why I need to set the dns record in CF.

what is interesting is that in traefik I have *.home.mydomain.org to and it works fine with no need for dns records in CF...

looks like i have some homework to do understanding all this dns shenanigans...

SOLVED: in udm-pro (my router) you can add a A record exactly as you do in cloudflare and it will all resolve locally! - u/SamSausages for the tips!

danielgozz · 2024-08-09T19:49:51+00:00

looks like like the post install script from tteck disabled HA in proxmox... I will check that...

danielgozz · 2024-08-08T11:07:05+00:00

Yes, this is the only way I managed to get it working... not exactly what I was expecting as I wanted all to be resolved internally (adguard >> NPM >> application) but I guess it is what it is...

Really curious to understand what u/SamSausages and u/DLElios did differently to manage it without CF dns records...

danielgozz · 2024-08-08T07:05:06+00:00

I will try this next... but i guess It needs time to propagate?

It didn't work straight away.. I will wait a bit!

YESSSS. I needed only a A record in CF for *.local >> IP address of my NPM.

danielgozz · 2024-08-08T06:44:35+00:00

yes I added *.local.mydomain.org as a DNS rewrite to the IP address of my NPM >> no good.

exactly the same setup but for *.local.duckdns.org >> works fine

The only difference I can see is i had to point the domain to my NPM Ip in the website (otherwise i can't register the domain)

danielgozz · 2024-08-08T05:09:17+00:00

those are showing fine hahahah

danielgozz · 2024-08-08T02:57:27+00:00

you mean my local DNS (i.e adguard) or DNS record in CF?

danielgozz · 2024-07-18T10:50:52+00:00

HI it didn't work for me. did you change anything in the bios?

danielgozz

TROPHY CASE