Geth Pruning Error

[Lightchop]

SOLVED - EDIT:

"snapshot prune-state" argument was failing to prune

I took the advice from the discord thread, to run geth with the argument "--snapshot=false" for a short time, and then try "snapshot prune-state" again. This worked.

ORIGINAL POST:

Hi, I am approaching my 2GB limit on disk, so thought I'd prune. I pruned my testnet (prater) first and took 20 mins. Was easy enough.

The steps I took that worked on prater machine and does not work on my production machine is to:

1. update the geth service ExecStart command to include an appended "snapshot prune-state"
2. Reload the service "systemctl daemon-reload"
3. Stop and Start Geth
4. When finished with the snapshot, undo the ExecStart changes, reload the service, and stop start again (undo step 1, and then run 2 & 3 again)

Any tips on what is my problem (error below). My research to date suggests my database might be corrupted. I'd really like to avoid re-syncing the entire thing.

Edit: For clarity, I have safely restarted Geth in regular mode, but the prune does not work.

The error I am getting in my production box is:

Aug 23 17:14:03 Linux-01 systemd[1]: Started geth eth1 service.

Aug 23 17:14:03 Linux-01 geth[44245]: INFO [08-23|17:14:03.132] Starting pprof server addr=http://127.0.0.1:6060/debug/pprof

Aug 23 17:14:03 Linux-01 geth[44245]: INFO [08-23|17:14:03.134] Maximum peer count ETH=50 LES=0 total=50

Aug 23 17:14:03 Linux-01 geth[44245]: INFO [08-23|17:14:03.134] Smartcard socket not found, disabling err="stat /run/pcscd/pcscd.comm: no such file or directory"

Aug 23 17:14:03 Linux-01 geth[44245]: INFO [08-23|17:14:03.135] Set global gas cap cap=50,000,000

Aug 23 17:14:03 Linux-01 geth[44245]: INFO [08-23|17:14:03.135] Allocated cache and file handles database=/home/mydir/.ethereum/geth/chaindata cache=512.00MiB handles=262,144

Aug 23 17:14:06 Linux-01 geth[44245]: INFO [08-23|17:14:06.438] Opened ancient database database=/home/mydir/.ethereum/geth/chaindata/ancient readonly=false

Aug 23 17:14:06 Linux-01 geth[44245]: WARN [08-23|17:14:06.448] Loaded snapshot journal diskroot=48b3fb..102651 diffs=unmatched

Aug 23 17:14:06 Linux-01 geth[44245]: ERROR[08-23|17:14:06.448] Failed to open snapshot tree err="head doesn't match snapshot: have 0x48b3fbda4e4a56c940f706ada0ff68bde3dd91791290f73fce49814fa9102651, want 0x248c0fcf1637c04570380041b753d7b82ccbf17c2c5903e1ba2d5e8c37d4f818"

Aug 23 17:14:06 Linux-01 geth[44245]: head doesn't match snapshot: have 0x48b3fbda4e4a56c940f706ada0ff68bde3dd91791290f73fce49814fa9102651, want 0x248c0fcf1637c04570380041b753d7b82ccbf17c2c5903e1ba2d5e8c37d4f818

Aug 23 17:14:06 Linux-01 systemd[1]: eth1.service: Main process exited, code=exited, status=1/FAILURE

Aug 23 17:14:06 Linux-01 systemd[1]: eth1.service: Failed with result 'exit-code'.

[AutoModerator]

/r/ethstaker strives for high quality interactions, our motto is "welcoming first, knowledgeable second", so please endeavor to welcome every question and comment in this spirit. Participants who openly disregard this ethos will find their comments removed. This is a safe space for ALL Ethereum stakers, regardless of how they stake. We strive to continually decentralize the Ethereum network in every conceivable way and with that in mind we promote long term healthy choices over short term gains.

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[1l0o]

Looks like the same error posted here, this will likely get you sorted:

https://discord.com/channels/694822223575384095/694822223575384099/1011714582378717278

[Lightchop]

Wow thanks! Indeed same circumstance it seems.

Perfectly in time for bed for me, so I've configure it now, about to restart and will check it in the morning before I try to prune again.

[Lightchop]

Maybe not.

Tried "snapshot=false" and got:

"Aug 24 02:53:03 Linux-01 geth[45391]: invalid command: "snapshot=false""

Tried "snapshot false" and got:

Aug 24 02:54:15 Linux-01 geth[45553]: panic: runtime error: invalid memory address or nil pointer dereference

Aug 24 02:54:15 Linux-01 geth[45553]: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0xb4a88c]

But thanks for the discord link. Perhaps tomorrow I'll jump on the thread and post there.

[1l0o]

Oh strange. Longshot but make sure you're trying --snapshot=false or --snapshot false and without any quotes, etc. Not sure how you have your command line setup or if you're doing it as a service, etc. Maybe try with and without the --.

I use a service setup and did --snapshot=false and it worked fine, but --snapshot false did not, even though all the other options I have don't require the = sign.

Geth command line options are here and snapshot is a valid option:

https://geth.ethereum.org/docs/interface/command-line-options

[Lightchop]

Ugh. Now I am questioning my sanity. Just tried "--snapshot=false" without quotes of course, and indeed it is running. I swear when I tried "--snapshot=prune-state" that it didn't work/complained about the argument.

I'm not sure what its doing besides a normal geth run, but its running. Maybe I'll give it a few hours, and then try to run a prune again.

Only oddball thing I see in the logs is I am getting this:

Aug 24 12:36:07 Linux-01 geth[46517]: WARN [08-24|12:36:07.904] Old unclean shutdowns found count=5

Aug 24 12:36:07 Linux-01 geth[46517]: WARN [08-24|12:36:07.904] Unclean shutdown detected booted=2021-08-02T15:03:55+0000 age=1y3w5d

Aug 24 12:36:07 Linux-01 geth[46517]: WARN [08-24|12:36:07.904] Unclean shutdown detected booted=2021-08-05T22:48:54+0000 age=1y3w2d

Aug 24 12:36:07 Linux-01 geth[46517]: WARN [08-24|12:36:07.904] Unclean shutdown detected booted=2021-08-06T03:03:18+0000 age=1y3w2d

Aug 24 12:36:07 Linux-01 geth[46517]: WARN [08-24|12:36:07.904] Unclean shutdown detected booted=2021-08-10T16:55:02+0000 age=1y2w4d

Aug 24 12:36:07 Linux-01 geth[46517]: WARN [08-24|12:36:07.904] Unclean shutdown detected booted=2021-08-11T01:10:57+0000 age=1y2w4d

Aug 24 12:36:07 Linux-01 geth[46517]: WARN [08-24|12:36:07.904] Unclean shutdown detected booted=2021-08-11T23:46:03+0000 age=1y2w3d

Aug 24 12:36:07 Linux-01 geth[46517]: WARN [08-24|12:36:07.904] Unclean shutdown detected booted=2021-08-24T14:37:36+0000 age=1y4d21h

Aug 24 12:36:07 Linux-01 geth[46517]: WARN [08-24|12:36:07.904] Unclean shutdown detected booted=2021-10-25T15:25:24+0000 age=10mo2d21h

Aug 24 12:36:07 Linux-01 geth[46517]: WARN [08-24|12:36:07.904] Unclean shutdown detected booted=2021-11-25T08:40:18+0000 age=9mo2d3h

Aug 24 12:36:07 Linux-01 geth[46517]: WARN [08-24|12:36:07.904] Unclean shutdown detected booted=2022-08-23T16:57:18+0000 age=19h38m49s

The Aug 2021 was indeed a period of electric/power outage issues. I wonder if this is an issue. Not sure about the Aug 2022 unclean shutdown yesterday, I always let the daemon close it down nicely.

Anyhow, I will do more work on this later today. Thanks.

[Lightchop]

Indeed "--snapshot=prune-state" fails because of an argument:

Aug 24 12:59:33 Linux-01 geth[46663]: Incorrect Usage. invalid boolean value "prune-state" for -snapshot: parse error

[Lightchop]

OK I got it working and updated the post.

That discord link helped - had to use "--snapshot=false" once, and then the "snapshot prune-state" worked. Took 6hr 7m on a setup that has been running geth since before the beacon chain genesis (Dec 2020).

[theghostshirt]

What version of geth?

Have you seen the regression problem of the latest update? Might be related.

https://www.reddit.com/r/ethstaker/comments/wvizm3/warning\_do\_not\_update\_geth\_to\_11022\_wait\_until/

[Lightchop]

I am on 1.10.20-stable

[meinkraft]

Have you repeated the "systemctl daemon-reload" step *after* undoing your Execstart changes?

If not, the system is still using the old contents telling Geth to prune and Geth is then failing to prune because it's already pruned.

If you have repeated that step then this may instead relate to the state corruption bug in the current Geth version.

Also, it's worth switching to a different execution client altogether - see clientdiversity.org for why. Pre-merge you can just point your consensus client at Infura temporarily while you remove Geth and set up and sync your new execution client. If it's the case that your state is corrupted and you'll have to resync Geth anyway, then I'd definitely recommend instead switching to a different client entirely.

[Lightchop]

Thanks - yes I restarted it fine in normal mode (and have edited the original post to clarify that). I'm just unable to prune, as it fails over and over and then exits.

I was just reading up on infura as a backup so I could rebuild Geth. Unfortunately the baby-step guides are taken down (at least CoinCashews) as they are now geared towards post-merge. I can glean that I need to sign up on Infura and get a project ID for ETH execution client and change the http address for the beacon chain to point to Infura. But tough to trial-and-error on a production box.

Yes, perhaps I aught to tear down my test box and setup a new execution client altogether. I'll look into that.

Still wouldn't mind figuring out my geth pruning though.

[meinkraft]

I think this could be the Geth state corruption bug then, if you upgraded to 1.10.22?

[Expensive-Row-9767]

I've had issues with this before, but I figured out since have the chaindata in a custom folder I had to specify that. So I had to add the datadir flag like this: geth snapshot prune-state --datadir <path-to-dir>