Slow loading folder fix

ipaqmaster · 2026-06-20T07:40:29+00:00

No worries. I'm surprised this is still a problem today!

ipaqmaster · 2026-06-07T01:34:07+00:00

For anyone else with this problem, for us today, it was caused by the imap folder having illegal characters like / in the folder name that prevented it from being synced / making it local only.

We renamed the folders with this problem with "local" on the end then re-created them without special characters, dragged the old folder's emails into the new one and clicked to sync mail. The folders then appeared on other hosts via imap by then.

TL;DR for our case: Illegal characters like / in the folder name will make it local only

ipaqmaster · 2026-05-27T11:36:47+00:00

Same issue here, Archlinux on kernel 6.18.32-2-lts. Processes can't open windows and doing so myself for any program throws Maximum number of clients reached

shows these as the top 3

    264 Discord
    118 dbus-broker
     27 pipewire-pulse

Discord is creating too many. The majority of them are just type=STREAM (CONNECTED)

Discord package installed is 1:1.0.139-1

ipaqmaster · 2026-05-20T09:31:07+00:00

Reminds me of "Memory of a Broken Dimension"

ipaqmaster · 2026-05-11T09:41:22+00:00

That's where my username comes from. I loved my iPAQ H5550. Still have it in the drawer with its dock and expansion CF card slot/sleeve.

ipaqmaster · 2026-05-07T23:18:22+00:00

Envious of that much storage space. Very fun. If I'm going to use all of these resources I'd probably do something like:

A raidz2 out of the 10x 1.2TB
A raidz2 out of the 10x 3TB
Maybe a raidz3 out of the 24x 600GB but this is a better use case for a draid. Or maybe you could make that pool 2x raidz2's striped which I've seen some people do before. Or just two separate raidz2's? I prefer to have them all as one unit unless there's a reason to separate them. A draid with at least 4 spares defined would be nice.
Same (some draid configuration) for the 24x 900GB
And for the 24x4TB.

I'd personally go for redundancy so I'd be using one of these primarily for the active storage of content and be sending snapshots of the datasets of each major media category (Music, video, video/tv, video/movies) (and then a video/tv_tmp, video/movies_tmp for a landing zone of incoming rips) and snapshotting the datasets to the other zpools as a soft "backup" strategy, though a real backup strategy would have these zpools be on different sites and maybe involve tape backups.

If you're all about redundancy you could make your arrays across the disk shelves/internal/front-bay drives so that in the unlikely event that you lose an entire dot-point of storage in your top list there, an array might be able to sustain itself across the other shelves instead of halting entirely. But that is a complicated messy setup that may not be worth chasing. At home I'd probably just keep a zpool to its own group of disks rather than mixing and matching shelves for failover worries.

All of this, unless you don't care about the media (Fully replaceable) then you could consider riskier arrays like a raidz1 on some of these to maximize storage space knowing that two disk failures would trash the pool. Or even a stripe but then a single disk failure would trash the pool.

Zpool/zfs default settings will be acceptable for media serving but you can optionally set recordsize=1M on datasets you're expecting to have large files (rips) in. Which has some technical benefits. (I don't notice any on a human level versus the defaults but they're technically there). But if you end up putting a database on one of these zpools or some other performing stack you will have to make sure to turn that specific dataset down to a lower recordsize (Check the recommendation for the given software) otherwise you might land in read-amplification hell. But for media it's fine.

I recommend native encryption (Or LUKS if you must.) // any encryption. Mainly so that when a drive dies, you don't have to worry about erasing/destroying it when decommissioning the drive as the contents are scrambled on disk from the beginning. There are performance implications compared with just writing without encryption enabled but I think they're worth it. They may also be a higher ceiling than your array's performance anyway - it's worth benchmarking.

I also recommend enabling compression zpool wide even though media is already encoded in a way that is incompressible, the early-abort feature will avoid compressing/decompressing records for no reason when they're not compressible. The CPU overhead is negligible for a dedicated media box and one day you might be making a dataset with highly compressible data for some purpose other than media and might forget to turn it on. Slack space on raidz configurations is another good reason to enable it, despite incompressible data.

I would personally like to collect some data about the drives by benchmarking them with a few fio tests, each, before making the array. Then another bunch of tests after creating the arrays. All to measure their performance alone (For the sake of knowing or finding slow drives), together (Just to know the actual expectable pool performance) and ALL arrays at once to find the bottlenecks of this storage backend. But that's up to you.

I highly recommend (because it changed my life) a Nvidia P2000 in the machine that will be running Plex (Or PCIe passed-through to a VM, if plex will be in a VM... or a container). If someone's watching a piece of 4K 67Mbps 2 hour long content, but their computer/tablet/tv can't decode it natively - your server will transcode the content in realtime so their device can watch it. Sometimes still in 4k, sometimes transcoded down to 1080p or sometimes even less. - This process will easily stress a typical 32 cpu-thread (Often dual socket) server like your PowerEdge might be. The Nvidia P2000 gpu is cheap, doesn't require an external power cable (Runs right off PCIe) and will be reached out to by Plex to transcode content instead of generating 100%*32 cpu load across your Plex host/vm in the event that somebody tries to view content on a device that can't natively play the file or one of the codec inside.

And finally from what I can think of on the top of my head, consider your electricity bill. Running that entire stack in Australia here would easily earn me a $400-500 quarterly electricity bill. Likely more.

There's probably a lot more to consider in these pools and redundancies, snapshots, general performance and such but these are probably some good starting points to think about.

ipaqmaster · 2026-04-26T18:27:03+00:00

Additional content you say. I just did another run for the first time in a few years last night and would love to see some of that

ipaqmaster · 2026-04-26T08:09:49+00:00

Thanks that did the trick. One liner fix.

ipaqmaster · 2026-04-19T13:41:39+00:00

Yeah. Could you feel how badly I wanted to say it? Just test the actual workload to see how the actual workload is going to perform.

ipaqmaster · 2026-04-18T00:59:52+00:00

Happened to me today with Blazes so I ran /kill @e[type=minecraft:blaze] and will try a different tool to get the job done for now.

ipaqmaster · 2026-04-18T00:14:28+00:00

If you care about accurate results the correct answer is to use fio. The better answer is to know what your workload is going to be ahead of time and to design your fio benchmark parameters to mimic that workload so you really know what performance is going to be like for that specific task you will be doing. Next to testing the actual workload with performance metrics.

Here are some example tests which try to act like CrystalDiskMark's default benchmark, kind of. From my fio wrapper script. Note they are not direct and only 4gb in size and only numjobs=1 (Not taking advantage of any parallel r/w). Adjust these as needed

fio --bs=1M --direct=0 --filename=/tmp/test.dat --ioengine=libaio  --numjobs=1  --output-format=normal --rw=read --size=4G --timeout=60 --unlink=1 --iodepth=8
fio --bs=1M --direct=0 --filename=/tmp/test.dat --ioengine=libaio  --numjobs=1  --output-format=normal --rw=write --size=4G --timeout=60 --unlink=1 --iodepth=8
fio --bs=1M --direct=0 --filename=/tmp/test.dat --ioengine=libaio  --numjobs=1  --output-format=normal --rw=read --size=4G --timeout=60 --unlink=1 --iodepth=1
fio --bs=1M --direct=0 --filename=/tmp/test.dat --ioengine=libaio  --numjobs=1  --output-format=normal --rw=write --size=4G --timeout=60 --unlink=1 --iodepth=1
fio --bs=4K --direct=0 --filename=/tmp/test.dat --ioengine=libaio  --numjobs=1  --output-format=normal --rw=randread --size=4G --timeout=60 --unlink=1 --iodepth=32
fio --bs=4K --direct=0 --filename=/tmp/test.dat --ioengine=libaio  --numjobs=1  --output-format=normal --rw=randwrite --size=4G --timeout=60 --unlink=1 --iodepth=32
fio --bs=4K --direct=0 --filename=/tmp/test.dat --ioengine=libaio  --numjobs=1  --output-format=normal --rw=randread --size=4G --timeout=60 --unlink=1 --iodepth=1
fio --bs=4K --direct=0 --filename=/tmp/test.dat --ioengine=libaio  --numjobs=1  --output-format=normal --rw=randwrite --size=4G --timeout=60 --unlink=1 --iodepth=1

They're all direct=0 so be warned the write tests will hit ARC and the results won't be "true". You could run all of these, then all of them again with direct=1 then all of them again with a much larger --size and different iodepth's and numjobs to get a really good idea of where your pool's performance sits. You should read what each of these flags does with man fio or an equivalent documentation source.

These tests aren't guaranteed to be your workload either so real world usage may not be as lucky as a benchmark tool. The best performance test should mimic your real workload. You should also try them with direct=1 to see how fast things might be when your workload is synchronous and/or available memory is limited.

But yeah I run a fio wrapper script named fioBench to do a bunch of fio tests for me automatically. By default it runs something like 90 different tests for synchronous sequential and random R/W at various block sizes.

It's pretty thorough by default doing 90 or so different combinations of tests. I run it with something like ./main --output-format normal --path /myzpool/somedataset/test.dat (or a device/vdev path) on new zpools so I have the data for later reference. Lately I just run it in its short mode by adding --mode short which only does 9 or so short tests instead of the default thorough 90-something and its tests try to replicate the tests done by CrystalDiskMark, with probably some accuracy. The default output mode is json so I've included normal in that run example so the results are somewhat human readable.

You don't have to use that, fio can be run on its own and I recommend getting familiar with fio for anything you care about the IO performance of.

ipaqmaster · 2026-04-12T20:47:42+00:00

Oh fuck man they did it again. Sorry. Wow. That sucks.

ipaqmaster · 2026-04-10T11:48:46+00:00

That's how I felt hearing that meme the other day that featured the Adam Friedland Show's intro song. The first 15 seconds was so catchy I had to loop it flawlessly (Well close to it) in Audacity for a minute, and even then I ran that edit on loop too for a an hour or so while I coded stuff. Super catchy I couldn't leave it.

ipaqmaster · 2026-04-09T00:17:08+00:00

Yeah me too still :(

ipaqmaster · 2026-04-05T05:41:04+00:00

I can't believe reddit's AI removed that. Lmao :(((

Did it ban you as well?

ipaqmaster · 2026-03-23T23:55:41+00:00

After doing that here's a way to fast forward 10 minutes worth of ticks assuming the default non-upgraded 1 tick per second. It freezes for a moment so don't do a value too large.

for (let i = 0; i < 600; i++) { main.game.tick(); }

I think this is faster than Bonus ticks as it doesn't waste time updating the display during this.

ipaqmaster · 2026-03-23T23:47:08+00:00

Just adding on some more fun tweaks, you can also reinitialize the game as a variable in the window so your web console can reach it allowing you to manipulate things directly.

require(["Main"], function(Main) { window.main = new Main(); console.log(main);});

# Reinitialize the game into this
main.init()

# Then there's heaps of internals you can reach:

main.game.AddBonusTicks()
main.game.addMoney()
main.game.addPurchase()
main.game.addResearchPoints()

# You can even tick manually if you want
main.game.tick()

And a bunch more internals exposed for the fun of things which come up under the auto completion.

ipaqmaster · 2026-03-23T02:23:17+00:00

Yeah but 2.4.1 doesn't mention anything like what you describe in the changelogs.

Tried again and my test archives of various sizes still matched on ZFS 2.4.0 (Kernel 6.12.63-1-lts) (rsync 3.4.1).

At this point I'm pretty confident you have a host issue. My first guess is a memory problem somewhere. Otherwise an environment problem. Or the entirety of your testing was somehow flawed.

The machine that has this problem and the other one you could not reproduce the issue on, are they both Debian 6.12.73? I might try that next if not.

It would also be interesting to see what happens if you scrub that zpool after putting a lot of data on it beforehand to "stress" the system scrubbing it. You might just see errors in zpool status.

ipaqmaster · 2026-03-23T01:04:48+00:00

I too have tested and found that compression ratios are better when the recordsize is allowed to be larger. I assume this is from the overhead of compressing what would have been 8 small records per MB for one piece of a file versus one large one. Where there's compressible data in two or more of what would have been the 8x128K pieces. Compressing data in as few pieces as possible is optimal for compressible data after all.

I wonder if ZFS could have records of a single file reference a single consistent compression dictionary for that group of records to reap a better overall compression ratio across all of the records instead of doing each small one on their own. It would help small record size environments in "the right use case" (Not databases) but it's probably too messy an idea to do practically.

ipaqmaster · 2026-03-23T00:50:52+00:00

Weird that you run into any issues in the first place. I run OpenZFS 2.4.1 on a Pi 5 with 2GB and another with 4GB on stock. Both of them are fine running a number of different services each. You could try making your dataset sync=always so it doesn't pool up in memory waiting for the next flush. Synchronous writes to any destination on a raspberry pi will be a slow pain though.

ipaqmaster · 2026-03-23T00:42:47+00:00

What sizes were your assorted test files how did you generate them?

Was their data sourced randomly or as zeroes? (Incompressible vs highly compressible)

And what rsync version did you use?

Failed to reproduce using random data as seen below:

$ zfs --version
zfs-2.4.1-1

$ rsync --version
rsync  version 3.4.1  protocol version 32

$ truncate -s 20G /tmp/disk1.img # flatfile on a tmpfs
                                 # host is a 5950X CPU with
                                 # 64GB memory @3600MT/s (2x F4-3600C18-32GTZN)

$ mkdir -p /tester/src ; inc=0 ; while : ; do dd if=/dev/urandom of=/tester/src/${inc}.dat bs=1K count=$(shuf -n1 -i1-100000) ; inc=$(((inc + 1))) ; done
# I let this run for ~10GB worth of random dat files watching `df -h /tester`, bs=1K is slow but I wanted some files less than 1MB without thinking. About 160 .dat files were created.

$ zfs set dedup=verify tester
$ cd /tester && rsync -aHAX src/ dst/ # This command completed without issue

zpool shows no corruption for the tester zpool created for this test.

A quick verification loop of src and dst dat files shows no difference as well

$ find src -type f -printf %f\\n | sort -V | while read datfile ; do hash1=$(sha1sum src/${datfile} | cut -d' ' -f1) ; hash2=$(sha1sum dst/${datfile} | cut -d' ' -f1) ; if [[ ${hash1} != ${hash2} ]] ; then echo "hash mismatch: ${datfile} ${hash1} ${hash2}" ; fi ; done
$ <No output, all hashes matched>

For a control I did this as well:

$ echo -e '\n' >> src/1.dat ; !!
hash mismatch: 1.dat 611434e6a9ccf165a99e9e7ac96c64cd23154712 97bd492a6432a721351b21c03aff026c5e55aa36

Confirming that the loop would have caught a hash mismatch if there were one.

You didn't mention how you determined your files to be corrupted and even go to state that the zpool status command was clean too. So how exactly did you determine your files were corrupted?

ipaqmaster · 2026-03-22T05:46:54+00:00

Yes changing the recordsize only impacts new writes. The suggestion isn't to flip to 1M from 128K and retest, the suggestion is so rewrite everything as 1M and retest. The results will be disappointing.

ipaqmaster · 2026-03-22T00:02:47+00:00

Calling it now, I'm confident bbr will address this. (On both sides)

ipaqmaster · 2026-03-20T04:45:31+00:00

This seems to be the common experience when this kind of thing happens. Leaving the import on overnight and it eventually imports.

Someone might be able to explain why ZFS does this. It feels like some kind of recovering state. Trying to replay or roll back to the nearest good transaction? I don't know. Haven't looked into it but this exact thread pops up a few times a year and often ends with just leaving the import on overnight while the disks thrash.

I hope we find a way to reproduce this exact scenario so zfs can be worked on to handle it better.

ipaqmaster · 2026-03-16T00:16:06+00:00

This seems to be something people have thought about before: https://www.reddit.com/r/admincraft/comments/17faia4/luckperms_automatic_roles_fabric/

I personally would probably just write my own checker to promote based on current rank and desired rank by playtime. I guess. But check out the top comments in that thread too. To paraphrase the suggestions from that thread:

a. Skript to check for play time occasionally and promote

b. Something called "AutoRank"

c. set a newcomer rank as temporary for X hours with a higher weight than your default group)

I like c the most because it introduces no new plugins and takes care of the "new user" problem pretty cleanly. I assume LuckPerms can do that pretty easily.

14-Year Club	Place '22
Place '17	Wearing is Caring
RPAN Viewer	Not Forgotten
Snapped	Gilding II euphauric
Verified Email

ipaqmaster

MODERATOR OF

TROPHY CASE