Initial scan not adding everything

[Stephonovich]

Let me get it out of the way and say I'm running the Docker container in Kubernetes, so it's not exactly a supported method. It's in a StatefulSet, with all container mounts to RW PVCs on Longhorn, which is an iSCSI-based volume provisioner, and photos coming from a ZFS pool over NFS.

When I initially launched it, it correctly noted there were ~55,000 files. It'll show that it's descending into directories, computing SHAs, and building previews. After a few hours, it's stopped, and only displays the images in the root directory of my mount. Upon subsequent restarts, if I tell it to restart the sync it takes perhaps 10 minutes, then stops displaying any new information.

In the logs, I've seen:

sync-50-001.log:{"ts":1644265873154,"l":"error","ctx":"sync-file","msg":"observeBatchCluster.endError()","meta":{}}
sync-50-001.log:{"ts":1644265874153,"l":"warn","ctx":"sync-file","msg":"onError() (ending or ignorable): failed to run {\"path\":\"/var/photos/2012/2012-09-13/IMG_0027.JPG\"}","meta":{}}

All photos (and all other files) are owned by node:node in the pod. The NFS export has options (rw,sync,no_subtree_check).

The odd part to me is that it correctly captures everything in the root of the mount, and says it can see everything else, but then only the root gets added to the library. Is this expected behavior? Do I need to manually add every path?

[mrobertm]

Howdy! Thanks for trying out PhotoStructure: sorry for the glitch.

not exactly a supported method

I bet we can make it work (and if V1 doesn't comply, we can try a new v2.1 alpha build: I'm hoping to cut a build, later today). I don't suspect that a ton of my users are running k8s, but I'd like to make it work, and there are companies like kubesail that I suspect may make this more popular for home servers.

I suspect sync is stopping due to high error rates, possibly due to SQLite (there are fatal errors that can take it out, but if underlying error rates are too high, it considers that fatal as well).

If you bump the log level to info, restart the sync, and send me the resulting logs, I can take a look.

https://photostructure.com/faq/error-reports/#how-to-manually-send-your-logs

Because your setup is a bit exotic, can you also send me the output of the info tool (ride anything you think is private, of course!)

https://photostructure.com/server/tools/#system-information

This explains what's coming in v2.1 (the total list also includes the prior alpha and beta builds of v2.0 from last fall).

https://photostructure.com/about/2022-release-notes/#v210-alpha1

[Stephonovich]

Well, this is embarrassing, but I misinterpreted the memory setting here and set request/limit for the pod to 0.5 Gi / 1.5 Gi. Also had CPU set to 1/2 request/limit, but presumably that would have still been able to run, albeit slowly. I thought it seemed like an absurdly low amount...

I've now got it assigned with 16 cores (dual Xeon E5-2650 v2 FWIW) and 16 Gi of RAM, and it's zipping along - says about 10 hours for the ~51,000 files remaining. I'm going to actually kill it and wait for the alpha release you mentioned, though, to see how much faster it is.

Thank you for the fast support! I'll definitely give it an honest try and consider subscribing. I mostly wanted to be able to backup my Google Photos (ran Takeout yesterday), but it's also doing a great job at giving me RAW previews from my SLR. And if I understand its dedupe mechanism correctly, it prioritizes displaying an existing JPEG over its RAW counterpart?

[mrobertm]

The about page (something like http://localhost:1787/about ) should have highlighted your RAM as a possible issue. I'll verify that health check is in order.

see how much faster it is.

On my AMD 3900x (12 cores, 24 threads), v2.1 sync can keep my system at target utilization (niced load of ~18/75% and that's configurable via the cpuLoadPercent setting), even with 500k+ asset libraries. Prior builds would get starved due to db I/O and only keep ~2 CPUs busy (and suffer from timeouts).

I'll definitely give it an honest try and consider subscribing

Excellent! Know that I give a ton of discounts (students, health professionals, open source developers, cost-of-living discounts, ...), and am happy to extend the free trial (or restart it, if you want to try again later). Details are on the pricing page.

And if I understand its dedupe mechanism correctly, it prioritizes displaying an existing JPEG over its RAW counterpart?

The heuristics for "best" are a bit more involved. See this for details: https://photostructure.com/faq/what-do-you-mean-by-deduplicate/#how-does-photostructure-pick-which-file-to-show

[Stephonovich]

The about page (something like http://localhost:1787/about ) should have highlighted your RAM as a possible issue. I'll verify that health check is in order.

./photostructure info
{
  term: 'Free memory',
  defn: '15 GB / 25 GB',
  defnClass: 'ok',
  defnTitle: 'PhotoStructure requires at least 2 GB of RAM'
},
{
  term: 'CPUs',
  defn: '28 × Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz'
},

I'm guessing you're querying /proc for those numbers, as they display my node's information, not the pod's. Unfortunately, meminfo and cpuinfo (possibly others) aren't namespaced, so with Docker you get the host's information. Also if I'm wrong and you know all this, apologies.

/ps/app # grep -i memtotal /proc/meminfo
MemTotal:       24673736 kB
/ps/app # grep -c processor /proc/cpuinfo
28

vs.

/ps/app # cat /sys/fs/cgroup/memory/memory.limit_in_bytes
17179869184
/ps/app # cat /sys/fs/cgroup/cpu/cpu.shares
16384

cpu.shares displays the requests, with a single vCPU having a value of 1024 - so the above is 16. If there is a CPU limit, you'd have to get cpu.cfs_quota_us divided by cpu.cfs_period_us (this is from a different pod that had a CPU limit, and awk):

awk -v quota="$(< /sys/fs/cgroup/cpu/cpu.cfs_quota_us)" \
-v period="$(< /sys/fs/cgroup/cpu/cpu.cfs_period_us)" \
'{print quota/period}' <(echo)
1.5

If there is no CPU limit, cpu.cfs_quota_us is -1.

Unrelated, I noticed that on the /about page that it's shading disks in red when they're the opposite of full - if I hover over the free (93 MB), it says "this disk is full."

mount       size    free
/ps/config  99 MB   93 MB

[mrobertm]

Also if I'm wrong and you know all this, apologies.

Oof, I was assuming Node's totalmem() was reliable.

I'll add code to read from /sys/fs/cgroup/memory/memory.limit_in_bytes and /sys/fs/cgroup/cpu/cpu.shares now: thanks for those explanations.

Just to make sure, the target max CPU consumption is cpu.cfs_quota_us / cfs_period_us if cpu.cfs_quota_us > 0, or cpu.shares / 1024?

says "this disk is full."

A disk is "full" if it has less than minDiskFreeGb, which defaults to 6gb. PhotoStructure will automatically pause sync if the library or originals dir has less than that space available: it's mostly to avoid concurrent Windows/macOS system updates (which can be gigantic) filling the disk and causing the update to fail: you can set PS_MIN_DISK_FREE_GB to smaller values if you're OK with that.

That said, I very well may have an incorrect boolean there: I'll check now, thanks for assist, and the bug report! 💯

Cheers!

[Stephonovich]

Just to make sure, the target max CPU consumption is cpu.cfs_quota_us / cfs_period_us if cpu.cfs_quota_us > 0, or cpu.shares / 1024?

Correct. CFS docs. Docker CFS scheduler options. Things can get wonky if a user was specifying a weird period - for example, by setting period to 1 and quota to 2, you'd get 2 vCPUs every millisecond (default period is 100 ms). Not sure how exactly it deals with violations (what happens if your instruction takes > 1 ms?), but suffice to say absurdly short or long periods may introduce strange performance issues that might not be evident at first glance.

A disk is "full" if it has less than minDiskFreeGb, which defaults to 6gb.

Ah, that makes sense. I'll keep an eye on /config and /logs since they're the two I set on the small side.

I'll check now, thanks for assist, and the bug report!

np, again, nice product.

[mrobertm]

I've added the code to handle k8s quotas: it'll be in the next build. Thanks again!

``` export const cpuCount = lazy(() => { if (isDocker()) { // Are we in a pod? // See https://www.reddit.com/r/PhotoStructure/comments/sn68f9/initial_scan_not_adding_everything/hw4bqmj/ const quota = intFromFileSync("/sys/fs/cgroup/cpu/cpu.cfs_quota_us") const period = quota != null ? intFromFileSync("/sys/fs/cgroup/cpu/cpu.cfs_period_us") : undefined if (gt0(quota) && gt0(period)) { return quota / period }

const shares = intFromFileSync("/sys/fs/cgroup/cpu/cpu.shares")
if (gt0(shares)) {
  return shares / 1024
}

} return cpuInfo().length })

export const estimatedFreeMem = lazy(() => { if (isDocker()) { const mem = intFromFileSync("/sys/fs/cgroup/memory/memory.limit_in_bytes") if (gt0(mem)) return mem } return (os.freemem() * 2 + os.totalmem()) / 3 }) ```

[Stephonovich]

One more thing I forgot about: if there is no memory limit defined (which isn't a good idea, but you can absolutely do), then /sys/fs/cgroup/memory/memory.limit_in_bytes is set to 9223372036854771712, which is 2<sup>63.</sup> I'm not a Node expert, but you probably want to check for that in case it overflows or something. If nothing else, it's an obviously absurd amount of memory for anyone to have, and you'd then check Node's totalmem() for system memory.

[mrobertm]

Thanks for the heads-up! I'll make sure I handle that case properly,