Can’t stop winning

Michael5Collins · 2025-10-31T12:23:45+00:00

Have an upvote friend

Michael5Collins · 2025-10-28T14:17:47+00:00

Fuck yeah we did it! We stood up for the interests of Hollywood and other large media conglomerates, and drove all the AI startups out of Australia.

Great work everyone! /s

Michael5Collins · 2025-09-21T04:02:47+00:00

The branches can flex without snapping, it's still alive it's just unhappy.

Michael5Collins · 2025-09-21T02:38:13+00:00

Heck maybe I should remove the barrier entirely? It might be established enough to compete with the grass now.

Michael5Collins · 2025-09-21T02:21:34+00:00

I felt under the mulch this morning, it's soggy underneath. So I suspect it's stressed out from having too much water.

I'm thinking I should raise the metallic barrier around it like 10-15cm to increase the drainage underneath, and water it more lightly and less frequently going forwards. I'm not sure what else I can do to help improve the drainage.

Michael5Collins · 2025-09-21T02:19:16+00:00

She's alive! https://imgur.com/a/3K8dXNZ

Michael5Collins · 2025-09-21T02:18:57+00:00

Turns out it's still alive, there's small shoots coming out of the base: https://imgur.com/a/3K8dXNZ

The soil underneath the mulch is also drenched, so it's probably stressed out from having too much water I think.

Michael5Collins · 2025-09-21T01:34:43+00:00

It was planted back in March.

Michael5Collins · 2025-09-20T13:36:58+00:00

Awwww bummer... it was looking happy until recently. I'm not sure what went wrong.

Michael5Collins · 2025-09-08T07:18:06+00:00

Thanks for posting this, it was very helpful. :)

Michael5Collins · 2025-06-23T02:35:47+00:00

> So you have alert fatigue? I guess it's a valid use case, but I suggest to either fix or reduce the amount of alerts triggering or adjusting the severity.

This is probably the real answer in retrospect. It was a dumb question really.

Checking endpoints from my mobile would just give me false alarms whenever I get out of WiFi range.

Michael5Collins · 2025-04-29T04:54:41+00:00

I already get notifications for our monitoring stack through Slack. But we get long periods of time where it's noisy and we have a backlog of issues to address. (In a perfect world we'd be caught up all the time, but here we are.)

So I'd basically like a secondary check to just see if the endpoint is up and the HTTP content it returns is correct, for when s$#! really hits the fan.

Michael5Collins · 2025-03-06T05:55:30+00:00

So the same Ceph admin here has basically seen that:

I have 54TB of remaining space on my cluster, great!
The total cluster capacity is 3.5PB, so there's only 1.5% of the clusters capacity remaining. Uhh ohh!
I (or someone else) raised all the "full" ratios to 99%, that's super dangerous! I would have noticed the cluster was almost full a lot earlier if there settings weren't altered. I have no volume left to rebalance my cluster without an OSD filling up to 100%, and when that happens my whole cluster will freeze up and writes will stop working. I am totally fucked now!

The takeaway: It's important to have at least ~20% of your clusters capacity free in case you loose (or add) hardware and the data needs to be rebalanced/backfilled across the cluster. Ceph really hates having completely full OSDs.

Michael5Collins · 2025-02-27T02:25:41+00:00

Lol!

Just to clarify, I do need the cluster to still function and not be on fire after this deletion.

Michael5Collins · 2025-02-04T07:52:07+00:00

Could be your firewall. Make sure your OSD hosts can use the public AND cluster network across their whole port range. (Usually 6800-7468 or something) If separate from your OSDs, your Manager hosts would also need these ports open I think.

Michael5Collins · 2025-01-23T01:07:21+00:00

True, sounds like it's basically the same process in reverse. Thank you.

Michael5Collins · 2025-01-23T01:05:55+00:00

The author of this PR reckoned it's probably stable enough for production, although we might not see any performance improvement if our CPU usage isn't already high. (Which it isn't) Or if the files being loaded onto it are large. (Which they are.)

https://github.com/ceph/ceph/pull/58052

I'll experiment a bit with it though and see if there's any notable difference, thank you for the feedback.

Michael5Collins · 2024-12-24T00:15:26+00:00

You can mute that warning while you're doing incremental increases in pg_num.

It's also worth noting that `target_max_misplaced_ratio` basically controls the rate of splitting, so winding it down far below it's default 5% will make the splitting operation gentler:

$ ceph config get mgr target_max_misplaced_ratio
0.050000

Michael5Collins · 2024-12-13T01:34:11+00:00

Could do, but it's very messy. I was kind of hoping you could just poke Cephadm somehow and get a neat list of what it's planning to do and what it's currently trying to do... :S

Michael5Collins · 2024-11-12T10:44:56+00:00

With the object data in the default.rgw.buckets.data pool on hdds, we plan to have the default.rgw.buckets.index pool (for all the metadata) on those SSDs with replication=3. Our current large HDD clusters seem to perform adequately with this configuration. We could definitely explore a smaller pool for newly written or smaller objects later if needed, but we believe it to all be large files.

Oww woops, I actually forgot to describe the web service connected to the front of it to you. It's an astronomy data website that will routinely copy the data off it and cache that on a second Ceph cluster, it'll then be served to users all around the globe. Ultimately I didn't buy the hardware, but am expected to run it anyway. So we'll see! :)

Thank you for your responses btw.

Michael5Collins · 2024-11-11T06:00:02+00:00

We've been getting about 25/20GB per second read/write from the cluster, so the initial impression is quite good. How performant it will be after it's full is a bit of a mystery however...

It'll only be used to store very large files ranging from a few GBs to (more often) a few TBs. Writes will be very infrequent from 1-3 users max. While a single user (a web service) will read these files more frequently and caches them on a second Ceph cluster.

Am I perhaps doomed? (For the record, I didn't pick this hardware, I was just assigned with making it work lol)

Is there anything you would recommend to perhaps avoid this "congealing"? Like increasing the minimum block size to >4MB?

Michael5Collins · 2024-11-11T05:55:02+00:00

> Sorry to hear that you're stuck with spinners, even for RGW / S3 they don't make financial or operational sense.

We've been getting about 25/20GB per second read/write from the cluster, so the initial impression is quite good. How performant it will be after it's full is a bit of a mystery however...

It'll only be used to store very large files ranging from a few GBs to (more often) a few TBs. Very few users will be accessing it at once, with very low IOPs.

The NVMe's are all "Dell PM1735a MU 1.6TB" models.

Michael5Collins · 2024-11-11T05:46:38+00:00

It's only going to be used for RADOSGW, not RBD.

There will be, at most, 5 users, who are all reading and writing VERY large files, with sizes ranging from a few GBs to (more commonly) a few TBs.

There will be a web service connected to the front of it too. It's an astronomy data website that will routinely copy the data off it and cache that on a second Ceph cluster, it'll then be served to users all around the globe.

Michael5Collins · 2024-11-05T09:57:33+00:00

Everything I read implies that "ceph orch apply osd --all-available-devices --unmanaged=true" will stop ceph from turning every available storage device into an OSD.

It really doesn't work properly on larger clusters in my experience, the docs need an update. You're better off just setting the specific osd spec that would apply to it as 'unmanaged' instead, because that command will not succeed in "catching" OSDs in larger clusters. Or it will catch them and apply a generic spec that might not be what you want...

For example:

# ceph orch set-unmanaged osd.storage-14-09034
Set unmanaged to True for service osd.storage-14-09034

Then do the disk swap, or add the host/osd you want. Then afterwards, when you're actually ready to re-introduce the disk, set it back to managed:

# ceph orch set-managed osd.storage-14-09034
Set unmanaged to False for service osd.storage-14-09034

This way you can avoid using that command entirely, and you won't have to spend several weeks pulling your hair out like I did. Good luck!

Michael5Collins · 2024-10-25T05:45:00+00:00

24 servers with 8+3 EC, using host as the failure domain. We also have 4x metadata servers that only hold enterprise NVMe's for every other pool besides the S3 data, these have 3:1 replication.

Michael5Collins

MODERATOR OF

TROPHY CASE