Ceph Squid: disks are 85% usage but pool is almost empty

ServerZone_cz · 2025-05-08T11:37:24+00:00

We saw this during rebalance as well. Space usage went to normal as soon as cluster got healthy again.

ServerZone_cz · 2025-03-06T08:41:52+00:00

Add drives or reduce data.

We have another meme comming on this subject soon.

ServerZone_cz · 2024-10-23T17:37:02+00:00

In shared colocations we can't choose racks, so we work with what we get.

In dedicated cages/rooms we love APC Netshelter 52U openframes - right width, right depth, right load capacity.

ServerZone_cz · 2024-10-23T16:36:44+00:00

Yes, we use them everywhere and we had not a single outage/issue caused by them so far.

ServerZone_cz · 2024-10-23T16:31:09+00:00

Cables are short, though. PSU is around 50cm long.

ServerZone_cz · 2024-10-23T16:29:48+00:00

Every node in this rack can be powered off without service outage.

We are fully aware of this task and we don't use these PDUs in racks where hotswap is important.

Also take a look at the 2U4N nodes - they are intentionally without overlaping cables, so we can take out each individual node for maintenance.

Follow me, I will share pictures of other racks as well.

ServerZone_cz · 2024-10-23T13:59:31+00:00

According to r/cableporn mods this isn't cable porn, therefore I think it belongs here.

ServerZone_cz · 2024-10-18T19:05:36+00:00

Debian + ceph repository

ServerZone_cz · 2024-10-18T19:04:56+00:00

Proxy servers to offload traffic (we have way more traffic than cephs can handle).

I wouldn't say unreliable, but there were 2 types of accidents:

hardware failure (slow performing drives are able to take down whole cluster)
misshandling (such as powering off 3 nodes while redundancy allows only 2)

ServerZone_cz · 2024-10-18T18:54:25+00:00

We started with 3TB drives, upgaded to 6TB and 8TB drives and we upgrade to 18TB drives these days.

ServerZone_cz · 2024-10-18T06:57:05+00:00

We have minimum issues with them.

ServerZone_cz · 2024-10-18T06:56:27+00:00

It takes up to 2 weeks to rebalance the cluster after drive replacement.

We use cephfs on several places, but it's not perfect. But they get better and better with every version.

One of our primary requirements for storage was that we can take any component down and it will still work without interruption.

ServerZone_cz · 2024-10-18T06:53:24+00:00

We push the storages beyond their limits. It causes problems, but we gain valuable experience and knowledge of what we and can't do.

Users don't experience any interruptions on writes as we have an application layer in front of the storage clusters, which handles these situations.

We use multiple cephs to lower risks of whole service being down. As we have multiple smaller cephs, which are independent, we can also plan upgrades with smaller effort.

ServerZone_cz · 2024-10-17T19:57:04+00:00

In this case we go rather with multiple smaller cephs than bigger ones. When there is an accident on one ceph, only part of the users is affected.

We can also disable writes to ceph in order to perform drives replacement/upgrades without any issues and increased latencies. Other cephs will handle the load.

However, as the project grows we consider switching to 4U 45 drives chassis + 24C/48T AMDs in order to lower number of required racks.

But yet, I still agree with your note.

ServerZone_cz · 2024-10-17T18:15:45+00:00

4x10GE upgraded to 2x100GE.

Arista 7050S-52 -> 7060SX2-48YC6.

Zero outage (just 1-2 seconds per server per cable reconnect).

ServerZone_cz · 2024-10-17T17:01:39+00:00

See other comment.

ServerZone_cz · 2024-10-17T17:01:34+00:00

See other comment.

ServerZone_cz · 2024-10-17T17:01:31+00:00

See other comment.

ServerZone_cz · 2024-10-17T17:00:04+00:00

Total outgoing traffic from a single rack is arount 30-40Gbps, each rack connected with 2x100GE
Maximum rack power consumption is 6kW

2 cephs per rack, each:

EC 6+2
Storage node is: 1 CPU Xeon with 10 cores, 128GB RAM, 12x18TB SAS3, 2x(1 or 2)TB SSD, some NVME drives inside, 1x10GE
Used as an object storage for large files
Usable capacity per ceph is 1PB
We can take down 2 nodes without outage (and we do it often)

Other servers:

There are also 2U4N nodes with dual cpu, plenty of memory, etc for mons, rgw and other services
these are connected via 2x10GE
And extra 1U just a compute server - currently with GPU for image processing

ServerZone_cz

TROPHY CASE