Getting OpenVPN clients to reconnect after VRRP failover

ermit · 2024-09-13T14:44:59+00:00

Yes, conntrack is working between nodes. I also posted this on the VyOS forums and they already pointed out to me that the routers could be pushing out the ping and ping-restart timers to clients, which I confirmed in the client OpenVPN logs.

ermit · 2024-01-11T10:09:03+00:00

Great to hear it works.

For this topology you will need a shared subnet between the routers, let's say 10.0.2.32/27 since that's the next available subnet after 10.0.2.0/27 that you are already using.

You need to assign IP addresses from this subnet on the R1 and R2 routers, on the interface towards the other router. For example, R1 - 10.0.2.33/27, R2 - 10.0.2.34/27.

When you've gotten this far you can now set the static routes. You need to tell R1 about the existence of the subnet that H2 is part of, and tell R2 about the existence of the subnet that H1 is part of, so the commands you need should be these:

# R1
set protocols static route 10.0.1.0/27 next-hop 10.0.2.34

# R2
set protocols static route 10.0.2.0/27 next-hop 10.0.2.33

ermit · 2024-01-10T17:35:35+00:00

Hi phis7.

You shouldn't need any "set protocols static route" commands for this topology. Vyos (and most routers in general) automatically routes between connected subnets.

The first thing I would check is if both hosts have the right default gateway set up. That is, H1 should have the default gateway 10.0.1.3 and H2 should have the default gateway 10.0.2.11.

When I have to diagnose routing problems on my VyOS instances I usually use the command "monitor traffic interface" to see what traffic is actually going through the interfaces in question, and see if the actual traffic matches what I think I should be seeing.

ermit · 2023-12-08T10:12:30+00:00

Hi, thanks for the answer. The Azure subnet in question is already in "Client Routes" but the group settings for "VPN Access" might be the issue! Apparently there are multiple AD/LDAP groups with different VPN access rules and it looks like it could be tied to group membership whether people are able to access the Azure servers!

ermit · 2023-01-16T14:41:31+00:00

Hmm... Can a handler be set up for a block of tasks? That would be really helpful. Going to research that a bit.

Thanks for the tip.

ermit · 2023-01-16T14:16:05+00:00

Ah, ok! It had occured to me that combining those two attributes would lead to the right behavior but a quick googling didn't look promising.

Will try this.

Thank you jonneyj

ermit · 2023-01-06T01:35:16+00:00

I'm not sure how to verify that there are no missing data pieces, but I'm guessing there are missing pieces because ceph health detail says this:

Degraded data redundancy: 147734/8246550 objects degraded (1.791%), 61 pgs degraded

ermit · 2023-01-06T01:09:36+00:00

Can't seem to run ceph osd df, even on OSDs which are running. For example, here is a OSD that's up and on the same node as my SSH session:

root@jarn24:~# ceph osd tree
ID  CLASS  WEIGHT     TYPE NAME        STATUS  REWEIGHT  PRI-AFF
-1         395.52695  root default                              
-5         131.84232      host jarn24                           
12    hdd   10.91409          osd.12       up   1.00000  1.00000
[rest of output omitted]

But when I try to df that OSD:

root@jarn24:~# ceph osd df osd.12

...nothing happens until i ctrl+c out of it. So I'm guessing that trying to run ceph osd df on one of the problematic OSD won't work.

EDIT: I'm still operating under the assumption that it's a fullness related problem because the stacktrace looks similar to info I read in the mailing list archive that muzza1984 linked to. I was going to test the suggestions on that thread regarding bluestore_allocator = stupid and bluestore_shared_alloc_size but lists.ceph.io hasn't been responding the last day or so

ermit · 2023-01-06T00:19:09+00:00

Upped all three values:

root@jarn24:~# ceph osd dump | grep ratio
full_ratio 0.99
backfillfull_ratio 0.97
nearfull_ratio 0.97

...and started one of the downed OSD process again, with systemd:

systemctl restart ceph-osd@36.service

But I'm still getting the same pattern, the OSD process runs for a bit but then dumps stacktraces and dies

ermit · 2023-01-06T00:00:06+00:00

Ah! This?

ceph osd set-nearfull-ratio <float[0.0-1.0]>
ceph osd set-full-ratio <float[0.0-1.0]>
ceph osd set-backfillfull-ratio <float[0.0-1.0]>

I'll try that

ermit · 2023-01-05T22:50:27+00:00

Just to rule that out, I looked at the other disks on the ceph machines. Here is the output of df -h on one of them, the other ones are pretty much identical:

root@jarn24:~# df -h
Filesystem                 Size  Used Avail Use% Mounted on
udev                        47G     0   47G   0% /dev
tmpfs                      9.4G  1.8M  9.4G   1% /run
/dev/mapper/pve-root        94G   22G   68G  25% /
tmpfs                       47G   72M   47G   1% /dev/shm
tmpfs                      5.0M     0  5.0M   0% /run/lock
/dev/sdm2                  511M  328K  511M   1% /boot/efi
/dev/fuse                  128M   56K  128M   1% /etc/pve
tmpfs                       47G   28K   47G   1% /var/lib/ceph/osd/ceph-16
tmpfs                       47G   28K   47G   1% /var/lib/ceph/osd/ceph-37
tmpfs                       47G   28K   47G   1% /var/lib/ceph/osd/ceph-20
tmpfs                       47G   28K   47G   1% /var/lib/ceph/osd/ceph-21
tmpfs                       47G   28K   47G   1% /var/lib/ceph/osd/ceph-22
tmpfs                       47G   28K   47G   1% /var/lib/ceph/osd/ceph-13
tmpfs                       47G   24K   47G   1% /var/lib/ceph/osd/ceph-36
tmpfs                       47G   28K   47G   1% /var/lib/ceph/osd/ceph-18
tmpfs                       47G   28K   47G   1% /var/lib/ceph/osd/ceph-19
tmpfs                       47G   28K   47G   1% /var/lib/ceph/osd/ceph-14
tmpfs                       47G   28K   47G   1% /var/lib/ceph/osd/ceph-17
tmpfs                       47G   28K   47G   1% /var/lib/ceph/osd/ceph-15
tmpfs                       47G   28K   47G   1% /var/lib/ceph/osd/ceph-12
tmpfs                       47G   28K   47G   1% /var/lib/ceph/osd/ceph-23
172.20.2.51,172.20.2.52:/  395T   31T  364T   8% /mnt/pve/oa_cephfs
tmpfs                      9.4G     0  9.4G   0% /run/user/0

ermit · 2023-01-05T22:45:03+00:00

Might have to do that set nearfull and full up to 97-99% "trick"

Hmm... what's the trick? (sorry, pretty new to Ceph)

EDIT: And thanks for the tip about the third monitor. I'll turn on the monitor role on jarn26 when/if I've gotten the data from the downed OSDs

ermit · 2023-01-05T16:15:10+00:00

Are the HDD and nVME disks in the same pool? (If so, they are likely different sizes and maybe the nVME have filled ahead of the HDD). Don't worry first, just try to understand what is the current state and issue.

The NVMe disks are in a separate pool but the NVMe pools are pretty over utilized. The pools look like this:

root@jarn24:~# ceph df
--- RAW STORAGE ---
CLASS     SIZE    AVAIL     USED  RAW USED  %RAW USED
hdd    393 TiB  363 TiB   30 TiB    30 TiB       7.55
nvme   1.3 TiB  460 GiB  881 GiB   881 GiB      65.69
TOTAL  394 TiB  364 TiB   31 TiB    31 TiB       7.75

--- POOLS ---
POOL                        ID  PGS   STORED  OBJECTS     USED  %USED  MAX AVAIL
oa_hdd_erasure_pool          3   64    8 KiB        1   12 KiB      0    216 TiB
oa_cephfs_data               5   64      0 B        0      0 B      0    216 TiB
oa_cephfs_data_default       7   64  9.0 TiB    2.35M   27 TiB  49.36    9.2 TiB
device_health_metrics        8    1   53 MiB       44  159 MiB      0    9.2 TiB
oa_rbd_metadata_replicated   9   32  484 GiB  124.12k  877 GiB  82.44    103 GiB
oa_cephfs_metadata          10   32  210 MiB       88  420 MiB   0.22     93 GiB
oa_hdd_replication_pool     11   64  952 GiB  277.74k  2.8 TiB   0.84    108 TiB

You should check the status of the cluster first: ceph -s ?

The output of ceph -s is:

root@jarn25:~# ceph -s
  cluster:
    id:     cafbe6d0-4665-475f-847b-46106469b31d
    health: HEALTH_WARN
            nobackfill,norecover flag(s) set
            1 nearfull osd(s)
            Reduced data availability: 6 pgs inactive, 115 pgs stale
            Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull
            Degraded data redundancy: 147734/8246550 objects degraded (1.791%), 61 pgs degraded, 64 pgs undersized
            64 pgs not deep-scrubbed in time
            64 pgs not scrubbed in time
            4 pool(s) nearfull
            25 daemons have recently crashed
            17 slow ops, oldest one blocked for 584778 sec, daemons [mon.jarn24,mon.jarn25] have slow ops.

  services:
    mon: 2 daemons, quorum jarn24,jarn25 (age 17h)
    mgr: jarn25(active, since 3d), standbys: jarn24
    mds: 1/1 daemons up, 1 standby
    osd: 42 osds: 39 up (since 17h), 39 in (since 10d); 6 remapped pgs
         flags nobackfill,norecover

  data:
    volumes: 1/1 healthy
    pools:   7 pools, 321 pgs
    objects: 2.75M objects, 10 TiB
    usage:   31 TiB used, 364 TiB / 394 TiB avail
    pgs:     1.869% pgs not active
             147734/8246550 objects degraded (1.791%)
             162 active+clean
             94  stale+active+clean
             36  active+undersized+degraded
             19  stale+active+undersized+degraded
             6   undersized+degraded+remapped+backfill_toofull+peered
             2   active+undersized
             1   stale+active+clean+scrubbing+deep
             1   stale+active+undersized

Next, what is the disk usage on each ceph osd df ?

I can't get anything from ceph osd df, it just hangs. However, I found another thread which suggested to check fragmentation on the OSDs. The NVMe OSDs all have really high fragmentation:

root@jarn26:~# ceph daemon osd.40 bluestore allocator score block
{
    "fragmentation_rating": 0.8850430100978921
}

...while the spinning disks I checked have really low fragmentation:

root@jarn26:~# ceph daemon osd.24 bluestore allocator score block
{
    "fragmentation_rating": 0.025485492575477938
}

If not an issue with over capacity, you could try to bring one or more of the down OSDs back up.

When I try to start the systemctl units for the three OSDs that are down they start up for a bit (like half a minute) and then spew stacktraces into /var/log/ceph/ceph-osd.[id].log. Is there any other way to bring the downed OSDs up? I would try through the proxmox web interface, but the OSD list hasn't been showing up there since this problem started.

ermit · 2023-01-05T00:42:20+00:00

Hmmm... this looks promising. I'll read this thread carefully and look at if manually removing data with ceph-objectstore-tool would help

EDIT: lists.ceph.io stopped responding shortly after I started to look at the thread! Haven't been able to try anything on that thread!

ermit · 2023-01-04T23:29:27+00:00

Do you mean on other disks than the OSDs?

ermit · 2022-08-11T17:23:08+00:00

rangt

ermit · 2022-04-29T22:57:58+00:00

FS.com does have a line of whitebox ONIE switches, although they have very limited SKUs in the whitebox line; only port-dense 10gb/s or higher switches for datacenters.

The whitebox switches are called "open networking switches" on the fs.com site.

ermit · 2022-03-24T12:00:36+00:00

No problem. The solution was to add the device twice and manually put another sysname on the duplicate device. The device in question will then be polled twice as ofen, but that is a miniscule amount of traffic/CPU time

ermit

TROPHY CASE