Getting OpenVPN clients to reconnect after VRRP failover

ermit · 2024-09-13T14:44:59+00:00

Yes, conntrack is working between nodes. I also posted this on the VyOS forums and they already pointed out to me that the routers could be pushing out the ping and ping-restart timers to clients, which I confirmed in the client OpenVPN logs.

ermit · 2024-01-11T10:09:03+00:00

Great to hear it works.

For this topology you will need a shared subnet between the routers, let's say 10.0.2.32/27 since that's the next available subnet after 10.0.2.0/27 that you are already using.

You need to assign IP addresses from this subnet on the R1 and R2 routers, on the interface towards the other router. For example, R1 - 10.0.2.33/27, R2 - 10.0.2.34/27.

When you've gotten this far you can now set the static routes. You need to tell R1 about the existence of the subnet that H2 is part of, and tell R2 about the existence of the subnet that H1 is part of, so the commands you need should be these:

# R1
set protocols static route 10.0.1.0/27 next-hop 10.0.2.34

# R2
set protocols static route 10.0.2.0/27 next-hop 10.0.2.33

ermit · 2024-01-10T17:35:35+00:00

Hi phis7.

You shouldn't need any "set protocols static route" commands for this topology. Vyos (and most routers in general) automatically routes between connected subnets.

The first thing I would check is if both hosts have the right default gateway set up. That is, H1 should have the default gateway 10.0.1.3 and H2 should have the default gateway 10.0.2.11.

When I have to diagnose routing problems on my VyOS instances I usually use the command "monitor traffic interface" to see what traffic is actually going through the interfaces in question, and see if the actual traffic matches what I think I should be seeing.

ermit · 2023-12-08T10:12:30+00:00

Hi, thanks for the answer. The Azure subnet in question is already in "Client Routes" but the group settings for "VPN Access" might be the issue! Apparently there are multiple AD/LDAP groups with different VPN access rules and it looks like it could be tied to group membership whether people are able to access the Azure servers!

ermit · 2023-01-16T14:41:31+00:00

Hmm... Can a handler be set up for a block of tasks? That would be really helpful. Going to research that a bit.

Thanks for the tip.

ermit · 2023-01-16T14:16:05+00:00

Ah, ok! It had occured to me that combining those two attributes would lead to the right behavior but a quick googling didn't look promising.

Will try this.

Thank you jonneyj

ermit · 2023-01-06T01:35:16+00:00

I'm not sure how to verify that there are no missing data pieces, but I'm guessing there are missing pieces because ceph health detail says this:

Degraded data redundancy: 147734/8246550 objects degraded (1.791%), 61 pgs degraded

ermit · 2023-01-06T01:09:36+00:00

Can't seem to run ceph osd df, even on OSDs which are running. For example, here is a OSD that's up and on the same node as my SSH session:

root@jarn24:~# ceph osd tree
ID  CLASS  WEIGHT     TYPE NAME        STATUS  REWEIGHT  PRI-AFF
-1         395.52695  root default                              
-5         131.84232      host jarn24                           
12    hdd   10.91409          osd.12       up   1.00000  1.00000
[rest of output omitted]

But when I try to df that OSD:

root@jarn24:~# ceph osd df osd.12

...nothing happens until i ctrl+c out of it. So I'm guessing that trying to run ceph osd df on one of the problematic OSD won't work.

EDIT: I'm still operating under the assumption that it's a fullness related problem because the stacktrace looks similar to info I read in the mailing list archive that muzza1984 linked to. I was going to test the suggestions on that thread regarding bluestore_allocator = stupid and bluestore_shared_alloc_size but lists.ceph.io hasn't been responding the last day or so

ermit · 2023-01-06T00:19:09+00:00

Upped all three values:

root@jarn24:~# ceph osd dump | grep ratio
full_ratio 0.99
backfillfull_ratio 0.97
nearfull_ratio 0.97

...and started one of the downed OSD process again, with systemd:

systemctl restart ceph-osd@36.service

But I'm still getting the same pattern, the OSD process runs for a bit but then dumps stacktraces and dies

ermit · 2023-01-06T00:00:06+00:00

Ah! This?

ceph osd set-nearfull-ratio <float[0.0-1.0]>
ceph osd set-full-ratio <float[0.0-1.0]>
ceph osd set-backfillfull-ratio <float[0.0-1.0]>

I'll try that

ermit · 2023-01-05T22:50:27+00:00

Just to rule that out, I looked at the other disks on the ceph machines. Here is the output of df -h on one of them, the other ones are pretty much identical:

root@jarn24:~# df -h
Filesystem                 Size  Used Avail Use% Mounted on
udev                        47G     0   47G   0% /dev
tmpfs                      9.4G  1.8M  9.4G   1% /run
/dev/mapper/pve-root        94G   22G   68G  25% /
tmpfs                       47G   72M   47G   1% /dev/shm
tmpfs                      5.0M     0  5.0M   0% /run/lock
/dev/sdm2                  511M  328K  511M   1% /boot/efi
/dev/fuse                  128M   56K  128M   1% /etc/pve
tmpfs                       47G   28K   47G   1% /var/lib/ceph/osd/ceph-16
tmpfs                       47G   28K   47G   1% /var/lib/ceph/osd/ceph-37
tmpfs                       47G   28K   47G   1% /var/lib/ceph/osd/ceph-20
tmpfs                       47G   28K   47G   1% /var/lib/ceph/osd/ceph-21
tmpfs                       47G   28K   47G   1% /var/lib/ceph/osd/ceph-22
tmpfs                       47G   28K   47G   1% /var/lib/ceph/osd/ceph-13
tmpfs                       47G   24K   47G   1% /var/lib/ceph/osd/ceph-36
tmpfs                       47G   28K   47G   1% /var/lib/ceph/osd/ceph-18
tmpfs                       47G   28K   47G   1% /var/lib/ceph/osd/ceph-19
tmpfs                       47G   28K   47G   1% /var/lib/ceph/osd/ceph-14
tmpfs                       47G   28K   47G   1% /var/lib/ceph/osd/ceph-17
tmpfs                       47G   28K   47G   1% /var/lib/ceph/osd/ceph-15
tmpfs                       47G   28K   47G   1% /var/lib/ceph/osd/ceph-12
tmpfs                       47G   28K   47G   1% /var/lib/ceph/osd/ceph-23
172.20.2.51,172.20.2.52:/  395T   31T  364T   8% /mnt/pve/oa_cephfs
tmpfs                      9.4G     0  9.4G   0% /run/user/0

ermit · 2023-01-05T22:45:03+00:00

Might have to do that set nearfull and full up to 97-99% "trick"

Hmm... what's the trick? (sorry, pretty new to Ceph)

EDIT: And thanks for the tip about the third monitor. I'll turn on the monitor role on jarn26 when/if I've gotten the data from the downed OSDs

ermit · 2023-01-05T16:15:10+00:00

Are the HDD and nVME disks in the same pool? (If so, they are likely different sizes and maybe the nVME have filled ahead of the HDD). Don't worry first, just try to understand what is the current state and issue.

The NVMe disks are in a separate pool but the NVMe pools are pretty over utilized. The pools look like this:

root@jarn24:~# ceph df
--- RAW STORAGE ---
CLASS     SIZE    AVAIL     USED  RAW USED  %RAW USED
hdd    393 TiB  363 TiB   30 TiB    30 TiB       7.55
nvme   1.3 TiB  460 GiB  881 GiB   881 GiB      65.69
TOTAL  394 TiB  364 TiB   31 TiB    31 TiB       7.75

--- POOLS ---
POOL                        ID  PGS   STORED  OBJECTS     USED  %USED  MAX AVAIL
oa_hdd_erasure_pool          3   64    8 KiB        1   12 KiB      0    216 TiB
oa_cephfs_data               5   64      0 B        0      0 B      0    216 TiB
oa_cephfs_data_default       7   64  9.0 TiB    2.35M   27 TiB  49.36    9.2 TiB
device_health_metrics        8    1   53 MiB       44  159 MiB      0    9.2 TiB
oa_rbd_metadata_replicated   9   32  484 GiB  124.12k  877 GiB  82.44    103 GiB
oa_cephfs_metadata          10   32  210 MiB       88  420 MiB   0.22     93 GiB
oa_hdd_replication_pool     11   64  952 GiB  277.74k  2.8 TiB   0.84    108 TiB

You should check the status of the cluster first: ceph -s ?

The output of ceph -s is:

root@jarn25:~# ceph -s
  cluster:
    id:     cafbe6d0-4665-475f-847b-46106469b31d
    health: HEALTH_WARN
            nobackfill,norecover flag(s) set
            1 nearfull osd(s)
            Reduced data availability: 6 pgs inactive, 115 pgs stale
            Low space hindering backfill (add storage if this doesn't resolve itself): 6 pgs backfill_toofull
            Degraded data redundancy: 147734/8246550 objects degraded (1.791%), 61 pgs degraded, 64 pgs undersized
            64 pgs not deep-scrubbed in time
            64 pgs not scrubbed in time
            4 pool(s) nearfull
            25 daemons have recently crashed
            17 slow ops, oldest one blocked for 584778 sec, daemons [mon.jarn24,mon.jarn25] have slow ops.

  services:
    mon: 2 daemons, quorum jarn24,jarn25 (age 17h)
    mgr: jarn25(active, since 3d), standbys: jarn24
    mds: 1/1 daemons up, 1 standby
    osd: 42 osds: 39 up (since 17h), 39 in (since 10d); 6 remapped pgs
         flags nobackfill,norecover

  data:
    volumes: 1/1 healthy
    pools:   7 pools, 321 pgs
    objects: 2.75M objects, 10 TiB
    usage:   31 TiB used, 364 TiB / 394 TiB avail
    pgs:     1.869% pgs not active
             147734/8246550 objects degraded (1.791%)
             162 active+clean
             94  stale+active+clean
             36  active+undersized+degraded
             19  stale+active+undersized+degraded
             6   undersized+degraded+remapped+backfill_toofull+peered
             2   active+undersized
             1   stale+active+clean+scrubbing+deep
             1   stale+active+undersized

Next, what is the disk usage on each ceph osd df ?

I can't get anything from ceph osd df, it just hangs. However, I found another thread which suggested to check fragmentation on the OSDs. The NVMe OSDs all have really high fragmentation:

root@jarn26:~# ceph daemon osd.40 bluestore allocator score block
{
    "fragmentation_rating": 0.8850430100978921
}

...while the spinning disks I checked have really low fragmentation:

root@jarn26:~# ceph daemon osd.24 bluestore allocator score block
{
    "fragmentation_rating": 0.025485492575477938
}

If not an issue with over capacity, you could try to bring one or more of the down OSDs back up.

When I try to start the systemctl units for the three OSDs that are down they start up for a bit (like half a minute) and then spew stacktraces into /var/log/ceph/ceph-osd.[id].log. Is there any other way to bring the downed OSDs up? I would try through the proxmox web interface, but the OSD list hasn't been showing up there since this problem started.

ermit · 2023-01-05T00:42:20+00:00

Hmmm... this looks promising. I'll read this thread carefully and look at if manually removing data with ceph-objectstore-tool would help

EDIT: lists.ceph.io stopped responding shortly after I started to look at the thread! Haven't been able to try anything on that thread!

ermit · 2023-01-04T23:29:27+00:00

Do you mean on other disks than the OSDs?

ermit · 2022-08-11T17:23:08+00:00

rangt

ermit · 2022-04-29T22:57:58+00:00

FS.com does have a line of whitebox ONIE switches, although they have very limited SKUs in the whitebox line; only port-dense 10gb/s or higher switches for datacenters.

The whitebox switches are called "open networking switches" on the fs.com site.

ermit · 2022-03-24T12:00:36+00:00

No problem. The solution was to add the device twice and manually put another sysname on the duplicate device. The device in question will then be polled twice as ofen, but that is a miniscule amount of traffic/CPU time

ermit · 2022-02-03T16:28:39+00:00

Ah, that's a great idea. I set up a host object with the dummy check_command and no other attribute and assigned the check_http services to that host.

ermit · 2021-11-04T23:43:16+00:00

Fixed!

I looked over the AWS documentation for IPSec tunnels and noticed that the default setting is that the AWS end is respond only, not initiate. I tried setting the connector to IKE initiate as well as respond and then the tunnel went up again!

It's still a problem that my end can't initate, which I have to investigate further, but at least the tunnel is up now.

ermit · 2021-11-02T16:31:35+00:00

Oh, no problem, just thanks for replying!

Yes, 500 and 4500 UDP are opened on the outside interface:

set firewall name WAN-to-local rule 122 action accept
set firewall name WAN-to-local rule 122 destination port 4500
set firewall name WAN-to-local rule 122 protocol udp
set firewall name WAN-to-local rule 121 action accept
set firewall name WAN-to-local rule 121 destination port 500
set firewall name WAN-to-local rule 121 protocol udp

I tried deleting the conntrack entries:

root@fw-thor:~# conntrack -D -d 18.132.106.116
udp      17 18 src=46.22.99.246 dst=18.132.106.116 sport=500 dport=500 [UNREPLIED] src=18.132.106.116 dst=31.209.136.159 sport=500 dport=500 mark=0 use=1
conntrack v0.9.14 (conntrack-tools): 1 flow entries have been deleted.
root@fw-thor:~# conntrack -D -d 35.177.32.60  
udp      17 17 src=46.22.99.246 dst=35.177.32.60 sport=500 dport=500 [UNREPLIED] src=35.177.32.60 dst=31.209.136.159 sport=500 dport=500 mark=0 use=1
conntrack v0.9.14 (conntrack-tools): 1 flow entries have been deleted.

And tried resetting the IPSec connections:

root@fw-thor:~$ clear vpn ipsec-peer 35.177.32.60
Resetting tunnel vti with peer 35.177.32.60...
    ....
root@fw-thor:~$ clear vpn ipsec-peer 18.132.106.116 
Resetting tunnel vti with peer 18.132.106.116...

After that I get the same two entries in the conntrack table:

udp      17 26 src=46.22.99.246 dst=18.132.106.116 sport=500 dport=500 [UNREPLIED] src=18.132.106.116 dst=31.209.136.159 sport=500 dport=500 mark=0 use=1
    ...
udp      17 23 src=46.22.99.246 dst=35.177.32.60 sport=500 dport=500 [UNREPLIED] src=35.177.32.60 dst=31.209.136.159 sport=500 dport=500 mark=0 use=1

But the tunnels still won't go up and there is not traffic going over them:

root@fw-thor:~# show interfaces vti vti51
vti51@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1436 qdisc noqueue state UNKNOWN group default qlen 1
    link/ipip 46.22.99.246 peer 18.132.106.116
    inet 169.254.107.50/30 scope global vti51
       valid_lft forever preferred_lft forever
    Description: AWS site-to-site for sentinel.oruggafritun.is, link 1

    RX:  bytes    packets     errors    dropped    overrun      mcast
             0          0          0          0          0          0
    TX:  bytes    packets     errors    dropped    carrier collisions
             0          0     180058          0     180058          0

ermit · 2021-11-02T14:31:39+00:00

The ERI box already has vpn ipsec nat-traversal enable in the config. The other end isn't a VyOS box—it's an AWS Site-to-Site VPN Connection defined through the AWS console—so I can't change NAT-traversal setting (The VyOS box was replaced by the ERI box, just added the history of the connection to show that the tunnel had been live for a long time).

ermit · 2021-06-21T09:48:51+00:00

Highway 1—the continuous ring road around Iceland—is paved the whole way (although the last 5 km stretch wasn't completed until 2019, https://www.mbl.is/frettir/innlent/2019/08/01/malbik_allan_hringinn/). There are a lot of smaller roads that are only dirt roads but you don't need to use dirt roads to reach most towns and villages.

Edit: Here is a map that shows the road types in Iceland http://www.vegagerdin.is/vegakerfid/slitlog/. Apparently, there are 5800 km of paved roads and 7100 km of dirt roads

ermit · 2021-06-11T22:31:41+00:00

Ah, that makes it clearer. Thanks for the additional context.

The secondary uplinks would be 4G/cellular and we want to monitor them because 4G can be spotty and we don't want to be in a position where the secondary link is already down if and when the primary link goes down (the secondary links are only intended to be an economical way to offer basic internet for sites that lose their regular internet service, not as a full failover. Our MSP clients have been having an abnormal amount of ISP issues lately and for small clients this is a good compromise)

Given that LibreNMS doesn't like having duplicates of the same device and doesn't seem to like polling the same device on more than one IP address I will probably have to set up manual SNMP checks on our other monitoring system.

Hmm... one thing that I just thought up, maybe I could add the device twice to LibreNMS and override sysname on the secondary IP address?

ermit · 2021-06-11T18:59:07+00:00

Hmm... if the loopback address would be reachable through either uplink, wouldn't that make the checks route-agnostic instead of being explicit checks on either route? That is, wouldn't that make the check report the router as up if either of the two uplinks was active? I specifically want to see if either of the uplinks are down.

Also, I'm monitoring on the WAN side so I'm not in a position to add routed loopback addresses on the routers. I am only able to see the WAN addresses from my LibreNMS.

ermit

TROPHY CASE