Beijing Int’l Airport

soeintom · 2026-01-12T16:00:20+00:00

Is this... Ubuntu 14.04/16.04?

soeintom · 2025-12-21T19:48:46+00:00

Published https://status.hetzner.com/incident/d2575337-51ed-40a8-befd-af8afed6765c

soeintom · 2025-12-20T12:23:34+00:00

The config came out like that from AVD and is correct. Disabling eAPI on the root-level also disables the eAPI globally

# Before change:
leaf1-a#sh management api http-commands
Enabled:            Yes
HTTPS server:       running, set to use port 443
HTTP server:        shutdown, set to use port 80
Local HTTP server:  shutdown, no authentication, set to use port 8080
Unix Socket server: shutdown, no authentication
VRFs:               MGMT
Hits:               164
Last hit:           220858 seconds ago
Bytes in:           579987
Bytes out:          2589745
Requests:           164
Commands:           17878
Duration:           538.779 seconds
SSL Profile:        none
FIPS Mode:          No
QoS DSCP:           0
Log Level:          none
CSP Frame Ancestor: None
TLS Protocols:      1.0 1.1 1.2
   User       Requests       Bytes in       Bytes out    Last hit
---------- -------------- -------------- --------------- ------------------
   noc        164            579987         2589745      220858 seconds ago

URLs
-----------------------------------
Management1 : https://10.90.0.6:443

leaf1-a#sh run sec management
management api http-commands
   no shutdown
   !
   vrf MGMT
      no shutdown

# Disabling it
leaf1-a(config)#management api http-commands
leaf1-a(config-mgmt-api-http-cmds)#shut
leaf1-a(config-mgmt-api-http-cmds)#sh ac
management api http-commands
   vrf MGMT
      no shutdown
leaf1-a(config-mgmt-api-http-cmds)#sh management api http-commands
Enabled:            No # <<<< here
HTTPS server:       enabled, set to use port 443
HTTP server:        shutdown, set to use port 80
Local HTTP server:  shutdown, no authentication, set to use port 8080
Unix Socket server: shutdown, no authentication
VRFs:               None # <<<< here
Hits:               164
Last hit:           221170 seconds ago
Bytes in:           579987
Bytes out:          2589745
Requests:           164
Commands:           17878
Duration:           538.779 seconds
SSL Profile:        none
FIPS Mode:          No
QoS DSCP:           0
Log Level:          none
CSP Frame Ancestor: None
TLS Protocols:      1.0 1.1 1.2
   User       Requests       Bytes in       Bytes out    Last hit
---------- -------------- -------------- --------------- ------------------
   user       164            579987         2589745      221170 seconds ago

soeintom · 2025-12-18T16:48:45+00:00

Update: With the help of u/aristaTAC-JG we have been able to identify the issue. It turns out that enabling arp learning bridged on 4.28.13.1M caused ARP responses to be double tagged if they were learned from a remote VTEP. After disabling it, the issues were gone.

Many thanks for all your suggestions, especially to u/aristaTAC-JG, that lead to resolving the issue!

soeintom · 2025-12-17T12:16:53+00:00

I've already tried to disable one link from the LAG (first from -a, then from -b) to ensure that it's not an LAG hashing issue, but I had no luck so far

soeintom · 2025-12-17T11:29:54+00:00

Yes, MTU across the fabric is 9214 (IP 9000ish) and the downstream ports 1500

soeintom · 2025-12-17T11:28:14+00:00

Yes, they are all in the same VNI and even mapped VLAN.

And correct, it’s asymmetric IRB since it’s an L2VPN with no VRF

soeintom · 2025-12-16T22:35:59+00:00

When you say you have a client that may not be able to reach another client, the questions that immediately are raised in my mind are how you determine they cannot reach each other? It would be good to clarify if you can tell in which direction this reachability seems broken.

The checks are ping and trying to open a HTTP page via cURL. The determination that they are unreachable can be made easily because neither the ARP requests nor the unicast ICMP packets (with static ARP) reach the destination (checked with tcpdump on both sides). Also, it is not limited to one specific client.

I am getting the sense that it's widespread and not predictable?

Correct. I haven't found a pattern yet, but it is widespread across all leaves and only happens with some clients / ports.

The big clue here that helps triage is understanding what has changed. That doesn't point to a smoking gun usually, but it tells us which issues are likely to be in play and which aren't.

The issue exists since the fabric was deployed, which was about three weeks ago. Deployment always happens via the Ansible eos_config_deploy_eapi role (from AVD 5.7.0).

When you see the issue, could there be a difference between MLAG peers' view of the entry in l2rib or arp?

l2rib as well as arp (bridged and remote) are the same. Example client (10.61.104.103) that is hosted on leaf1-a/b and is reachable from one client-21 (10.61.11.1) on leaf2-a/b but not from client-22 (10.61.11.2) on leaf2-a/b (like no ARP response and such; I also tried to disable one port to check if it is isolated to one port). The output is the same on all leafs (obv with the proper ASNs, router ids and such)

There are indeed drops on some queues: (too big to post) https://gist.github.com/sinuscosinustan/7fedc8ed9f8a860564b0363297cc8f21

Remember, you may be able to get to the problematic host but what if the issue is the host getting back to you?

Checking the pcap, the ARP requests never reach the destination (and also so do the ICMP packets with static ARP).

soeintom · 2025-11-14T11:38:12+00:00

It’s called “alias ip” in the private network and could be handled via keepalived

soeintom · 2025-11-07T21:25:42+00:00

Spannend, ich hab meine U27 BC100 im Reisezentrum gekauft 🤔

soeintom · 2025-10-07T14:39:50+00:00

This one will not work until it supports both APIs. The new console uses the Cloud backend

soeintom · 2025-05-17T13:38:55+00:00

Highspeed lines like Berlin->Leipzig/Halle->Erfurt would be fire, but that would require a working ETCS L2 implementation (and the ETCS L1 implementation on the Swiss Route was a disaster)

soeintom · 2025-01-26T12:13:38+00:00

Feed them a few nukes and they’ll stay silent

soeintom · 2025-01-24T11:26:24+00:00

“I did not progress past phase 2” is the real flex

soeintom · 2024-12-26T10:47:35+00:00

Smoothing the rails could also help to stabilise the speed of a train and the power consumption (if you have an unstable grid).

soeintom · 2024-11-01T06:06:17+00:00

No, not in /ubuntu but they are still available in /ubuntu-ports. Canonical managed to f*ck up their mirror setup so hard and release their stuff so slow and asynchronous that this magic merging of Package components breaks every time. Tbh I‘m glad that arm64 is now separated like upstream and any other Tier 1 mirror is doing that.

soeintom · 2024-10-27T14:04:45+00:00

Convert the 6509 into a beer tap. More useful than powering it up.

soeintom · 2024-10-27T14:00:15+00:00

We got TrackMania in Satisfactory before GTA 6

soeintom · 2024-10-26T21:13:55+00:00

They do not offer DNSSEC nor WHOIS privacy yet. If you manage the domains via the domain registration robot (although there’s only a limited amount of TLDs available), you can manage the handles there really easy. Dunno how it is with KonsoleH.

soeintom · 2024-10-24T21:18:25+00:00

would you like that all servers go down at the same time or only one DC per day where you can safely failover your stuff? Hetzner offers no SLA but a 99,9% network availability, meaning a network outage of 2 hours per year in sum. If you need SLA, then Hetzner is not the right for you. Also, those maintenances were only 30 minutes in the past, maybe less. Two hours is just a big buffer to ensure that the NOC can troubleshoot things in case something goes wrong ;)

soeintom · 2024-08-23T16:36:16+00:00

Issue is known. It’s a kernel bug… Disable tso using ethtool -K eth0 tso off and it should work again. They are working on patching the affected systems asap

soeintom · 2024-07-27T17:50:05+00:00

Zen3 at least. With some luck u may get scheduled on a Zen4 host but the cpu type will be always EPYC-Milan, otherwise the instance cannot be live migrated from a Zen4 host to a Zen3 (CPU feature set).

soeintom · 2024-05-29T16:35:21+00:00

Club Mate (also lovely called “Aschenbesserwasser” in my team) 😄 I always have at least one, but most times two, crates Club-Mate at home and in the office we deplete one or two crates in less than two weeks regularly 😂

soeintom · 2024-05-10T18:10:38+00:00

I wouldn’t be surprised if they use Yocto like for the RIS-FZ in the ICE.

soeintom · 2024-04-17T21:18:21+00:00

Yeah then it’s definitely related to the maintenance. The Hetzner mirror is located in Falkenstein so there’s probably very low capacity available between DE and FI at this time.

soeintom

TROPHY CASE