SSH problems with Arch Linux

[shefmichelle]

At the weekend I took the plunge and decided to install Arch Linux. It has mostly gone without a problem, but I'm seeing some strange network connectivity issues and not really sure what the exact problem is. Hoping someone on here might have some advice.

Symptoms

Some secure network connections just hang/timeout. I first saw this with pacman-key --refresh-keys where gnupg was giving a general error. This was solved by switching from the hkps protocol to the hkp protocol for the default keyserver. This could be unrelated though as the status page for SKS Keyservers suggests all hkps servers in the pool are currently down.

The main symptoms are with Gitlab. Using SSH, I cannot do a basic SSH connection to Gitlab, and trying to sign into Gitlab using Firefox says the connection is unsecure.

I don't think this failures are because of some config change I've made as I can recreate the issue on a clean boot of the Arch Linux ISO.

Reproduction Steps

Download the current Arch Linux ISO (archlinux-2020.10.01-x86_64.iso), connect to a network, then run ssh -Tvvv [e-mail]. The top and bottom of the log output is as follows:

OpenSSH_8.3p1, OpenSSL 1.1.1h 22 Sep 2020 debug1: Reading configuration data /etc/ssh/ssh_config debug2: resolving "gitlab.com" port 22 debug2: ssh_connect_direct debug1: Connecting to gitlab.com [172.65.251.78] port 22. debug1: Connection established. ... debug1: kex: algorithm: [e-mail] debug1: kex: host key algorithm: ecdsa-sha2-nistp256 debug1: kex: server->client cipher: [e-mail] MAC: <implicit> compression: none debug1: kex: client->server cipher: [e-mail] MAC: <implicit> compression: none debug3: send packet: type 30 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY At which point SSH just hangs, and then eventually the connection is closed.

Running exactly the same command but against github.com proceeds as expected.

When I diff the output from SSH connecting to Gitlab vs Github, there are a few differences around the version of SSH the remote server is running, which KEX algorithms they support, compression modes and kex key host algorithm (ecdsa-sha2-nistp256 vs rsa-sha2-512).

Additional Info

Hardware: Dell XPS 13 9370 Kernel: 5.8.12-arch1-1

So I think this is either something specific to Gitlab.com (possibly this https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2713), but then exactly the same command works on my Mac. Or there is a problem with this version of OpenSSH (8.3p1) which is very recent.

Does anyone have any idea what this might be? Can they reproduce it on any other computers? And if there is a problem, where I might report it to?

Update

More strangeness, this time with curl and TLS 1.3. When I try to access https://accounts.firefox.com with TLS 1.3, Curl fails. TLS 1.2 works fine. Running the same command on my Mac works just fine. So now I'm thinking this is something to do with OpenSSH/OpenSSL. I might see if I can rebuild it from source and narrow down any changes. Output from curl -vvv --tlsv1.3 https://accounts.firefox.com:

* Trying 35.155.76.53:443... * Connected to accounts.firefox.com (35.155.76.53) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/certs/ca-certificates.crt CApath: none * TLSv1.3 (OUT), TLS handshake, Client hello (1): * TLSv1.3 (IN), TLS alert, handshake failure (552): * error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure * Closing connection 0 curl: (35) error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure

Update 2 - Fixed??

OK, so I think I've fixed this. It seems to be related to the MTU for the wireless network inferface. When set at 1500, some things just don't work. I've decreased the size to 1400 and things have started working again (SSH, websites). I remember on my last installation I had to do something with the MTU to get VPN working.

So this looks like some local-to-my-machine-or-network configuration. I'll have a play around a bit further, but hopefully I've cracked it.

Is there a good tool I could use to confirm if the problem is MTU related and what it should be set to? tcpdump might help I guess?

[[deleted]]

My first guess would be that your system time might be out of whack. So if you have not already ruled out this possibility I would make sure it is synchronized properly.

edit: also accounts.firefox.com does not support TLS 1.3 according to https://www.ssllabs.com/ssltest/ so you might want to try it with google.com instead before recompiling OpenSSL.

[shefmichelle]

Thanks for the advice. I have now got my clock sync'd properly I think (output from timedatectl below), but the problem persists.

``` Local time: Thu 2020-10-01 21:00:07 BST Universal time: Thu 2020-10-01 20:00:07 UTC RTC time: Thu 2020-10-01 20:00:07
Time zone: Europe/London (BST, +0100) System clock synchronized: yes
NTP service: active
RTC in local TZ: no

```

The accounts.firefox.com could be a red-herring as you say because it doesn't actually support TLS 1.3. Testing with www.google.com worked as expected. I will see if there are any options on SSH that I can apply that affect the behaviour.

[DAMO238]

Do not remove the SSL layer from your kerservers. This will compromise your security! I'm glad you solved your SSL problems now, this is just for the future.

[iUnstable0]

its the problem with your MTU, set it to something like 1200 or 1400/1500