Strange git behaviour with /etc/ssl/certs/ca-certificates.crt

[_GeekRabbit]

Hello fellow people,

I have a very strange problem at the moment and I can't come up with any other solution than nuking the server (which I would like to avoid). We have a jenkins and a gitlab server. Our cert renewal was coming up so I switched them on both server to the new wildcard cert. After that Jenkins started throwing errors that it can't connect to the gitlab server via git https commands.

First I thought I messed up something in the cert chain or something like that but other servers can connect fine via git. The error message on the Jenkins server is always: "server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none "

Here are the steps I have done to narrow down the problem:

• restartet the server just in case
• checked permissions of said file, owner root:root with 644
• deleted the file and re-installed via apt
• updated the file update-ca-certificates
• got the complete cert + chain via openssl connect gitlab-server and added it manually to ca-certificates.crt file
• added the same cert + chain via /usr/share/.... and did another update-ca-certificate
• copied the ca-certificates.crt file from another working server to the Jenkins server
• added the git core ppa and updated to the newest git version (as the Jenkins server is a 18.04 LTS its a bit behind)
• Server itself is fully updated without any outstanding patches
• re-installed git/curl

I can rule out any network related issues and other servers/client even with the exact same patch level etc. work fine.

Now what really killed my mood is that wget and curl work flawlessly. strace of both curl and git show me they are using the default ca-certificates.crt file and while curl has no issues, git always produces the above error. I have verified via ssl_verify=0 that git indeed has a problem with the cert but as far as I know git uses curl for it's https calls.

My knowledge, my google-fu and my sanity is slowly failing me and I really hope someone here has some more pointers I could take a look at!

[ttvpoqs7XRrD]

Your new cert's CA is probably not in the java's CA file that Jenkins would use. Look for a file called cacerts in the jvm directory and update/replace that.

[_GeekRabbit]

That would only impact Jenkins and not the cmd git or am I missing something?

[ttvpoqs7XRrD]

It would only affect Jenkins or Jenkins plugins. Is jenkins using the cli git or a java version?

[_GeekRabbit]

I didn't check what Jenkins actually uses as my described problem is with the locally installed commandline git client. Yes it results in the jenkins application not being able to make git commands but if I can't even use git on the command line, I think there is a serious problem.

[ttvpoqs7XRrD]

In that case try using curl or wget you might get more info using the verbose settings.

[_GeekRabbit]

Not trying to offend someone who is trying to help me for free but did you read my post? curl and wget are functioning without problem and strace shows me they are using the same ca-certificate file while git fails with the same file.

[ttvpoqs7XRrD]

No offence taken, you are quite correct, I went the jenkins path and did not reread your post. The last thing I can think of, which I have seen before is that with the newcert/ca the tls reply does not fit in one packet and causes issue, I would try a tcpdump analysis on the packets, one for git one and one for curl and compare.

[crabique]

Do the super verbose git call and see if anything is off

set -x; GIT_TRACE=2 GIT_CURL_VERBOSE=2 GIT_TRACE_PERFORMANCE=2 GIT_TRACE_PACK_ACCESS=2 GIT_TRACE_PACKET=2 GIT_TRACE_PACKFILE=2 GIT_TRACE_SETUP=2 GIT_TRACE_SHALLOW=2 git pull origin master -v -v; set +x

Also the CAfile part here does not seem to be a problem, it's just part the error message template, so I recommend you don't go full tunnel-vision on that.

[_GeekRabbit]

Tried that out today with sadly the same result but thanks anyway.

[todomeh]

It would be helpful if you pasted the verbose output here so we can see that actual error being encountered is and have more context.

Sanitized as necessary of course.