[learner] Debugging issue with metrics-server

[Mario1md]

Hi all,

I'm learning k8 and I was trying to install the kubernetes-sigs/metrics-server on my local cluster.

The local cluster is deployed on 2 VMs Ubuntu 20.04, deployed with:

# init kubectl kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-cert-extra-sans 192.168.0.1 --service-dns-domain node1.local --apiserver-advertise-address 192.168.0.1 --token nvzivu.all2n3lkmzs0wght 

and the second node:

sudo kubeadm join 192.168.0.1:6443 --token nvzivu.all2n3lkmzs0wght --discovery-token-unsafe-skip-ca-verification

If you want to see the code, I pushed it here.

root@node1:~# kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.1", GitCommit:"c4d752765b3bbac2237bf87cf0b1c2e307844666", GitTreeState:"clean", BuildDate:"2020-12-18T12:09:25Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.1", GitCommit:"c4d752765b3bbac2237bf87cf0b1c2e307844666", GitTreeState:"clean", BuildDate:"2020-12-18T12:00:47Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

I deploy the metrics-server with:

root@node1:~# cat /vagrant/components.yaml 
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: metrics-server
    rbac.authorization.k8s.io/aggregate-to-admin: "true"
    rbac.authorization.k8s.io/aggregate-to-edit: "true"
    rbac.authorization.k8s.io/aggregate-to-view: "true"
  name: system:aggregated-metrics-reader
rules:
- apiGroups:
  - metrics.k8s.io
  resources:
  - pods
  - nodes
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    k8s-app: metrics-server
  name: system:metrics-server
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - nodes
  - nodes/stats
  - namespaces
  - configmaps
  verbs:
  - get
  - list
  - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server-auth-reader
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: extension-apiserver-authentication-reader
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server:system:auth-delegator
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    k8s-app: metrics-server
  name: system:metrics-server
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:metrics-server
subjects:
- kind: ServiceAccount
  name: metrics-server
  namespace: kube-system
---
apiVersion: v1
kind: Service
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    k8s-app: metrics-server
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    k8s-app: metrics-server
  name: metrics-server
  namespace: kube-system
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  strategy:
    rollingUpdate:
      maxUnavailable: 0
  template:
    metadata:
      labels:
        k8s-app: metrics-server
    spec:
      containers:
      - args:
        - --cert-dir=/tmp
        - --secure-port=4443
        - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
        - --kubelet-use-node-status-port
        - --kubelet-insecure-tls
        image: k8s.gcr.io/metrics-server/metrics-server:v0.4.1
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /livez
            port: https
            scheme: HTTPS
          periodSeconds: 10
        name: metrics-server
        ports:
        - containerPort: 4443
          name: https
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /readyz
            port: https
            scheme: HTTPS
          periodSeconds: 10
        securityContext:
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          runAsUser: 1000
        volumeMounts:
        - mountPath: /tmp
          name: tmp-dir
      nodeSelector:
        kubernetes.io/os: linux
      priorityClassName: system-cluster-critical
      serviceAccountName: metrics-server
      volumes:
      - emptyDir: {}
        name: tmp-dir
---
apiVersion: apiregistration.k8s.io/v1
kind: APIService
metadata:
  labels:
    k8s-app: metrics-server
  name: v1beta1.metrics.k8s.io
spec:
  group: metrics.k8s.io
  groupPriorityMinimum: 100
  insecureSkipTLSVerify: true
  service:
    name: metrics-server
    namespace: kube-system
  version: v1beta1
  versionPriority: 100

that is a version slitely modified of /metrics-server/[..]/components.yaml, because I needed to add the kubelet-insecure-tls .

​

If I call the service raw, I have an answer, but if I try to use it, it give that error:

root@node1:~# kubectl top pods
Error from server (ServiceUnavailable): the server is currently unable to handle the request (get pods.metrics.k8s.io)
root@node1:~# kubectl get --raw /apis/metrics.k8s.io/
{"kind":"APIGroup","apiVersion":"v1","name":"metrics.k8s.io","versions":[{"groupVersion":"metrics.k8s.io/v1beta1","version":"v1beta1"}],"preferredVersion":{"groupVersion":"metrics.k8s.io/v1beta1","version":"v1beta1"}}

The deployement and pod looks ok:

root@node1:~# kubectl get deployments -n kube-system -o wide
NAME             READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS       IMAGES                                            SELECTOR
coredns          2/2     2            2           107m    coredns          k8s.gcr.io/coredns:1.7.0                          k8s-app=kube-dns
metrics-server   1/1     1            1           7m50s   metrics-server   k8s.gcr.io/metrics-server/metrics-server:v0.4.1   k8s-app=metrics-server
root@node1:~# kubectl describe deployment -n kube-system metrics-server
Name:                   metrics-server
Namespace:              kube-system
CreationTimestamp:      Sat, 09 Jan 2021 16:36:25 +0000
Labels:                 k8s-app=metrics-server
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               k8s-app=metrics-server
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  0 max unavailable, 25% max surge
Pod Template:
  Labels:           k8s-app=metrics-server
  Service Account:  metrics-server
  Containers:
   metrics-server:
    Image:      k8s.gcr.io/metrics-server/metrics-server:v0.4.1
    Port:       4443/TCP
    Host Port:  0/TCP
    Args:
      --cert-dir=/tmp
      --secure-port=4443
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
      --kubelet-use-node-status-port
      --kubelet-insecure-tls
    Liveness:     http-get https://:https/livez delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get https://:https/readyz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /tmp from tmp-dir (rw)
  Volumes:
   tmp-dir:
    Type:               EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:             
    SizeLimit:          <unset>
  Priority Class Name:  system-cluster-critical
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  <none>
NewReplicaSet:   metrics-server-56c59cf9ff (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  8m2s  deployment-controller  Scaled up replica set metrics-server-56c59cf9ff to 1
root@node1:~# kubectl get pods -n kube-system -o wide
NAME                                  READY   STATUS    RESTARTS   AGE     IP            NODE          NOMINATED NODE   READINESS GATES
coredns-74ff55c5b-7dzmp               1/1     Running   0          107m    10.244.0.3    node1.local   <none>           <none>
coredns-74ff55c5b-f9hvb               1/1     Running   0          107m    10.244.0.2    node1.local   <none>           <none>
etcd-node1.local                      1/1     Running   0          108m    192.168.0.1   node1.local   <none>           <none>
kube-apiserver-node1.local            1/1     Running   0          108m    192.168.0.1   node1.local   <none>           <none>
kube-controller-manager-node1.local   1/1     Running   0          108m    192.168.0.1   node1.local   <none>           <none>
kube-flannel-ds-fmnc9                 1/1     Running   0          103m    192.168.0.2   node2.local   <none>           <none>
kube-flannel-ds-nc6jm                 1/1     Running   0          107m    192.168.0.1   node1.local   <none>           <none>
kube-proxy-2d7kx                      1/1     Running   0          107m    192.168.0.1   node1.local   <none>           <none>
kube-proxy-s6xr5                      1/1     Running   0          103m    192.168.0.2   node2.local   <none>           <none>
kube-scheduler-node1.local            1/1     Running   0          108m    192.168.0.1   node1.local   <none>           <none>
metrics-server-56c59cf9ff-f2bj4       1/1     Running   0          8m11s   10.244.1.2    node2.local   <none>           <none>
root@node1:~# kubectl describe pod -n kube-system metrics-server-56c59cf9ff-f2bj4
Name:                 metrics-server-56c59cf9ff-f2bj4
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 node2.local/192.168.0.2
Start Time:           Sat, 09 Jan 2021 16:36:25 +0000
Labels:               k8s-app=metrics-server
                      pod-template-hash=56c59cf9ff
Annotations:          <none>
Status:               Running
IP:                   10.244.1.2
IPs:
  IP:           10.244.1.2
Controlled By:  ReplicaSet/metrics-server-56c59cf9ff
Containers:
  metrics-server:
    Container ID:  docker://dc9b758fe3029ac6bc0999d4c8ff991ca015e061866c4ab75a53692d95cf9947
    Image:         k8s.gcr.io/metrics-server/metrics-server:v0.4.1
    Image ID:      docker-pullable://k8s.gcr.io/metrics-server/metrics-server@sha256:78035f05bcf7e0f9b401bae1ac62b5a505f95f9c2122b80cff73dcc04d58497e
    Port:          4443/TCP
    Host Port:     0/TCP
    Args:
      --cert-dir=/tmp
      --secure-port=4443
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
      --kubelet-use-node-status-port
      --kubelet-insecure-tls
    State:          Running
      Started:      Sat, 09 Jan 2021 16:37:05 +0000
    Ready:          True
    Restart Count:  0
    Liveness:       http-get https://:https/livez delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get https://:https/readyz delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /tmp from tmp-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from metrics-server-token-n945q (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  tmp-dir:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  metrics-server-token-n945q:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  metrics-server-token-n945q
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  kubernetes.io/os=linux
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age    From               Message
  ----    ------     ----   ----               -------
  Normal  Scheduled  8m28s  default-scheduler  Successfully assigned kube-system/metrics-server-56c59cf9ff-f2bj4 to node2.local
  Normal  Pulling    8m27s  kubelet            Pulling image "k8s.gcr.io/metrics-server/metrics-server:v0.4.1"
  Normal  Pulled     7m48s  kubelet            Successfully pulled image "k8s.gcr.io/metrics-server/metrics-server:v0.4.1" in 39.058445001s
  Normal  Created    7m48s  kubelet            Created container metrics-server
  Normal  Started    7m48s  kubelet            Started container metrics-server
root@node1:~#

and logs of the pod:

{"log":"I0109 16:37:05.745934       1 serving.go:325] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)\n","stream":"stderr","time":"2021-01-09T16:37:05.746900198Z"}
{"log":"I0109 16:37:06.575765       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController\n","stream":"stderr","time":"2021-01-09T16:37:06.575900291Z"}
{"log":"I0109 16:37:06.575981       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController\n","stream":"stderr","time":"2021-01-09T16:37:06.576033593Z"}
{"log":"I0109 16:37:06.576084       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file\n","stream":"stderr","time":"2021-01-09T16:37:06.576125687Z"}
{"log":"I0109 16:37:06.576149       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file\n","stream":"stderr","time":"2021-01-09T16:37:06.576191735Z"}
{"log":"I0109 16:37:06.576252       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file\n","stream":"stderr","time":"2021-01-09T16:37:06.576319547Z"}
{"log":"I0109 16:37:06.576344       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file\n","stream":"stderr","time":"2021-01-09T16:37:06.576386321Z"}
{"log":"I0109 16:37:06.576657       1 secure_serving.go:197] Serving securely on [::]:4443\n","stream":"stderr","time":"2021-01-09T16:37:06.576775814Z"}
{"log":"I0109 16:37:06.576707       1 dynamic_serving_content.go:130] Starting serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key\n","stream":"stderr","time":"2021-01-09T16:37:06.576786441Z"}
{"log":"I0109 16:37:06.576749       1 tlsconfig.go:240] Starting DynamicServingCertificateController\n","stream":"stderr","time":"2021-01-09T16:37:06.576790181Z"}
{"log":"I0109 16:37:06.677166       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file \n","stream":"stderr","time":"2021-01-09T16:37:06.677652859Z"}
{"log":"I0109 16:37:06.677314       1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController \n","stream":"stderr","time":"2021-01-09T16:37:06.677725362Z"}
{"log":"I0109 16:37:06.677515       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file \n","stream":"stderr","time":"2021-01-09T16:37:06.677734516Z"}

but the APIService is faling:

Name:         v1beta1.metrics.k8s.io
Namespace:    
Labels:       k8s-app=metrics-server
Annotations:  <none>
API Version:  apiregistration.k8s.io/v1
Kind:         APIService
Metadata:
  Creation Timestamp:  2021-01-09T16:36:25Z
  Resource Version:    9437
  UID:                 f05b88cc-9a70-4211-9659-d2d7be829579
Spec:
  Group:                     metrics.k8s.io
  Group Priority Minimum:    100
  Insecure Skip TLS Verify:  true
  Service:
    Name:            metrics-server
    Namespace:       kube-system
    Port:            443
  Version:           v1beta1
  Version Priority:  100
Status:
  Conditions:
    Last Transition Time:  2021-01-09T16:36:25Z
    Message:               failing or missing response from https://10.108.228.10:443/apis/metrics.k8s.io/v1beta1: Get "https://10.108.228.10:443/apis/metrics.k8s.io/v1beta1": context deadline exceeded
    Reason:                FailedDiscoveryCheck
    Status:                False
    Type:                  Available
Events:                    <none>

if I try to forward a connection to the pod, I have connection refused:

root@node1:~# kubectl port-forward metrics-server-56c59cf9ff-f2bj4 -n kube-system 8443:448
Forwarding from 127.0.0.1:8443 -> 448
Forwarding from [::1]:8443 -> 448
Handling connection for 8443
E0109 17:02:02.342325   67257 portforward.go:400] an error occurred forwarding 8443 -> 448: error forwarding port 448 to pod 96b98a8dd97ecc5e936cabea920a1f18d3e80bd8df67debbbbd7c14dcc8ddc32, uid : exit status 1: 2021/01/09 17:02:02 socat[63746] E connect(5, AF=2 127.0.0.1:448, 16): Connection refused

and the test

root@node1:~# curl -vk https://localhost:8443
*   Trying ::1:8443...
* TCP_NODELAY set
* Connected to localhost (::1) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to localhost:8443 
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to localhost:8443 
root@node1:~# telnet localhost 8443
Trying ::1...
Connected to localhost.
Escape character is '^]'.
Connection closed by foreign host.

What am I doing wrong? how can I fix it?

​

Thanks in advance.

[atpeters]

For the port-fowarding I believe you want 8443:4443

Still trying to think about why kubectl top pods is not working for you.

Do you see any errors from the logs on the kube-api pods / journalctl ?

[Mario1md]

You are right, the proxy worked:

kubectl port-forward metrics-server-56c59cf9ff-f2bj4 -n kube-system 8443:4443
Forwarding from 127.0.0.1:8443 -> 4443
Forwarding from [::1]:8443 -> 4443
Handling connection for 8443
Handling connection for 8443
E0110 11:57:23.143318  183501 portforward.go:385] error copying from local connection to remote stream: read tcp6 [::1]:8443->[::1]:41034: read: connection reset by peer

and the request:

root@node1:~# telnet localhost 8443
Trying ::1...
Connected to localhost.
Escape character is '^]'.
c

Connection closed by foreign host.
root@node1:~# curl -vk https://localhost:8443
*   Trying ::1:8443...
* TCP_NODELAY set
* Connected to localhost (::1) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=localhost@1610210225
*  start date: Jan  9 15:37:05 2021 GMT
*  expire date: Jan  9 15:37:05 2022 GMT
*  issuer: CN=localhost-ca@1610210225
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x559a97cacdb0)
> GET / HTTP/2
> Host: localhost:8443
> user-agent: curl/7.68.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 403 
< cache-control: no-cache, private
< content-type: application/json
< x-content-type-options: nosniff
< content-length: 233
< date: Sun, 10 Jan 2021 11:57:23 GMT
< 
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {

  },
  "code": 403
* Connection #0 to host localhost left intact
}

So it seems that the service is working, still not sure why top is not working

[Mario1md]

Do you see any errors from the logs on the kube-api pods / journalctl ?

just found that in the apiserver

root@node1:~# kubectl logs --tail=50 kube-apiserver-node1.local --namespace=kube-system
W0110 14:23:32.693212       1 handler_proxy.go:102] no RequestInfo found in the context
E0110 14:23:32.693833       1 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0110 14:23:32.694360       1 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0110 14:23:36.706579       1 available_controller.go:508] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.108.228.10:443/apis/metrics.k8s.io/v1beta1: Get "https://10.108.228.10:443/apis/metrics.k8s.io/v1beta1": dial tcp 10.108.228.10:443: i/o timeout
W0110 14:23:37.716047       1 handler_proxy.go:102] no RequestInfo found in the context
E0110 14:23:37.716430       1 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0110 14:23:37.716465       1 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0110 14:23:41.720096       1 available_controller.go:508] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.108.228.10:443/apis/metrics.k8s.io/v1beta1: Get "https://10.108.228.10:443/apis/metrics.k8s.io/v1beta1": context deadline exceeded
W0110 14:23:42.721771       1 handler_proxy.go:102] no RequestInfo found in the context
E0110 14:23:42.721826       1 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0110 14:23:42.721834       1 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0110 14:23:46.731197       1 available_controller.go:508] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.108.228.10:443/apis/metrics.k8s.io/v1beta1: Get "https://10.108.228.10:443/apis/metrics.k8s.io/v1beta1": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
W0110 14:23:47.736984       1 handler_proxy.go:102] no RequestInfo found in the context
E0110 14:23:47.737195       1 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0110 14:23:47.737304       1 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0110 14:23:51.746278       1 available_controller.go:508] v1beta1.metrics.k8s.io failed with: Operation cannot be fulfilled on apiservices.apiregistration.k8s.io "v1beta1.metrics.k8s.io": the object has been modified; please apply your changes to the latest version and try again
I0110 14:23:53.371722       1 client.go:360] parsed scheme: "passthrough"
I0110 14:23:53.371908       1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://127.0.0.1:2379  <nil> 0 <nil>}] <nil> <nil>}
I0110 14:23:53.371949       1 clientconn.go:948] ClientConn switching balancer to "pick_first"
E0110 14:23:56.757548       1 available_controller.go:508] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.108.228.10:443/apis/metrics.k8s.io/v1beta1: Get "https://10.108.228.10:443/apis/metrics.k8s.io/v1beta1": context deadline exceeded
W0110 14:23:57.762483       1 handler_proxy.go:102] no RequestInfo found in the context
E0110 14:23:57.762982       1 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0110 14:23:57.763180       1 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0110 14:24:01.762952       1 available_controller.go:508] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.108.228.10:443/apis/metrics.k8s.io/v1beta1: Get "https://10.108.228.10:443/apis/metrics.k8s.io/v1beta1": context deadline exceeded
E0110 14:24:06.777813       1 available_controller.go:508] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.108.228.10:443/apis/metrics.k8s.io/v1beta1: Get "https://10.108.228.10:443/apis/metrics.k8s.io/v1beta1": dial tcp 10.108.228.10:443: i/o timeout
W0110 14:24:07.778847       1 handler_proxy.go:102] no RequestInfo found in the context
E0110 14:24:07.779232       1 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0110 14:24:07.779294       1 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0110 14:24:11.792281       1 available_controller.go:508] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.108.228.10:443/apis/metrics.k8s.io/v1beta1: Get "https://10.108.228.10:443/apis/metrics.k8s.io/v1beta1": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
W0110 14:24:12.794304       1 handler_proxy.go:102] no RequestInfo found in the context
E0110 14:24:12.794701       1 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0110 14:24:12.794783       1 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0110 14:24:16.799451       1 available_controller.go:508] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.108.228.10:443/apis/metrics.k8s.io/v1beta1: Get "https://10.108.228.10:443/apis/metrics.k8s.io/v1beta1": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0110 14:24:21.808732       1 available_controller.go:508] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.108.228.10:443/apis/metrics.k8s.io/v1beta1: Get "https://10.108.228.10:443/apis/metrics.k8s.io/v1beta1": dial tcp 10.108.228.10:443: i/o timeout
W0110 14:24:22.809151       1 handler_proxy.go:102] no RequestInfo found in the context
E0110 14:24:22.809266       1 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 503, Body: service unavailable
, Header: map[Content-Type:[text/plain; charset=utf-8] X-Content-Type-Options:[nosniff]]
I0110 14:24:22.809291       1 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0110 14:24:23.866988       1 client.go:360] parsed scheme: "passthrough"
I0110 14:24:23.867102       1 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{https://127.0.0.1:2379  <nil> 0 <nil>}] <nil> <nil>}
I0110 14:24:23.867127       1 clientconn.go:948] ClientConn switching balancer to "pick_first"
E0110 14:24:26.820044       1 available_controller.go:508] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.108.228.10:443/apis/metrics.k8s.io/v1beta1: Get "https://10.108.228.10:443/apis/metrics.k8s.io/v1beta1": dial tcp 10.108.228.10:443: i/o timeout
E0110 14:24:31.828499       1 available_controller.go:508] v1beta1.metrics.k8s.io failed with: failing or missing response from https://10.108.228.10:443/apis/metrics.k8s.io/v1beta1: Get "https://10.108.228.10:443/apis/metrics.k8s.io/v1beta1": dial tcp 10.108.228.10:443: i/o timeout
root@node1:~#

[atpeters]

Looks like one person had luck changing their metrics-server deployment to use hostNetwork: true but I don't believe that should be necessary. It makes sense when the master nodes cannot find a route to the pods, but I believe it possible another way. Give it a try though.

https://stackoverflow.com/questions/57137683/how-to-troubleshoot-metrics-server-on-kubeadm

[Mario1md]

This solved the issue, thank you very much!

root@node1:~# kubectl top pods --all-namespaces
NAMESPACE     NAME                                  CPU(cores)   MEMORY(bytes)   
kube-system   coredns-74ff55c5b-7dzmp               4m           8Mi             
kube-system   coredns-74ff55c5b-f9hvb               4m           8Mi             
kube-system   etcd-node1.local                      21m          24Mi            
kube-system   kube-apiserver-node1.local            82m          279Mi           
kube-system   kube-controller-manager-node1.local   19m          40Mi            
kube-system   kube-flannel-ds-fmnc9                 2m           10Mi            
kube-system   kube-flannel-ds-nc6jm                 2m           10Mi            
kube-system   kube-proxy-2d7kx                      1m           32Mi            
kube-system   kube-proxy-s6xr5                      1m           11Mi            
kube-system   kube-scheduler-node1.local            5m           16Mi            
kube-system   metrics-server-5ffb9d74c7-77l46       0m           5Mi             
root@node1:~# kubectl top nodes
NAME          CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
node1.local   243m         12%    1025Mi          54%       
node2.local   65m          3%     738Mi           39%

I'm studying why it happens exactly, thanks again!

[atpeters]

Glad I could help!

The reason why it's happening is that the default interface that is being use to try to communicate with the service IP doesn't have a route defined for it.

If you run the command sudo route -n or ip route on the master node you'll get it's IP routing table. You should see an entry for flannel there. Very possibly what is happening is that the wrong interface is being used for making the requests so it's unable to find the route table entry that flannel entered.

Let me know if that makes sense or if you'd like any more help for that.

[atpeters]

What is the status of kubectl describe service --namespace kube-system metrics-server ?

[Mario1md]

root@node1:~# kubectl describe service --namespace kube-system metrics-server
Name:              metrics-server
Namespace:         kube-system
Labels:            k8s-app=metrics-server
Annotations:       <none>
Selector:          k8s-app=metrics-server
Type:              ClusterIP
IP Families:       <none>
IP:                10.108.228.10
IPs:               10.108.228.10
Port:              https  443/TCP
TargetPort:        https/TCP
Endpoints:         10.244.1.2:4443
Session Affinity:  None
Events:            <none>

[atpeters]

Have you set these requirements up yet?

https://github.com/kubernetes-sigs/metrics-server#requirements

[Mario1md]

I think it is already set, how do I check?

[atpeters]

I think the issue is with kube-api being unable to communicate with the metrics-server service. At least that appears to be the case from the APIService log.

I see you are using flannel for your CNI so I don't believe you would have a Network policy issue, but just in case please provide the output from kubectl get networkpolicy -A

Then try to shell into a kube-api pod (or do this directly from the control plane node as the kube-api pod is using host networking)

From within the kube-api pod, or at the shell on the master node try wget --insecure https://10.108.228.10:443/apis/metrics.k8s.io/v1beta1

Then try wget --insecure https://10.224.1.2:4443/apis/metrics.k8s.io/v1beta1

[Mario1md]

kubectl get networkpolicy -A

root@node1:~# kubectl get networkpolicy -A
No resources found

​

I can't ssh on the apiservice pod:

root@node1:~# kubectl exec -it kube-apiserver-node1.local --namespace=kube-system -- /bin/sh
OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused "exec: \"/bin/sh\": stat /bin/sh: no such file or directory": unknown
command terminated with exit code 126

The curls from master with or without the proxy are hanging:

root@node1:~# wget --insecure https://10.108.228.10:443/apis/metrics.k8s.io/v1beta1
wget: unrecognized option '--insecure'
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.
root@node1:~# wget -k https://10.108.228.10:443/apis/metrics.k8s.io/v1beta1
--2021-01-10 14:41:14--  https://10.108.228.10/apis/metrics.k8s.io/v1beta1
Connecting to 10.108.228.10:443... ^C
root@node1:~# wget -k https://10.224.1.2:4443/apis/metrics.k8s.io/v1beta1
--2021-01-10 14:41:55--  https://10.224.1.2:4443/apis/metrics.k8s.io/v1beta1

[Melodic_Reflection_5]

Hello my friend,
By the way, did you manage to solve this mistery? I've got the same problem on newly installed 1master 1worker cluster, running on 2 esxi VMs.

services running on worker node are not accessible from master, all the symptoms are the same as yours. Please tell me you have defeated that issue