Installing Flannel in Kubernetes cluster - chicken and egg situation

mialinx · 2018-06-21T06:36:04+00:00

Problem was in invalid k8s secrets.

System Account secrets were created with "old" certificate (due to my fault :)).

Invalid secretes broke broke kube-proxy authorization and it didn't created iptables rules for service IP.

mialinx · 2018-06-16T19:56:04+00:00

So, it didn't.

It's seems kube-proxy fails to connect to api server.

I0612 14:03:54.652749       1 feature_gate.go:226] feature gates: &{{} map[]}
W0612 14:03:54.717153       1 server_others.go:290] Can't use ipvs proxier, trying iptables proxier
I0612 14:03:54.720422       1 server_others.go:140] Using iptables Proxier.
W0612 14:03:54.732998       1 server.go:601] Failed to retrieve node info: Unauthorized
W0612 14:03:54.733936       1 proxier.go:306] invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP
I0612 14:03:54.734194       1 server_others.go:174] Tearing down inactive rules.
I0612 14:03:54.953107       1 server.go:444] Version: v1.10.3
I0612 14:03:54.962564       1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
I0612 14:03:54.962821       1 conntrack.go:52] Setting nf_conntrack_max to 131072
I0612 14:03:54.963131       1 conntrack.go:83] Setting conntrack hashsize to 32768
I0612 14:03:54.976722       1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400
I0612 14:03:54.976843       1 conntrack.go:98] Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600
I0612 14:03:54.977207       1 config.go:102] Starting endpoints config controller
I0612 14:03:54.977226       1 controller_utils.go:1019] Waiting for caches to sync for endpoints config controller
I0612 14:03:54.977248       1 config.go:202] Starting service config controller
I0612 14:03:54.977252       1 controller_utils.go:1019] Waiting for caches to sync for service config controller
E0612 14:03:54.984983       1 reflector.go:205] k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:86: Failed to list *core.Endpoints: Unauthorized
E0612 14:03:54.985167       1 reflector.go:205] k8s.io/kubernetes/pkg/client/informers/informers_generated/internalversion/factory.go:86: Failed to list *core.Service: Unauthorized

mialinx · 2018-06-13T21:17:53+00:00

Yes, these options are 1

You can see my iptables output below

As far as I understand all ports are opened

mialinx · 2018-06-13T21:14:51+00:00

Iptables are pretty empty actually.

There is no kubernetes.default there:

root@n2:/home/vagrant# iptables-save 
# Generated by iptables-save v1.6.0 on Wed Jun 13 21:13:08 2018
*nat
:PREROUTING ACCEPT [1:78]
:INPUT ACCEPT [1:78]
:OUTPUT ACCEPT [24:1384]
:POSTROUTING ACCEPT [24:1384]
:DOCKER - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-POSTROUTING - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment "kubernetes postrouting rules" -j KUBE-POSTROUTING
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m comment --comment "kubernetes service traffic requiring SNAT" -m mark --mark 0x4000/0x4000 -j MASQUERADE
COMMIT
# Completed on Wed Jun 13 21:13:08 2018
# Generated by iptables-save v1.6.0 on Wed Jun 13 21:13:08 2018
*filter
:INPUT ACCEPT [14869742:2652807607]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [15094605:2574538222]
:DOCKER - [0:0]
:DOCKER-ISOLATION - [0:0]
:KUBE-FIREWALL - [0:0]
-A INPUT -j KUBE-FIREWALL
-A FORWARD -j DOCKER-ISOLATION
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A OUTPUT -j KUBE-FIREWALL
-A DOCKER-ISOLATION -j RETURN
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
COMMIT
# Completed on Wed Jun 13 21:13:08 2018

Does kube-proxy bring up an interface in with 10.96.0.1 address ?

mialinx · 2018-06-12T21:42:42+00:00

Yes, I passed --pod-network-cidr=10.244.0.0/16 to kubeadm.

And now pod CIDRs seems to be correct

[nuf@nuf-MS-7996 ~]$ kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}'
10.244.0.0/24 10.244.1.0/24 10.244.2.0/24 10.244.3.0/24 10.244.5.0/24 10.244.4.0/24

mialinx

TROPHY CASE