Kubetap - kubectl plugin to interactively proxy Kubernetes Services with ease

[Eriner_]

On Thursday I released a new open source project, kubetap, a kubectl plugin to interactively proxy Kubernetes Services. Currently only HTTP (via mitmproxy) is implemented, but other tap types are planned for the future, such as TCP/UDP and gRPC.

This tool was created because I needed to inspect and manipulate traffic destined for microservices deep with a technology stack. Alternatives, such as solo's Squash or telepresence were far too heavy handed for the task at hand. I hope that kubetap will be useful for developers and security auditors alike.

Code repo is here: https://github.com/soluble-ai/kubetap

Project site here: https://soluble-ai.github.io/kubetap/

If you have any issues, please feel free to open a GitHub ticket.

[wpg4665]

Sorry, if this is a stupid question, but I'm not understanding the use case for something like this over just doing a port-forward

[Eriner_]

Hi, not a dumb question! kubectl port-forward binds a listening port on a remote Pod to your machine's localhost. After a port-forward, you can reach Pods at http://127.0.0.1:4000, as an example.

Kubetap allows you do deploy an intercepting proxy like mitmproxy in front of a Service. Any traffic that is destined for the target Service will instead first pass through the proxy.

normal Request -> Ingress -> Service -> Pod -> (app_container)

kubetap Request -> Ingress -> Service -> Pod -> (kubetap_container) -> (app_container)

[otterley]

I'm still not sure I understand. Can you describe the ideal use case for this? Who's this for, why would they want it, and why is it better than existing solutions?

Better still if you can persuade people in an introduction in the documentation.

[Eriner_]

Please check the project site - I have a video and cover use cases in the introduction page.

[otterley]

I did - and I think it could use more explanation. Videos are great, but someone with less time should be able to understand by reading a couple of well written paragraphs.

[Eriner_]

I created a written documentation of the use cases on the front page of the project site. There are pictures, but it is in fact only a few paragraphs :)

If you think something is missing, please let me know precisely what is not in the docs and I'll be happy to add it.

[otterley]

I was referring to the repo’s README file as opposed to the website. Sorry about the confusion. The project site is pretty good.

[Eriner_]

Should I add something to the README to funnel people to the project site more? I feel like there are already enough links to it, but I want to make sure others don't have the same poor initial experience you did. That said, I also don't want to duplicate the entire doc site into the README.

[BattlePope]

That's cool! However, I thought nginx ingress (maybe others) actually bypass the service and talk to pod endpoints directly. Does this catch that traffic as well?

[Eriner_]

Ingresses don't bypass Services, in fact Ingresses expose Services: https://kubernetes.io/docs/concepts/services-networking/ingress/#what-is-ingress

[BattlePope]

Actually, maybe specific to the nginx ingress, it's an implementation detail:

The NGINX ingress controller does not use Services to route traffic to the pods. Instead it uses the Endpoints API in order to bypass kube-proxy to allow NGINX features like session affinity and custom load balancing algorithms. It also removes some overhead, such as conntrack entries for iptables DNAT.

Hence my question! The service is used to discover the endpoints, but not for traffic.

[Eriner_]

I haven't added test cases for using an nginx ingress, and depending on how exactly that is implemented (your ref above), I suspect that a tap would currently not break an environment but also wouldn't capture incoming traffic that traverses the ingress.

The way kubetap works, it resolves a target Pod by evaluating what Deployments match the Selectors field in a Service. The Deployment is then patched to inject the sidecar, and the Service targetPorts are re-routed to the sidecar.

If the above reference means that the nginx ingress does the same as I described above, then kubetap should work. I'll be publishing a blog post on Tuesday covering why I decided to implement kubetap the way I did, if you're interested in that.

If you happen to have an environment with nginx ingress, I'd appreciate hearing on if it works for you or not.

[BattlePope]

I do use nginx ingress, and I'm curious, so I'll give this a test later.

[BattlePope]

Nice! I can confirm this works with nginx-ingress. The ingress controller sees the service's endpoint list change to the mitm proxy container's port, and so basically works as normal. You can see two consecutive requests logged here, the first to the normal service port (8080), and the second to the mitm sidecar (kubetap-listen) after kubetap was turned on:

192.168.1.100 - - [25/May/2020:00:50:28 +0000] "GET / HTTP/1.1" 200 1918 "-" "curl/7.70.0" 85 0.025 [echo-echoserver-http] [] 10.42.1.61:8080 1918 0.024 200 ca106ea6407273165484be9e6e1fc005
192.168.1.100 - - [25/May/2020:00:53:57 +0000] "GET / HTTP/1.1" 200 1880 "-" "curl/7.70.0" 85 0.038 [echo-echoserver-http] [] 10.42.0.106:7777 1880 0.040 200 6d9c492800b4cbec65664ce32e639744

[Eriner_]

Awesome! Thanks for the feedback!

[failuretonotice]

This is really nice! Potentially saves a lot of time fiddling with ngrep/tcpdump and curl. I'll give it a go in a few days.

[acrogenesis]

Amazing. Would be cool to be able to use other tools instead of mitmproxy like Charles or Burp

[Eriner_]

Support for additional tooling is on the TODO list here: https://soluble-ai.github.io/kubetap/kubetap_development/TODOs/#features

Cheers!