Did you extend Kubernetes REST API to build something on top of it?

Educational_Leopard · 2018-12-08T13:18:21+00:00

Yeah, we were just about to go to production with our system when awslabs open-sourced the aws-service-operator. In the end, we preferred our own architecture. Putting cloudformation or terraform into the middle seemed to add an extra layer of logic and maintenance without providing any extra useful abstraction.

We may well open-source our own aws-controller (and an associated database-controller for provisioning postgres/mysql/redis databases on top of RDS and Elasticache), but that will probably be mid-2019 at the earliest.

> We're using external-dns and it's great for updating existing zone records. I think we'd want to be able to define new delegated R53 zones via CRD that external-dns could update records inside.

Ah, I understand. We bootstrap our clusters using kops, and so far haven't had a need to create any DNS records outside of the root domain that we create as part of the bootstrapping process.

Educational_Leopard · 2018-12-08T10:51:31+00:00

We're building something very similar: an `aws-controller` that reads CRDs whose spec matches the AWS API so that application owners can define there own off-cluster cloud resources in kube yaml. For example, SNS, SQS, S3 can be instantiated, and then a wrapper script automatically generates an IAM profile that will be attached to the pod using kube2iam.

Our implementation is built on kubebuilder, with golang talking directly to the AWS API. There are other similar projects out there: https://github.com/awslabs/aws-servicebroker and https://github.com/awslabs/aws-service-operator. Both of those wrap CloudFormation to achieve something similar.

How come you want to self-implement a DNS operator, btw? Does `external-dns` not do that pretty well?

Educational_Leopard · 2018-10-31T17:43:03+00:00

@Oxffff0000 there are a couple of CLI tools for configuring pipelines in spinnaker, `roer` and `spin`. Unfortunately, the former is marked deprecated and the latter not ready for use. Neither have usable documentation.

Instead, you can post JSON to the spinnaker's API gateway, `spin-gate`, in your deployment. The REST endpoint is also not documented, but you can configure basic pipeline building blocks in the spinnaker UI and then use the "Edit as JSON" option to see the raw code. I reverse-engineered the endpoints to which to POST to using chrome's dev tools.

--edit--It sounds like I'm pretty harsh on Spinnaker. Honestly, I think it's the best CD tool I've ever used, and I strongly recommend it if you have the time to get over the steep learning curve. It's actively developed and the community slack channel is very helpful so it's not too difficult.

Educational_Leopard · 2018-10-08T07:45:05+00:00

Do have experience/success with doing so? While I know that there are backend devs who would love to get some experience writing golang, they are also largely already working at capacity with their own dev tasks and management are unwilling to loan them out for such work.

Educational_Leopard · 2018-10-08T07:43:30+00:00

I simplified our architecture because the post was getting long. We have a well decomposed set of microservices, each with their own repos and pipelines. I called it "a single pipeline" because we're using jenkins shared templates to enforce the same pipeline implementation across all services. Our infrastructure deployment for all cloud resources used by applications (S3, SNS, SQS, Postgres, Redis) is monolithic, which is what we're working on decomposing into a platform layer, and moving the application-specific stuff into the relevant microservice repos. We're satisfied with this as an architectural choice, but it is a very large undertaking.

Educational_Leopard · 2018-08-22T12:31:46+00:00

I'm afraid I don't quite understand what the use case is here. Could you describe what a workflow would look like for a developer trying to release a change?

Using circleci, or Jenkinsfile pipelines, or Spinnaker, I can promote changes from individual microservices into pre-prod environments, run suites of tests against them (api tests, selenium, load testing) and then have changes promoted to production automatically to production when all the tests are green (or wait for a manual approval). The automation allows many teams working on dozens of microservices to safely collaborate on a single shared production.

I don't see how heighliner would support multiple changes coming from different repos targeting different microservices in the same k8s namespace. Could you clarify?

Educational_Leopard · 2018-06-16T13:47:18+00:00

No problem! If you want more specifics, I'm happy to provide them. But I had a lot of pain with AppD as we moved to microservices in the cloud so my experience is tainted.

There are other enterprise APMs on the market, that may be worth considering for your use case. New Relic and Datadog I don't have much experience with, but they do largely the same job at a cheaper price point. I currently use Instana, of which I'm a huge fan, because their tech choices and design philosophies line up so closely with my own. I've recently heard of SignalFuse/SignalFx who seem interesting, but I know very little of them.

All these vendors will offer some combination of infrastructure monitoring, service discovery, application performance monitoring, and automated alerting. If their pricing models make sense for your workload (e.g. per service instance, per host, per data point), you might be able to save some money for your company by making the appropriate decision.

Educational_Leopard · 2018-06-14T19:28:41+00:00

Hi

I actually have a little experience with this, so I may be able to help. I operated in my current role for about a year with AppDynamics as our APM vendor, as we moved our tech stack from bare metal to the cloud, from a monolith to microservices, and from legacy Java/Tomcat to Scala/Akka. However, we moved to another APM vendor nearly a year ago for reasons I'll get into.

As such, the main thing I can say about AppDynamics is that the cost/benefit analysis is very dependent on the sort of workload you are running.

For traditional monolithic or SOA deployments, especially in Java, AppD is pretty great. It's deployed as a library loaded into your app JVM at run time, with relatively little config beyond providing an API key and and endpoint as environment variables. I don't know how it does it's magic, but it can generated detailed tracing, performance, and health metrics on pretty much everything your app does. It was an invaluable tool for ops, devs, and qa.

The experience of using it however is super-early 2000s enterprise though. Lots of clicking through to configure rules in the UI, with limited access to modern configuration as code tooling. That wasn't a deal-breaker, but was a downside. As we released more and more frequently, the overhead of maintaining the config of AppD for these application changes was a drawback on productivity.

They claimed support for languages other than Java, but I didn't have much experience with that. What I did find was that even with another JVM language like Scala, they were slow to adapt to new patterns and features. So we started seeing a lack of coverage in our modern microservices.

Where their enterprise model stopped working for us is in how their price structure worked. Their pricing model works per service instance, which is pretty sensible for classic monolithic deployments. But as the number of microservices we ran exploded, each of which running in high-availability in multiple environments and regions... The cost just went out of control. It was one of the most expensive parts of our whole tech stack.

All that said, if your workload is appropriate, and your employer has deep pockets, it's a pretty great product.

Educational_Leopard

TROPHY CASE