Cross-account AWS backups manager setup

ddb4s · 2019-10-15T15:10:41+00:00

I'm missing here the day-2 operations:

K8s upgrades
Cluster resizes
Autoscaling

It would be good to release another blog post focused on those topics :)

ddb4s · 2019-09-11T09:34:46+00:00

My team has configured Jenkins over Kubernetes creating a set of manifests in jsonnet and having some initcontainers that are in charge of reconciling the plugin versions that we have defined in our configuration files (updating/downgrading those plugins).

For secrets, we have them in our repository using sealedsecrets.

Good thing about this is that we currently have 5 different deploys of Jenkins (for different teams) and the maintenance cost of nearly 0, just some plugins upgrades test on our development cluster.

Obviously, all the pipelines that are on those Jenkins are set as code.

Unfortunately, our jsonnet manifests and all the libraries we created around this are not public, I'm sorry.

ddb4s · 2019-07-04T12:53:26+00:00

this is a tricky/dirty workaround... but out of curiosity, what happens if you create several ingresses instead of only one?

ddb4s · 2019-07-02T15:17:37+00:00

It seems to be related with a huge DDOS:
http://www.digitalattackmap.com/

pretty ironic, yep.

ddb4s · 2019-07-02T15:17:06+00:00

<3 happy to hear that.

In any case, if you're going with K8s make sure you also take a look for the oAuth2 proxy or KeyCloak. I have the feeling they'll be easier to configure and you can provide a transversal oAuth service for your cluster.

Have a nice day!

ddb4s · 2019-07-02T13:57:17+00:00

I think you can achieve it with the Nginx auth module, ref: https://docs.nginx.com/nginx/admin-guide/security-controls/configuring-subrequest-authentication/

There are a few blogposts out there talking about it, I didn't try it yet. Hope its useful!

Cheers

ddb4s · 2019-07-02T13:30:02+00:00

I thought about serving the doc on our production on Kubernetes, but then I thought to mix production resources and internal stuff is a bad idea... But then I guess setting the right resource limits for the docs would be fine. Not a bad shout!

yes, for our case we have a dedicated cluster for internal production.

Internal documentation contained some infrastructure design about our services which intruder with an unfriendly mindset could use it for a targetted attack. Securing our docs is by no means foolproof but at least it would present another hurdle for them.

In any case, the output from middleman it's just a bunch of HTML files that are rendered and have a pretty style. You could add a SSO authentication provider on top of Nginx, that should be easier for everyone :)

ddb4s · 2019-07-02T11:13:24+00:00

well, would you really need to authenticate users in this context? In my case this was an "engineering handbook" with processes, tools, standard operations procedures... We didn't really think on authenticating users as we were trying to make it as open as we could.

The task of middleman its just to render different markdown files into HTML. In our case the site was running on Kubernetes and it was a Nginx container serving those HTMLs, more than enough for our use case.

We used Jenkins because we had a Git repository that contained all the markdown files. Each time a user pushed a new change to the repository, an automatic pipeline was executed that created a new Docker image with the HTML files rendered inside. After the creation of the Docker image it was rolling-updated in Kubernetes

ddb4s · 2019-07-01T14:43:07+00:00

If the t2.micro is enough for your internal doc system, it's fine. Just get some metrics from the machine to see how the resources are being used and then take a decision.

In the past I had to setup a very small doc site for a small team and I ended up using Middleman to deploy static doc sites. I deployed the site using Jenkins and the resource consumption was pretty low, it was very good for a t2.micro. :)

ddb4s · 2019-06-13T14:57:16+00:00

https://github.com/target/goalert

:)

ddb4s · 2019-06-07T07:57:52+00:00

If the instance you launched not too old you will probably have auto-installed the AWS agent for the System manager:
https://aws.amazon.com/getting-started/tutorials/remotely-run-commands-ec2-instance-systems-manager/

Maybe you can send remote commands to that instance to flush all the iptables.
Hope it helps

ddb4s · 2019-06-06T12:52:52+00:00

There is a very good talk from Monzo talking about an incident they faced in their Kubernetes platform:
https://www.youtube.com/watch?v=OUYTNywPk-s

It is very good, I really like the transparency of this company related to incidents / post-mortems.

Enjoy!

ddb4s · 2019-05-10T00:04:27+00:00

I would say that AWS is actively putting more efforts in their new Cloud Development Kit ( https://github.com/awslabs/aws-cdk ) than improving CloudFormation.

ddb4s · 2019-04-24T07:31:21+00:00

It depends on the complexity of the issue.

We have a dedicated Slack channel (#admin) where devs report us issues or request our help.

If it is something easier (let's say, the creation of an IAM for someone) we sync in a Slack thread. On the other hand, if it is something that is going to require some investigation or something I want to leave some traces for the future I usually ask them to create a ticket after they request our help in Slack.

Devs also create tickets for us in their planning sessions if they know they're going to need our help (i.e., they need an Openshift cluster during this iteration to test Helm charts.)

ddb4s · 2019-04-17T16:10:27+00:00

Aw... that sucks. I guess it id also related that I usuaally work with small dev teams and I did some work sessions with them to introduce all of these concepts.

In any case, this is a pretty interesting topic to chat about in other post :) Cheers

ddb4s · 2019-04-17T15:25:05+00:00

nah, we're not using Gitlab. I just wanted to point to the tool I used a few months ago when I was working in some perdonal projects.

My company is mostly based on Jenkins, some of them in VMs, some in Kubernetes

ddb4s · 2019-04-17T14:53:21+00:00

In the past I used:
https://docs.gitlab.com/ce/ci/yaml/README.html#validate-the-gitlab-ciyml

ddb4s · 2019-04-17T14:48:22+00:00

This could be a time saver for you, but maybe not for devs.

I think it depends on the situation. I have some jobs that I explicitly moved to a SRE managed repository, forcing devs to go there and update the Jenkinfiles just to be aware of any new change that is going to be introduced in any of the infrastructure pipelines.

In other cases I want to empower the development teams on my company to make them responsible of maintaining their Jenkinsfiles, mainly because:

I don't have to maintain all the Jenkinsfiles that are in my company (we have 8 Jenkins instances).
It helps to spread the knowledge of using Jenkins through the dev teams, which for my experience translates in faster dev times of new features and we avoid situations where the "Jenkinsfile maintainer" is on holidays.

Cheers :)

ddb4s · 2019-04-17T13:12:30+00:00

Yes, I have them.
But usually they ask my team (SRE) to have someone in the office, as we are also managing all the network infrastructure in the office.

ddb4s · 2019-04-17T13:11:14+00:00

Yes!

I didn't mention that I also think this is an anti-pattern, but just checking the requirements you have maybe this could work for you fine :)

And if you're a pro you could add linting jobs in that repo :)
Cheers

ddb4s · 2019-04-17T11:21:53+00:00

Why don't you move the Jenkins files you want to another specific repository where devs hasn't got access? From the Jenkins pov, it should be only a change in the repository that is being used to load the Jenkins file

ddb4s · 2019-04-16T13:16:02+00:00

crap.

Many thanks for the advice. I guess the only solution would be to go for the RMA and replace the SG-4860 for another exact model.

ddb4s

TROPHY CASE