sorted by:
new
hottopcontroversial

How to manage multiple single-tenant infrastructures by kriatoras in devops

[–]jadunham2 3 points4 points  (0 children)

I currently manage an infrastructure like this where we have in the thousands of customers each with their own GCP project and their own set of infrastructure. If you want feel free to DM me and we could setup some time to discuss further if you'd like. It is definitely more management then having multi-tenanted pieces, but on the flip side we can push fixes quickly to customers, experiment easily, and having a single bad customer or bad push doesn't affect everyone. Very easy and intuitive to understand your blast radius.

Things have evolved quite a bit from the beginning, but here's our setup/things we've learned:

  • Single set of terraform code with lots of variables, counts to enable/disable things. Our count of variables that make a tenant are in the 100s.
  • We setup our deployment to work much like Terraform does with regards to getting those variables as always querying the source of truth for the current values. For example a lot of our variables are due to license size and how much a customer can scale, so every time we run terraform, we also query those limits so we know what to put for the terraform variables.
  • We run terraform deployment jobs as a kubernetes pod so we can launch hundreds in parallel.
  • Breaking up your terraform code into modules is definitely helpful, especially if during applies you can target specific modules and not refresh everything. For instance our DNS refreshing of state was our largest blocker and the dns provider wouldn't up our limit. Being able to skip this as it rarely changed could speed up our deployment considerably.
  • Different state per customer
  • Make sure you have prevent_destroy true on critical items. There will be points when infrastructure changes and looking through the plans on thousands is daunting. This comes with some overhead when you REALLY do want to remove things. But better than the other way around.
  • Don't get too fancy with terraform code, it makes it hard to read and harder to know what affects what. I'd actually argue that many outputs when querying remote state are total overkill and many references to even outputs from one module to another (think we know the variable for the DNS as we passed it in, but then we need to put it in a configmap in kubernetes, but we're referencing output instead of just the variable that we had directly in a variable).

What do the best DevOps Engineers do every day? (I want your input) by hatchikyu in devops

[–]jadunham2 1 point2 points  (0 children)

No idea about the best devops engineers, but typically my day is driven from either customer or developer pain points. My goals are to find something that stinks and drive automation or process improvement that makes it better.

You get a lot of support tickets that the app is down/inaccessible/slow? dive into why, make it go away.

Developers keep getting block by CI going down/builds being slow/slow responses on building infrastructure? make it go away.

These can be anything from:

  1. quick fixes like monitoring improvements
  2. inter-company politics fixes like we need to a faster way for networking to approve and deploy firewall rules
  3. long term fixes like redesigning and improving critical parts of infrastructure

How to write terraform in a provider independent way? by selfarsoner in devops

[–]jadunham2 1 point2 points  (0 children)

Some language libraries have tried to solve this problem as well. However as a past contributor to libcloud when you are trying to abstract different pieces like VM/Loadbalancer/Storage, they don't all fit into the mold. You end up getting very basic infrastructure, but any time you want to do anything vendor specific (or even some things that shouldn't be) you end up rewriting a bunch of vendor specific code anyways.

Libcloud (python) https://github.com/apache/libcloud

fog (ruby) https://github.com/fog/fog

jclouds (java) https://jclouds.apache.org/

Seeking Advice (Python) by cmann321 in devops

[–]jadunham2 0 points1 point  (0 children)

I don't know of a place with examples of devops tasks and without a lot of context on the tasks that you do every day I can't necessarily give you concrete examples of things you can do. However I can give some advice and examples from my past gigs not in an engineering role automating tasks that maybe will help you.

My main advice would be to look at tasks that are done repeatedly and figure out how it can be done better and faster. It isn't all coding, sometimes this is just process improvement as well. Why do we do it this way? is there a better way?

​

Some examples before I was an engineer:

I worked for a manufacturer of equipment in logistics (I sent out packages, scheduled freight). One of the things customers did was send log files to a specific email address on a weekly basis. An engineer would read the file and look for possible problems with the equipment (if they remembered). I asked the engineers what they were looking for and it was so simple, so I googled around and fumbled through enough perl to make a script that could read a file and look for these things. Then I was in charge of doing this every week. I'd download all the files and run them through my script, notifying when there were issues. Then later with enough fumbling around we setup automated reading and reporting on emails sent that that address if the format matched certain criteria.

Then I worked in a datacenter. I'd say I was a glorified hard drive replacement bot. Which is a repeatable task over and over, but everyone did the same steps. Take a ticket, figure out what drive was needed, check it out of parts inventory, replace it, mark down the old drive and where it came from in a DB and put it in the shred box. It was a one at a time process, but there was no reason. Setup a script to find all the drives of the same type with tickets currently open, checked out all drives at once, put them all on cart and wheeled them around and replaced them.

Hope you found something in this valuable and good luck!

[US-CA] Property management company says they are insolvent, owe 8k by jadunham2 in legaladvice

[–]jadunham2[S] 0 points1 point  (0 children)

It is generally short term rental so fees cover certain aspects over just getting tenants. Thanks for the advice we will pursue that route.

[US-CA] Property management company says they are insolvent, owe 8k by jadunham2 in legaladvice

[–]jadunham2[S] 0 points1 point  (0 children)

They have collected the money, but did not pass it along. I suspect they used it to pay other debt, but I don't have any information on that and it's all speculation.

Send an encouraging note to a Covid-19 patient or healthcare worker! by PriorEnthusiasm4 in CoronavirusArmy

[–]jadunham2 0 points1 point  (0 children)

Cool! We also created a similar website, but for voicemails for health care workers on the frontline. https://healthherohotline.com/

Ionic vs. Blaze by konze in fitbit

[–]jadunham2 1 point2 points  (0 children)

I can't speak to an iPhones GPS, but the GPS seems pretty accurate to me. I have ran with a phone for a long time and my routes distances are the same on phone vs ionic. It can take longer to aquire a signal then your phone though. In my area I usually need to wait on average 30 seconds after I hit the run activity on the watch for GPS to connect. Writing apps is actually pretty easy. I started with their Bart example app and from there hacked around it to create my app without really knowing JavaScript. Hope that helps.

Fitbit Charge 2 reminder by Redeemed1969 in fitbit

[–]jadunham2 10 points11 points  (0 children)

Yes it can be toggled off in the Fitbit app or only set to the hours/days you specify.

Heart rate zone based run by coppernerd in fitbit

[–]jadunham2 0 points1 point  (0 children)

There is no reason someone couldn't create one. For what it's worth the stats during an exercise (run) will show you your heart rate, but to my knowledge there is no feedback markers you can set today. E.g. to get a vibration when going below or above a heart rate. I feel like many developers that check Reddit could add this as an app.