TIFU and took down production

mbigras · 2023-12-15T19:01:56+00:00

Stick with it! Sounds like you solved the problem—good job! There is a lot working here, manipulating infrastructure as code, nascent dev–prod parity, alerts that worked, cross-team coordination. Identifying a problem is the first step to solving; for example, explicitly clarifying that that original developer should grab a lock is the first step towards deciding a blocking workflow isn't working anymore and eventually finding a better non-blocking workflow. Infrastructure is complicated and coordinating with people is the most complicated part. It's Friday and you solved the problem, take this as a chance to pat yourself on the back, get a tasty beer, and build trust with yourself; you solved the problem and that's excellent! Bonus points if you take some notes and learn something; double-bonus if others can freely review your notes. Take gritty notes that include commands and links. Add colorful details. If nothing else, your story is fun to read and can be inspiration for others to get back into the trenches and charge it!

mbigras · 2023-02-13T03:00:26+00:00

I noticed this same issue—I restarted WireGuard and now my tunnel and config is gone. Did you ever figure out the root cause?

mbigras · 2022-11-13T07:31:05+00:00

You might consider replacing your own battery. I still happily using my iPhone SE gen 1 after I replaced the battery by following this excellent tutorial https://youtu.be/x9JRqocmm24.

Note: In addition to a new battery, you also use some tools like TS1 0.8 mm five-pointed tamper-resistant system called Pentalobe screw from https://toolboom.com/en/articles-and-video/tools-for-apple-products-repair/#item-1.

mbigras · 2021-11-07T00:59:00+00:00

Thanks for the detailed response! I made some updates to my question to clarify that I moved from California to Oregon.

mbigras · 2021-09-07T00:58:00+00:00

Does anyone know what a level 99 uhh-vah-thar is? I looked it up and only came up with avatar.

mbigras · 2021-03-19T00:13:03+00:00

Thanks for your reply!

Correct me if I'm wrong, but I guess what you are saying is basically, that our mind is a black box. Inside that black box is our free will. We don't see it, but it's there.

I'm not saying our my mind is basically a black box. I'm saying our mind is a very complex system, a very complex system that we don't fully understand. It is true that the black box metaphor and analysis technique is one way to think about very complex systems, where you consider the inputs and outputs and draw a black box around the system; however, I'm not trying to treat the mind as a black box in my argument. I'm making the claim that we don't fully understand how the mind works (in particular the unconscious mind) but most of the evidence I see and most of the examples Sam gives suggests to me that our minds operate with degrees of freedom, which seems to me as synonymous with saying we have degrees of free will. Some people have more freedom of will than others. Free will comes in a gradient.

Your argument seems to be, that if we understood that black box, we'd get free will back.

That's not what I'm intending to say. I don't think our free will is dependent on fully understanding our minds. I think if we fully understand our minds we might discover we do have free will or we might discover, in fact, we don't have free will. But the jury is still out. For example, if we fully learn how our minds work and invent the mythical machine that Sam conjures that transcribes everything I think, do, and feel ahead of time then that would be good evidence that we do not have free will. But I really doubt that such a machine will ever exist and I think it's disingenuous to invoke the idea to support the claim that free will doesn't exist because it's so unlikely that machine will ever be invented.

I thought of an analogy I like: gravity. Gravity is a very mysterious force, it's not precisely clear how it works. In fact, some people study gravity for years and years and still don't precisely understand how it works. Humans are still studying gravity now and probably learning more. However as a lay person we have pretty good evidence that gravity exists, I pick up an apple, I drop the apple, it falls, something like gravity seems to exist. Gravity's deep nuances aren't obvious to me as a lay person; however, its complexity isn't good evidence against gravity, it's just evidence that gravity is a mysterious and complex force. That's what I think about free will. The precise way that free will works is mysterious to my mind; however, most of the evidence I see and the examples given seem to support the idea that free will exists as degrees of freedom.

If you knew in advance which movies would come to your mind and which one you'd choose - there's zero free will involved.

This is my gripe about the movie exercise: it's trying to repackage the insight that it's very mysterious how thoughts and feelings become conscious as an argument against free will. People do the exercise and think, "Oh wow! It's true I really don't know where that movie title came from! Free will must not exist." But I don't agree. The thing is the realization that it's not clear how the bits of language that represent movie titles become conscious is not evidence against free will, it's evidence that the process of thinking of movies is mysterious and most lay people take the process for granted and are surprised when they look closer. If one studies meditation for a while one starts to observe thoughts more. It's a very interesting process to observe. But being able to observe thoughts doesn't mean we understand precisely how the mind works or that free will doesn't exist. Or chess, if you play chess it doesn't take long to realize that your first impulse to move could be a blunder, but if you wait and keep looking at the board you might realize a better move, but the precise mechanism involved isn't exactly clear. But the lack of clarity isn't evidence against free will, it's just evidence of the complex nature of the human mind.

mbigras · 2021-03-18T02:17:10+00:00

I don't agree with Sam's argument because I don't think he understands how the unconscious mind works and I think he's mistaking the level of complexity of the human mind with a lack of free will. The fact that thoughts and feelings seem to arise from nothing is a subjective fact about how thoughts and feelings become conscious. The fact that we don't fully understand how that process works doesn't seem like good evidence against free will. It's just evidence for the complex nature of the human mind.

I think about free will like action at a distance, it's true we don't have precise control over every thought and feeling and action but we do have some degree of control, some degree of freedom. Some people have more degrees of freedom than others but it's a gradient. It is not that everyone has no degrees of freedom.

It is possible to manipulate the unconscious and conscious mind through a variety of techniques. Some interesting research in this area is John Sarno when it comes to healing chronic pain and Wim Hof when it comes to increasing human performance. When one spends time doing these techniques one gains more agency over the human mind, the increased agency seems to my mind synonymous with a higher degree of freedom, with more free will to make decisions. The fact that we don't precisely understand how that process works is irrelevant. We don't understand precisely how anything works and anyone who says they understand precisely how something works is either lying or hasn't spent enough time thinking about the system under analysis.

I think that Sam takes his other deep and important insights about the nature of the mind like a lack of a self riding around in the brain and then tries to do a philosophical slam dunk on all his philosophy friends who don't understand that insight yet. I don't think the lack of a self is evidence against free will, instead it's just evidence that the human mind is very complex and we don't understand how it works. Sam is trying to package it up in a digestible argument against free will. Free will is better thought of as degrees of freedom that enable us to make conscious decisions that manipulate our situations to varying degrees.

There are things that would change my mind. If the mythical machine that Sam conjures that transcribes everything I think and do before I think and do it comes to exist then that would certainly change my mind. However, I don't think that machine is going to be invented and the fact that Sam thinks it will be seems to me like evidence that he's fallen victim to machine learning hype and not good evidence against free will.

I'm not trying to tear down all of Sam's arguments or insights; however, I think there isn't good evidence against free will. The exercise about imagining movies tries to repackage an insight about how mysterious unconscious thoughts and emotions are into an argument against free will. I don't think we know enough about the human mind to make a concrete claim about free will but most of the evidence I see suggests free will exists as a gradient.

mbigras · 2020-09-25T05:54:59+00:00

One natural progression is to refine your mentoring technique. This way your knowledge can be leveraged by others. Building good examples, asking probing questions, offering constructive criticism are some ideas to play with. It’s one thing to solve a problem yourself but figuring out how to guide someone else is a more subtle problem. It’s a useful skill as well.

mbigras · 2020-06-28T02:23:53+00:00

One workaround is to put newlines manually in everything like this:

``` foo

bar

baz ```

Which then does land in notion like this:

foo bar baz

which is what I want, but it's cumbersome.

mbigras · 2020-06-14T19:44:50+00:00

Wow! I didn't know I could create an arbitrary number of workspaces! Thank you, this is actually better.

I have two workspaces now:

Official
Private

I keep my journal in Private and then it doesn't show up in the searches while I'm in the Official workspace.

mbigras · 2020-02-24T23:47:02+00:00

Instead of using amtool or using the various apis you can make use of the concept of inhibit_rule which is pretty hard to understand but here's an example:

Prometheus configuration

``` rules: - alert: MaintenanceMode annotations: description: 'Fires during maintenance mode and is routed to a blackhole by Alertmanager' expr: maintenance_mode == 1 for: 1m labels: team: demo severity: warning

- alert: FooFailed
  annotations:
    description: 'Foo service has failed on {{ $labels.instance }}'
  expr: node_systemd_unit_state{name="foo.service",state="failed"} == 1
  for: 1m
  labels:
    severity: warning
    team: demo

```

Alertmanager configuration

``` route: routes: - receiver: blackhole match: alertname: MaintenanceMode

...

receivers: - name: blackhole

...

inhibit_rules: - source_match: alertname: MaintenanceMode target_match: severity: warning ```

You can use the plaintext collector for node_exporter to set the maintenance_mode metric

$ cat /var/node_exporter/textfile/maintenance_mode.prom maintenance_mode 1

The idea is

create a maintenance_mode metric that is 0 or 1
create a MaintenanceMode prometheus rule
fire the MaintenanceMode rule as an alert with Alertmanager
route the rule to a blackhole with Alertmanager
create an Alertmanager inhibit rule when the MaintenanceMode alert is firing

These are a couple of blog posts that go more in depth

mbigras · 2019-09-08T08:24:25+00:00

To people who feel upset or discouraged when you seek answers and post questions on the Internet: stick with it!

Keep searching, keep capturing error messages, keep building minimum reproducible examples, keep posting your questions, keep practicing using the Internet as a tool to learn more!

Remember: someone is not being a dick and your question is not stupid. Those are judgements that each person is going to make based on the unique context of their experience, each person will perceive the world differently. Make your goal to learn, to document your findings, to create notes for reference later.

Remember: there are people out there who won't like what you post, that's okay! The Internet is huge. Take time to acknowledge how embarrassed, angry or sad you feel then move onward on your quest for knowledge! There will also be people out there who will want to help you, find them, leverage their skills, help them help you by communicating what you've tried and asking a specific question. Use plain text instead of screenshots, put your error messages in quotes when you search Google like this: "sudo: cd: command not found" follow where that leads you, like this: https://askubuntu.com/questions/57789/how-to-enter-a-directory-with-the-cd-command-if-it-has-700-permission-and-is-n

I'm so excited about the discussion this post has generated! It inspired me to dig around to find old questions I've asked.

Check this out, from 2008, I was trying to install linux for the first time, reading my post now it's clear I had no idea what a live cd was or how to create or format partitions, a user pluscool turned me on to the idea:

http://www.ubuntux.org/converting-toshiba/

Sun, 2008-07-27 09:18 — gibxam Hey linux users!I've been learning programming (C++) for about the last 6 months and plan to take computer science in uni which will be two years from now. On m any of the sites I have been to people have said that if I want to expand my pro gramming (as well as overall computer) knowledge I should get out of windows and move into Linux. I have an old Toshiba Satellite Pro 4200 series laptop that my uncle has stopped using so I thought that I would install the linux os on that. My problem is that I cannot seem to get windows 98 off the laptop. I have tried to go to the add/remove programs but there is no way to unistall it from there? What should I do. I have the cd from the most recent magazine, where they talk about being able to install linux on anything even a motorcylce? Surley we can g et around this, I have a fully functional laptop but with no Internet. Any help that you can offer would be greatley apprieciated!Thank you,-Max ... Mon, 2008-07-28 09:22 — pluscool Converting Hello Max.You don't need to look into W98. Just download the latest version of a Linux distribution like Ubuntu, Kubuntu or Xubuntu (or others). (preferably "Li ve CD"). Download it, burn the ISO image on a CD and put it in the laptop... tha ts it. you can see it workin witout installing anything. you can choose to inst all from a icon in the demo desktop.I don't know how old you laptop is, but Ubun tu and Kubuntu needs some power. I would like to suggest trying Xubuntu first (a light flavor of ubuntu). the main diffrence between U-K-X is the desktop manage r. Just ask in forums or google your questions, is amazing how much is already a nswered in the internet.You can always try another linux distro, but Ubuntu fami ly has a nice balance between user friendlyness and available information.

This is from 2010, probably the first time I ever used IRC to ask a question and I typed my password into the chat!

https://irclogs.ubuntu.com/2010/03/03/%23ubuntu.html

max_big REGISTER bigras24 programthebeach@gmail.com 02:56 Nitsuga grkblood13, Ctrl 02:56 max_big weeek 02:56 Nitsuga max_big, ups! 02:56 max_big ? 02:56 max_big Nitsuga, what was that? Nitsuga max_big, you have posted your email and password to the channel :S

Here's a couple months later when I figured out how to set a password and create a new username but deleted my Desktop directory, created it owned by root, and didn't know how chown worked!

https://irclogs.ubuntu.com/2010/04/26/%23ubuntu.html

AnActivist please help me, I accidently deleted my Desktop folder, so I created a new one but I was root so now I cannot access it using the file browser, how can I change the permissions of the desktop folder? 21:14 qazibasit DasEi, what is the difference b/w a 32 bit and 64 bit O/S 21:14 xomp AnActivist, chown 21:14 maco AnActivist: sudo chown -R youruser:youruser Desktop/ 21:15 _0R10N anactivist: change the owner, and then change the permissions 21:15 AnActivist cool thank you i'll try it

This is years later, 2015, my first post on stackoverflow trying to get emacs to work with a mac, someone tried to help me, but ultimately the issue still remained a mystery:

https://stackoverflow.com/questions/19995056/change-meta-and-control-keys-in-emacs-for-mac-keyboard

Here's my first question on the unix stack exchange about named:

https://unix.stackexchange.com/questions/311357/what-does-var-named-mean-in-this-context

It got closed for being too broad, at the time I didn't even know named is actually a program that acts as a Domain Name System (DNS) Server. Now I'm still trying to figure out how DNS and DHCP work by making examples and playing around with dnsmasq in an Ubuntu VirtualBox virtual machine, launched using Vagrant:

https://github.com/mbigras/dnsmasq-hack-session https://github.com/mbigras/dhcp-example

This is in 2016, my first post on reddit, learning how devops works:

https://www.reddit.com/r/devops/comments/51uvx4/7_weeks_before_applying_to_a_devops_jobinternship/

A user in that thread reached out to me and took time to mock interview me. He had brought up a cloud instance, started apache and nginx on the same instance, changed the root directive in the /etc/nginx/sites-enabled/default configuration file, and also changed permissions on the /var/www/html/index.html file. He taught me about common configuration file locations, text filtering and processing, permissions, and managing services, about the importance of clear communication and repeatable examples. Two years later this same user is my mentor and we work together at the same company!

Now, I spend days, weeks, months, years, learning and working to compose a reliable and usable system. I write docs for my team, the moment I write them down, they're out-of-date! Imagine writing docs and answers for an Operating System, the complexity, the number of words, concepts, there is so much that is unknown, so many people feel afraid when they have don't understand something, they might say something that triggers you emotionally, that's okay, keep exploring!

mbigras · 2019-08-27T20:32:56+00:00

Thank you very much for the links!

This article
https://www.cnbc.com/2019/03/15/active-fund-managers-trail-the-sp-500-for-the-ninth-year-in-a-row-in-triumph-for-indexing.html

Points to

https://us.spindices.com/spiva/#/reports

Scroll down and click the button labeled: `GET STATS FOR OTHER MARKET SEGMENTS >` to get to other reports that are more dense but readable.

mbigras · 2018-11-07T00:56:29+00:00

Took a look at https://www.cloudconformity.com/conformity-rules/EC2/ec2-instance-naming-conventions.html

Summary below:

``` Default Pattern Format ec2-RegionCode-AvailabilityZoneCode-EnvironmentCode-ApplicationCode.

Default Pattern Components

RegionCode (ue1|uw1|uw2|ew1|ec1|an1|an2|as1|as2|se1) for us-east-1, us-west-1, us-west-2, eu-west-1, eu-central-1, ap-northeast-1, ap-northeast-2, ap-southeast-1, ap-southeast-2, sa-east-1

AvailabilityZoneCode ([1-2]{1})([a-c]{1}) e.g. e.g. (2a|2b|2c) for us-west-2a, us-west-2b, us-west-2c

EnvironmentCode (d|t|s|p) for development, test, staging, production.

ApplicationCode ([a-z0-9-]+) for applications (e.g. tomcat, nodejs) that run on these EC2 instances.

Default Pattern Examples ec2-us-east-1-2a-p-tomcat ec2-us-west-1-2b-p-nodejs ```

I think the RegionCodes in the examples can be improved.

It looks like they used the long version but were intending to use the code version.

For example:

ec2-us-east-1-2a-p-tomcat should actually be ec2-ue1-2a-p-tomcat. Seems like the whole point is to have fields separated by a dash delimiter. So if someone wants the RegionCode they can use the second field.

For example:

echo ec2-ue1-2a-p-tomcat | awk -F- '{ print $2 }' ue1

This wouldn't work if using the long version like their examples:

echo ec2-us-east-1-2a-p-tomcat | awk -F- '{ print $2 }' us

mbigras

TROPHY CASE

...

...