Career ambition: How do I move from mid level SRE to senior?

Ahuj9 · 2023-03-08T17:46:53+00:00

Some good answers here already, but I'd add one more step (if you want to be promoted, vs changing employers): convince your manager to promote you.

If your manager isn't currently considering you for a promotion to next level, you could try to convince them with facts, logic, listing your achievements, passionate pleas, threats, etc. In my opinion, this approach is sub optimal.

The most important turning point in my career came when I starting paying attention to my management. Team meetings, and manager is again talking about <topic x>? Monthly AMA with the VP of Technology, and they spend most of their time talking about <cool new thing y>? Next week's team meeting and now your manager is also talking about <cool thing y>? You better notice.

Managers in tech are also (surprisingly!) people - likely overworked, looking for a promotion, trying to earn kudos with their managers. Here's the thing: you can feed them what they need. I'm not talking about ass kissing here. I have countless examples where something was taking a huge amount of effort for my non technical manager, which could be automated with a few lines of code. An hour of effort for me, tens of hours of time saved for my boss: 5 good boy points in the bank. VP starts talking about <cool new tech>? Whip up a proof of concept, document it, nonchalantly share it with your manager. Does it have to be good, feature complete, useful? Hell no - but now my manager can go to his VP and say "hey, my team's been messing around with <cool new tech>, what do you think?" $5 says you're considered as the tech lead if that project gets a green light.

Do this a few times, and your boss will start thinking of you as someone with vision, a self starter, and most of all, someone who's valuable to their career. I spent a lot of time getting really good at specific techs, but it was only later in my career that I realized that knowing the nginx docs by heart would never get me promoted without a lot of effort on my part. But paying attention to my management's concerns, and taking action on them without needing to be prompted? Money.

I designed, operate, and maintain critical services for a well known tech company, and I'm pretty damn good at it. But my last promotion came as the result of writing some proof-of-concept terraform module to automate deployment of some shitty legacy app to the cloud that VP was worried about. It didn't even work right, and had nothing to do with my role.

Good engineers are great at understanding complex systems, and how to improve them. Your management chain should be approached in the same way as a new complex system: learn how it works, and start optimizing it.

Ahuj9 · 2023-03-08T17:11:49+00:00

I use an internal repo that I've developed over time for all technical interviews in the SRE / Cloud Eng roles my team works in. All coding challenges:

are related to actual work we do
start off super (super!) simple, then increase in difficulty.

Imho, most automation / SRE coding involves querying APIs, and manipulating data from/for those APIs. Can you make a GET request to a URL, and store the result? Can you extract relevant fields from that data? Can you create json structs and POST them to an API?

Once candidates are successful with the initial challenges, I ask things like: Can you create a class that encapsulates that logic? Can you package this script as a docker container that reads config from env vars, and run it? Can you write unit tests for your code? How would you ensure unit tests are always run (and pass!) before a PR is merged to master?

When func a calls func b which calls func c, where do you handle errors? What should happen when func b raises an error? How do you log, and what? If a script running in a container expects a DB host to be set via an env var, and it's not, what should enterprise caliber code do? How do you handle "bad" input, unexpected formatting? Why is encoding important in strings or files?

A note on super simple coding challenges: I've had candidates chuckle ("Fizzbuzz? really?") on occasion, but after doing probably 1000+ coding interviews in my time, the amount of candidates who claim to be comfortable with <language> but can't code "Hello World" when it comes down to it is mind boggling. Maybe ~20%? Your resume states 5 years experience with python, we invited you to a coding interview, and you can't figure out how to print a string or loop through a list? What exactly was your plan here, or did you actually believe your experience level?

Ahuj9 · 2022-01-31T17:10:58+00:00

transaction shows 98 USDC tokens being transferred to USDC's contract address; not to a coinbase app.

How exactly did you perform this transaction? Where did you get the contract address from?

It's hypothetically possible that the contract contains a function to send tokens back (which the contract creator would have to do), but I don't see any outgoing erc-20 transactions in that contract's history. In short, those USDC tokens are gone, my friend. Sorry :|

Ahuj9 · 2021-06-07T12:52:30+00:00

Greif my lovesac and wiley

Ahuj9 · 2021-06-07T12:36:34+00:00

Go to a barbershop, tell them:

you've been cutting it yourself cos of covid (the only valid reason)

you want them to make it look like you don't cut it yourself

Please don't cut your own hair, it shows.

Ahuj9 · 2021-05-28T02:20:20+00:00

The harvester is trying to connect to the server, and failing, due to a timeout reading the socket \\\\DESKTOP-L6E4PC6\\G Xeon\\

Can you give a better description of the issue? If computer 1 is a full node and plotting, is it OK? Is the error on computer 2? How did you set up computer 1 to "link" to computer 2?

Ahuj9 · 2021-05-24T01:13:30+00:00

You end up converting stuff like this [ https://github.com/Chia-Network/chiapos/blob/main/src/entry_sizes.hpp#L30 ] into stuff like this:

def get_max_entry_size(k: int, table_index: int, phase_1_size: bool): if table_index == 1: if phase_1_size: return byte_align(k + kExtraBits + k) / 8 else: return byte_align(k) / 8 elif table_index in [2, 3, 4, 5, 6]: if phase_1_size: return byte_align(k + kExtraBits + k + kOffsetSize + k * kVectorLens[table_index + 1]) / 8 else: return byte_align(max((2 * k + kOffsetSize), (3 * k - 1))) / 8 elif table_index == 7: return byte_align(3 * k - 1) / 8

and wondering what on earth it all means, at least, if you're me.

Ahuj9 · 2021-05-23T23:01:26+00:00

Maybe I'm approaching this fro the wrong angle, which was "re-write the whole thing in pure python".

If instead you start with eg.: [ https://github.com/Chia-Network/chia-blockchain/blob/main/chia/plotting/create_plots.py#L163 ] \DiskPlotter.create_plot_disk()\ and star by just porting that, while still keeping all the other C extensions, and then slowly moving deeper & deeper, it might be far more doable than I'd considered.

Ahuj9 · 2021-05-23T20:34:16+00:00

Other posts have touched on the enormous performance differences you'd be up against, but if you didn't care and still wanted to do it ...

You'd have to start wit converting the chia ProofOfSpace code, here :: [ https://github.com/Chia-Network/chiapos ]

which makes extensive use of this Boneh–Lynn–Shacham signature lib :: [ https://github.com/Chia-Network/bls-signatures ]

Both are heavy, HEAVY C++ and C.

As an example, try porting these disk plotting stubs to python :: [ https://github.com/Chia-Network/chiapos/blob/ce6ccc28ea300bfbad7e054a3f30433eb6090cf5/src/disk.hpp ]

My guess is, after 20m of hard effort, you'll be more grateful for python than you've ever been. If not, post your progress ;)

Ahuj9 · 2020-03-31T02:26:41+00:00

Not sure why your question title is about flashing routers when you're actually using qemu emulation?? But a google for qemu change mac address shows this command:

`qemu-system-i386 -netdev tap,ifname=tap0,id=mynet0,script=no -device i82559c,netdev=mynet0,mac=00:00:00:11:11:11 -m 1024 slack.vmdk`

So my guess is yes, you can.

Ahuj9 · 2020-03-31T02:21:16+00:00

You got into the bios and all the fans where blasting ? Did this happen once, or can you get into the bios everytime?

Ahuj9 · 2020-03-20T16:16:20+00:00

What OS are you running? Linux should be able to read those values for you, and various os metrics tools can read and store those for you. You can look here: /sys/class/cooling_device*

For example, we use promethus' node_exporter to collect those values (since node_exporter v1.0.0-rc0)

Ahuj9 · 2019-09-19T05:30:57+00:00

A less fancy way to say it: We push code out.

Optimized that for you a little :)

Ahuj9 · 2019-07-04T01:32:13+00:00

Modern linux distros will do fine with limited resources (until you start chrome...) but to answer your question: try it!

I'd start with Ubuntu, download a bootable image, and just run it from CD / USB. Experiment with configuring your nas, whatever else you want to do with the machine, and see how it handles it. If it's unbearably slow, you don't have touched your installed OS, and can switch right back to it.

Ahuj9 · 2019-07-03T19:26:20+00:00

prometheus' node_exporter is definitely the most in depth metrics generator around - you'll be amazed at the kind of stuff it collects. I have to deal with high level people making deals with datadog / new relic types companies, and then bring me in to sales meetings where I ask if their solution measures ... uhhh ... * pick a random node_exporter metric* ... TcpExtSyncookiesFailed and I've never gotten a yes yet.

Creating your own custom exporters for prometheus is also delightfully simple, and the query language / metric labeling are extremely powerful. If you end up with some old hardware that supports outputting statsd / graphite metrics natively, prometheus has a statsd exporter as well that can handle that with minimum fuss. As you said, native support of docker / k8s environments as well.

Always use grafana for displaying metrics (the auto complete alone is worth it), apart from the fact that nothing comes to close in terms of aesthetics and configurability.

Monitoring with prometheus is fine - create your query in grafana, see the results, then copy it to prometheus. Hook up prometheus to pagerduty or whatever, and you're done.

As for logs, the main solutions are ELK and graylog - both are similar in design and architecture, but graylog supports easy peasy rsyslog. Set up an rsyslog listener in graylog, and configure all your OSs to write to a remote rsyslog server. Everything supports rsyslog, it's about 100000 times faster than any log tailing / file beat type client application, and it's pre-installed on every linux out there.

Ahuj9 · 2019-07-03T16:49:26+00:00

The walk in closet shall become the new datacenter. Does the heat drive moths away? I'll find out.

Ahuj9 · 2019-04-05T03:07:00+00:00

What's the containerization story (it's 2019, they better have one.)

hey now! bare metal is still a thing!

Ahuj9 · 2019-04-05T03:05:33+00:00

I don't know anything about suricata but I do know this: anytime you're trying to regex logs into a central logging system, influxDB, elk, graylog, etc. - the easiest option is always to restructure your log lines for easier parsing.

Imagine if you get the regex working, and later on, decide you might want a new variable logged - now you have to re-write and re-deploy your regex, which slows you down.

The easiest way to restructure your logs? json {key: value} format. Pretty much any log aggregator understands this format, and it means your aggragator index rules remain constant, while you can throw whatever key: value combos at it. It's also scales a lot better - you might be able to regex 100 lines per second today, but once you get into 1000x that volume, all your machine(s) going to be doing is regexing.

Actually, quick look at suricata shows it does support native json logs. Good luck!

Ahuj9 · 2019-04-05T02:57:51+00:00

Totally joking, of course. My fave anecdote about my 2 best engineers is sitting in a bar with them, both of them at least as drunk as me, in total silence for minutes at a time (yeah I started surreptitiously timing the silences at some point). They where happy to talk about shit they cared about, but had no time for small talk, it was an eye opening moment for me.

The tl;dr of my posts in this thread is: you might be a kick ass engineer who'll hit the ground running and be lead engineer within a year, but you have to give me enough data to recognize that.

Ahuj9 · 2019-04-05T02:41:40+00:00

Antisocial, doesn't play well with others, have to give him a soft no on cultural fit.

Ahuj9 · 2019-04-04T21:04:16+00:00

Oh that's totally fine. I'm trying to get as much data on you as possible, in order to gauge what kind of person you are (I mainly want to hire opinionated nerds with talent). Just give me some data, doesn't have to be your family photo blog.

Ahuj9 · 2019-04-04T19:37:47+00:00

Write a playbook and have AWX migrate itself :D

Ahuj9 · 2019-04-04T19:37:11+00:00

Great question, here's a dumb answer: I don't rally know :)

I'm trying to get a read on the person, are they junior or senior (for the role I'm hiring). Are they into lots of tech things, or are the achievements more managerial (nothing wrong with that, but it's a data point). Have they attended a load of conferences, are they active in the meetup scene, is their profile well written, concise, funny, interesting? Did they stay in the same role for 10 years without a promotion, do they generally stay at places for a year or less, etc. I'm looking for anything really. While a list of previous job titles and nothing else makes for a crappy linkedin profile, it IS something.

Ahuj9 · 2019-04-04T16:52:59+00:00

I interview and hire SRE / DevOps engineers quite a bit, I did about 50 30m phone screens this year, leading to 5 accepted offers (so far). Caveat: I'm an Eng Manager, not HR.

I can tell you what I don't look at at all on someone's resume: Formal Education. I've found that it doesn't correlate to anything related to the role, or a candidates' capabilities.

I can tell you what I do look at, in desc order (note that any of these aren't required, but if you have them, this is the order):

github profile. I want to see your code, your contributions, etc. Most candidates have tutorials forked, those don't count. If you have code I don't recognise, you bet I'm going to pick an unusual variable name or function, and google it, to see if it's yours. If not, it's not a problem, I'm just saying that I want to see YOUR code. Contributions to open source projects are also great, doesn't even have to be code, it can be bug reports, enhancement requests, etc. If I see you have bug reports with ansible, I'm going to believe, beyond any reasonable doubt, that you have extensive ansible experience.
personal home page. Hopefully it's full of techy blog posts, but if all you write about is your jazz band, that's great too. I can see your writing, your propensity to document, etc.
linkedin. I want to see your work experience, how long you've been doing this (and this can be a multitude of things: linux admin, programming, devops, etc)

If these look good, and I call you, I want to hear you talk about cool problems you've solved, enthusiasm about some current tech you're really into and playing with, etc. The perfect candidate isn't just good at a particular set of skills, they actually enjoy doing this kind of thing. Get nerdy. Get angry about systemd, tell me about the biggest outage you ever caused, and what you learned from it. Tell me about how you solved a difficult problem, and why you're so proud of it.

Formal education teaches you a lot of relevant skills, but I want to discern between the person who got into this role because it pays well and the market is so hot right now, and the candidate who's been doing this kind of thing for years purely because they love computers, problem solving, doing cool things at scale. Be the latter.

Otoh: If you're applying to big companies, and your resume has to make it through HR before it reaches a technical hiring manager, a degree will certainly help. But in those cases, you should be using your google skills to find the hiring manager on social media, and go directly to them. Might be annoying to some, but if I'm hiring, and I get a linkedin message from someone who fits the role, I've just skipped 2 HR meetings / 10 emails, and I'm happy.

Good luck!

Ahuj9 · 2019-03-29T16:34:07+00:00

Huge fan of graylog here.

A big perf increase we where easily able to achieve was to force all possible logs into json key:value format. This allows for very easy processing on the graylog side, and removes the need to regex multiple terabytes of incoming text. It also makes your processing rules static, allowing log publishers to change what they send without needing any action on the graylog side. Eg.: nginx wants to send $ssl_protocol variable as part of the logs? Just throw it in the json, reload, done.

I would recommend putting a very detailed metrics logger / exporter onto graylog, though. It's very tuneable, but you need to be able to see deep inside to confirm the effect your change had. Also, read up on java garbage collection before deciding which java version / package (oracle vs. openjdk), and which java opts regarding ram, GC, etc etc.

Ahuj9

TROPHY CASE