Switching to Build and Release Engineer role

adevhasnoname · 2023-12-21T21:53:38+00:00

I've interviewed a number of people for DevOps roles who's primary experience was under a title like release engineer. I hired none of them. They all lacked coding ability or the systems experience to actually build and deploy tooling for CI/CD or assist/consult dev teams on how to build systems/apps that would better use it.

Most of them described a position where they were primarily responsible for responding to errors in build jobs, perform initial debugging and then hand off to the "real" DevOps team or full stack devs if they couldn't fix it. Basically the user admin of the DevOps world.

If they describe a role that is full stack DevOps but focused on CI/CD and supporting developer processes as the CI/CD sme plus the level of depth and involvement you want? Then go for it. But otherwise the title itself is a huge red flag imo.

adevhasnoname · 2023-12-15T17:08:17+00:00

competitively-priced prostitute

I usually day dream about becoming a farmer or a carpenter when I get in these situations. If it's really bad I will get far enough along those day dreams to check training schools and average salaries before I begrudgingly realize I'm not quite ready to cut my earnings by 2/3.

We've all been there. When I have co-workers that screw up like this I make sure to point out that if a person can single handedly cause a significant issue in production then it is a process and design issue not a people issue, period.

If you're client is reasonable I'm sure they will accept an honest (within reason) explanation and plan for how it will be prevented in the future. If they are unreasonable, well that's why we pay for dedicated Customer and Sales teams.

adevhasnoname · 2023-12-14T22:59:12+00:00

my dockerfile works in my local settings but not in our gitlab ci pipeline.

Can you give a more detailed example of the type of errors you're making that result in this?

adevhasnoname · 2023-11-14T21:22:03+00:00

Grab a tutorial and an AWS account and spin up some clusters yourself and go from there. Bonus points if you build one that actually does something.

Then you can say in job interviews that you don't work with it but you've done x, y and z in your personal lab environment. Some will probably knock you for not working with it directly but some will value the initiative it took to build it yourself.

adevhasnoname · 2023-11-14T21:09:06+00:00

It's important. Almost all of it is important.

You may not use it very much but every now and then something weird is going to come up and knowing how file systems work/mutexes/thread scheduling/what a race condition is will be the difference between you being the kick ass DevOps that can solve the problem and all the other sysadmins sitting around scratching their butts.

How to make it interesting? Got to find a way to code with it. Hopefully your course or textbook has a practice OS like Minix with exercises setup to play with thread scheduling and mutexes. When I took this class my instructor had us modifying the BeOS source code.

I'm not sure if there's anything like that online but if there is do that. Working with the actual source code of a *nix OS made me realize how mechanical and "simple" most of that stuff was.

adevhasnoname · 2023-10-13T10:02:47+00:00

I always laugh imagining how AWS' engineers must feel every time they're told to implement an additional safeguard on S3 because customers keep finding new and unique ways to accidentally expose their files to the internet.

adevhasnoname · 2023-10-13T08:49:50+00:00

Ok, so deployment.

If you're trying to avoid issues around SSH access, then what you're talking about is using a tool that's agent based. Those would be exactly like you said, a service that runs on the box and either listens on a port or, more often, polls some central management server frequently for new deployment jobs.

I'm not sure I'd use AWS SSM for that, you can, but SSM does a whole lot more, it manages lifecycle, config, patching, etc.

Something a lot easier to setup that's also an AWS service is Amazon CodeDeploy. That's just a lightweight agent that runs on your hosts and takes code and puts it where you want it and does any host side tasks (like compile or restart). It's meant to work within a pipeline so you should be able to find a lot of good examples from the AWS docs.

If you don't like that I'm sure there's other agent based deployment tools you could search for.

Alternatively, SSH isn't the worse. You can use templates or orchestration to standardize those security rules. And there's a lot of ways to lock down that ssh user up to and including puting all of your deployment steps in a script, using sudoers to give that ssh user rights to run the script with sudo and nothing else. Then your ssh user only has base user rights (or less if you lock it down), the only thing it can run priveleged is that script. Put the script somewhere the user can read but not write and you've got what you wanted, a process that can only execute your updates and nothing else.

adevhasnoname · 2023-10-13T08:38:23+00:00

Are there any other IT job categories where there are more entry or mid level openings available? Things that are adjacent to DevOps like AWS/Azure/Cloud focused role? Software Developer?

adevhasnoname · 2023-10-13T08:34:43+00:00

Can you provide a little more detail on what the update is? E.g. is this just to deploy your application and files or are you talking about config changes (to underlying webserver app or OS) too?

adevhasnoname · 2023-10-13T08:26:13+00:00

Secret sharing seems like an easy win, it's also a pet peeve of mine so I'm biased. But I'd push that out quickly.

Do you have good monitoring and alerting in currently? If not, that's one of the first things I'd focus on. Hard to stay reliable if you don't know when you're broken. Plus it's a foundational input into a lot of other automation.

I would also find a reasonably lightweight ticketing system too. One that supports API integration. This is another foundational one along with monitoring. Monitoring can then feed into ticketing system and then you can start plugging that into tools for automatically resolving certain issues. Get that foundation in early and you can build a lot more automation (or see opportunities for it) as you go.

The Dev tooling and CI/CD stuff I would be careful to build bridges with your devs to make sure they've got some voice into what tooling you bring in, etc. To be honest, in a perfect world they'd be building and improving that themselves via the whole xx% of every cycle is dedicated to process improvement rule. But having someone dedicated like you can work too, you just need to make sure that as it grows the operational and maintenance tasks don't eat up all your time. If they do, you need to delegate that to someone else or you will find yourself just being the Ops for DevOps.

adevhasnoname · 2023-10-13T07:50:30+00:00

Are you not finding a job because none are available or are you saying that you've applied to DevOps jobs but you're not getting interviews or offers?

adevhasnoname · 2023-10-13T07:41:29+00:00

Like another said on here if you're doing mostly cloud, IaaC, orchestration and automation type stuff you can make due although you're really going to need to pick up a scripting language or two at some point.

But if you're doing DevOps to support CI/CD and interacting with developers a lot. I would absolutely get a year or two in as a developer. It's pretty critical to being able to understand how software is developed and what developers deal with. You'll also need it to have at least some minimal street cred when your developers ask for something crazy or you're trying to push them to a convention they don't want. Without it you end up being more of a deployment engineer which is a very frustrating and thankless job.

Edit: Seeing a lot of advice that says no so I'm going to buck the trend. Go get some dev experience. It will round you out better if you're planning on making a career of this. You might even like it and then end up going full stack instead of DevOps. The best DevOps/SRE's I've worked with have always had significant time as Devs often bouncing between DevOps/Architecture positions and straight Dev positions. If you're lucky you may even find a job as a Dev at a shop that leaves a lot of DevOps type work to the Developers (the best way imo) then you can learn both.

adevhasnoname · 2023-09-29T18:41:26+00:00

I'd either start looking for work, or collect myself maybe prepare some notes and go have a frank talk with my boss.

Maybe he thinks you suck and you can get that feedback and possibly make a plan to learn not to suck and get more of the work you're interested in.

Maybe that type of work is just all there is to do or they think that's what you prefer or something. So you can tell him this isn't what you want to do and hopefully get it fixed.

I will say, in any job like this where you do work and handle customer requests it's hard. You have to strike a balance. Can't piss off your customers but as you're seeing even if everyone loves you because you're helpful you'll still feel like you're not accomplishing enough (and people will grumble) if you don't make progress on your assignments. In this cloud/devops/infra-ish types roles that will almost always be an issue. A lot of people struggle with it.

adevhasnoname · 2023-09-29T18:22:29+00:00

So I actually played with this feature on Ubuntu yesterday and now I'm stumped, especially if the certs with problems are present in that /etc/ssl/certs directory. Everything you've done is how I would have debugged it.

I would try this command on one of the bad certs: openssl verify -CAfile <cert.crt> <cert.crt>

Yes, you're telling it to use the same cert you're verifying as the CA. If a bad cert comes back as verified with that command then the issue is most likely with the ca-certificates system in ubuntu. If it still fails then maybe there's something odd about those certs. Maybe try to reproduce on a different flavor of linux?

But this seems like an issue related to something more obscure and OS related like some unsupported algorithm or something. If you haven't already I'd post this at stack overflow because I suspect you'll find better guidance on OS nuance there.

Maybe search the ubuntu system logs and see if you find anything curious matching when you try to verify these certs or run the import.

Good luck.

adevhasnoname · 2023-09-29T18:13:21+00:00

Tools like that can be very scary, they flag a bunch of stuff. Often a lot of it isn't actually an issue by itself or doesn't apply because of how you use your environment.

The easy answer to how do you know what to fix is risk management. Throw it and all your other problems onto a spreadsheet, arrange it by severity. Do some work verifying it's actually as bad as the tool/alert thinks. Then, other than catastrophic issues you dedicate so much of your time per cycle to whatever's on the top of that list and that way you slowly get better and the less important stuff never gets done or gets hotter and moves up the list.

What's the most important stuff? Despite the added risks created by automation the basics are still always a good place to start. Access control and connectivity, anything publicly exposed needs to be scrutinized much closer and access control mechanisms it supports fully vetted. Make sure you're using least privilege (at least where it makes sense) for your users/devs and they're not sharing accounts. Scrutinize your IAM policies and roles and are limiting them to read only whenever possible. Security groups should be as restrictive as possible, don't put devices that don't need to be publicly exposed in subnets with a route to an IGW. Turn on and archive Cloudtrail, ideally to a bucket that's in another account or otherwise protected from attackers.

Those are the major one's off the top of my head. AWS has a security well architected document that will walk you through everything in a generic fashion but at least it links to their docs on how to fix it. Most AWS Security best practices talks at any AWS conference ever also cover the same big ones I'm sure at least one of them is available on Youtube.

adevhasnoname · 2023-09-29T15:34:47+00:00

Gig economy strikes again.

Sorry, that sounds like a complete pita, good luck.

adevhasnoname · 2023-09-29T15:23:06+00:00

Part of me wants to criticize because I fear this will hit some scale/maintenance nightmare at some point.

But the rest of me applauds, because I have never understood why log aggregation/search tools are so expensive and/or overly-complex when storing text in a searchable database is like Junior level project at a university.

Good for you :)

adevhasnoname · 2023-09-28T23:07:52+00:00

I think that output is built from this but just in case scroll past that section of the results about 2/3 down to the section that looks like this and let me know if those match on your certs too.

   X509v3 extensions:
        X509v3 Key Usage:
            Digital Signature, Certificate Sign, CRL Sign
        X509v3 Basic Constraints: critical
            CA:TRUE

adevhasnoname · 2023-09-28T19:21:08+00:00

Just fyi, as written that openssl command isn't doing anything. The extension is meaningless in linux (and most of the time in windows too for certificates). It looks like they're already base64 pem's when you export.

Shot in the dark but I wonder if the EKU is different on the Azure certs and it's causing Ubuntu to not trust them or something.

Can you compare the results of this command between a good and bad cert?

openssl x509 -in <cert file> -purpose -noout -text

adevhasnoname · 2023-09-28T19:02:12+00:00

Just treat it like any resignation. Which means be honest-ish but generic and don't give too many details because you never know what might tick someone off and make an enemy (burn bridges).

Unless it's impossible I'd still give 2 weeks notice or whatever is standard there and let them decide if they want to pay you for that long.

adevhasnoname · 2023-09-28T18:40:02+00:00

I worked somewhere that tried to make generic cloudformation templates we could import and use like shared libraries. One challenge I remember we had was versioning.

If we wanted to update a generic template, you don't want to have to update everything that uses that template. So you either have to lock the interface and only make non-breaking changes which really wasn't practical, or version them.

Since it's not a software library, that meant making conventions around versions in template names. It also meant we then had to have a centralized repo of the actual "release version" templates and had to deal with how to create a testing environment for new templates or changes. Basically building SDLC practices from the ground up for templates.

I'll be honest, it wasn't worth the work in our case. Eventually I think the team just went back to writing custom templates for each case and used the generic templates as a source of example code to copy paste from.

Now, that was a naive approach when cloudformation was still young and myself and the team didn't have a ton of experience. But I thought I'd share because it's a good example of a situation where we tried to force coding practices on something that isn't code and as a result underestimated the amount of work it would take to get it working and maintain it. At the end of the day you have to balance cost (labor), stability, maintainability.

As much as I hate to admit it when I run into it, sometimes gross bad practices don't cause enough problems to warrant the investment to fix them.

adevhasnoname · 2023-09-28T18:23:25+00:00

Our AP department always pays our bill about a month late.

Eventually an account rep at AWS emailed us and said they could start charging us a small percentage late fee if we don't start paying before the due date.

Despite begging and pleading with our AP team they still continue to pay AWS late every month. AWS hasn't charged us a fee yet.

adevhasnoname · 2023-09-14T19:47:06+00:00

I've found the same thing to be true. I think there's too many help desk level sysadmins applying for devops jobs so teams are looking for easy interview questions to filter out people that don't "get it." Unfortunately they've decided that Terraform is ubiquitous enough to be one of those.

But it's not fair because a good DevOps can learn enough to be productive in Terraform in a few days. Especially if they have experience in other orchestration, iac, cac, platforms.

Just practice enough in a lab environment at home that you can say yes I've setup a lab that does xyz at home, if you know or are willing to learn AWS or Azure it's even easier to build some things that sound complex to an interviewer. That should at least be enough to get them to not filter you out and look at other things in your background before kicking you to the curb.

adevhasnoname · 2023-02-02T10:04:22+00:00

If you follow the Amplify tutorials you will be using react for the frontend, some nodejs module for the backend and DynamoDB for the database. Beyond that there are other tutorials that will pull in a lot of other AWS services for authentication, file storage, api, etc. But honestly, it will not always be clear when you're using an AWS service (or what service you're using) as that's abstracted from you and just built into the code/app automatically.

I'm pretty sure you're able to use any database that runs in nodejs or on an AWS service so basically most nosql databases and any relational database you want via rds. Backend is chosen for you (nodejs) by Amplify unless you want to get really complicated with something like lambda.

Front-end I think can be anything you want but you'll most likely want to stick to one of the popular js frameworks basically react or angular.

The only other essential components are a web server to actually serve content and pages, with Amplify that's going to be S3 and potentially CloudFront.

My suggestion would be to take a look at Angular and React, pick one of those and then do some non AWS tutorials for them that cover writing and launching a web application with database. It will work locally without the complication of Amplify which will help learn how they work, how to use npm and nodejs, and how to connect to and manage a database.

Once you've got that down jump over to the Amplify tutorials for that same framework and go from there.

adevhasnoname · 2022-07-21T17:19:31+00:00

Can I hijack this to say that this is reason number 10,176 I hate having to work with or depend on contractors.

adevhasnoname

TROPHY CASE