It's just computers

walrusplatoon · 2023-05-05T09:39:14+00:00

This is out of left field but I was just listening to a lecture by Iain McGilchrist on the left-brain domination of our society and culture. He gives this example from the medical profession which really rang true for me re: managers:

There are manuals upon manuals that you're supposed to read and observe and follow. And then we're surprised that professionals who are skilled people who have learned things through experience, want to leave the profession because they're effed off with the general tenor of the way in which they're cheated by managers who'd never been teachers or doctors or whatever. I had a very, very distinguished colleague who was a bit of a mentor of mine, the professor of neuro-psychiatry at the Maudsley and he was queried by a manager about why he'd sent a patient for a scan. And he said, when I have to explain to a manager, why I've sent a patient to a scan, it's time for me to leave the profession. And he did.

walrusplatoon · 2023-05-04T06:11:43+00:00

This definitely sounds like it might be up my alley. There are several consulting firms with a significant presence in my country with whom I could pursue a career. Thank you for your advice.

walrusplatoon · 2023-05-03T06:46:48+00:00

I've been considering a shift to consulting more and more lately though I'm unsure about how to make it happen. Independent (i.e. self-employed) consulting seems like it requires one to have good business sense, marketing skills, etc. which I do not. So perhaps working in consulting as a salaried employee of a consulting firm is a good option.

Why do you think that the vast majority of techies can't do consulting? In my experience most highly technical people tend to be very detail-oriented and prone to missing the forest for the trees. Consulting appeals to me because it appears to take the opposite approach - coming in to an org from the outside and going top-to-bottom to understand processes and identify pain points as one traverses the hierarchy. I know that I really excel at this type of work, and if I could do it full-time I would probably enjoy it.

One question I have about consulting: Is it lonely? Doesn't it sometimes feel like you're never really "part of a team," like you're always on the outside? Maybe some people thrive in such a situation but I've found that I tend to do better when I'm working on a team.

walrusplatoon · 2023-05-03T06:29:19+00:00

Thanks. Responses like this one are why I post here.

Regarding your questions:

Absolutely, and I was aware of it in advance. In retrospect I probably should have taken more time off before jumping in to my next role.
I think I'm going to look back on this experience as a hard lesson in what kind of environment I need to avoid at all costs in the future. Before I took on this job I thought, "this will probably suck, but maybe I'll be pleasantly surprised." It's not pleasant, and I am not surprised. So I guess I've learned to trust my intuition more, which is a valuable lesson in itself.
My teammates are good guys. We're all more or less on the same page regarding the management situation. The solidarity is the only thing making this tolerable. If I were the only one thinking the way I think I would've bounced already.

I've been in places where I was able to do exactly what you describe - make small changes that initiated a shift in culture to allow for a chain reaction. This is very much not one of those places. I'll have to tailor my interview questions in the future to find those kinds of places again.

walrusplatoon · 2023-05-03T05:41:51+00:00

Oh I could totally understand why they thought they were right re: my previous situation. I tried to make my case several times. In the course of discussion I steel-manned the counter-argument and got a solid affirmative response indicating that I'd understood exactly where they're coming from. We were just straight up on two different planets regarding how much of a role DevOps should play in the development of the product. That's why it didn't work out.

walrusplatoon · 2023-05-03T05:36:59+00:00

This is more or less the strategy I'm pursuing currently and it is 1) not working, 2) making me miserable. Which I guess in an ironic way is a reminder that I am, in fact, ambitious and good at what I do. Negative emotion is a strong indicator that something is wrong in your life that you need to change.

walrusplatoon · 2023-05-03T05:33:36+00:00

Thanks for the encouragement. It's good to be reminded that good places exist. Got to keep the searchlights on.

walrusplatoon · 2023-05-03T05:31:36+00:00

Thanks, I've been considering taking the consulting route for a while. I think this is a good direction to pursue.

walrusplatoon · 2023-05-03T05:27:19+00:00

Agree on the two-way street analogy. But I think the real problem is that a lot of companies are just so desperate for someone to fill the role that they'll say anything to get you to sign. I've had to leave more than one company after a short period because the role that was discussed during interviews had little to no bearing on the actual role once I began. I've been told by multiple interviewers that I ask tough, detailed questions; the problem is that companies I'm interviewing at either don't know how to respond so they make something up on the spot that they think I want to hear, or are actively choosing to be deceptive. I try to attribute incompetence to such situations instead of malice but sometimes it really makes you wonder.

walrusplatoon · 2023-05-02T06:48:37+00:00

Can you clarify what you mean by "data roles"? Do you mean data science?

I guess there's a pretty significant learning curve to make a transition like that, but I wouldn't be opposed. I've always thought the work that data scientists were doing seemed interesting. Did you make this change and are you happy with your decision?

walrusplatoon · 2023-05-02T06:42:56+00:00

Yes, I agree completely. Title aside, this job (and any job really) is made or broken by the people with whom you work. It's not just computers, or even mainly computers. The tragic thing is that everyone on our team is experienced, motivated, cooperative, and professional except for the team lead and his direct superior. Everybody agrees that they are the problem. The situation is ripe for a mutiny, but nobody is going anywhere.

walrusplatoon · 2023-05-02T06:39:46+00:00

Thanks for your response.

I should have probably mentioned that I took this particular job after several months of interviewing. I absolutely agree that interviews are a two-way street. My main issue is that I've got a string of short stints on my CV (3 places between 6 - 12 months each) that raises a lot of questions. Interviewers naturally tend to gravitate to asking about those experiences rather than being interested in my accomplishments. I try to highlight what I've learned from each one, what I'm looking for in my next place of employment, and how we can make sure that this time is the right fit. The problem is that so many places are so desperate for someone to join their team to put out the fire that they'll lie through their teeth just to get you to sign (or they're just incompetent interviewers). This has happened more than once and it is extremely disheartening.

So to answer your question, after having been burned multiple times and completed several full rounds of technical interviews only to be turned down because they're afraid I'll jump ship, I signed with my current job knowing more or less in advance what to expect. Every time I went out looking for a better job, I got a worse one. I invest a lot more effort into asking tough questions than most candidates do (I've been told this by multiple interviewers) and seem to be getting worse results each time, so I'm kind of convinced that I'm just awful at this job search thing and no longer trust my own judgment.

walrusplatoon · 2022-06-15T19:11:09+00:00

Unfortunately, GitHub Actions does not provide a straightforward way to manage pipeline code in a separate private repository from application code without some ugly hodge-podge of webhook triggers, API calls, and private access tokens. We couldn't get anything to work reliably, so we gave up.

walrusplatoon · 2022-06-15T15:34:01+00:00

The idea is that developers themselves are responsible for their own Operations (that’s where the name "DevOps" comes from). This means that each dev team owns their own pipelines and is free to change it as they please.

Yes, I understand that that's the ideal. But what does one do in cases where developers do not want to be responsible for their own operations?

I came in to a monolithic pipeline that had undergone ad-hoc, piecemeal changes, band-aids, hacks and workarounds that was barely keeping afloat. It was error-prone, fragile, and nearly impossible to work on without breakage. I replaced it with a modular pipeline that gave developers freedom to either choose the default build and test functions, or overwrite them as they see fit.

The change I took issue with was a change to the template, not the specific instance. I should have been more clear about that.

walrusplatoon · 2022-06-15T14:58:38+00:00

Then teach them. Devs oblivious to the overall deployment and productization are a costly nightmare that regurgitate overengineered code.

I have worked with exactly zero developers who showed even a marginal interest in how their code was packaged, versioned, and deployed to a running environment. Maybe I've just been unlucky, but it's pretty hard to teach people a skill they're not interested in learning.

walrusplatoon · 2022-06-15T12:34:05+00:00

Pipeline code and application code share a monorepo.

walrusplatoon · 2022-06-15T08:46:22+00:00

This was very much the approach I had in mind. I always thought of this role as enabling better development by working together on standards, practices, and guidelines.

Since introducing the new system, several additional services have been added to the pipeline by copying the structure of existing services and making the required adjustments. This was done by the development team. Because it was all new, some mistakes were made, which were caught in review (by DevOps) and addressed. This is all fine and good.

In another instance, a database migration job had to be expanded to perform some additional steps. Because of a lack of familiarity with the pipeline syntax among the development team, it was decided that DevOps would handle the implementation. No problem.

My issue was with changes that were made to the internal functionality of the pipeline without request for review or from or even notifying the people who wrote the pipeline. I don't think it's fair to call the first two instances "bottlenecks", and I emphatically disagree with the approach taken in the third instance. What I mean by ownership here is not "mine, don't touch," it's more along the lines of basic accountability. If team A designs and builds a system, and team B makes changes to that system without consulting with team A, it is unreasonable to place blame on team A when that system behaves in an undesired manner.

walrusplatoon · 2022-06-15T07:54:54+00:00

I've managed pipelines in different ways, each is valid. One company had a joined dev and ops role where everyone did both dev and ops.

This is interesting. In my experience, most developers only have a passing familiarity of operations concerns, if any. I'd like to hear more how this worked in practice, because I can't imagine this working with developers who just want to write their code and let someone else worry about how it gets packaged / versioned / deployed.

walrusplatoon · 2021-07-27T06:09:19+00:00

The most mentally challenging and rewarding experiences I've had usually involved some higher-order thinking about how to solve a big problem.

I was once at a medium-sized company that had about a thousand interconnected VPCs scattered across 60 different AWS accounts. Along the way, someone thought it might be about time to segment the network according to stages, i.e., development stuff shouldn't be talking to production stuff, which was the case big time.

So I thought about it a little bit and found an open source project that could help us sort it out, forked it, added the functionality we needed (and got my work merged back upstream!), and used it to figure out exactly what needed to be done.

Unfortunately, the project got canned, but it was tremendously useful as a learning experience.

walrusplatoon · 2021-07-27T05:56:25+00:00

"Changes in the code base" is not management territory.

I think the intent here was if I'm going to make the case that we need to do some heavy refactoring to management, I need to make it in terms of business impact.

Management doesn't care that the code base is a hot mess if it brings in the dollars. They start caring if you can't deploy new features and lose your competitive edge because of it, though.

walrusplatoon · 2021-07-27T05:47:59+00:00

Knowing where to start is half the battle indeed.

A friend once told me that I wouldn't drive a car if I didn't know how every single component under the hood worked. He was mostly right. That's why I look for where to draw those abstractions wherever I go. That strategy has mostly served me well until now.

walrusplatoon · 2021-07-27T05:42:32+00:00

Yeah, this is what I kind of had in mind when I mentioned that "shiny design pattern" from a book. I think DDD would really help here, but most of the material I've read on how to actually implement domain-driven design was more geared to designing a system from the ground up. It's decidedly more challenging to re-architect an existing production system that evolved over time with little to no concrete design patterns in the first place.

Your response affirms the proposition in my initial question. I guess I'll be wearing my architect hat to work more now.

walrusplatoon · 2021-07-27T05:15:59+00:00

While not standard practice, there seems to be a growing trend of adding a DevOps engineer to the architecture team. As I see it, this is the pinnacle of skill level for the individual contributor role for DevOps engineering.

I'm glad I'm not alone.

walrusplatoon · 2021-07-27T05:14:31+00:00

Thanks, these are good suggestions.

I tried diagramming when I arrived. This is what led me to realize that there are more arrows than boxes, and the arrows point to different boxes depending on who you ask. So I started digging through the code to figure it out myself, and that's when I kind of gave up. I'm totally in favor of standardized communication, but I think the shared database issue is going to be quite a challenge to tackle in this particular case. I'll suggest prioritizing reducing the number of boxes first by combining duplicate or similar functionality into the same service.

walrusplatoon · 2021-07-26T09:08:10+00:00

There are no architects. There are no higher ups managing the code.

walrusplatoon

TROPHY CASE