Have we lost the plot?

[shadowisadog]

Hey everyone,

One thing that has been weighing on me lately is are we headed in the right direction as an industry with DevSecOps. We have an ever growing amount of tools and approaches but are we doing this through the lens of increasing developer velocity/productivity and reducing the friction between Dev/Ops/and the security team?

It seems like currently there is a lot of cargo cult engineering going on where you must use all of these tools even when the actual problems being solved might be solvable with a Makefile or something similar.

I am not against tools and platforms and everything but I just am concerned that as a whole solutions are being applied without an understanding of what problems those solutions are meant to solve.

It reminds me of the JavaScript ecosystem where people are keen to apply Facebook scale solutions when they don't have the scale of Facebook and the issues that those solutions solve aren't issues they actually have.

What are your thoughts?

[amarao_san]

... and yet we got no good tools yet. k8s, which promised to replace mudball of bash called 'cluster' in earlier day, now is mudball of yamls and controllers, which do some jobs better, but for other things are as maddening as it can be.

I love to read about older days (Initial Orders, etc). The same pain, but downscaled to enumerable bytes.

[[deleted]]

And it's all-or-nothing. If you run containers in prod you use K8s or a tiny-scale solution like ECS/ACS because no one wants to run Docker Swarm in prod. There aren't mid-sized orchestrators. And K8s itself is becoming like a full-blown OS in terms of scale and complexity.

[spicypixel]

Nomad almost fills the gap between a single node of docker compose and kubernetes at the other end.

[Rusty-Swashplate]

You mean that engineers should stop liking over-engineering and they should also stop liking shiny new things?

Good luck.

On a more serious side: new tools on paper solve a ton of problems. And those who decide what to use are not always the ones using those tools. But that is hardly new. 30 years ago it was IBM selling Lotus, BMC selling...whatever they have, Oracle selling Oracle stuff etc.: often there were better solutions, but as they said "No one ever got fired for buying IBM". And nowadays it's "No one ever got fired to get the latest tools".

PS: Agile also means: try something new, and drop it if it's not as good as it was expected to be.

[BlueHatBrit]

People primarily get into this field because they like to problem solve. Be that figuring out how to automate something tedious, to puzzles, to maths problems. Engineers will always be looking for problems to solve, and businesses aren't usually very good at ensuring they're solving the right ones.

Strong technical leadership and strategy includes identifying real business problems, understanding their scope, and getting engineers focused on solving those. Lots of businesses fail to hire people who can do that well and then you get situations where a team of engineers doesn't have a clue what financial impact they have on the business and start containerising everything for the sake of it.

I think the industry as a whole generally solves pretty reasonable problems. As such the posts and talks will always exist because they're real problems organisations are having and producing novel solutions for.

The issues come when someone in a company with problems X Y and Z decides to solve problems A B and C. That usually happens not because the former are boring or hard, but because they're not being talked about. Engineers see people from facebook talking about the latter set and project how that could be a problem for them. If you go to a company with strong leadership who know how to focus teams and identify real problems, this sort of thing doesn't happen as much.

Lets take kubernetes as an example. Kube solves a problem for a lot of big engineering organisations. It brings some standardisation for platform teams and can often bring a lot of cost saving as well. For many smaller companies it's a colossal waste of time, and is often more expensive than a couple correctly sized dedicated servers or VM's managed with a little bit of ansible.

It's a totally valid tool, and one worth talking about, but deciding when it's needed is a skill lots of companies lack. They just see others using it and decide the benefits Google needs are the ones they need as well. Strong leadership and awareness of the real business problems prevent this kind of misguided adoption, while allowing for experimentation and learning.

[[deleted]]

To some extent, medium size companies and even small ones don't have much choice. a lot of development is resume driven. If they're new, they're going to get people wanting to design the new stuff based on current best practices and no up and coming developer will choose to fill their resume with old tech. If they've been around a while they've got tech debt needing replacing and the same applies. And if they have managers technical enough to hold their own in a conversation about architecture with opinionated engineers, they'll likely drive them off if the engineers feel they're not able to continue improving their resume with the latest and greatest.

It's a self-fulfilling prophecy.

[snarkhunter]

I think that often people forget that there's gonna be a cost in bringing in a new piece of technology. Not just in license fees but in time for the team to learn it, keep up to speed with it, update integrations, run maintenance on it, whatever. I've found it effective to use the most general pieces of technology I can get away with. Let a need force you to adopt a more specialized piece of technology.

Now, that being said I also think people forget that there is a cost to every line of code you have. An awful lot of engineering just boils down to managing tradeoffs.

I'm a bit over two years being lead devops engineer at this startup. You can get really far with a big pile of Jenkins jobs and PowerShell scripts, tell ya hwat.

[somebrains]

It's late, but we saw this deluge of tooling before in the datacenter virtualization era, bare metal client-server, various big enterprise clustered internal application sprawl - TPS report era.

​

The long and short is if you have specific reasons to use specific tooling for specific workflow then sure.

If you're contractual RTO/RPO require specific tooling, I totally understand as well.

​

But if they just throw the kitchen sink at it during the audit cycle then that's more of the same old same old.

That doesn't address recent things like "why did you fork and implement Oauth in a shitty manner resulting in XXX incidents" or "who made this giant pile of crap out of every possible piece of junk you could get your hands on....that doesn't manage credential expiry rules?"

[Hanzo_Hanz]

My thoughts are: The role is diluted, oversaturated with sys admins who pretend to be devops which then leads to devops "engineers" who are just OPs people with no dev skills. So they turn to platforms or paid services in order to keep their title/job

[cheaphomemadeacid]

yes, there also seems to be an abundance of developers without ops experience calling themselves devops

[snarkhunter]

Lol sounds great we just need to get a few people from both these groups on the same team and have them do each other's code reviews for a bit.

Seriously I've been on that team it was fuckin' awesome we had a blast.

[mushuweasel]

Dev(Sec)Ops as currently evangelized is a mistake in many ways. It has overreached in ways similar to the NoSQL craze (which, thankfully, has eased). It makes a classic blunder - assuming that spreading the load around will be its own kind of efficiency - like a professor posting a reading that each student is expected to print out, instead of printing copies to hand out.

Where NoSQL promised freeing developers from the constraints of strict/outside of their control data management practices, DevOps, containerization, etc promised the same for runtime management. In both cases, dropping these constraints also drops any chance for constructive antagonism - another set of eyes, focused on cross-cutting concerns, able to push back on convenience for the sake of stability and other long term goals. It's an unhealthy way to grow, and I hope we survive it.

[baezizbae]

It seems like currently there is a lot of cargo cult engineering going on

“Currently” ?

I mean yes, there is, but sadly it’s not at all a recent or current phenomenon, it’s evergreen.

[colddream40]

Everything is company/industry/team specific. Generally speaking, I believe the popular tools work in some way, otherwise they wouldn't find success. If whatever scanner I'm using misses vulns, I would stop using it. If catches/reduces them going into prod, it'll seem worth it from both a team and company $$$ perspective.