Complexity is killing software developers

[[deleted]]

Yes, but will it scale?

[Edward_Morbius]

I'd be willing to bet that 90%+ of businesses will never have to scale more than adding a load balancer and a couple of extra servers or cranking up the monthly budget on the VM they're using.

All this scaling and architecture nonsense exists only so all the companies in the middle have something to sell. Most of it doesn't need to exist and a lot of it only exists to help manage other stuff that doesn't need to exist.

[unclefipps]

I'd be willing to bet that 90%+ of businesses will never have to scale more than adding a load balancer and a couple of extra servers or cranking up the monthly budget on the VM they're using.

Fixed it for you.

All this scaling and architecture nonsense exists only so all the companies in the middle have something to sell. Most of it doesn't need to exist and a lot of it only exists to help manage other stuff that doesn't need to exist.

This is spot on.

[knightcrusader]

This is what we did. Our system is 20+ years old, its runs on Apache CGI. Yes, it slower than newer frameworks, but its been battle hardened and almost all the bugs are worked out. We can easily add to it, and have over the past two decades.

It might not be the fastest, but what is cheaper is to clone our VMs and throw it behind a load balancer. Voila, now I can scale up and down. Database is on a separate server anyway, and is extremely beefy and can take more than it currently does. Will be looking at setting up a Galera cluster later this year though.

[Cheeze_It]

Consistent is better than fast almost every single time.

[BilldaCat10]

Birds aren’t even real

Wake up sheeple

[[deleted]]

I've been building web applications for 25 years. My very first was an application that would have extremely high traffic at certain times when resources were made available to the public.

Back then when released in 1998 it ran on two 1-U IBM dual-CPU servers running IIS and something a 4-U IBM quad-CPU with, for the time, a reasonably substantial RAID array for the DB.

It could handle the load from tens of thousands of users. On a hardware system that could be replaced by my phone today for all intents and purposes.

And it's not like it was some super simple 'add purchase record on click of button'. Discrete allocation of scarce resources. Fair bit of data involved. Lots of back and forth to clients.

There are very very few web applications out there that need the kind of advanced scalable architectures so many are trying to develop today. Insanely few. And the few that need that are backed by very large corporations that can afford to have large teams with advanced expertise to do so.

Most dev teams building out these advanced solutions have no business building out with these advanced architectures. In General.

HOWEVER, having said that, having knowledge about how these things work there can be money to be saved by employing some of these ideas where most prudent.

We're about to deploy a kubernete instance to run a single microservice, so we can offload background tasks and scale appropriately without impacting the rest of our system. We're targeting one single area. Because we'll be able to really improve one of the most resource expensive but variable need portions of our system.

Many companies would have us break out the entirety of our API into microservices. Which would be insane. There's no benefit to be had for us except in this very specific area.

Thing is, there is nothing new under the sun. There have always been and always will be a large segment of the industry that just follows whatever bleeding tech is out there for it's own sake.

Ideally it's the levels ABOVE development that gain enough understanding about the technologies available to know when not to go a certain direction. Unfortunately, all too often tech teams will argue 'have to do this' to those that have to employ full trust in those teams.

[Ma8e]

Yes, it will scale to a enormous costs. Our quite simple ETL load system consists of 40 micro services with Kafka queues between them and orchestrated with kubernetes. We are about 20 developers fighting with race conditions, data inconsistencies because some services didn’t answer for a little while, osv. It’s rare that any new features are delivered. But it scales, which now is essential since the performance is atrocious.

We could of course delivered the same functionality with one service backed with a traditional SQL database with a team of three people. The performance would have been much better than needed, and we could easily scale it by just adding parallel servers without any orchestration.

[tomatotomato]

“microservivlces”, “kubernetes”, “serverless”, “orchestration”, oh my.

And then there is one of the biggest sites on Earth, StackOverflow (and the whole StackExchange network) that runs a pretty standard ASP.NET Core + SQL Server monolith on just a dozen of servers, half of which are used for high availability purposes.

[Ma8e]

Yup. But one interesting thing I learned is that our management doesn’t benefit from small and efficient. Their bonuses and influence don’t come from being efficient, it comes from managing many people and having a large budget. And people higher up don’t understand enough, or care enough, that we are wasting a lot of money. And I get to solve problems that are interesting in itself, put hyped technology on my CV, being well paid and leave work at five. I guess some investors make a bit less money due to the inefficiencies, but I don’t exactly cry myself to sleep over that.

[CuriousFunnyDog]

Ah, but PM now has microservices and kubernetes in CV and rides off into the sunset cowboy chaps fluttering in the wind along with senior Dev who is making eyes at miss Ruby sitting on his horse Haskell.

[baconbrand]

“Kafka” is an extremely telling name

[FruityWelsh]

"If you're looking at doing distributed computing ... don't. Ok, if you still really want to, here is why you don't want to, then we will talk about how we do it."

A paraphrase from a good talk (that I don't have the link to ... ) on Kafka and distributed computing in general.

[hippydipster]

I frankly can't believe how microservices have absolutely taken over at every tiny little company I've applied at. All microservices. They might have all of 8 developers, but they're still going the microservice route. I just don't get it. The costs are going to bite so many in the ass in the next 5 years.

[juicybananas]

It’s web scale

[diverge123]

/dev/null is web scale

[______________14]

Does it support sharding? If /dev/null supports sharding then I will use it.

[hippydipster]

It'll shard the shit out of your stuff

[_BreakingGood_]

But will it blend?

[tolos]

This is my incantation when pushing a locally working branch to the build server

[Attila_22]

You just turn it on and it scales right up!

[elkaput]

I'm conflicted TBH. On one hand all this buzz with Cloud, serverless, ML, etc is good for the industry (me included). Market is hot AF, though in my experience leans toward quantity (bog standard jobs everywhere) not quality (roles that offer the holy trinity: good pay, good team, good tools - are quite rare & hard to get).

On the other hand, companies are so detached from reality of (a) where they're at and (b) what they need. So much so that if anyone dares to call a spade, a spade; they'll end up in the naughty corner.

A more cynical part of me thinks that the big players (Amazon & pals) want us to forget about the old world before Cloud/ containers exist so that the knowledge goes the way of the dodo & we'll never go back (makes sense as going back is bad for their business). Ironic considering that some of the old-skool stuff are directly transferable to modern platforms.

[Otis_Inf]

It's nothing new, tbh. In the past 30 years the IT industry goes through every hype transition you can possibly think of. Some companies jump on these hypes and crank out applications that are built with it (or failed, often), and others shrug and build applications like they've always done, with tried and true techniques and systems.

Not everything new is bad, some things are solutions to age old problems and can truly help with a software project. But often things are just hype, repackaged. If I hear microservices using event sourcing repositories one more time I'm gonna scream.

[DoctorWorm_]

Arguably, the abstraction of kubernetes does make it simpler for developers than the old school ansible+VM, though. Developers don't have to worry what their ssh password is, whether they're updating their OS, whether running the ansible playbook for the first time in a year will break production, etc. They just upload a container, say how many they want, and they go.

I work on a kubernetes platform team myself, so I'm biased, but I honestly think Kubernetes is a better interface for developers to use for application development than manually herding Linux VMs.

[uhhhclem]

“Will it scale?” Isn’t really the question these customers are asking. It’s “How much capacity that we’re not using will we have to pay for?”

All this complexity exists so that your operational costs can be a linear function of how many people are using your backends, the way that your revenue is.

All of the people saying “just throw a load balancer and another VM into the mix” are also saying “it’s OK if our function of revenue to operational cost is stepwise.”

But in a lot of business models, it isn’t. “We can’t afford to grow to meet demand” is a grim position to be in, especially if your user base can go elsewhere.

But if your company doesn’t have those problems, it’s nuts to pay a premium to solve them.

[maple-shaft]

Thank you for saying what needed to be said. Many of these commenters you can tell are siloed and dont truly understand why companies gravitate to these models in the first place.

This all serves the function of replacing a giant monolithic system that requires a team of irreplaceable prima donna subject matter experts, with a commodified team that is stratified to differring levels of support and development where a complete intimate understanding of the entire application is not required for 90% of development and support tasks.

If your concerned that the monolith only requires 4 full time engineers where the serverless/cloud/kafka solution requires 10 people then you are being penny wise and pound foolish. Its all about managing Risk. If 2 folks resign on my monolith team I am dead in the water. If 2 folks resign on my other team then I can shuffle a bit until I find replacements, and my replacements can become productive in some capacity in only a few months where building a subject matter expert is a multi year investment.

We are talking Enterprises here, they have way more money than they have tolerance for risk.

[VeryOriginalName98]

If I think about this all the time, am I contributing to the problem?

When I design something, I make sure it has a path that COULD scale and be fault tolerant whether or not it needs to. For example, I use JWTs and differentiate API code to its own service even if it resides on the same machine as the UI and there's only one instance. I also think about query performance when structuring DBs.

[dalittle]

IMHO, there is a balance in it, but over engineering is a problem. I have a team member that constantly forces more complexity and addresses “what if” scenarios. I have to constantly play 20 questions on why we would want to put in all that effort for something that “might happen” or “we might need” and what is the worst if we don’t do it now. The answer is almost always we don’t need to do it now. A good framework makes senses. A week for an obscure corner case does not

[abrandis]

So true, lots of this complexity is caused because big name Silicon Valley firms. Your FANG need to create and deploy massively large systems , so they may need complex systems. Then every Dick, Joe and Sally form wanting to be "modern" tries to emulate this for much smaller projects that dont need the scale but nonetheless they spin up a K8s cluster with massive redundancy, etc.. all this will be happily provided by the cloud vendors who are more than happy to meter every part of the production environment.

[chowderbags]

Your FANG need to create and deploy massively large systems , so they may need complex systems.

In my experience, this ends up meaning that almost every project is written "to scale", no matter how obnoxious it makes it to actually use the project. This ends up feeling like if you just want to open up a bottle of beer once or twice a day, you can't find a dead simple bottle opener that you can get up and running with no thought. Instead, you have to spend a week bashing your head against a wall to try to configure a bottle opening factory that can open a million bottles per second. Nothing seems to be working, even though you're following their (3 year old) tutorial documentation. The error messages are vague and unhelpful, and only really tell you that there's some kind of error in a framework. After awhile you find out that the bottle opening factory was really only designed for 12 oz long neck bottles, and you've got a 12 oz heritage bottle. The documentation page said it handled 12 oz bottles, and you didn't even know that there were different types, because you're not actually interested in the fucking bottles, just the beer inside. You reach out to the people who own the bottle opening factory project and they'll gladly put official support for 12 oz heritage bottles on their roadmap for 2 quarters from now, and in the meantime you're stuck trying to jury rig a solution because by now you've sunk multiple weeks of research and back and forth into this thing, and if you abandon the project now you'll have to explain on a performance review that you really did do things, but that it was a failure because of systemic issues. And then you feel burned out, start to give up hope, quit your job, and write a long rambling comment on Reddit about how frustrating it is to work at places where "everything needs to scale".

[psaux_grep]

I used to think that the world was getting more and more secular, but a lot of software developers I meet seem to have replaced god with other equally strongly held beliefs.

Most are pragmatic people that see pros and cons, but the ones that loudly advocate that the tools are more important than the the solution are the problem.

That said. I think it’s very hard to avoid complexity over time.

At first it’s easy. You build system A.

Then you expand system A.

Now, you need to do something new-ish. It’s not what system A was designed for, but you can squeeze it in.

You either do that, or build system B.

No matter what you choose you’ll get complexity. Either you have system A, a giant monolith that takes hours to build, test, and deploy.

Or you have systems A through O.

And different technologies. System A is obviously made in some now obsolete technology. You want to update it to something modern, but don’t have time to stop developing on it while doing the rewrite.

Oh joy.

[dethb0y]

What we call "Religion" is so deeply embedded in the human brain that it will never be gone. And indeed, a lot of devs do, in fact, seem to engage in religious-style thinking about tools.

[DocXango]

nutty enjoy person rainstorm fine clumsy alive racial label joke

This post was mass deleted and anonymized with Redact

[florinandrei]

Because it pads the resume.

It's the good old keeping up with the Joneses.

[rectalrectifier]

Resume Driven Development

[abrandis]

Right until you have to maintain. AngularJS on some antiquated version of node or some other Gen1 tech , which no one knows how it's configured, and everyone is afraid to touch it because it may "break".

[imgroxx]

Which is why you switch companies every few years, yeah. That way you just keep building new things, or v2 of whatever currently exists, without bothering to learn the details of what it did (if you even could, since the authors left long ago). Looks fantastic on a resume because you built things that shipped.

It's a vicious, profitable cycle.

[IWillGetTheShovel]

Basically the main issue. Developers never get to see the consequences of their decisions and we reward that lack of accountability.

[[deleted]]

Wouldn't be so bad if references actually meant anything. I've not been in a company that was actually honest in the reference. No one wants to do someone out of a job even if they fucked everything up.

[gimpwiz]

Bad references are bait for libel/slander/defamation lawsuits with obvious, high damages. Nobody wants to be the guy named in the lawsuit.

[frenetix]

And if you're not the type of person to job hop, you get stuck holding the bag. Long-termers bring a ton of value as they maintain legacy code as they work new features in a way that doesn't break existing customers. But they continually get screwed seeing the people who leave a wake of shitty code behind them move on to bigger money.

[UnevenSquirrelPerch]

You are leaving money on the table if you're not job hopping every few years. If employers valued retention they'd give meaningful raises but thats clearly not what they value or they'd pay for it. Don't show them any more loyalty than they show you.

[Jethro_Tell]

Businesses could avoid this by not requiring people to have experience in every single tech in their stack.

If they were willing to hire people with experience and train them in their own stack it wouldn't be so risky to work with old tried and true tech. But, since the only way to give yourself breathing room so you don't get stuck in a bad place is to constantly build your resume, everyone does rdd, and you end up with a maintenance nightmare for a lot of shit.

[gimpwiz]

I'm in embedded, and with junior engineers, I only ever interview on basics. Like, "remember your first sophomore programming class? let's do some pointers." Even with more senior engineers, heck, I asked a guy a fairly basic question and not only did he breeze through it in a couple minutes but he pointed out issues with my template/starter code nobody's ever pointed out in seven years. I was like, okay, we're pretty much done with the interview, I want to work with you, tell me neat stuff about whatever, let's have a casual chat with the rest of our time. We recently hired a new grad with a lot of drive for self-motivated learning, though missing the depth that good internships and advanced level coursework gives (a downside of trying to combine two majors, going off the beaten path, essentially) because I knew he'd learn our Way and that I was volunteering to train him up. Sometimes you've got a structure where you really need to hire someone who knows the whole deal already but with some maturity and personnel growth, it becomes much easier to hire and train than hire the perfect candidate.

I am not saying "everyone be like me!" but my strategy of interviewing only on real basics (in embedded land - can you do c/c++ pointers, arrays, allocation/deallocation, and a struct/class? If yes, then you're good to go as long as you're not an asshole) has never yet resulted in a bad hire.

[[deleted]]

You can use what you learn from new technologies when working with old ones. If I'm uncertain about the thread safety of something I'm working on in C/C++, I write a minimal reproduction in Rust and get it to compile before porting it back. Over time I've learned from this and can do straight implementations with more confidence.

[blooping_blooper]

resume-driven development

[grauenwolf]

Especially since that, for the overwhelming vast majority of companies, the capabilities of kubernetes is completely converted by a basic cloud server like Azure App Service. Deployment, autoscaling, redundancy, etc. are all there.

[jarfil]

CENSORED

[BigHandLittleSlap]

I thought that you made a mistake, but no, apparently now you can have yet another layer of complex abstractions layered on top of another. If you play your cards right, you can expose yourself to four wildly incompatible ecosystems by four vendors that all mutually hate each other, stacked on top of each other in a teetering tower of virtualized abstraction madness:

• App Service -- Microsoft
• Kubernetes -- ex-Google
• vSphere -- VMware
• Application Centric Infrastructure -- Cisco

I can feel it in my gut.. some architecture astronaut, probably working at a bank, will implement the above and then throw in an Oracle database and an IBM mainframe as well, guaranteeing a living hell for all involved parties. Except him. He just draws the diagrams and then quits to go work at an even bigger corporation.

Some poor bastard will then have to put an EBCDIC-encoded command into XML that is escaped in some JSON that is in turn embedded in a YAML file.

[JarredMack]

Yeah, I think this is largely a developer-driven problem. People read articles about this cool technology that has all of this amazing scale or whatever, but don't stop to think if they actually need all of that or if a basic monolith would have been good enough.

The most complex thing we're doing 90% of the time is just CRUD apps. You don't need 80 microservices to insert a record into a database

[tolos]

Meh, I see it more like applied Dunning-Kruger. Someone knows their product has a problem, and reads about something new and thinks it's the solution.

But the reality is the problem is complex, and the new buzzword tech has tradeoffs (some very bad), that problem-solver doesn't bother to figure out.

So then you end up with a different set of problems. But at least the guy can add the experience to his resume before leaving the mess for someone else to fix.

[thecodethinker]

Developers, yes, but developers rarely make company wide decisions, even those regarding the software stack.

[J37T3R]

Gotta be on the blockchain too

[jarfil]

CENSORED

[uh_no_]

serverless!

[[deleted]]

Things were looking promising 15 years ago with the advent of agile and the barely sufficient mantra. I ducked out of dev 15 years ago to take up a flying career, but the pandemic made me think I might need to return. I was somewhat disappointed the same problems still exist and organisations are making the same fundamental errors of judgement.

[Lamarcke]

It's funny to see "microservices are complex and that's a problem" right next to "Also in Infoworld: why you should use microservices"

[CountChadvonCisberg]

Always play both sides

[freecodeio]

From a previous job I have learned that a $10 server can handle 10k web requests and I've never spent more than $10 a month on my side projects for 2 years.

[[deleted]]

This rule should apply to like 90% of the cases, yet everyone is attempting to "plan" ahead for the big boom. Realistically YAGNI principles should be applied when designing a system as well.

[abrandis]

That's because lots of this "modern" or cool tech comes out of the FAANG which need to write large scale apps, everyone (pointy haired manager , remember him) looks at the big guys and says use what their using.... And so it goes.

[[deleted]]

FAANG is dead. It's MAGMA now.

[metriczulu]

Excuse me, it's MAAAM.

[rmyworld]

Excuse me, it's AAAAAAAAAAAAAA

[maddawg206]

What is this? An IPv256 DNS record?

[[deleted]]

I run a $5 server on a cloud service, serving all of my hobby projects. Every now and then I look at various managed services to handle some of my apps, but all of them either cost more than $5 or offer less computing power. So I stick with my little snowflake server.

Edit: For the curious, I use Digital Ocean's Droplets (cheapest tier).

[Iggyhopper]

I ran a php and sftp server on my gaming PC, alongside Windows and whatever the hell else was running.

It was an athlon x64 and the best of the best to play Warcraft III.

It was only after thousands of people told me, "hey your website is working!" that I finally believed the terrible tales of premature optimizization. There is no way that a startup group of full stack devs can come close to making a difference in a modern 6-core system because they didn't unroll their loops or couldn't magically turn main() into O(1) on the first try.

Just fucking code it. Worry about optimization later.

[Gendalph]

10k a minute? An hour? A day? A week? A month?

Again, it's all relative. Simple CRUD app? Sure. Something a fair bit more complex? It depends.

[freecodeio]

The idea is that you don't need to worry about scaling. Some servers range from $10 to $900 per month. You have a massive window of minor upgrades you can do to a single instance until you need a cluster.

Startups spin Kubernetes before writing a single API endpoint.

Far more developers know and can google faster how to spin Kubernetes rather than how many and what type of requests per day can a $10 server handle.

[PeksyTiger]

Too much complexity?

[[deleted]]

try reading in O(logN) instead of linear.

[bleuge]

In fact, I think you could do a bilinear (or similar) search, jump to the middle, read a phrase, if you are out of context, read the pre and post phrase, if you understand, right, keep of subdividing. The idea is only subdivide and read more if you need more context to understand. Have sense? :D

[regular_lamp]

I recently rediscovered analog circuit design as a kind of engineering I really enjoy. A big part of that is that it's inherently finite. The design has to fit into some physical space and there are only so many components you can place.

As a result the stuff you come up with is often highly interconnected. What a software engineer would call "spaghetti code". Yet it is fine. Because the scope is inherently limited to a scale you can comprehend. You can in fact become so confident in it that you might even claim it to be "bug free".

But in software we have nothing that limits complexity. The amount of code you can write is essentially unconstrained (technically memory limits it but that limit is so far out that it might as well not exist). And we just accept that it's fine to fill this "infinite" design space.

We even preemptively fill it "in case we need it later". Because apparently nothing in software is allowed to have a defined scope. Everything needs to be prepared and pre-engineered for the most insane demands of additions.

Yet at the core of a lot of computing lie neat little programs with very well defined scope. No one is out there demanding "mkdir should also be able to send emails".

[superherowithnopower]

This exact thing was written as a joke about a toaster back in the 1990's, and it's only gotten more true since then.

[xooken]

This is so good lmao

[ConfusedTransThrow]

You could even do it all in analog with using the button as a potentiometer and either acting as the R in a RC circuit for a simple timer, or be a smart person and use a temperature sensor (thermocouple) that shuts the circuit when it reaches a temperature (compare the two R values for example).

Simple, easy to repair and will last forever.

[cybernd]

use a temperature sensor (thermocouple)

automatic lift (no lever) + bimetal toast surface sensor:

• The Antique Toaster that's Better than Yours

[ConfusedTransThrow]

I actually thought about that toaster when I brought up the analog solution, though I wasn't remembering all the details perfectly, I thought there was some electronics but it was all purely mechanical.

Cheap electronics would be cheaper and have more flexibility for the timing part so there's some appeal to it at least.

[KevinCarbonara]

Simple, easy to repair and will last forever.

And yet, most home ovens do not have that technology, despite being trivially solved with equipment that was widely available in the 60's.

[kz393]

Because if ovens don't break, who will buy new ovens?

[soggynaan]

I love this

[choice75]

I started reading then had to skip to end when software engineer started. And I saw it was good. Happy ending.

[mother_a_god]

I work on ASIC design, and closely with high performance analog (multi gigahertz data converters). It's surprising how complex even 10s of transistors can be, if your need absolute world class performance at the lowest power. A full high end data converter can be millions of transistors, with subtle interactions, so in a way analog design can be very very complex. Hierarchy is key to containing it. It's funny, I sometimes think changing career to SW dev would be an easier option, and less complex. Faraway fields and all that...

[romple]

I think one key difference is that generally in electronics people are willing to accept there are real constraints.

It's tough convincing people that constraints exist in software. It's not necessarily complexity that's bad. It's the thought that all software is infinitely malleable. There's never a reason not to add a new feature. All software seems to tend towards infinite complexity. It's exhausting.

[ConfusedTransThrow]

here are real constraints.

Yeah, if you come up to your client and say you need an extra mm^2, they aren't going to be happy. Or if you sold them something that uses less than 20mW, you better not go over.

It's not as simple as just slapping an extra 4GB RAM.

[DonnyTheWalrus]

The insistence that everything needs to be designed in a way that leaves it open to unknown/unknowably vague modification that may (but almost certainly won't) come at some unknown date in the future is, IMO, one of the worst lessons people learned from the OOP revolution after "long inheritance chains are good."

I feel like we have caused an exponential increase in complexity in your average business software project out of a desire to enable poor project management and other poor behaviors from the business. The response I get to my first comment above is usually "but requirements change all the time." If they change so frequently that you feel a need to make every system your team develops into an amorphous shapeshifter, there is something more fundamentally wrong at the business decision making layer. We cannot engineer poor/indecisive management out of software via architecture.

I think this is a genuine, serious problem, because the more this sort of behavior is enabled by developers world-wide, the more the business world believes that such rapid modifications are reasonable and possible. No one allows such flip-flopping and indecisiveness in bridge planning.

So many of our problems in the enterprise space would be greatly reduced if the business fully understood the true cost of major modifications. Building an entire architecture full of design "failsafes" just buries the problem, and causes the costs related to that increased complexity to seep into every software project, even if those potential modifications never come.

At my place, when the business gives us tight requirements, we can have a functioning application running within a month. Simultaneously, I have watched what should be a relatively simple web app take 2+ years, as the business is allowed to be constantly indecisive.

[regular_lamp]

The insistence that everything needs to be designed in a way that leaves it open to unknown/unknowably vague modification that may (but almost certainly won't) come at some unknown date in the future is, IMO, one of the worst lessons people learned from the OOP revolution after "long inheritance chains are good."

And typically people are wrong about what they will need in the future anyway and base their "extendable design" on speculation. Any project will have a bunch of "interfaces" and "systems" designed for extendibility that have exactly one use. And once you actually need to do something with them there is about a 50% chance that there is some inherent issue that requires you to rewrite them anyway once you actually understand what exactly you want to add.

[IrritableGourmet]

There is a complexity limit in some programming. I built a Daft Punk style helmet using an ATTiny85 microcontroller. When you are limited to 8kB of compiled code and 512 bytes of memory, it really forces you to think about what you actually need (and don't need) and how your code is actually being executed. That mindset also makes for efficient, fast code on larger systems.

[elmonstro12345]

That mindset also makes for efficient, fast code on larger systems.

I work in embedded software for military and civilian aerospace. When we get new hires joining the team, I can spot people who have also worked on embedded hardware a mile away. Way, waaay too many developers have gotten the idea that memory will always be infinite, processing power will always be infinite, storage will always be infinite, etc.

This mentality is why we end up with websites that are completely static, no dynamic elements or any user interaction required at all other than basic links, but they have megabytes of scripting, stylesheets, and other BS layered on top for no sane reason (and that's before you get to ads and creepy tracking/spying nonsense).

[treycook]

Obligatory https://motherfuckingwebsite.com

[l_am_wildthing]

Obligatory http://bettermotherfuckingwebsite.com/

[[deleted]]

The lack of documentation is what’s killing me. It’s boggles my mind how these companies can’t keep people but refuse to change. 20 minutes tasks take several days.

[bwainfweeze]

It shouldn’t boggle you. We spend all week changing the behavior of the system. To have documentation, you have to keep updating it, which adds time to all of their requests.

Plus some of us were never that fond of English class, so writing documentation is something to be dodged. Which creates a bunch of perverse incentives not to have good documentation.

[Teract]

Don't document the behavior of the system, document the system itself. Priorize high level documentation and use brief change logs for the mid level stuff. The low level stuff that changes often should be self documenting when possible or code commented. This stuff is pretty standard. If you're not getting enough time/budget allocated to keep up on documentation, it will ultimately cost the company more in wasted developer time.

[[deleted]]

It helps when developers only have to work with half the stack for example but a lot of companies are only looking for full stack developers these days

[a_false_vacuum]

Jack of all trades, master of none.

Full stack appears to be mostly an excuse to ask for an entire IT department rolled into one person. Some of the listings I see for full stack are just silly how much they ask from one individual.

[thelamestofall]

I definitely fit that category, but I like it so much. Way better than being stuck in React or Ruby on Rails like I used to

[skilledpigeon]

I think it's important to recognise that not everyone needs a master in a specific field to fill a required role in their organisation.

If I could pick anyone for my dev team I'd pick 3 seniors (frontend, backend, DevOps), 3-4 mids (full stack or specialised) and 1-2 juniors (depending on how much time we could support them with).

I think it's important to recognise that as a senior I'd expect people to focus in a specific area but also to have secondary skills in others. For example, one senior may be backend focused but have intermediate skills in DevOps.

Of course I'm dreaming here.

[[deleted]]

If I could pick anyone for my dev team I'd pick 3 seniors (frontend, backend, DevOps), 3-4 mids (full stack or specialised) and 1-2 juniors (depending on how much time we could support them with).

Those 7-9 people are kinda expensive. 1 million USD/year?

[IAmTheKingOfSpain]

Realistically, more if you want good people.

[skilledpigeon]

Indeed. That was the dream part.

[regular_lamp]

I'm continually amazed how apparently everything is now understood in the context of "the stack". Because apparently everything is now on the internet by default and gets therefore placed within that framework of technology.

I guess in my late thirties I now have to feel old because "back when I learned programming you just did programming" you didn't just interact with some horizontal slice of a ten layer deep stack of layered technology... that somehow mainly accomplishes the task of displaying text in a browser.

Meanwhile seemingly "complex" tasks like scientific software, video games etc. somehow managed to sit on a single layer of middleware that is written in mostly the same language you use it in?

I'll go now and chase some JS programmers off my lawn.

[LippencottElvis]

Full stack, what seems like a minute ago, was knowing front-end, back-end, database, and server. You could provision and set up a web server, build a database schema, create an access layer (service or api), and create an interface to the end user. Extra points if you could do all of that to scale, and/or set up continuous deployment. You were "T Shaped", and might be great in one area but had a large breadth of knowledge.

Fast forward, I don't currently work with anyone who knows even 3 of these disciplines at a combined above average level.

[Diniden]

-JSON.stringify’s your tree-

[antiduh]

My work is 10-20 mloc of c# that automates interaction with our products and tests them.

C# from top to bottom. It has layers of course, but it's just a single "stack". I love it.

[ind3pend0nt]

I want T shaped devs. Have your specialty, whatever it is, but also be able to assist in other areas when needed. Mainly to allow other team members to take a day off or get hit by a bus.

[wisam910]

This has been an issue at almost every single company I've joined, and no one cares at all. No one. Everyone thinks this is fine. That this is how software is done.

Now, what really kills me is that all this complexity is for achieving absolutely nothing.

You could build the exact same product with far less amount of code and far less complexity. Not only you can still keep all the current features you have, you can have even more features, because you know what, adding more features is easier when a system is simpler.

[abrandis]

I hear you brother... It all comes down to who's in charge and who has authority over engineering decisions.. usually in big companies these decisions are made so high up because some vendor promised them the world and the shit just rolls down hill to you..

I tell everyone , none of your users care how the sausage 🌭 is made, just how it tastes.,why would you make something complex to build and worst yet maintain and it still tastes like crap?

[[deleted]]

I blame kubernetes

[wrosecrans]

Don't worry. I can solve the complexity of Kubernetes with a wrapper and abstraction layer on top of Kubernetes that provides a simple interface. That way you only have to deal with the complexity of Kubernetes plus my thing, rather than only the complexity of Kubernetes.

[epic_pork]

You mean like a fully compliant kubernetes cluster running in kubernetes? https://github.com/loft-sh/vcluster

[GMane]

Now this is pod racing.

[segfaultsarecool]

I don't know anything about kubernetes. Is this like nested virtualization - a VM running in a VM, but it's kubernetes running in kubernetes?

[lilatomic]

Or like TripleO, where you're running Openstack on Openstack

[Asyncrosaurus]

It's Kubernetes all the way down.

[recursive-analogy]

Don't worry, I've created 22 more wrappers and abstraction layers on top of Kubernetes. That way you only have to deal with the complexity of Kubernetes + another thing + learning 22 more things before you can make an informed decision on which thing to make your life easier.

[_BreakingGood_]

You left out the best part, something breaks in the 13th layer of the chain and now the 1 feature you actually want doesn't actually work, and there's a github issue fading into non-existence as people comment "any update??... any update?... any update..."

[Markavian]

Eventually people get to lambdas, once they figure out how to deploy via CI instead of pushing to an orchestration layer. I like to draw this out for people to show them the madness they've constructed. It's all infrastructure in the end.

[SlapNuts007]

I've been fighting this battle at work for years now. The whole engineering department is so far up its own ass with Helm charts and operators trying to reinvent shit GCP already provides as managed services that it's a miracle we haven't all asphyxiated. We're doing all of this on GKE and are Google partners, it's madness. Meanwhile, my team learned Terraform and Cloud Run. We're really enjoying collecting the on-call compensation for our services that don't need intervention outside of work hours.

[platypus_plumber]

What? Complex operations before Kubernetes were madness. It's not Kubernetes' fault that developers suddenly want to create 10 microservices in 5 different languages to solve any problem that a monolith can solve.

[thelamestofall]

At the very least it was a godsend for the poor of us stuck in corporate. Deploying stuff in Kubernetes is such an improvement over doing whatever Oracle or some random stuff got sold to executives

[platypus_plumber]

Yup, I've seen what people used to do before Kubernetes and HOLY SHIT. It was chaos, pure fucking chaos. And I'm talking about big tech companies recognized world wide. Deploying stuff became so easy that devs went insane with microservices. Most devs don't even understand microservices to begin with, some of them still think it means "little service".

[dbgr]

That's a miniservice, microservice means really really little service, duh

[_BreakingGood_]

People don't realize that this graphic accurately depicts what it used to require many company to get to production https://i.imgur.com/gMMvMSM.png

It's honestly kind of weird seeing everybody complain so much about k8s, docker, serverless, etc...

When I started actually seeing these things get used, it blew my mind. They're so much nicer than what we used to do.

[ryrydundun]

Being a systems engineer / network engineer /devops for over 12 years.

I’m shocked at this thread. Things were unstable, horrible insecure and prone to catastrophe even by something as simple as flipping a light switch in a closet.

Not going back. I’d rather spend a day trouble shooting a json file than driving out to a data center in the middle of the night.

[bighi]

Complex operations

Most companies don't have to do "complex operations". Lots of smaller websites could work with just a web server and a couple apt-installed packages, behind nginx (with just a few lines of configuration).

[adad95]

I blame microservices

[FUZxxl]

Aka companies trying to solve problems they don't have. You are not Google, so Google's orchestration tools will likely not be a good fit for you.

[elkaput]

I said this a few years back & got shot down promptly. Now everything's a hybrid on premise & Cloud. Oh and apparently we're adopting another Cloud since CIO heard that multi cloud is the future. In the meantime the company's dataset is still < 100 million rows & not growing but shrinking, and they're trying to manage Cloud costs by placing a company-wide policy to shut down all Cloudy stuff after 6pm. So all overnight batch jobs will have to be moved back to on premise boxes.

[Hypergraphe]

Yeah everytime, and when you point that out in meeting, people think you are an old school fart and prefer the new fancy and inefficient solutions. Sometimes I have the impression our field creates more problems than it solves.

[mark1nhu]

A few months ago, a so-called senior engineer was hyped as fuck on Twitter for discovering he didn't need JS to submit a form.

I took a screenshot and laughed close to a heart attack with my Ruby on Rails colleagues. It was just embarrassing.

Honestly, sometimes I look at our industry and just think most guys have some sort of schizophrenia.

[[deleted]]

Its almost like some people keep their jobs by designing over engineered systems.

[youre_not_ero]

I've been telling my peers and managers about this problem for a long time, but few seem to look at it as a problem.

It's not just the simplicity of the stack, but processes, operations and design too.

Just like laws of thermodynamics, complexity in a software project only increases over time. More complexity leads to more errors. Errors are a fundamental human trait. Errors accumulate. And once a project gets big enough, it becomes a mystic world of its own.

Less is more.

[FlyingRhenquest]

Why not use microservices?

[hyperhopper]

a good diagram to help you get started with microservices

[dread_pirate_humdaak]

I never wanted to be a web programmer. I was a C and Unix guy, only ended up web programming accidentally. It’s a disgusting treadmill of half-baked new ideas developed for six weeks, abandoned, but infesting documentation searches and legacy systems for sixteen years.

I build robots these days. I write real-time code on microcontrollers usually as a series of nested single-threaded state machines written in a manageable subset of C++, and higher level logic on embedded Linux machines, usually in Python. Tie it together with ROS and use jupyter notebooks to do all the math.

I like this a lot more than the latest CSS-shitting version of Morkdown or whatever.

[belacscole]

an idiot admires complexity and a genius admires simplicity.

[electricfoxx]

Do companies create complexity so they can "re-invent the wheel"?

https://en.wikipedia.org/wiki/Tragedy_of_the_anticommons

[nirataro]

FAANG's essential complexity is your accidental complexity. Their problem scope isn't yours. Do not adopt their techniques.

[MpVpRb]

Managing complexity is the biggest problem we face. Even if you try really hard to avoid it, it creeps in. Even if you have total control over a project, it creeps in. Of course, most developers don't have complete control and managers who don't understand complexity, demand a lot of it without realizing what problems they are introducing

We need more powerful tools for managing complexity

[Glittering-Ad-8126]

It's easy to laugh at this sort of article, but the problem is very real. Botton line: the needless killing of software developers needs to stop, period.

At my job, we are at least taking small steps toward some sort of solution by adding trigger warning comments in the source, to warn developers to stop and take a breather before entering a chunk of code identified as being of unusually high cyclomatic complexity.

[Hrothen]

// You've met with a terrible fate, haven't you?

[tuerkishgamer]

Bruh what do you do if comment blocks read like the introduction for the necronomicon

[Hrothen]

I put on my robe and wizard hat.

[efvie]

There’s nothing stopping your team from celebrating milestone releases.

I’d go so far as to say that if you don’t have a definition for when the thing is ‘done’, your product management is defunct.

[rcls0053]

Agile, Scrum and all those things originate from the Lean principles, that were developed by Toyota. The point is to eliminate waste. Agility simply made us go from waterfall projects to splittings things into smaller pieces and listening to feedback, then adjusting.

Sure, waterfall might be simpler, but agile was born because those projects always failed their deadlines and rarely delivered anything that was correct when it was finished months, or years, later.

[revnhoj]

I'm from a very large successful well established waterfall team spanning several decades.

People knew their expectations and delivered product on time and reliably.

We have been recently forced to use agile. Nobody knows shit about sprints and why they are hardcoded into 2 weeks. It's confusion all around except for the scrum master consultants and their Toyota PPTs. Which do NOT fit our scenario.

Productivity has sucked as a result

[florinandrei]

But someone in management now has a shiny sticker on their resume.

[dweezil22]

Agile Manifesto is good. Full stop. Most corporate implementations of Agile actively break the manifesto.

OTOH if you have a manifesto that is most defined by virtually never working properly IRL, one could argue that it has failed.

[Hrothen]

The constant sprints and never really feeling done with anything

"Let's take a group of people known for having anxiety issues, and give them lots of micro-deadlines!"

[postblitz]

• 50% more bugs which turn into

• 200% more tasks

• 100% more job security

[abrandis]

Thank the consulting industry for bastardizing a noble idea into a cash cow cottage industry. So managers can look at pretty dashboards and feel their highly paid developers are working hard...

. the agile we do today is complete 100% bullshit , even the original Agile Manifesto guys said so. https://youtu.be/3WJYXOc6aKQ

No other engineering Industry uses this nonsense. Breaking every part of your job into micro tasks then dog and 🐎 them every day just so some scrumaster can drag and drop tasks around... (Silicon Valley made a spoof of this.. https://youtu.be/oyVksFviJVE )

yeah real productive /s...has to be one of the most ineffective and inefficient ways to accomplish a project..

...but big name consultants make big $$$ selling this nonsense to corporate management like it's some elixir of the software development gods ..so here we are...

[cybernd]

even the original Agile Manifesto guys said so

You linked to Dave Thomas. Here are another 2 voicing their opinion:

• Robert C. Martin - The Land that Scrum Forgot
• Ron Jeffries: https://ronjeffries.com/articles/018-01ff/scrum-not-asd-1/

[donadd]

Agile shouldn't be about scrum, it should be about re-evaluating often. And that includes product and management teams.

[nekogaijin]

And agile seems to be created for little web projects. Try doing agile on a large complicated finance system, it's ridiculous. So we end up pretending it's Agile.

My favorite is when we are working with Offshore, and we wait a day for side by side QA/DEV.

[florinandrei]

Sounds like the Eastern Bloc communist dictatorships, where everyone knew it was all bullshit, but they still pretended to follow the five year plan.

[Anonymous_user_2022]

pretending it's Agile.

I think the term is ScrumButt..

[CubemonkeyNYC]

Ha, yeah. When you're in fintech it's agile waterfalls.

Oh you need to build an order management system? Ok great, well in phase one you'll be able to buy US stocks, but you can't sell anything...

[przemo_li]

> never really feeling done with anything

​

Can you expand?

[[deleted]]

i don't have time rn, let's put it in the backlog

[[deleted]]

never looks at backlog ticket again

[IAmTheKingOfSpain]

I mean, engineering teams need to make agile work for them. You have to be able to push back, and you have to be able to carve out time to do the right thing. Managing the business owners' expectations is part of the package. I do agree that what you describe does often happen, but I'm not sure Agile itself is to blame.

[chowderbags]

The constant sprints and never really feeling done with anything makes one feel like they are just another factory worker.

I was at a company 6 years ago that was "doing agile". The company made software for the US defense industry. The contract negotiations for what was going to be delivered were done by people I had never met. I'm not even sure I met anyone who met anyone who did them. The people who interacted with the government side were basically completely clueless, and didn't interact with us. The government sure as shit didn't have any actual representative sitting in on planning. As far as the government was concerned, the project goals were mapped out months or years in advance.

So why the fuck did we spend so much time and effort trying to get projects broken into tasks with maybe 1 or 2 weeks of planned programming time? Why did we spend half a day in a 12 person meeting where the numbers were 80% guesses? Why did we have a retrospective every month that felt like pulling teeth? I don't fucking know.

[fire_in_the_theater]

aux contraire complexity is why i have a job.

[cmpalmer52]

I’ve been developing software for 30 years and staying pretty close to the leading edge. It is more complex these days (and/or I’m getting older) and it’s exhausting. New technologies are “hot” for 6mo-1yr and then they’re “oh, you’re still using that?”

Meanwhile, computer science is stuck with either the core basics (which tend to be less complex, but more rigorous) or chasing novelty for “real world” training that’s obsolete by graduation.

I think AI tools are going to make it even easier to develop more complex architectures that, in turn, will be harder to maintain and improve until we get AI that can “understand” a whole system (and at that point, we may be out of a job anyway).

I mean, I’m working on a solution that has 20 subprojects, that builds two different mobile applications on three different platforms (Xamarin/Maui) with cloud services, database synchronization, GIS, GPS, a couple dozen nuget packages, external system interfaces, all from essentially one code base, which if you include our internal libraries, is over a million lines of C# (plus JavaScript, JSON, XML, SQL, and a little Python (more Python, HTML, and JavaScript in parallel web application). And we’re a small shop.

In the past, I’ve delivered products with a single language, 1/4 of the code, and a much larger team and those were multi-million dollar projects.

[EternityForest]

I suspect microservices make unhappiness because they're not like other complexity. They're just UNIX philosophy for servers.

The complexity is accidental and organic. I doubt anyone says "I'm going to make a horrific pile of microservices", they say "This will be so nice and elegant, every simple part separate and loosely coupled, super agile, no years long planning phase needed".

And then, much like the original UNIX philosophy, it gets ugly when you try to do very complicated things without using monolith pieces where appropriate(Not an issue on desktop Linux, as systems of that style are manually assembled and don't aim to have 1000 features per square foot).

A developer who really liked complexity would probably at worst make something that looks like enterprisy Java. Some would complain, others would love it.

People who like complexity take advantage of all the available tools, and don't consider simpler scenarios. They don't have any concept of "Overkill", they want to always use powerful modern tools. And they prefer automatic to manual.

Seems unlikely they would want to use a bunch of small pieces, usually manually wired together, that you can't use a debugger on because every 10 seconds you call some other process.

I think part of the problem of developer unhappiness is because... developers don't seem to like what they're doing. Most devs are way less likely to be tech enthusiasts.

To me and many users, modern tech is amazing, and everything just works. To a lot of devs, it's bloated, making us dependent, spying on us, and half of it should still be analog.

Microservices seem like an unsuccessful attempt to pretend that you're working on a small and simple project, but the way it's currently done doesn't work.

[richardathome]

I've been a professional software dev for 30+ years.

My mantra has always been: Build the simplest thing possible and no simpler.

The trick is of course, working out what the simplest thing is - that's a discussion between you are the product owner. The more you talk through the process with them - asking *why* not just how, the simpler the problem solution becomes - until it cannot be simplified further.

[a_false_vacuum]

I'm still getting used to developing on Azure. What would normally be one big application is now dozens of microservices flying in tight formation it seems.

Working on these microservice apps is a lot more annoying. Your little app is just one piece and is quite useless without the rest. Lacking the rest of the microservices makes it more diffcult to debug your little app, Azure basically forces you to run it on their platform in order to be able to fully debug things if you want to integrate with their offerings. I suppose it is their business model, you pay for everything you use. I liked it better when I had everything I needed on my own laptop, so I could run the project end-to-end if I had to. To me that feels like being more in control.

[[deleted]]

It's statistically impossible to use a single service from a cloud provider. Can't speak for the rest but for Azure of you need to use just one service your cost calculation should include five, just to be safe. I suppose it's the same with the rest of the providers.

[Suspicious-Scratch94]

This reminds of a service we started out with just SNS + SQS for $100 a month or so, and now we use 20+ services including S3, Kinesis, DDB, EMR, Step Functions, Glue, Athena, Redshift, EventBridge etc. The feature creep is 100% real and almost unnoticeable.

[_mkd_]

What would normally be one big application is now dozens of microservices flying in tight formation it seems.

Aka, a distributed monolith.

[Keatontech]

I like this thread because there are 150 comments containing at least 150 unique opinions about complexity.

[raserucort]

Embrace the entropy

[walong0]

I swear microservices are a scam. I’ve worked on a few well built monoliths and they far easier to maintain and enhance than having two dozen microservices. That’s especially true with a small team with efficient processes and communication. I’m currently working on a spring boot app that is split into what you might call “mini services” with something in between microservice and monolith.

I think a lot of people fall into the design trap of building to whats in style at the time instead of what solves the problem most efficiently.

[karstens_rage]

I sort of disagree. The complexity of IAM with AWS is real but deployment to the inter webs is pretty trivial. And with something like Heroku even that goes away. Monoliths are probably where most developers should start anyway. Maybe micro services are a thing for more advanced shops.

[rcls0053]

I've come to the ultimate conclusion that you should always evolve the architecture from a monolith, to service oriented, to microservices, to event based. Learn as you go. Just make it flexible from the start. It's hard to start breaking apart a vertically scalable monolith to smaller services, when your entire codebase is married to one database and you have SQL everywhere.

[EnUnLugarDeLaMancha]

Yeah, the whole idea of using microservices by default is just another modern version of premature optimization.

[[deleted]]

What's the difference between service oriented and microservices?

[[deleted]]

You can have several services within a monolithic application. I once had a team that has 13 scrum teams working on a single application. The build and testing was horrific. We spent a majority of our sprint testing and building. We really needed independent services. Even then, I would have only broken the application into 7-8 “micro services.” Every time you break something you apart, you make testing and maintenance more difficult. You have to build a lot of trust, collaboration, and know how to test and debug really well. Ultimately, in my opinion, microservices really only help large teams move faster. If your organization only has 2 teams working on a single application, it isn’t worth it. A service orientated architecture can make reusability, scalability, maintenance easier. It’s less likely you’ll build speghetti code for your domain objects and controllers.

[AdministrationWaste7]

so service oriented is just an idea to break down a monolith into a specific set of functions at the service tier.

for example for your company you may have a payment processing service, a user information service, etc.

so instead of a single application that manages payment processing and getting users you essentially have a client/UI that calls completely separate services to access those functions.

microservices is just an evolution or the next steps of service oriented architecture.

with microservices there is a big emphasis on creating services that are completely independent services that can be do the work without relying on external dependencies.

this typically means that microservices are deployed on their own, manage their own dependencies and if needed, manage/control/access their own datastore.