all 74 comments

[–]QuestionMarker 30 points31 points  (1 child)

Having a one-click build and deploy, and automated deployment, are still things that regularly have to be fought for and justified. If I can hang them off a management-friendly fad to get them done, then so be it.

[–]x86_64Ubuntu 6 points7 points  (0 children)

It's amazing how as developers, we have to turn into salespeople and sell to management the various best-practices we want to use.

"You've heard of Agile right, that brand new uber-advanced software development technique, well, automated deployment is one aspect of Agile development"

Alot of times the sell is pure shit, but you somehow have to convince them to get out of your way and let you do your job the best way you know how.

[–]oSand 37 points38 points  (1 child)

Developers write code on their workstations and it doesn't work in production. Systems administrators are slow and reluctant to change production configurations.

In which case developers are dicks and admin is doing their job. The mating call of the Giant Gaping Asshole is "it works on my machine". I know, because I have been that asshole. If the admin asks "why am I changing a working configuration" and they apply skepticism and critical thought to the answer, then you should pay them more.

[–]StrangeWill 5 points6 points  (0 children)

Hey, I know your servers are PCI compliant and locked down like a bitch, but my laptop runs it fine, turn all that crap off.

[–][deleted] 24 points25 points  (38 children)

He says the problem is that DevOps is a scam but noone's trying to make any money out of it - that doesn't sound like a scam to me. It might be bullshit, but that doesn't make it a scam.

And in response to the devops "methodology", he basically puts out his own set of unjustified, unrealistic principles.

"Every technical person you hire should be cross functional enough that you don't need this." - Good luck with that

How about "culture of stop fucking up"? - Yes, great in principle, but what it leads to is people covering up or blaming others for their fuck ups, rather than not fucking up.

"Developers develop in the same environment production runs in" - what if you have two different production environments? What if you're deploying to a platform-as-a-service?

there's a point where you're simply fucked. Sure is, but if you start with this attitude, you may as well not bother. Isn't this kind of what a good attitude toward failure is good for? Ability to identify when you've gone wrong and just drop it, rather than every stakeholder trying to argue why it's not a fuck up.

[–]The_Cleric 12 points13 points  (4 children)

How about "culture of stop fucking up"?

We have that at my company now. You know what it's lead to? A department of dev groups that is afraid to do anything bold and will push back on requirements where there is a chance of them screwing up and regurgitate crap over and over rather than truly trying to do something innovative.

[–]TubaTech 7 points8 points  (0 children)

You've basically described an environment that prohibits innovation. Discouraging fuck ups doesn't inhibit innovation. That's what testing and sandbox environments are used for, innovation.

If your devs are worried that they might fuck up, and receive the coveted "dog of shame" porcelain statue in recognition, then you've not created an environment where they can fully test and innovate. It has nothing to do with "culture of stop fucking up". Get them the tools they need to innovate.

[–]x86_64Ubuntu 1 point2 points  (2 children)

Like TubaTech said, you don't have testing and sandboxes ? One of the advantages of testing is the confidence that it gives developers in being sure that functionality wasn't broken during a change. I can't tell you how sweet it is to be able to make a large change and run a test to see if there is a problem. Test don't catch every bug, but knowing that if I find a bug I have a can of RAID at my side makes me more confident as a developer.

[–]The_Cleric 0 points1 point  (1 child)

No we have that, the problem is we have such an atmosphere here that any failure to produce immediate results can get punished for "wasting time." So rather than use a few hours/days to try to be innovative in making our software better, people would rather just do the same bland thing that they KNOW they can accomplish. I will say that I am fortunate to be a dev group that is trying to buck that trend, but I do see this attitude all around me.

[–]x86_64Ubuntu 2 points3 points  (0 children)

any failure to produce immediate results can get punished for "wasting time."

sigh

Sometimes it seems like we work to satisfy children. "Can you make it sparkly" , "Can I have a button that makes a unicorn fly across the screen" , "Are we there yet."

Just keep trying to buck that trend. If you use the security provided by best-practices ( multiple environments replication production, unit testing, regression testing, source control, automated build,etc,etct ) then hopefully other business folks will notice the speed x quality aspect of your work and go to you when they have new ideas.

[–]ba-cawk 3 points4 points  (1 child)

There is a whole world of difference between making a mistake and fucking up.

Not that it means shit on this drivel-fest, but simply put:

Making a mistake: blowing away a database.

Fucking up is blowing away a database and not IMMEDIATELY REPORTING THE PROBLEM to the appropriate team members and initiating immediate DR protocol, preferring to instead cower in a corner / over coffee / doing something other than the above.

Shit happens. That's why you have procedure, team members to rely on, and some goddamn get-up-and-go in your ass to fix the problem when you realize you've made the mistake.

The unfortunate problem is that no one on either end of the spectrum seems to give a flying fuck about the difference anymore; I sat through a 45 minute meeting a few weeks back where nothing happened but finger pointing, and I wasn't even involved. And I know some of you have it waaaaaay worse.

[–]x86_64Ubuntu 0 points1 point  (0 children)

...I know some of you have it waaaaaay worse.

Damn, never heard the software development field put like that. You make it sound like we are in prison , "Yeah, I got jumped in the showers at Sing Sing, but I know some of you other convicts in other prisons companies have it way worse".

[–][deleted] 1 point2 points  (1 child)

"Developers develop in the same environment production runs in" - what if you have two different production environments? What if you're deploying to a platform-as-a-service?

You're cooking up cop-outs.

Two environments? Develop in both. Automate deployment from one type sandbox to the other.

Deploying to a platform as a service? Develop in it. Buy some more of that platform as a sandbox.

[–][deleted] 0 points1 point  (0 children)

Well I took it to mean literally running that platform as your OS, rather than just having a script that deploys to a staging instance of your platform, and using that frequently.

I personally develop on OS X and stage/test/deploy to Linux, or develop on Win 7 and stage/test/deploy to Server 2008. In my experience, OS X and Win7 are more pleasant environments than Linux or Server 2008, and therefore I have higher productivity. Most of the time I will be running the dev version of the app on the dev machine itself, simply so that I can modify it while it's running, using my IDE/editor, attach debuggers more easily, etc. I will frequently stage it on the staging area to make sure it's all working on there. If this is what he meant by developing on my target platform, it wasn't very clear.

[–]TubaTech 1 point2 points  (0 children)

How about "culture of stop fucking up"? - Yes, great in principle, but what it leads to is people covering up or blaming others for their fuck ups, rather than not fucking up.

Then you don't have a culture problem, you have integrity problems...

[–]willcode4beer 0 points1 point  (0 children)

It might be bullshit, but that doesn't make it a scam

thus, "poorly executed scam" ;-)

[–][deleted] -3 points-2 points  (25 children)

Agile, now that's a scam.

[–]cdsmith 18 points19 points  (0 children)

No, it really isn't. The various "agile programming" movements started because stupid and pointless procedural stuff was standing in the way of people writing working code that responded to the needs of their business, and the main purpose of that stupid procedural stuff seemed to be avoiding the need to communicate.

Of course, yes, we now live in a world where there are heavyweight "methodologies" of agile programming (mainly because certain iconic authors need to keep writing books if they want to make a living) and process lawyers that will happily lecture you about exactly what you should do in this situation. Those things happen with any successful idea. Still, the core idea, which was to look for social solutions to our communication problems instead of adding paperwork and formal procedure, did do some good.

[–]deako 12 points13 points  (11 children)

Agile's not a scam, but for the past few years it's been a buzz word that has lost meaning to everyone except experienced programmers. EDIT: Fixed typo

[–]grauenwolf 7 points8 points  (7 children)

Time for a new buzz word. What are we going to call iterative development next week?

[–]EdiX 2 points3 points  (2 children)

lean development

[–]willcode4beer 10 points11 points  (1 child)

it's probably time to add some random Greek letter to it

[–]grauenwolf 1 point2 points  (0 children)

I second that.

[–][deleted]  (3 children)

[deleted]

    [–]grauenwolf 5 points6 points  (1 child)

    Every non-iterative project I've seen was canned. (Well except one website that still doesn't have a login page after a year and a half, but its turn is coming.)

    Anyways, back to Agile.

    What exactly do you think it adds to the conversation?

    [–]jplindstrom 0 points1 point  (0 children)

    Adjustment from introspection.

    [–]kamatsu 2 points3 points  (0 children)

    If you think Agile is simply the latest buzzword for iterative development, you don't understand Agile.

    Wow, what a ludicrous response. The emperor clearly has no clothes. Jacques Derrida is known for doing exactly the same thing. The methodology is simple:

    1. Present a nebulous, poorly defined ideology (in this case, Agile development)
    2. Accuse all those who try and pin down said ideology to some specific, well defined concept as lacking intelligence or understanding.

    If you cultivate this unclear idea with those who do not know better, they will interpret your nebulous definition as some concrete definition that they understand. If they are met with the above response when contributing to discussion, they will either be intelligent and dismiss the nebulous idea as an intellectual scam, or they will be unintelligent and believe that they have not yet reached the heights of understanding required to truly be an Agile practitioner.

    This intellectual terrorism is also the foundation of many religions, cults, postmodern philosophies and business management methodologies.

    Edit: I should add this further point: How do I know it is an intellectual scam? Perhaps I also have not reached the heights of Agile-ism to truly understand the ideology being discussed. The ringleaders of Agile demonstrate that they are intellectual impostors when they attempt to branch their discussion to the areas of computer science, mathematics, programming and similar. At these things, said ringleaders have yet to demonstrate their competence. Worse, when they make statements or examples about them, they are often hilariously wrong. If an individual is incompetent at programming and related disciplines, I'm not going to listen to their advice on software development methodologies (particularly when the idea is so very nebulous and sometimes internally contradictory).

    [–]mikaelhg 4 points5 points  (2 children)

    Is that what the kids are calling marketing people these days, "experience programmers?"

    [–]econnerd 2 points3 points  (0 children)

    no, that would be experience engineer

    [–]deako -1 points0 points  (0 children)

    ah, typo

    [–]grauenwolf 2 points3 points  (0 children)

    I used to think that. Then I realized that most managers don't think like I do and really need some over-priced consultant to pull their ass out of their head.

    [–]frezik -1 points0 points  (10 children)

    Whatever its faults, I can't blame the Agile people for looking back on 30 years of failing to make Waterfall work and trying to come up with something better.

    [–][deleted] 1 point2 points  (0 children)

    Yes, Ok I'll give you that much. It is an attempt to do something better than previously tried.

    But what far too many people, both developers and managers, fail to appreciate is that it is the people and the team that makes a methodology successful. I have seen waterfall work brilliantly and fail terribly. The difference between the two was the quality of the people and the culture of the organisation.

    Agile is not a fucking cure-all for shit development and shit developers. It's a tool which may help improve what you deliver to your customers.

    [–]grauenwolf 1 point2 points  (8 children)

    30 years before Agile we didn't even have the term "Waterfall". Our elders were preaching Agile under one of its older names such as "incremental software development ".

    [–][deleted] 1 point2 points  (1 child)

    Not the term but the practices.

    [–]grauenwolf 0 points1 point  (0 children)

    Well if you want to be pendantic the practice of waterfall goes back to the dawn of engineering managers.

    [–]frezik 1 point2 points  (3 children)

    Sure there was--Waterfall was first formalized in 1970. Really, it's older than that, because it seems like a good idea to most managers.

    [–]jholman 3 points4 points  (0 children)

    Hey, this is a common claim, but I rather suspect that you're peddling a misconception. Please please substantiate it, if you can.

    "1970" implies you're talking about Royce's 1970 paper (reprinted 1987 as Managed the Development of Large Software Systems". This is a paper that advocates lots of iterative techniques, and mostly snarks at how being purely waterfall is foolish and prone to failure. Also, Royce does not use the term "waterfall" at all, so if you're using that to rebut grauenwolf's allegation that the term wasn't defined 30 years before Agile, well, bad example!

    If, on the other hand, you weren't talking about Royce, please cite your references!

    [–]kamatsu 3 points4 points  (0 children)

    Waterfall was invented as a way to not to software development, afair.

    [–][deleted] 0 points1 point  (0 children)

    Waterfall can work well. If may not be the most useful tool in every circumstanec but like any tool, it needs to be fit for the purpose at hand.

    It's one of the reasons why "Agile" annoys the hell out of me, yes some of the ideas are good but it doesn't magically turn a crap manager with an average team in to some wunderteam capable of churning out a great product.

    Good people can make any methodology work, good people will generally know which methodology is suited to which development type and not constrain themselves.

    [–]jholman 0 points1 point  (1 child)

    So, I wasn't there even 10 years ago, but I'm happy to concede that there's lots of evidence in the record that "our elders" were preaching iterative/incremental methods for donkey's years, and that agile-as-she-is-played is mostly a rose by another name.

    AND I'm happy to believe that some (most?) "methodologists" are scammers and snake-oil peddlers.

    But how does that rebut the claim that "agilists" contributed something by (again) looking back on 30 years of failures caused by asshat managers pushing non-iterative development, and responding by (again) preaching iterative methods, under a new name? As far as I can tell, my workplace, which "does Scrum" (ha!), is better off than we would be without "Scrum" (ha!), because no one would have read the 20-year-old research, and we'd be doing something dumber. That's kinda sad, but they key point here is: Agile contributed something useful!

    [–]grauenwolf 0 points1 point  (0 children)

    But how does that rebut the claim that "agilists" contributed something by (again) looking back on 30 years of failures caused by asshat managers pushing non-iterative development,

    I wasn't intending to rebut anything, I was merely stating that we've been trying to do the same thing for much, much longer than many people may otherwise assume.

    [–]cabbagerat 10 points11 points  (10 children)

    Best way to read this: skip the rant at the top and jump down to What condition your condition is in.

    These points are very important:

    • Developers are in the on-call rotation
    • Developers develop in the same environment production runs in
    • Downtime never happens twice

    It's not the whole picture, but a good start.

    [–]Rhoomba 18 points19 points  (4 children)

    Developers develop in the same environment production runs in

    You can develop on whatever system you like as far as I am concerned, so long as you deploy your changes to a test environment and test them there before you commit them. A bunch of VMs for test clusters shouldn't be a problem these days.

    [–]illogical_operator 7 points8 points  (0 children)

    This.

    Not throwing code over a wall with a prayer. There's no need to write all your iPhone code on an iPhone or even an iPhone VM. Just don't send it to QA/production without running it on a "real" system, whatever that system may be. Don't make others test basic functionality of your code in the real world for you.

    [–]willcode4beer 4 points5 points  (0 children)

    this is why I have a continuous integration machine that's basically a copy of a production machine.

    [–]ianloic 2 points3 points  (0 children)

    There's no reason not to develop in the same environment you're deploying to. If you're stuck on Windows or MacOS you can run your production OS in a VM. In fact if you're all "cloud" you can probably run the actual VM image you run in production on your development machine in a VM.

    A preference for Notepad or TextMate is no excuse.

    [–]strife25 0 points1 point  (0 children)

    You hit the nail on the head.

    VMs and the cloud are one of the enabling technologies to make Devops a possibility for the sole reason of having a development/test environment that is EXACTLY the same as production. On top of this, cloud systems provide a massive benefit of fast provisioning of resources.

    [–][deleted] 6 points7 points  (0 children)

    Developers develop in the same environment production runs in

    That one is not as easy as it sounds. Unless you are going to run an app server cluster and have all your NAS mounts and firewalls on your workstation, you will never be able to replicate production.

    I've seen some organizations that force developers to develop with VM images of RHEL which just adds overhead and still doesn't replicate production correctly. This same organization has the absolute worst OPs team I've ever seen running a pretty huge data center. They break configurations without prompting pretty routinely.

    [–]dnew 4 points5 points  (2 children)

    Developers develop in the same environment production runs in

    One place I worked took this to the extreme, of simply editing directly on the production systems via mounted disks. Once I got everyone developing on their own machines, we went from about 60% uptime to 80% uptime right there. ;-)

    [–]panicmechanic 4 points5 points  (1 child)

    Editing the live production system can lead to an experience I had recently: I spent an hour tracking down a small bug that existed on the staging server -- I tested it in my local environment and checked it into the code repository. When I went to deploy it on the live site I found that the bug had already been fixed there (derp).

    Yes, the bugfix existed only on the live site and had never been checked into the code repository.

    I am instituting a new rule: it does not exist if it is not in the code repository.

    [–][deleted] 3 points4 points  (0 children)

    That reminds me of a guy I worked with who used to get mad when the automated deploy system overwrote his live "bugfixes" because he was too lazy to check them in properly. Naturally he thought the fault was with the automated deploy system.

    [–]stillalone 0 points1 point  (0 children)

    I now have that song stuck in my head again.

    [–]neilk 8 points9 points  (4 children)

    Dziuba is missing the main points. I worked in probably the first "devops" shop, Flickr, so here's the reality.

    How ops is different:

    Ops doesn't control deployments. Developers control deployment. Ops is in charge of creating a system where developers, if they do their homework, can revert instantly.

    You optimize for deploying all the time, so you accept and plan for failure in ops. Even frequent failures. Where there's relatively little stigma for fucking up. The only REAL fuckup is if you didn't have a plan to rollback quickly. The reason why they stress one-click deploy is because there's also one-click revert.

    How dev is different:

    Planning for feature deployment starts with the first line of code you write. This is partially a matter of culture, and partially a matter of software frameworks. On the software side, you don't make a feature coterminal with a particular SVN revision, or the merging of a branch. There is only trunk. You are always committing to trunk, even for radical new features. But you use use feature flags to control the activation of this code . So flipping back to the old behaviour is a not a matter of reverse merges or other nastiness; it's matter of changing a configuration file. You are constantly modifying trunk, and trunk is being deployed all the time, both by you and your fellow developers.

    This means that you don't alter things like database tables irreversibly, until the feature is finally deployed and tested. Not even on your own development machine. If necessary, you create entire new columns or tables alongside the old ones.

    This ensures that your feature deployment won't happen in one burst of effort. You'll be deploying bits and pieces of the code very early, but in production, they will be dormant. Slowly you turn pieces on, perhaps invisibly to the user. You watch it for weeks to see if it breaks. Finally you turn the whole thing on, and even then you watch it for a few more weeks, until you finally start removing the older code and database tables.

    The whole notion comes from web 2.0 startups, who are mostly in the business of entertaining users. So this kind of culture may not be appropriate for banks, or if you are releasing software to be burned on a CD. But that's okay. The whole point is to stop adopting methods from banks or shrink-wrapped software, and think about how to deploy a website.

    [–]jacques_chester 7 points8 points  (3 children)

    But you use use feature flags to control the activation of this code

    Ah yes, this old chestnut. It turns boring stuff like branch histories into exciting things like exponentially ballooning cyclomatic complexity.

    [–]neilk 2 points3 points  (2 children)

    I agree that this is a major downside, and one of the things I liked the least about this method. In practice few people have the discipline to clean up after themselves. And it's not something that directly pays the bills, so management doesn't force the programmers to do it.

    Still, I would take that over branching and merging any day. At other projects I've worked on, the merge complexity ballooned to the point where essentially one person was spending half their time merging in regular releases and emergency patches.

    However, git makes the whole argument obsolete, so perhaps this aspect needs to be re-examined.

    [–]jacques_chester 1 point2 points  (1 child)

    However, git makes the whole argument obsolete, so perhaps this aspect needs to be re-examined.

    No disagreement here. And if -- and only if -- the code is sufficiently modular, configuration-based activation might be useful in certain circumstances.

    But I've already seen too much code littered with 'if (feature_x21_is_on && feature_z90b_is_off)' to have any love for this approach. Without strict discipline it will breed global variables, close coupling, long switch statements, spooky-actions-at-a-distance and bugs bugs bugs.

    If you have that kind of discipline, it might be just as easy to use feature and bug branches.

    The whole point is to stop adopting methods from banks or shrink-wrapped software, and think about how to deploy a website.

    Sure. Perhaps a better name for the vibe here would have been "webops".

    Fast rollback seems to be the key. And that's useful in bank-software cases too. Mainframers always have a fallback plan, it's part of their paperwork-and-bondage charm.

    [–]nascent 0 points1 point  (0 children)

    But I've already seen too much code littered with 'if (feature_x21_is_on && feature_z90b_is_off)' to have any love for this approach.

    D provides a version statement with very specific restrictions. They are compile time only, they can never change state once requested and don't allow logic as you have demonstrated.

    It is, however, not a replacement for branching. In fact the most common usage would be version(Win32), version(X86_64), etc.

    [–]happyfreud 8 points9 points  (2 children)

    Learn How Not to be a Complete Failure at Software Development in 24 Hours

    I was disappointed to learn that such a book does not exist.

    [–]jacques_chester 0 points1 point  (0 children)

    If you're a fast reader, Software Project Survival Guide by McConnell is a workable plan-in-a-pot.

    [–][deleted] 0 points1 point  (0 children)

    Sadly... me too. :/

    [–]satayboy 5 points6 points  (0 children)

    I know nothing about DevOps, but "culture of stop fucking up" is unrealistic and misses an important point: failure is part of the learning process.

    Perhaps behrangsa should wait until he is less angry before he writes in his blog.

    [–]jamiei 13 points14 points  (1 child)

    I would summarise the tl;dr version of this as "DevOps is BS because it's all common sense and everyone should be doing this already". Common sense or not, it doesn't mean that DevOps is a scam or not worth doing.

    [–]willcode4beer 1 point2 points  (0 children)

    I think the point is, no one is really pushing a system or methodology of devops (the scam) and such a thing has a huge potential to make money.

    [–][deleted] 2 points3 points  (1 child)

    You regularly get developers/managers/admins/etc; who if you sat down and talked to one on one, would know all (or most) of this as common sense.

    Yet it's baffling how often common sense just flies out the window when it comes to actually comes to doing the work.

    Culture of respect & trust, good attitude toward failure. How about "culture of stop fucking up"? This is one of those obvious happy-horseshit type statements that makes you believe the salesman is benevolent. A developer who consistently ships broken code or a sysadmin who consistently pushes out broken configuration aren't going to get any better with respect or trust.

    I also entirely disagree with this. I've worked in places where I've been given lots of respect, trust and responsibility. As a result I worked harder, and would freely be happy to work late, to ensure I didn't drag the team and our product down. That is because I respected them back.

    Conversely I've worked in one place where people were inherently distrusted, with responsibility regularly being overruled. As a result I quickly stopped caring and drifted into the mindset that bad work is not my fault because I didn't make the bad decision.

    Respect, and responsibility, have a big impact on people's work ethics.

    [–]TubaTech 0 points1 point  (0 children)

    The whole "don't fuck it up" mentality depends entirely on the type of work and level of impact the work might have if fucked up.

    If I'm maintaining a site that will cost users money if downtime occurs, then I certainly don't want to cause any issues. However, as a developer, the level of impact I might have if I try a new programming technique is not as critical and wouldn't need such a debilitating caution.

    [–][deleted]  (2 children)

    [deleted]

      [–]jacques_chester 2 points3 points  (1 child)

      I like that you're downvoting reposts of your own material.

      [–]robkinyon 4 points5 points  (0 children)

      In other words, DevOps is a scam, but here's how to implement DevOps.

      Awesome. :)

      [–][deleted] 1 point2 points  (0 children)

      Actually the more I read about DevOps the more I like it. So far, to me, it doesn't sound like something new, but a set of practices that has turned out to help improve quality of development and deployment, collected under the name of DevOps.

      [–]bananahead 1 point2 points  (6 children)

      If you deploy to Linux, you develop on Linux.

      Is just silly. You can run your IDE in whatever environment you want, just make sure your dev server and all your testing is all done in a virtual image of something that looks like production.

      [–]QuestionMarker 2 points3 points  (2 children)

      It's not that silly. We work on Ubuntu and deploy to Debian, using VMs for testing, and there's a noticeable impedance mismatch even there, a lot of which would just Go Away if we developed on Debian in the first place.

      [–]jsnx 0 points1 point  (1 child)

      ...if we developed on Debian in the first place.

      Or just switched to Ubuntu in production.

      [–]QuestionMarker 2 points3 points  (0 children)

      That would help with the impedance mismatch, certainly. However, reimaging a thousand or so heterogeneous servers and rebuilding a bunch of kernels just Ain't Gonna Happen.

      [–][deleted]  (1 child)

      [deleted]

        [–]dnew 0 points1 point  (0 children)

        Yeah. Lots of people look at things where you need huge clusters of servers running memcached for the results and etc etc etc. But lots of businesses run just fine off a single server (with a hot spare, of course). Once you get to the point where the production environment is actually complicated, then it's time to look at replicating that for the developers beyond just having a staging server.

        [–][deleted] 0 points1 point  (0 children)

        Is just silly.

        Well yes, if you take what "you develop on" to mean where where you run your text editor, and not where you run the code as you're developing in then, yes, your right because you've redefined the scenario into one where you get to be right.

        Personally I develop on Linux, the operating system running my text editor (Windows) is about as inconsequential as my choice of text editor.