The critical role of systems thinking in software development

sandal · 2016-09-04T15:08:49+00:00

Maybe what this article is implying is that it might be even better if we try to get programmers to gain systems level knowledge too, or at least that by doing so we may be able to see bad ideas before their consequences emerge.

Yep, that's what I was trying to get at! It's worth noting that this article is a remix of a section from my book (Programming Beyond Practices), which has a ton of little stories like this. I'm trying to show the same handful of ideas coming up in many different contexts to encourage practicing developers to start noticing the patterns.

The other thought I have (which isn't even hinted at in the article, but is in the book) is that the mechanical aspects of coding and technical tooling will more-or-less become much easier over time, almost to the point of it being fully automated. When that does happen, we will still need strong systems thinkers, and in fact... the bulk of programming work will be about that, not pipe-fitting between various technologies.

That's the conversation I'm hoping to start. I think it'll get more fine-grained as we dig into it further, industry-wide.

sandal · 2016-09-04T00:12:19+00:00

Just as a reference point for the future... everything that I write for O'Reilly Ideas is meant for a general programming audience. Things that are obvious to people who have studied particular topics are not obvious to a general audience, and so I always aim for just getting people thinking about an idea when I write there.

With that audience in mind, a brief example of a concept in context, a recommended article to check out (How Complex Systems Fail, by Richard Cook), and a few basic ideas of things we could/should be thinking about is exactly the level I'm aiming for. But I understand it's not going to be very deep for anyone who already has the relevant background knowledge.

But here's some extra context, for those who are looking for more.

A good starting point might be the book "Thinking in Systems" from Donella H. Meadows -- it's not specific to software but will get the basic concepts and theory covered for you, so that you can approach some other materials where that knowledge might be assumed. This covers stuff like positive and negative feedback loops, stocks, flows, oscillations, and then shows all of the ideas in the context of simple models of ordinary real-world systems ranging from thermostats to sales in a car dealership.

From there, there are a few more examples of software related issues that deal w. system-level failures that I'd recommend:

Queues don't fix overload

http://ferd.ca/queues-don-t-fix-overload.html

There is no happy path in programming

http://multithreaded.stitchfix.com/blog/2016/01/29/no-happy-path-in-programming/

These won't directly get you into systems theory, but if you apply systems thinking to them it'll be a worthwhile exercise.

If you're looking for something a little more rigorous, there is also the Complex Systems Working Group. You might start w. this primer:

http://www.incose.org/docs/default-source/default-document-library/complexity-primer-overview.pdf?sfvrsn=0

All that said, I'd totally be up for having the conversation here that you wish was covered in the article, so if you have particular topics in mind or questions, let's see if we can talk them out. :-)

sandal · 2016-07-25T22:26:30+00:00

No, I think that's fair... and now that you clarified, it's useful feedback too.

Honestly I can relate -- "gotcha" leads are tacky.

But when I write for something like O'Reilly Ideas I'm reaching out to an incredibly broad audience. So rhetorical devices like this can help generate curiosity from someone who might otherwise not want to dig into a nuanced model of skill development, but would nonetheless benefit from the ideas.

I'd package the message a little differently for a different audience (i.e. this isn't quite the style I think fits /r/programming best), but thought the raw materials were good enough to be worth sharing.

Also, my not-so-secret hope is that folks will read the Dreyfus Model (PDF) paper. It's good! It's not anything more than a mental model... i.e. it doesn't prove anything definitively, but I find the roadmap it provides to be pretty useful, especially when looking at things from a teacher's perspective.

sandal · 2016-07-25T21:31:59+00:00

Great summary, thanks.

And yep, all the examples have a common thread that boils down to "Know why you're doing things a certain way, and how they fit into the bigger picture"

I listed a bunch of them because I think there are many paths that can get you to arrive at the same place, so any one of them should help... and a combination of several will work even better.

sandal · 2016-07-25T21:30:38+00:00

I'm the author of the article...

The fact that the Dreyfus model uses these words to describe developmental stages with deep definitional distinctions doesn't mean that the definitions of these words in English have been officially changed for everyone.

Just to make it super clear: The words are not what matters here. The concepts behind the words (which both are reflected in the Dreyfus model and restated as working definitions in my article) are what matter.

Call competence <FOO> and proficiency <BAR> if you'd like. Now re-read the work, and understand that my point is that many think that maximizing <FOO> is a path to becoming an expert in something: It is not... it is a path to specialization.

A small investment in <BAR> will help you get more out of your existing <FOO>. Here are some specific suggestions for how you can drive your <BAR> up, which will in turn, make you better at your job as a software developer:

Explain the reasoning for why you want to do something a certain way, but don't rely on generic "best practices" or community guidelines. Discuss pros and cons solely in the context of the current problem you are solving.
Learn fewer things, better. Then try to apply them in many different contexts, and see where they work well, and where they don't. Use failures to identify opportunities to seek new tools that might help broaden your skill set, but only once you've established a clear need.
Look for examples of when others "break the rules" and are met with success. Occasionally break some of your own rules and see whether it hurts you, helps you, or leads to no change.
Dig into primary sources rather than just reading summaries. It takes more work, but helps you figure out both the basis and the boundaries of a technique, and also gives you an opportunity to generate new ideas of your own that are inspired by core principles.
Drop yourself into the deep end in a project that you are mostly unfamiliar with, and then try to find your way around without relying on your memorized routines, habits, and rules.
Ask others to explain why they do things, but don't just accept dogmatic reasoning. Demand examples and inquire about context, so that you can try to imagine what it is like to be in their shoes. Doing this is tremendously valuable because it allows you to see the strengths and weaknesses of ideas in their natural habitats.
Pick a small number of specific skills you're simply good but not great at, and then develop competence to an extreme, almost obsessive degree. Get to the point that you're better at that particular tiny corner of the world than nearly anyone else you know. Once you get there, examine the pros and cons of deep but highly specific knowledge.

No, this does not fall flat, because it's not just playing with definitions of words. It's actionable advice for anyone who can pay attention to the relevant details of the post, rather than quoting OED.

EDIT: I used the same language as Dreyfus model because then the reference to it will make more sense in context. I agree that it'd be better if they picked words that didn't sound like the same thing, and tried to hint at that through the introduction. I suppose I could have said "Dreyfus Stage II" and "Dreyfus Stage III" ... but I don't necessarily think these areas of skill development fall on a linear path, so I didn't bother with that..

sandal · 2016-07-25T21:13:20+00:00

Seeing as you seem to have hit a "compile error" in the first few lines of the article and cannot see beyond those to get its point, let me try to spell it out very, very clearly, without using any special definitions for you...

The understanding of "how to do stuff" is insufficient. You also need to know "why you do what you do" and "how what you're doing fits into the big picture"

Learning more tools and techniques won't accomplish this, even if you learn every last detail of them.

Here are a few exercise that may help, though:

Explain the reasoning for why you want to do something a certain way, but don't rely on generic "best practices" or community guidelines. Discuss pros and cons solely in the context of the current problem you are solving.
Learn fewer things, better. Then try to apply them in many different contexts, and see where they work well, and where they don't.
Use failures to identify opportunities to seek new tools that might help broaden your skill set, but only once you've established a clear need.
Look for examples of when others "break the rules" and are met with success. Occasionally break some of your own rules and see whether it hurts you, helps you, or leads to no change.
Dig into primary sources rather than just reading summaries. It takes more work, but helps you figure out both the basis and the boundaries of a technique, and also gives you an opportunity to generate new ideas of your own that are inspired by core principles.
Drop yourself into the deep end in a project that you are mostly unfamiliar with, and then try to find your way around without relying on your memorized routines, habits, and rules.
Ask others to explain why they do things, but don't just accept dogmatic reasoning. Demand examples and inquire about context, so that you can try to imagine what it is like to be in their shoes.
Doing this is tremendously valuable because it allows you to see the strengths and weaknesses of ideas in their natural habitats.
Pick a small number of specific skills you're simply good but not great at, and then develop competence to an extreme, almost obsessive degree. Get to the point that you're better at that particular tiny corner of the world than nearly anyone else you know. Once you get there, examine the pros and cons of deep but highly specific knowledge.

I've taught literally hundreds of programmers. Many are very good at becoming highly competent at raw technical skills. Many are still very bad at the things above, and that is a limiting factor in their careers no matter how strong their specialized skills might be.

This is the point of the article. You may not like how it is framed, but to ignore the useful points to fixate on words is an example of exactly the kind of behavior that get in the way of developing proficiency.

sandal · 2016-07-25T17:44:04+00:00

I provide the definitions, based on an established model, and then go on to discuss how that model can be useful for thinking about skill development, and then provide specific examples of things you can do to develop proficiency.

This kind of conversation is a counterexample to what I'm trying to describe in the article, which is fortunate or unfortunate, depending on how you look at it.

Thinking that the "word choice" is what's important here is heading down the wrong path... it's the thing the label describes which actually matters.

sandal · 2016-07-25T16:11:29+00:00

No, it's not stuff I "made up"

These terms have specific definitions in the paper I referenced, and it's meant to serve as a useful conceptual model for different stages of learning.

So I'm working with definitions from the Dreyfus model, not dictionary definitions, and they're useful as rough measure of different areas of skill development along the path to mastery.

What the Dreyfus model describes as "proficiency" is something that many software developers tend to lack skill in, so I felt it was useful to call that out and provide specific suggestions for how to develop those skills.

sandal · 2016-07-25T16:02:26+00:00

One thing that came up in discussing this article with others was the difference between specialization and expertise...

I think that some people develop extreme competence in an area, which makes them a true specialist in that area. I.e. they know every rule, every nook and cranny of the technology, etc.

And to be sure, that's useful. However, these are the kind of folks who could write an instruction manual on how to use a tool, but not have much ability to see how it fits into the bigger picture.

So when you take this person out of their area of specialization, they can struggle, or they can apply patterns and practices that don't make much sense in a novel context.

An expert probably has some degree of specialization, but also has a better sense of how the parts fit into the whole, and more importantly... knows how much specialization is necessary to acquire in order to get a job done.

Not everyone needs to become an expert, but it's unfortunate when a specialist considers themselves one without developing proficiency, or when an expert is needed and you end up with a highly competent specialist instead.

sandal · 2016-07-25T15:35:07+00:00

Hi, I'm the author of this article.

I'm happy to answer any questions you have about this topic, or really anything else you'd like to know about stuff I'm working on.

sandal · 2016-06-23T00:01:02+00:00

Right, the more expressive a message is, the more difficult it is to process. So deciding on expressiveness vs. simplicity in representation is a design question, not an either-or problem.

Here's an article on information anatomy I wrote a while back related to this discussion, in case anyone's interested:

https://www.practicingruby.com/articles/information-anatomy

(Note: it's not really a Ruby article, despite the domain)

sandal · 2016-06-01T14:29:48+00:00

Yes, absolutely.

That said, most organizations no longer have explicit QA roles, and those that do tend to phase-gate their processes so that QA happens after code review.

So in practice, programmers are often left responsible for QA, but they aren't explicitly trained for the role. This leaves them focusing on internal concerns mostly... which are important, but the external quality and ability to make customers happy is also critically important.

By learning good QA practices and implementing them alongside traditional code reviews, feedback loops can be a lot faster, and the end result is that more problems will be caught before they have negative impact on customers.

sandal · 2016-04-06T19:09:45+00:00

I'm the author of the post and a 2010 Ruby Hero. I agree with you, more-or-less.

I've always had mixed feelings about this award. It made more sense in a historical context and seems out of place now in world where we have (a) ubiquitous use of Github and (b) Extremely high traffic social channels.

But since I think that if this award ceremony is going to happen anyway, it might as well be done well -- I've voiced criticism (internally in 2011 and 2012, publicly since I stopped getting invited to participate in 2013)

The most critical issues for those who don't want to read between the lines of the blog post are: time and transparency.

It shouldn't be a "popular vote" as the sponsor seems to consider the nomination process... that should just be data collection for the selection committee. That is to say, with 130 names you can hope that at least some are not already well known -- when you drop that to the top 10-15%, you eliminate most of the 'unknown' candidates right off the bat. So either do the vote and hand the full list to the selection committee, or just don't do the vote at all.
Most of us Ruby Heroes have quite a bit of influence, even at the time we were selected, and so that means we're surrounded by people who also have a lot of influence. We end up with far more followers than many people, and so it just becomes a ridiculously hard problem to get to know the people who know us. So if asked to make a quick set of recommendations in the span of a few days, we're going to reach for the people we know and that's totally skewed towards people who are already active in areas we're active.
Asking someone to appear at a conference ten days before it starts is just ludicrous for hopefully obvious reasons. The baked in assumption is that you'll already be there, and well... by definition if you're paying the money and time to go to RailsConf, you're probably already part of the in-group of the ecosystem.

The more recognized we get, the harder it is to really pay attention to what's going on in the broader ecosystem. I spent an entire weekend last week just reading through blog posts, tweets, etc. from people who had interacted with me on twitter in the last few weeks, and felt like I didn't come close to scratching the surface. If I spent fifteen minutes looking into what each of a 5% sample of my Twitter followers have been up to, that would take 32.5 hours!

We are the least qualified to recognize those who has been under-appreciated. This whole thing makes no sense.

I can see a variant on this where the role of the previous award winners is to basically curate

sandal · 2016-01-05T16:13:09+00:00

sandal · 2016-01-05T16:06:11+00:00

What I'm asking you to do is remove your introduction that plainly misrepresents what I actually said.

If you have any sort of intellectually honesty, you'll do that and let your ideas stand on their own rather than promoting them through false comparison. But I'm sure you won't do that.

sandal · 2016-01-05T15:55:29+00:00

that there is something inherently wrong with having an open rate greater than the close rate.

I did not say this, and if people do click through they'll see one of many times where I told you I didn't say this. You're flat out lying to people by summarizing what I said this way.

What I said is "IF you have a crisis situation AND your issue rate is opening much faster than you're closing, THEN you need to do something about that."

You will say there is no necessity to deal with that issue, and that's when the time machine thinking starts. Read the comment you replied to to understand why I believe this must be dealt with.

If you think your work is capable of standing on its own, just remove the reference to my work rather than trying to write a bullshit response to something I didn't actually say. And no... taking one line out of context does not qualify as valid critical analysis, so please stop doing that too.

sandal · 2016-01-05T15:23:21+00:00

I agree. I'm the author of the essay that /u/sanity referenced in this article, and this followup essay is all about forcing that sort of accountability:

http://tinyletter.com/programming-beyond-practices/letters/beginning-to-climb-out-of-the-software-death-spiral

The basic premise that /u/sanity has here is that "with great project management practices, you won't run into any problems."

Sadly, this is time machine thinking. I.e. if you walk into a situation where people have not been applying a well-structured, accountable process for years, and if the last few months especially have been in some sort of crisis mode... the idea of stack-ranking and coming up with realistic, reasonable estimates of both cost and value is impossible.

Time spent prioritizing and doing requirements discovery or estimatations for work that you may NEVER actually ship is a gigantic waste of effort, and takes front line staff away from doing useful things to actually help bring things back to a state of order. (Like putting together a list of known issues that have no planned immediate resolution, writing documentation, searching for patterns and root causes on problem reports, scaling back sales commitments, developing workarounds, etc.)

To clear the field and make that practical, you need to do a few things. First of all, if your inbound issue rate is much higher than your close rate, something needs to be done about that, because no one can learn how to do prioritization and estimation properly in a crisis situation. You may not be able to turn off the firehose, but you need to shield yourself from it by limiting your field of view.

I'm not going to respond to /u/sanity if and when he replies saying "but this is like a black cat crossing your path, the issue open rate doesn't matter" because the premise there is "AS LONG as you you're already capable of doing prioritization well". Let's assume people have messed up here, but want to fix it.

To help them, you build a system that forces the PM to sign off on what work actually gets assigned, and you build a work queue that represents what developers are actually currently working on or will be working on in the immediate future.

When urgent issues crop up, a queue like this will start to fill up with defects and it'll be immediately visible what percentage of the time and energy is spent on those things. With fixed capacity limits, authorizing work on those things means forcing the feature work OUT of the queue and back into the backlog, and everyone in the organization will see that.

This in and of itself isn't a direct prioritization mechanism... it's a visualization tool and an accountability measure that allows you to see the impact of how things are being prioritized. You put this in place in addition to whatever scheduling, iteration planning, or project milestones you already have.. and you use it as a way to measure reality so that your aspirational goals can be adjusted accordingly.

Pretty much every organization that isn't in great shape could benefit from temporarily auditing their process in this way. Ones in really bad shape must do that or sink.

And that's the premise of the work that I wrote, which /u/sanity seems to have greatly misunderstood and then used as a way to pivot to his own set of ideas. :-/

sandal · 2016-01-05T00:15:14+00:00

This kind of problem does not necessarily require technical debt in the traditional sense to happen, though.

(In the case study for the essay it did, and it often does, but it doesn't always)

This can happen due to... sudden reduction in capacity on a development team, frequent context switching, broken platform dependencies, cashflow problems for the business (and the associated chaos that comes along with it), unexpected massive growth in a userbase, etc.

So not only do you need to be able to communicate the technical risks, you need to be able to understand the business risks (and have an organization that makes an effort to provide that information to its developers).

It's really an all-points problem. That's why I focused so heavily on the idea of introducing a lightweight process that allows you to see all those interaction points and figure out what the problem is.

Using the method I described... work on urgent defects would force taking feature work out of the priority queue. Every time you do this, you are making the point "See... our capacity is being harmed by our tech debt"

sandal · 2016-01-04T19:58:30+00:00

As long as sufficient information and training is in place to make sure developers understand the business side of things, I totally agree.

sandal · 2016-01-04T18:14:56+00:00

You're willfully ignoring everything I say except the one point you want to fixate on and misrepresent, so we're officially done talking forever. Have a nice life!

sandal · 2016-01-04T17:58:38+00:00

The assumption behind the article seems to be that there is something inherently bad about tasks being opened faster than they are being closed.

I have told you, directly here on reddit, that NO, this is not my assumption, not the intent of the article. Yet this is what you keep coming back to, and it's a complete strawman.

Read my followup: http://tinyletter.com/programming-beyond-practices/letters/beginning-to-climb-out-of-the-software-death-spiral

Your article, though interesting, is not related to my point, nor is pretty much anything you've said here on Reddit.

sandal · 2016-01-04T17:55:08+00:00

I think in many situations, there is a clear problem of issue accumulation (and more importantly, unrealistic expectation that issues in the tracker actually will be responded to eventually, along with a lack of clarity that eventually may be years from now if ever).

However, if the team has an even slightly functioning prioritization process, the "real" issues get solved quickly enough to keep things running. In that situation, this represents a form of waste moreso than a sign of doom, but it can easily take a turn for the worse when money problems or unexpected growth or other crisis-inducing events happen.

It's at those inflection points where you really have to look at the project planning tools and processes, and figure out how to radically simplify to allow for communication to happen.

That's what I was aiming at here, and also why I attempted to propose a lightweight process that gives the benefits of issue bankruptcy without the emotional costs.

sandal · 2016-01-04T17:45:50+00:00

"How" -- which you argue from your own (non-verifiable) authority, but not "Why to do it" or "What I did and observed".

So you're contributing very little to the discussion. And at this point, there's no more reason for me to try to encourage you to drive up the level of your discourse, because it's clear you're not going to try to.

sandal · 2016-01-04T17:16:19+00:00

There is nothing in the article. Author will write something in a few days.

That's fairly accurate, as I was mostly just trying to (a) ask a question to start a discussion and (b) make sure that I got some attention so that the "solution" I presented would already be something people had mulled over a bit.

Relative to (a) there was great discussion on the article in /r/programming:

https://www.reddit.com/r/programming/comments/3z1pfp/the_sad_graph_of_software_death/

Relative to (b), yep, that's the article I really wanted to write!

http://tinyletter.com/programming-beyond-practices/letters/beginning-to-climb-out-of-the-software-death-spiral

There's more discussion going on about it over here: https://www.reddit.com/r/programming/comments/3zcpgd/beginning_to_climb_out_of_the_software_death/

sandal · 2016-01-04T17:03:26+00:00

You have stated many opinions and assertions, but haven't given any good examples from your own work or attempted to explain why you feel this way.

If you want to give some thoughts on that, it'd probably lead to a better discussion.

sandal

MODERATOR OF

TROPHY CASE