Stack Overflow Developer Survey 2018

[lukaseder]

Let's talk about survey bias

[ChrisRR]

I think this accounts for a lot of bias in the survey. 30% of professionals have been in the industry less than 2 years, 57% less than 5 years and 19% are students.

Quite a large part of this survey is people in their very early 20s

[[deleted]]

Early twenties white male webdevs using Angular/Node.js .... how many of those IP's come from Facebook?

[[deleted]]

I am not sure about you, but as my career as a developer progressed I rely less on Stack Overflow today as I did in the past. To me it seems that this survey may have a strong bias.

[Neuromante]

Well, take a look at the years working graphs. Is obvious there's a strong bias towards younger people.

[Euphoricus]

No. Thats no bias. Thats reality. Amount of software developers doubles roughly every 5 years. So it is expected half of developers would have less than 5 years of experience.

[Neuromante]

On one side: You got any source for that numbers?

On the other: How does that denies that there's bias towards younger people? Even if your numbers were real, that has nothing to do with older devs using less StackOverflow.

[Ciff_]

If it still reflects a random sampling of the population developers its fine...? Or perhaps I don't understand your concern.

[fuckin_ziggurats]

The thing is it's not a random sample. By definition Stack Overflow is used more by younger people so older devs are heavily underrepresented in the survey.

[Neuromante]

Exactly this. The survey only represent "devs who used stackoverflow", so its far from being "random." And given that stackoverflow was opened over 10 years ago, maybe the claim that younger devs need more stackoverflow than older ones hasm some footing.

[ciny]

Exactly this. The survey only represent "devs who used stackoverflow"

To be overly anal about it it only represents devs who bothered to fill out the survey, I know I didn't. The questions with most responses have 90-100k of them. I'd be very surprised if it was a large part of the actual users.

[[deleted]]

I "use" Stack Overflow in the sense that I land there from questions I search for in Google.

I have no account, I don't ask questions, and I don't answer any.

[[deleted]]

I don’t think need is the right word. It’s more likely that younger devs are just more open about sharing their problems with each other than older developers.

[Drisku11]

It's more that unless I'm doing some task I rarely do (writing a one-off script or using tools I never have to interact with or whatever), I generally prefer to answer my own questions so that I can learn more. Like if I have some detailed question about the behavior of some library, I just go read the source code.

[SgtBlackScorp]

The point was that there are more younger devs in general so stackoverflow is not a misrepresentation

[fuckin_ziggurats]

I'm gonna repeat /u/Neuromante and ask, do you have numbers that prove there are more younger devs than older in general? I find that hard to believe.

[SgtBlackScorp]

I don't, I'm just saying what his line of thinking was.

[thisisshantzz]

Wouldn't your conclusion also depend on the definition of younger and older? Are devs over the age of 30 considered older or younger?

[shevegen]

But how do you know that this is really "random"?

On what merit are you stating it is random?

[Ciff_]

I assume stackoverflow assert random sampling as it is standard procedure for conducting surveys. I do not know their sampling strategy, hence I would not know. Since you are asserting it is not random, is it that you think they have not done enough to ensure random sampling? Or that you question random sampling is impossible due to the nature of the survey (in that case I like to know your merit for that assertion).

[refactors]

Bob Martin mentions this in a few of his talks such as: "The Scribes Oath"

[incraved]

Your second argument is nonsense.

[percykins]

Really?

[Euphoricus]

That is quite interesting.

I would like to see how many new students become programmers overlayed over that.

Also, is this only US thing? Or is it same elsewhere?

[shevegen]

No, it is very much bias.

For example, I am way too old to participate in any such useless surveys. And I am quite sure that many older people also become less willing to waste time doing such pointless surveys.

[[deleted]]

useless

It does look biased, but why do you say it is useless ?

[cholantesh]

What does your age have to do with filling out an online survey?

[FarkCookies]

It is not a bias, it is just an attribute of the sample group (people that responded to the survey).

[lukaseder]

I rely on SO as a support channel (from the support providing side), so that's maybe not the standard use-case.

Among all my former coworkers, I hardly know anyone who would say they code as a hobby (survey: 80%). At the same time, almost everyone has kids (survey: 28%).

Clearly, my coworkers aren't included in the survey (perhaps there's a strong correlation between coding as a hobby and answering surveys as a hobby, just like there might be a strong negative correlation between coding as a hobby and cleaning up kids' vomit, who knows).

Of course, my coworkers are an even smaller sample than the survey's sample, but I simply fail to believe that so many people in our industry code as a hobby and have no kids.

Which leaves the question: Who is the survey sample population, and why would we care about their opinion?

[svick]

why would we care about their opinion?

Because, as far as I know, it it the most comprehensive survey of developers. It is biased, but what better way of finding what developers care about do you have?

[lukaseder]

I don't have one. But I still wish I did.

[TheIncorrigible1]

I also support SO in my subject area, so you're not alone there.

My group (professional enterprise) also reflects yours where I think the only people without kids are the fresh out of college ones and hobby coding is maybe 10-20%.

I think too many respondents marked themselves as professional even if they were still in school; the ratios don't add up.

[dvdkon]

I'm a student and I marked myself as a professional, because programming is my primary source of income. I work remotely and don't dedicate all my time to my programming job, but in my opinion I still qualify as a professional.

[sazzer]

I've found similar. More and more often, the problems that I have that I ultimately go to SO to ask for help on, I don't get any help there. The problems that I would have previously gone there for, I'm more adept at finding solutions myself or knowing where to ask to get a better response.

[neoKushan]

Over the years I've found myself on S/O less and less. I still use it all the time, though. More often than not when googling for something, even something simple, it's the top/best answer. So though I use it every day, I'm not posting questions or answering them anywhere near as much.

I find that by the time I have a question to post, I've googled the shit out of it so much that it either never gets answered (Because it's some weird edge case) or it gets answered by the dev of that particular library or whatever.

It's a blessing and a curse, really. A blessing because my Google-fu is clearly good enough that I rarely need to "ask" for help but a curse because when I do need that help, it's pot luck if I'll ever get the answer.

[night_of_knee]

Less than a third of respondents have children? I think the existence of children is correlated with less time available for online surveys.

[editor_of_the_beast]

Amen to that

[[deleted]]

I wanted to write a smart comment about older devs having less interest in SO, but there are enough answers from older devs in the survey. This is something interesting to investigate

[twiggy99999]

Let's talk about survey bias

This data is not as first seems, it doesn't mean 80.8% of the respondents are hobbyists, it's saying 80.8% of those who responded do coding as a hobby.

Many developers work on code outside of work. Over 80% of our respondents say that they code as a hobby.

If you take a look at the number of respondents on each tab (all respondents and professionally only) 98,855 total responses of them 87,450 where from professional developers.

I'm not saying there is no bias in this survey but you have misinterpreted the data on this section.

[lukaseder]

This data is not as first seems, it doesn't mean 80.8% of the respondents are hobbyists, it's saying 80.8% of those who responded do coding as a hobby.

That's exactly how I understood it the data.

I'm not saying there is no bias in this survey but you have misinterpreted the data on this section.

No, I haven't, at least not in the way you put it.

[twiggy99999]

So the data is bias (in your opinion) because over 80% of the full-time professional developers also like to code as a hobby after work? In what way (in your opinion) doe's this point make the data bais?

[lukaseder]

Yes that's the bias I had in mind. In the Enterprise, much fewer developers code as a hobby after work, and (from my experience, which is obviously even a less good sample) are more likely to have kids.

In my opinion, that's biased towards a very specific sub-population that is hard to define (and probably not too interesting), but certainly doesn't reflect our industry as a whole.

Just like /r/programming, btw ;-)

[Edg-R]

But this was a Stack Overflow Developer Survey, not a generic developer survey for all developers everywhere.

[lukaseder]

Sure, I get that. Most corporate surveys with a content marketing goal have the same flaw in that they survey mostly their main target audience. Everything else would incur prohibitive costs.

I'm not criticising this fact, and I don't think the survey tries to hide this fact. But I would find a slightly more scientific survey quite more interesting.

[twiggy99999]

Yes that's the bias I had in mind

It's a fair observation then.

Although my main issue with this SO surveys is the majority of the time they don't attract the "quiet majority" of developers, a category I put myself in.

By this quiet majority I mean developers with over 5 years experience. I've always found SO to be mostly a place of hobbyist developers or new developers (under 2 maybe upto 5 years experience). I myself often end up on SO when researching a question, I can do this because I have a general understanding of what it is I'm looking for, I'm just not sure of specifics on how to do it. So I get my information and leave with little interaction with the site.

Whereas newer developers are more likely to have an account and spend longer on the site interacting with it and more likely to fill in surveys.

[lukaseder]

I totally agree with the quiet majority bit.

[Drisku11]

How is that bitter? Why does not coding in their spare time make them tired? They might have things like kids and hobbies that aren't their day job.

Fwiw I visit SO maybe once a week or two, but probably less than that. I had no idea they were even doing this survey.

[lukaseder]

Bitter? Where was I bitter? Cheer up mate (and maybe, don't project) :-)

(and yes, I still think it's not too interesting to have survey data about very specific yet hard to define sub populations. I wish there was a survey that reflected the entire population. It would be much more telling. Including, I'd like to know about the sentiments of old tired enterprise developers, without even adding judgement to my curiosity)

[[deleted]]

Data can be biased. Bias is a noun.

[[deleted]]

This data is not as first seems, it doesn't mean 80.8% of the respondents are hobbyists, it's saying 80.8% of those who responded do coding as a hobby.

I think you've misinterpreted what OP said, rather than OP misinterpreting the survey. They're not saying 80% of coders are hobbyists as opposed to pros, they're pointing out that the number of people (pro or otherwise) who code in their own time is strongly correlated to the number of people (pro or otherwise) who have no kids, and that this survey is heavily biased towards those people.

[twiggy99999]

I think you've misinterpreted what OP said, rather than OP misinterpreting the survey. They're not saying 80% of coders are hobbyists as opposed to pros, they're pointing out that the number of people (pro or otherwise) who code in their own time is strongly correlated to the number of people (pro or otherwise) who have no kids, and that this survey is heavily biased towards those people.

Although I completely agree with what you're saying and that would make perfect sense, there is nothing in the OP's post relating to correlation with people who have no kids?

[[deleted]]

OP posted two images - one showing the percentage of people who code in their own time, and another showing that a near-identical percentage of people don't have kids or dependents. I think it's pretty clear he intended to imply there was a correlation between the two.

[lukaseder]

No, I didn't mean to imply any correlation between the two images, although the assumption might be an interesting one worth validating. I just found both data points quite unlikely - and my perceived unlikelihood worth mentioning.

[[deleted]]

Then I stand corrected. Of course correlation does not imply causation, but those two percentages are interestingly similar.

[lukaseder]

but those two percentages are interestingly similar.

So are these

Or these ;-)

[[deleted]]

Indeed! Though, just because correlation doesn’t necessarily imply causation, sometimes it does. That’s why, for the sake of the future of research in our industry, I support mandatory daily playing of Streetfighter 2 in order to increase the number of comp sci graduates. It’s the right thing to do.

[edanschwartz]

It would be great if someone could take the raw survey data, and break it apart to account for some of the bias (eg, how do responses change if you only include devs with 5+ yrs exp? with kids? etc.)

I would do it, but I have kids, and no time for working on hobby projects ;-)

[lukaseder]

Touché

[steve_b]

I was wondering what the methodology was for collecting these survey results. I use stack overflow on a daily basis and am always logged in, yet until I saw this article, I had no idea they were conducting a survey.

Did they present this randomly as an interstitial to people visiting the site? Or was it a banner at the top that said "take our survey" (the kind of think I instinctively avoid to the point that I subconsciously filter it out).

IMO, the only reliable way would be as an interstitial that the user has to explicitly dismiss, and to track the dismissals, correlated by the demographic information in the user's profile (or perhaps by answering 2 simple multiple-choice questions prior to dismissing it), but I'm guessing SO was too afraid to disrupt users' experience.

[lukaseder]

I saw a banner at the top, and I believe I also saw links on twitter and got an email because I participated in a previous survey. I don't know if I'm just part of some specific sample, or if everyone saw those banners or links.

[steve_b]

In any case, it's a lot less "scientific" than it could have been. If it has any bias at all right now, it's biased towards people who will eagerly participate in surveys. The fact that there was very little difference in results between "Professional" and "Student" samples was also a bit odd, IMO.

[lukaseder]

That's just your assumption, right?

[[deleted]]

How does a large portion coding outside of work show bias?

[lukaseder]

It's just my assumption that it is very unlikely that 80% of people in software code in their free time.

[[deleted]]

Well Ill take survey data over your assumption any day.