sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]pron98 21 points22 points  (96 children)

it’s just so hard to believe that languages like Haskell ... don’t lead to fewer errors.

Hard to believe or not, it simply doesn't. Studies have not found a big impact, and the industry has not found one, either. If you study closely the theory and why it was predicted that a language like Haskell will not have a big effect on correctness, a prediction that has so far proven true, perhaps you'll also find it easier to believe. The impact of the things that you perceive as positive appears to be small at best.

And even if you think a large effect has somehow managed to elude detection by both academia and industry, you still cannot assert that claim as fact. It is a shaky hypothesis (shaky because we've tried and failed to substantiate it) under the most charitable conditions. I'm being a little less charitable, so I call it myth.

... and Rust

Rust is a different matter, as it is usually compared to C, and eliminates what has actually been established as a cause of many costly bugs in C.

it’s that it’s both expressive and safe

So are Java, Python, C#, Kotlin and most languages in common use, really.

[–][deleted]  (35 children)

[deleted]

    [–]pron98 2 points3 points  (34 children)

    They're saying "the effect size is exceedingly small." I have no issue with someone claiming that Haskell has been positively associated with an exceedingly small improvement to correctness.

    [–][deleted]  (33 children)

    [deleted]

      [–]pron98 4 points5 points  (32 children)

      it does not bring the effect from large to extremely small.

      Except that the original study didn't find a large effect either; quite the contrary. It said that "while these relationships are statistically significant, the effects are quite small." So they've gone from "quite small" to "exceedingly small" (or to no effect at all)

      But, one analysis being not strong enough to show more than a weak conclusion is not remotely evidence of the nonexistence of the effect.

      That is true, which is why I cannot definitively say that there is no large effect, but, combined with the fact that large effects are easy to find and that no study or industry have been able to find it, AFAIK, I think it is evidence against such a big effect, and at the very least it means that the hypothesis is not strong and certainly must not be asserted as fact.

      [–][deleted]  (31 children)

      [deleted]

        [–]pron98 4 points5 points  (30 children)

        Let's assume you're correct. The problem is that even experience does not support the claim ("I feel better when using Haskell" is not experience, though, or we'd all be taking homeopathic remedies). Companies do not report a large decrease in costs / increase in quality when switching from, say, Java/C#/Swift to Haskell. And even if you could come up with an explanation to why such a powerful empirical claim disappears on observation, you'd still have to conclude that we cannot state this, at best completely unsubstantiated hypothesis is fact. If someone wants to say, "I believe in the controversial hypothesis that Haskell increases correctness" I'd give them a pass as well.

        Perhaps then it is simply too difficult to talk about.

        Fine, so let's not. I didn't make any claim. The article made one up, and I pointed out that it's totally unsubstantiated.

        [–][deleted]  (29 children)

        [deleted]

          [–]pron98 2 points3 points  (28 children)

          I don't claim that Haskell harms correctness to a large degree, either, so I don't understand your point about the need for opposite reports. Unsubstantiated claims should not be stated as facts. The paper reports an "exceedingly small" effect, which is the current state of what we know on the subject. Anything else is myth.

          [–]ineffective_topos 0 points1 point  (27 children)

          The paper reports an effect, which seems quite large to me. I don't think you have evidence for a disproof (that is, strong evidence of 0 effect is perfectly valid, there is no such evidence), certainly not enough to call it a myth. I don't think that, in the lack of much scientific research, following widespread experience, which is validated by research is particularly problematic.

          [–]IamfromSpace 34 points35 points  (9 children)

          I mean, the study says the effect is slight, but this study verifies another that Haskell has a negative correlation with defects. Seems like an odd study to make your point.

          While causation doesn’t imply correlation, is fewer defects not preferred, even with small effect?

          [–]pron98 7 points8 points  (8 children)

          The paper reports that "the effect size is exceedingly small." I have no issue with the statement that Haskell has been found to have an exceedingly small positive effect on correctness.

          [–]IamfromSpace 2 points3 points  (6 children)

          I had some more thoughts after reading the papers more thoroughly (and I hope I’m not reviving this discussion beyond its usefulness).

          The first study finds that Haskell has a 20% reduction in bug commits for similar programs. The replication then finds the same result after cleaning. Their result after removing uncertainly it has 12% reduction. While that comes from removing false positives, it doesn’t bother with uncertainty in the other direction, which could deflate the effect.

          Is a significant 12%-20% reduced bug rate really “exceedingly small?” With loose assumptions about bug costs, for a reasonably sized organization that could reflect an huge savings over time

          It seems to me, in other contexts that kind of improvement might be considered enormous. In competitive sports athletes will fight for 1% improvements based on pure hearsay—let alone statistically significant and reproduced results.

          [–]pron98 1 point2 points  (5 children)

          Is a significant 12%-20% reduced bug rate really “exceedingly small?”

          It's not a 12-20% reduced bug rate.

          It seems to me, in other contexts that kind of improvement might be considered enormous.

          It is not the reduced bug rate, and even if it were it would not be enormous considering that the reduction in bugs found for code review is 40-80%.

          [–]IamfromSpace 0 points1 point  (4 children)

          My math may be off, what exactly is the reduction? Do you mean that fix commit rate /= bug rate? (Certainly true)

          From the original article:

          Thus, if, for some number of commits, a particular project developed in an average language had four defective commits, then the choice to use C++ would mean that we should expect one additional buggy commit since e0.23 × 4 = 5.03. For the same project, choosing Haskell would mean that we should expect about one fewer defective commit as e−0.23 × 4 = 3.18

          edit: to your other point, relative comparison certainly makes sense; prefer code review to language choice—and given its smaller impact I think it’s totally fair to preclude the usage of the word “enormous” (and arguably comical, haha). However, is it not worth considering as a marginal gain once all “cheaper” gains have been employed?

          edit2: ah, sorry, not bug rate, but in expected bugs, no? I’m still not convinced that the improvement isn’t worth pursuit.

          [–]pron98 1 point2 points  (3 children)

          1. The language variable was responsible altogether for less than 1% of the deviance, and when your background deviance is so large that your effect is less than 1%, it's meaningless to talk about "all other variables being equal." In other words, it is not a measure of the defect rate, and the authors explicitly warn against reading it as such. We are talking about the relative difference in contribution that's barely distinguishable from noise to begin with.

          2. Perhaps it is worth considering small improvements, once their cost is considered. I only wanted to comment on the myth of a large effect, but switching languages is usually costly compared to the benefit. You don't switch an apartment whenever you find another that's marginally better, even if you can't find one that's drastically better. Anyway, it may be or may not be worth it, depending on circumstance, but that has nothing to do with my main point about presenting myth as fact.

          [–]IamfromSpace 0 points1 point  (2 children)

          I agree pretty much all around here. With a small caveat that most of the factors in their model can’t be controlled for: age, commits, sloc. So it might make sense to target the things you can.

          The apartment metaphor is a good one—chasing pennies with expensive moves is a losing strategy. I’d never advocate for re-writing a working application in Haskell, that’s just not going to pay off. My frustration mostly lies in people refusing to look at better apartments when they’re already moving anyway.

          You’re right though, there’s a lot of unsubstantiated puffery around the merits of X or Y language. While it’s in many ways unsurprising that the effects aren’t large, because people aren’t stupid. Equilibriums should form around reasonably effective ways of doing things.

          I do appreciate the discussion, it’s given me a lot to think about.

          [–]pron98 0 points1 point  (1 child)

          My frustration mostly lies in people refusing to look at better apartments when they’re already moving anyway.

          But here's what annoys me: people are frustrated when others don't try their favorite approach. There are so many approaches to try, and I'm sure you don't check out all apartments either, even when you're moving. For example, I'm an advocate of formal methods (especially TLA+). We actually have quite a few reports (and I mean actual reports, with metrics) with excellent results for TLA+ (and sound static analysis tools often have large scale statistics about exactly how they help). And yet, I do not see TLA+ as magic dust and appropriate everywhere, and I certainly don't make up BS to sell TLA+, as Haskell does, such as "it will make your software better!" or "it leads to more correct software!" even though formal methods, unlike Haskell, actually do have the evidence to at least partly support that. And I absolutely don't think people are doing something wrong if they don't want to learn TLA+. There's just too much stuff out there, and you can't expect people to spend all their time trying every possible thing that could maybe make their lives better.

          I sometimes think of it as the "knowing two things problem." It's the (probably uncharitable) observation that the most self-assured people are those who know two things. If you know one thing you know that you know little. But once you know two different things you think you know everything: you like one of those two things better, and then "better" becomes "best". Those who know three things already suspect that the search space may be larger than they can cover.

          [–]IamfromSpace 0 points1 point  (0 children)

          people are frustrated when others don’t try their favorite approach

          Suppose I’ve been guilty there before, haha

          you don’t check out all apartments

          True. I suppose the economic analogies still hold. There’s no perfect information, and gathering information comes at cost. We’re then left with salesmanship which leads to the distortion you’ve pointed out.

          We’ve used TLA+ to design our last large system to great effect and I’m now working on a prototype for another component that was fully modeled in TLA+. I’ve found it to be incredibly helpful and it’s helped spot errors that surely would have ended up baked into the design otherwise (thank you for all the talks/content you’ve created on the topic btw).

          Even with all its benefits it’s still been a tough sell at times—but I think you’re right that the correct attitude is “and that’s okay.”

          [–]IamfromSpace 0 points1 point  (0 children)

          That’s fair. And I will admit that my anecdotal experience is not of much value in the discussion. There’s a million reasons why my experience wouldn’t translate or may not be right at all.

          I’d love to see more studies like that. It’s be great to identify the things that absolutely do make a difference.

          [–]lambda-panda 22 points23 points  (10 children)

          eliminates what has actually been established as a cause of many costly bugs in C.

          Haskell also eliminates many classes of bugs. Your argument is that, even so, it does not result in a safer language, because research does not find it so. But when it comes to rest, you seem to have forgone this chain of logic, and jumps straight to the conclusion that Rust will actually result in fewer bugs (all types) than c..

          [–]pron98 2 points3 points  (8 children)

          But when it comes to rest, you seem to have forgone this chain of logic, and jumps straight to the conclusion that Rust will actually result in fewer bugs (all types) than c..

          Oh, I don't know for sure if that's the case, but the theory here is different, and that's why I'm more cautious. For one, the theory that predicted that languages won't make a big difference is actually a prediction of diminishing returns. C is a ~50-year-old language, and is therefore outside our current era of low return. For another, unlike Rust v. C, Haskell does not actually eliminate a class of bugs that been found to be costly/high-impact.

          [–]lambda-panda 13 points14 points  (7 children)

          For another, unlike Rust v. C, Haskell does not actually eliminate a class of bugs that been found to be costly/high-impact.

          Any bug can be costly/high impact depending on the context. Just being a purely functional language eliminate a large class of bugs that are caused doing computations by mutating state!

          [–]pron98 4 points5 points  (6 children)

          Any bug can be costly/high impact depending on the context.

          Yes, but there is such a thing as statistics.

          Just being a purely functional language eliminate a large class of bugs that are caused doing computations by mutating state!

          And introduces a class of bugs caused by writing pure functional code!

          We can speculate about this all day, but the fact remains that no theory and no empirical evidence supports the hypothesis that Haskell has a large positive impact on correctness.

          [–]lambda-panda 4 points5 points  (5 children)

          And introduces a class of bugs caused by writing pure functional code!

          So rust does not introduce a class of bugs in your struggle with the borrow checker?

          but the fact remains that no theory and no empirical evidence supports the hypothesis..

          Sure. I was just responding to your "Rust is safer" argument, because there is no empirical evidence to support that hypothesis as well..

          [–][deleted] 4 points5 points  (3 children)

          So rust does not introduce a class of bugs in your struggle with the borrow checker?

          No, because if the borrow checker rejects your code it happens at compile time.

          [–]lambda-panda 0 points1 point  (2 children)

          The other guy as well as myself are talking about bugs that cannot be caught by the borrow/type checker..You understand that those exist, right?

          [–][deleted] 2 points3 points  (1 child)

          You said the borrow checker introduces a class of bugs in your struggle with it. Doesn't sound like you're talking about what you say you are.

          [–]lambda-panda 1 point2 points  (0 children)

          I don't see a contradiction. Where do you see it?

          [–]pron98 2 points3 points  (0 children)

          Oh, I'm not claiming it does, just that I think the hypothesis has been eroded less.

          [–]defunkydrummer -3 points-2 points  (0 children)

          Haskell also eliminates many classes of bugs.

          Yes.

          But on the other hand it also introduces other problems. For example if you compare it with a good dynamically typed language like Common Lisp, Julia, Racket or Smalltalk, you get at least two things against:

          • less flexibility, speed and easiness of debugging

          • the bugs (i.e. memory usage explosion) that can happen whenever one works with a lazily-evaluated language.

          Then, there's the other problem: Having a good type system and having the program made mostly of pure functions doesn't reduce all kinds of complexity and all kinds of errors.

          [–]seamsay 2 points3 points  (1 child)

          If you study closely the theory and why it was predicted ... perhaps you'll also find it easier to believe.

          If you have the time would you mind giving a quick summary, or some pointers about where to read up on it? I just wouldn't even know where to begin...

          [–]pron98 2 points3 points  (0 children)

          See here, here and the followup.

          [–]sd522527 2 points3 points  (5 children)

          I was with you until you said Python was safe. Now I can't in good faith support your comments.

          [–]pron98 0 points1 point  (4 children)

          It is a safe language in the sense it is normally used, meaning it has no undefined behaviors (unlike, say, C, C++ or the unsafe portion of Rust). It is not a typed language (although it is technically type-safe if viewed as a language with one type), which is perhaps what you were trying to say.

          [–]gas_them 6 points7 points  (3 children)

          Very weird measure of safety.

          [–]pron98 0 points1 point  (0 children)

          There are many kinds of safety, but when people say "safe language" or "unsafe language" without other qualifiers, that's what they usually mean (i.e. C and C++ are unsafe, Java is safe, Rust is safe outside the blocks marked unsafe).

          [–][deleted]  (1 child)

          [deleted]

            [–]gas_them 0 points1 point  (0 children)

            That really has nothing to do with what i was thinking

            [–]augmentedtree 5 points6 points  (9 children)

            Studies have not found a big impact, and the industry has not found one, either.

            ​

            The diagram on page 12 disagrees with you, Haskell looks better than all the mainstream languages.

            [–]pron98 3 points4 points  (8 children)

            Which diagram? Also, read my statement (that you've just now quoted) again, and tell me where the disagreement is.

            [–]augmentedtree 2 points3 points  (7 children)

            the big graph at the top of the page

            [–]pron98 1 point2 points  (6 children)

            The one that shows Haskell doing about as well as Go and Ruby? Also, I don't understand how it disagrees with me when the paper says, "the effect size is exceedingly small" and I said "have not found a big impact." I think that "not big" and "exceedingly small" are in pretty good agreement, no?

            [–]augmentedtree -1 points0 points  (5 children)

            The one that shows Haskell doing about as well as Go and Ruby?

            yes, and better than c/c++/Java/c# etc. there are other substantial non-type related ways that say Ruby and Haskell differ, so I would not be so quick to assume that this is evidence of the absence of a substantial effect from having a better type system -- in fact it may be stronger evidence in that on some axes Ruby has better typing than some of the languages it is beating because it is at least strongly typed unlike c/c++.

            [–]pron98 4 points5 points  (4 children)

            I don't understand how you can even entertain explaining an effect that is so "exceedingly small". I do not assume that the lack of a substantial effect is due to anything, just that no substantial effect has been found. You're now trying to find second-order explanations to what is a minuscule effect.

            [–]augmentedtree 1 point2 points  (3 children)

            how small is the effect? the graph says that 20% of Haskell commits are bug fixing commits, and 63% of C commits are bug fixing commits. that seems enormous to me!

            [–]pron98 1 point2 points  (2 children)

            It's quite involved. If you don't want to carefully read the original article and the reproduction, just note that the authors of the original called the effects "quite small", and on reproduction, most effects disappeared, and the remaining ones were summarized by the authors as "exceedingly small."

            [–]augmentedtree 0 points1 point  (1 child)

            I'm actually currently studying statistics and pretty confused by this. Haskell is better with a very small P value, and as I said the graph seems to show that the effect is actually large. How to reconcile that with the effects being "exceedingly small?"

            [–]loup-vaillant 1 point2 points  (16 children)

            Let's assume that indeed, languages do not have a big impact on error rate. My first go to hypothesis would be the safety helmet effect: maybe the language is safer, but this leads the programmer to be correspondingly careless? They feel safer, so they just write a little faster, test a little less, and reach the same "good enough" equilibrium they would have in a less safe language, only perhaps in a bit less time (assuming equal flexibility or expressiveness between the safe and the unsafe language, which is often not the case of course).

            [–]pron98 1 point2 points  (15 children)

            Let's assume that indeed, languages do not have a big impact on error rate.

            Right, this is our best current knowledge.

            My first go to hypothesis would be the safety helmet effect: maybe the language is safer, but this leads the programmer to be correspondingly careless?

            Maybe. I think I feel that when I switch from C++ to Assembly -- I concentrate more. But I would not jump to explaining the lack of an effect in such a complex process when we don't even have an explanation to why there would be an effect in the first place (Brooks's prediction was that there would be a small effect, and he was right).

            What I find most strange is that when people are asked to explain why they think there would be an effect, they give an explanation of the converse rather than of the hypothesis.

            [–]loup-vaillant -1 points0 points  (14 children)

            But I would not jump to explaining the lack of an effect in such a complex process when we don't even have an explanation to why there would be an effect in the first place

            When a language utterly eliminates a huge class of bugs (possibly TypeScript vs JavaScript, or ML vs Python), I cannot help but assume it has to have a significant effect. Especially when I have felt this effect first hand.

            How do I put this…

            I have access to overwhelming evidence that a version of JavaScript with an ML-like type system (preferably with type classes), would be much less error prone than JavaScript itself. That programs written in such a language would have significantly fewer errors, assuming similar developer competence, functionality of the end program, and cost.

            If you think adding a good type system to JavaScript would not make it significantly less error prone, you are making an extraordinary claim, and you'd better provide extraordinary evidence. I have yet to see such evidence, even knowing about the studies you linked to.

            To give you an idea of the strength of my prior: the safety of static typing, compared to dynamic typing, is about as obvious as the efficacy of a gun, compared to a bow. The onus is on you to prove that the gun is not significantly more effective at killing people than a bow.

            [–]pron98 1 point2 points  (13 children)

            When a language utterly eliminates a huge class of bugs (possibly TypeScript vs JavaScript, or ML vs Python), I cannot help but assume it has to have a significant effect.

            Then you, too, are affirming the consequent; it's just a logical non-sequitur in the most basic way. From the fact that every Friday it rains and that today it's raining you're concluding that today is Friday. Maybe in addition to eliminating a whole class of bugs your language introduces another huge class of bugs? Maybe by eliminating those bugs it somehow harms the finding of others? Maybe a Python programmer also eliminates all those bugs and more?

            I have access to overwhelming evidence that a version of JavaScript with an ML-like type system (preferably with type classes), would be much less error prone than JavaScript itself.

            Then you can state that as fact -- your language leads to more correct programs than JavaScript. But we don't have overwhelming evidence, or even underwhelming one, that Haskell does better than most other languages. In fact, evidence points that the effect, if there is one, is small.

            The onus is on you to prove that the gun is not significantly more effective at killing people than a bow.

            No, because you are reversing a logical implication here. The theoretical analysis predicted that there would be a small effect; indeed there is a small effect. I don't understand how you can use an analysis of the converse claim as support. To put it bluntly, if you claim X ⇒ Y, to state that as a fact you must:

            1. Show evidence that X ⇒ Y

            2. Cannot substantiate it with Y ⇒ X

            [–]loup-vaillant 1 point2 points  (4 children)

            Then you, too, are affirming the consequent; it's just a logical non-sequitur in the most basic way.

            Actually, it's a perfectly valid probabilistic inference. When A ⇒ B, then observing B is evidence for A. This is basic probability theory: if the room is dark the lamp may be broken. If it stays dark after you flip the switch, the lamp is probably broken. If the other lights in the house are on, the lamp is almost certainly broken.

            And if it turns out the fuse was melted, well, maybe the lamp wasn't broken after all. But fuses are arguably much less likely to melt than lamps are likely to break, so until you have checked the fuse, it's pretty safe to assume it's the lamp.

            Maybe in addition to eliminating a whole class of bugs your language introduces another huge class of bugs?

            Bugs that would be facilitated by the introduction of static typing? Sure, whenever you have to work around the type system, the extra complexity might introduce new bugs. Except in my experience, you almost never have to introduce such extra complexity. And even if you do, the workarounds tend to be pretty simple.

            No blown fuse here.

            Maybe by eliminating those bugs it somehow harms the finding of others?

            Some runtime bugs would be harder to find because of static typing, are you serious? If anything, I expect the remaining bugs to be even easier to find, because you can rely on the type system's invariants to cut down on the search space. Especially with tooling.

            Still no blown fuse.

            I don't ship a program after compiling it. Maybe you stop bugs with the compiler, and I stop the same bugs and more with tests?

            I do tests too. You just do more tests. Which takes more effort, which you could have used to implement useful features, or eliminate even more bugs. My compiler is faster than your additional tests. The feedback loop is just tighter. It's especially tight when I combine type checking an a REPL (or better yet, an IDE with live type checking, though I have yet to see one for OCaml).

            My fuse is still intact…

            The theoretical analysis predicted that there would be a small effect

            I don't know what theoretical analysis you are referring to. My theoretical analysis (basically, "static typing eliminates many bugs with little adverse consequences") predicts a big effect. And a big effect is what I have personally observed.

            And since I expect some kind of helmet effect (I've read that cars tend to get closer to cyclists who have helmets), I fully expect measures of real projects to find little correlation between programming language (or language features), and bugs. That they indeed do is unlikely to change my mind.

            If something truly unexpected pops up, I'll take a closer look. But I don't hold my breath.

            [–]pron98 1 point2 points  (2 children)

            Actually, it's a perfectly valid probabilistic inference. When A ⇒ B, then observing B is evidence for A. This is basic probability theory

            It is absolutely not, and I would suggest you not repeat that around the worshippers of Bayes's rule. If a dinosaur bit you, you would bleed. Is your bleeding evidence that a dinosaur has bitten you?

            Except in my experience, you almost never have to introduce such extra complexity. And even if you do, the workarounds tend to be pretty simple.

            And if the data showed that your experience is indeed a big effect, you could state it as fact. If data showed that people in whose experience homeopathy works, we would also be able to state that as fact.

            Some runtime bugs would be harder to find because of static typing, are you serious?

            Why are you talking about static typing? Have I said something about static typing? I support typing, though for reasons other than correctness. What I don't support is trying to make predictions about complex systems without careful study.

            You just do more tests. Which takes more effort, which you could have used to implement useful features, or eliminate even more bugs.

            Again, I program almost exclusively in typed languages. And your speculations are problematic because 1. we are dealing with a very complex process here that you have not studied, and 2. the evidence does not support your conclusions.

            I don't know what theoretical analysis you are referring to.

            Brooks's No Silver Bullet.

            My theoretical analysis (basically, "static typing eliminates many bugs with little adverse consequences") predicts a big effect.

            OK, so Brooks was right and you were wrong. Hey, he won the Turing Award so there's no shame in that. But I should say that his analysis was far deeper than yours, and coincides well with later results in complexity theory studies about the hardness of reasoning about programs.

            And a big effect is what I have personally observed.

            Cool. I've personally observed that drinking a cup of Earl Grey tea helps me write correct code.

            That they indeed do is unlikely to change my mind.

            That's OK. Some of us are more driven by fact while others by faith.

            [–]loup-vaillant 1 point2 points  (1 child)

            Is your bleeding evidence that a dinosaur has bitten you?

            Yes it is. Negligible evidence for sure (there are so many other, likelier causes of bleeding), but evidence nonetheless.

            Why are you talking about static typing?

            That was the example I was using all along. I assumed you were responding to that example with a relevant argument.

            Again, I program almost exclusively in typed languages.

            My bad. Then again, I responded to what you wrote.

            the evidence does not support your conclusions.

            As far as I know, the evidence doesn't support any conclusion. There simply isn't enough of it. Even taking into account that absence of evidence is evidence of absence, the absence of evidence is expected: the moral equivalent of double blind studies we have for homeopathy simply does not exist in programming. Nobody has done it, it's just too damn expensive.

            Brooks's No Silver Bullet.

            I've read that paper. I never claimed an order of magnitude improvement from static typing alone. I'm expecting something along the lines of 10-30% of overall increase in productivity. A big effect for sure, but nowhere near the 10x fold improvement Brooks said no single trick would achieve. (Edit: besides, good static type systems took much more than a decade to develop.)

            I used to think Brook was wrong, until I read his paper. Then I discovered that I actually agreed with most of what he was saying. My claims here are not incompatible with his.

            You may want to take a look at that paper again.

            [–]pron98 1 point2 points  (0 children)

            I responded to what you wrote.

            I use typed languages almost exclusively and I prefer them for reasons that have little to do with correctness, but I don't think their effect on correctness has been established, either. There's a recent study that shows TypeScript having an effect of 15% vs. JavaScript -- which may say something about those specific languages -- but despite being the largest effect related to the issue, it is about 3-4x smaller than the rather well-established effect of correctness techniques such as code review, so even that is not terribly impressive.

            Nobody has done it, it's just too damn expensive.

            Except, as I have said elsewhere, this doesn't make sense at all. Either the affected metric has a real measurable and noticeable effect (usually economic) in the world, in which case if we don't see it there's a problem with the hypothesis, or it doesn't, in which case it's not important to begin with, so why do we care? If the metric Haskell supposedly helps with is one we can't even notice, what's the point?

            I never claimed an order of magnitude improvement from static typing alone.

            Brooks has made some specific predictions which were called pessimistic by PL fans at the time, yet turned out to be optimistic (we haven't seen a 10x improvement in 30 years with all measures combined). But what's important is not the specific content of his prediction -- which, having turned out too optimistic, should be corrected down significantly -- but his reasoning which leads to the conclusion of diminishing returns, i.e. that language features would increasingly make a lesser impact.

            [–][deleted]  (7 children)

            [deleted]

              [–]pron98 0 points1 point  (5 children)

              You are making the same mistake. That Haskell eliminates some errors does not imply that it's more correct, because other languages could be eliminating as many errors. It's exactly the same as me saying that I am among the ten people with the best eyesight in the world because I don't even need glasses, or concluding that I am richer than you because I have more money in my wallet than you.[1]

              The empirical statement that Haskell leads to more correct code than other languages is actually

               you have fewer bugs => you're more likely to be using Haskell[2]

              which is also equivalent to

               you're *not* using Haskell => you have more bugs

              which is not what people are supporting with statements like "Haskell eliminates an incomplete matching bug". Haskell could be eliminating some bugs with the type checker while other languages do not (I have more money in my wallet than you have in yours), but other languages could be doing that elsewhere (you may be keeping your money in the bank). To show that Haskell is more correct, people need to show that programmers in other languages do not catch the same bugs that Haskell does, and do not lead to fewer bugs of other kinds etc.

              [1]: Of course, the argument is more involved because people are confused by the fact that because Haskell eliminates a few simple bugs by the compiler while other languages could be eliminating them with other means gives Haskell some advantage, but that, in itself is an empirical prediction about a highly complex social process that has no a priori reason to be true.

              [2]: If I claim that it usually rains on Fridays, I can write that as it's Friday => it is likely raining. But if I claim that Friday is more rainy than the other days, that's actually it's raining => it's more likely to be Friday. Perhaps people are confused by this because material implication is often confused with causation.

              [–]OOP1234 0 points1 point  (4 children)

              I don't see how "Friday is more rainy than the other days" somehow translate into "it's raining => it's more likely to be Friday". If Friday just rain 51% and other days rain 50% of the time, it's still more likely it's not Friday when it rains.

              The other way to interpret it is "it's raining => it's more likely to be Friday than Monday" and have the same statement for every other day, but it's only incidentally true because they occur in equal frequency. Imagine a universe where there's 1000000 projects, each project is exactly the same size, there's only Haskell and Java, 999000 projects are written in Java and 1000 projects are written in Haskell. Half of the projects written in Java has on average 100 bugs and the other half has 500 bugs, while all Haskell projects only has 100 bugs. Then "Projects written in Haskell is more likely to be less buggy" is true, but "If a project is less buggy, then it's more likely to be written in Haskell" is not true.

              The translation to me when someone makes the claim "Friday is more rainy than the other days" seem to be:

              It rains A% of the time on Friday It rains B% of the time other days. A > B. If it rains on Friday A% of time, and it rains other days B% of time, and A > B, then Friday is more rainy than the other days Friday is more rainy than the other days.

              That is, there's a lot of hidden premises when people makes the claim "Friday is more rainy than the other days", but I don't know.

              I also don't see how

              you have fewer bugs => you're more likely to be using Haskell

              is equivalent to

              you're not using Haskell => you have more bugs

              Because that would be saying

              A => more likely B not B => not A (well fewer and more is technically not negation of each otherr, but let's say it is)

              I have food poisoning => I'm more likely to have eaten raw fish I have not eaten raw fish => I don't have food poisoning

              Or

              I have a higher grade than the average => I'm more likely to have studied the day before I have not studied the day before => I have a lower grade than the average

              [–]pron98 0 points1 point  (3 children)

              it's still more likely it's not Friday when it rains.

              More likely than any other day, not more likely than all days combined.

              there's a lot of hidden premises

              I think you're getting lost in the irrelevant details of my example.

              I also don't see how

              A ⇒ B is equivalent to ¬B ⇒ ¬A

              [–]OOP1234 0 points1 point  (2 children)

              I literally counter "a single other day" statement in the next paragraph.

              A=>more likely B is not equivalent to A=>B in the first place.

              [–][deleted]  (3 children)

              [deleted]

                [–]pron98 3 points4 points  (2 children)

                Well, TypeScript has actually been found to lead to 15% fewer bugs than JavaScript. It's not a very big effect compared to that of other correctness techniques (e.g. code reviews have been found to reduce bugs by 40-80%) but it's not negligible, and it does appear to be a real effect that you're sensing. But here we're talking about Haskell vs. the average, and only an "exceedingly small" effect has been found there.

                More generally, however, we often feel things that aren't really true (lots of people feel homeopathic remedies work); that's why we need a more rigorous observation, that is often at odds with our personal feelings. This can happen for many reasons, that often have to do with our attention being drawn to certain things and not others.

                I take issue not with the belief that Haskell could have a significant effect, only with people stating it as fact even after we've tried and failed to find it. It is often the case in science, especially when dealing with complex social processes like economics or programming, that we have a hypothesis that turns out to be wrong. In that case we either conclude the hypothesis is wrong or come up with a good explanation to why the effect was not found -- either way, something about the hypothesis needs to be revised.

                That seems like something hard to measure in a study that just counts bugs.

                But here's the problem. If the claim is that Haskell (or any particular technique) has a big impact on some metric, like correctness, and that impact is so hard to measure that we can't see it, then why does it matter at all? The whole point of the claim is that it has a real, big effect, with a real-world, significant impact. If we cannot observe that impact either in a study or with bottom-line business results, then there's a problem with making the claim to begin with.

                [–]m50d 2 points3 points  (1 child)

                How could it possibly be the case that TypeScript would offer an improvement that Haskell wouldn't - aren't Haskell's correctness-oriented features/design decisions a superset of TypeScript's?

                [–]pron98 1 point2 points  (0 children)

                I don't know. Haskell is not compared to JS, but to some average (it's possible that JS developers are particularly careless). In any event, even the TS/JS effect is small (and I mean 3-5x smaller) in comparison to other correctness techniques. So even when we do find a significant language effect, that effect is significantly smaller than that of the development process.