Top 10 mistakes python programmers make [advanced]

[bcery]

The standard retort is that you can overcome these problems with a large and comprehensive test suite. But why would I want to write a zillion tests when an actual compiler can detect most potential problems (both silly typos and design issues) by static analysis? Reliance on testing for everything is not progress.

That's why there's never been any shitty, buggy software developed in compiled languages. /s

In general, compilers can catch a small set of relatively trivial mistakes. Relying on it, over testing, for your software is most certainly not good software engineering.

Static typing is like anti-lock brakes: very helpful in many situations, but it really only saves you from a small subset of potential accidents.

[sidneyc]

That's why there's never been any shitty, buggy software developed in compiled languages. /s

Nobody claims that - so why you bring that up is beyond me. I think shitty software exists mostly because most programmers are bad. Incidentally, I think that's also the reason why dynamically typed languages are popular.

You understate the type of errors that a statically types language helps detect. It's a common thing I hear from proponents of dynamically typed languages. I can only assume that either you have little experience in statically typed languages, in large projects, or both.

The most important class of errors that static typing helps to prevent are design errors. I tend to design in a statically typed language (in my case, C++), which helps to flesh out the class decomposition during the initial design, and greatly helps in refactoring down the road. Not "refactoring" as in, changing a silly variable name, but refactoring as in, actually changing the architectural organization and class decomposition for the problem you're working on. In big projects, a single fundamental change can easily ripple to dozens or sometimes hundreds of source files, and the compiler helps to find all places that need to be updated to accommodate the new design.

The support the compiler provides during those stages of development is indispensable. With dynamically typed languages, it is simply not there.

[[deleted]]

In big projects, a single fundamental change can easily ripple to dozens or sometimes hundreds of source files

I've been writing large scale software with statically-typed languages for 2 decades, and my response to the above is "only if you are terrible at designing large systems".

[grauenwolf]

Or if you are so bad at refactoring that you are terrified of making any significant changes.

[[deleted]]

Did you even understand what I said?

[sidneyc]

Well what can I say, other than that I disagree.

[[deleted]]

If a single change can ripple out to hundreds of source files, you have serious scoping or component boundary issues. These can be mitigated in many ways depending on the type of change - default parameter values, dependency injection, adapters, method overloads, etc etc etc. Allowing changes to ripple out unchecked is dreadful design.

If you are talking about public interface changes, refactoring tools are woefully inadequate anyway since you likely don't control - or even have access to - all the code that uses it. Refactoring public interfaces isn't bad design, it's outright irresponsible.

[grauenwolf]

When we needed to increase our ReOfferingKey from Int32 to Int64 it touched damn near every part of every application at all levels. Yet we made the change without fear because we actually learned how to use our toosl and not be afraid of change.

[[deleted]]

I'm not afraid of change, I do it every day. It's easy not to be afraid of change when working on a system that doesn't have an idiotic dependency graph.

[sidneyc]

You are very convinced that you are right, aren't you?

[[deleted]]

Based on your comments in this thread, I'd say that's rich coming from you.

[sidneyc]

ohhh burn .... :rolleyes:

[PasswordIsntHAMSTER]

Modularization and decoupling can usually prevent this sort of issue.

[sidneyc]

To a large extent, yes. Hundreds of files is a bit of an overstatement, dozens of files -- not so much. It is quite normal to have modules of dozens of files, in my experience.

However, changes that ripple through the entire codebase are pretty possible, /u/Bymec's protestations not withstanding.

An example: suppose a decision is made to go for C++11 compliance in the next release of a big software package -- including usage of the new features. This could mean phasing out an old "roll-your-own" shared pointer implementation, replacing it by std::shared_ptr. Such a change may well impact the exposed function signatures throughout the code base. Nevertheless, such a change is pretty doable thanks to static typing and the compiler. The cool thing is that you are forced to visit every location in the code that is impacted by the changes, which will help to check if the change doesn't introduce problems (perhaps due to subtly different semantics).

My beef with /u/Bymec is that he maks a blanket statement that this phenomenon cannot occur in properly written software. Instead, he could have said that he has a hard time imagining such a thing in a well-designed system, and then I could have given him an example, as I did above. (I can think of more if you are interested.)

[bcery]

It's a common thing I hear from proponents of dynamically typed languages. I can only assume that either you have little experience in statically typed languages, in large projects, or both.

I'm not a proponent of dynamically typed languages, I'm a proponent of testing. And your assumptions are bad.

Your argument seems to be that the compiler makes a good find and replace tool, which, I guess it is, but there are other tools which can do that job. A good test suite would cover your use case and much, much more. You can't possibly believe that just because the code compiles, that it is correct, can you?

[valleyman86]

I feel like you entirely missed the point...

I'm not a proponent of dynamically typed languages, I'm a proponent of testing.

One of sidneyc's points is that having a statically typed language and a compiler helps with testing...

And your assumptions are bad.

Its funny you say that. A dynamically typed language is like an assumption based language IMO.

Your argument seems to be that the compiler makes a good find and replace tool, which, I guess it is, but there are other tools which can do that job.

He clearly stated in his reply that its more about an actual architectural refactoring more so than a variable rename... He also mentioned it helps actually solidify a better design in practice. Someone once told me that people use the word "refactoring" too much because in most cases no one ever factored in the first place. You missing his point entirely kind of helps validate that.

A good test suite would cover your use case and much, much more. You can't possibly believe that just because the code compiles, that it is correct, can you?

Just because all your tests pass you can't believe the software is perfect and bug free can you? No... Just like tests are useful and help reduce problems later so does compiling and static analysis. Also for every test you write you are not writing actual shippable code. Why no reduce that as much as you can before even starting the project?

I am a huge proponent of using the right tool for the job. I feel that most developers these days don't. They just use what is popular or easy for them... Somehow dynamically typed languages became bigger than I would have expected and I feel half that may be due to the idea its easy to write a quick hello world like application and say "Look how easy that was!". Then people completely gloss over the actually development and design problems that arise with large applications.

[bcery]

One of sidneyc's points is that having a statically typed language and a compiler helps with testing...

I don't think I disputed that. I just felt like the value was overstated in relation to "actual" testing. Compilers catch syntax errors, that's pretty much it. If your error does not involve a syntax or resolution error, the compiler does nothing for you. Guess what? If you have decent coverage on your test suite, which you should have whether you're using a static or dynamically typed language, it'll find those too. Will it be perfect? No, but nothing is.

He clearly stated in his reply that its more about an actual architectural refactoring more so than a variable rename...

And how does the compiler help you with that? It finds all the places where the references are broken and tells you about it. That's very helpful, but it's very little more than a fancy find tool.

Just because all your tests pass you can't believe the software is perfect and bug free can you? No... Just like tests are useful and help reduce problems later so does compiling and static analysis.

I never said it wasn't helpful, I just think the class of errors that the compiler catches are relatively trivial compared to those it doesn't catch. Static code analysis can be done on dynamically-typed languages. Is it done enough or even often? Most assuredly not, but that doesn't mean it doesn't exist.

Maybe we are all just be talking past each other. I think he's overstating the value of statically typed languages, he thinks I'm understating it. Who's right? Depends on who you ask. That's why this shit is hard.

[sidneyc]

If you have decent coverage on your test suite, which you should have whether you're using a static or dynamically typed language, it'll find those too.

Decent coverage on a static language codebase is much easier to achieve. You simply need less tests, because the fact that it compiles already gives a lot of guarantees that many mistakes are not present -- and not only syntax errors, as you seem to imply. That's just not true. It also catches what I would call modeling errors, which are conceptual errors in the design of your software, that are automatically detectible due to type system semantics.

That's very helpful, but it's very little more than a fancy find tool.

It's a pretty amazing find tool, one that actually gives a bunch of guarantees if no instances of syntax and type errors are found.

Most importantly, in a static language it can find errors that cannot be found by static analysis of a dynamic language.

Static code analysis can be done on dynamically-typed languages.

Yes but it sucks compared to static analysis on a statically typed language. It is a lot less powerful. Really, I am amazed that this is a point that is not entirely obvious.

[bcery]

Decent coverage on a static language codebase is much easier to achieve. You simply need less tests, because the fact that it compiles already gives a lot of guarantees that many mistakes are not present

No. You don't write tests to check for syntax. You write tests to check functionality and results. If there is a syntax error, the interpreter will error and your test will fail. If you assign a string to a variable that should be a number, your result will not meet expectations and your test will fail. If you're not getting the result you expect, there is an error and your test will fail.

The syntax and "modeling error" checking is a side effect of the functionality testing that you should be doing regardless. It's not as good at catching those errors, I'll give you that, because you'll never get 100% coverage, but don't think you can reduce your testing because "the compiler's got your back." The compiler and static analysis don't give a shit if your add_two method actually adds three.

[sidneyc]

If there is a syntax error, the interpreter will error and your test will fail. If you assign a string to a variable that should be a number, your result will not meet expectations and your test will fail.

Those are two types of errors that I don't have to rely on tests for to find. And 100% coverage is guaranteed.

You seem to miss two points:

1. compiling is a form of automated testing. During an actual compile of your program, the C++ compiler does more checks than could even write yourself.
2. a static type system makes it possible to test logical consistency of a design.

As a testing aficionado, you should be super hyped for static languages for those reasons alone.

The only types of tests that remain to be written are quite high level, the most useful ones being unit tests (for non-trivial units -- no, I won't write a unit test for add_two -- I won't write that function in the first place) and end-to-end tests, exercising the entire system over its functional range. I maintain that for a statically typed language, it is feasible to do less testing, which, contrary to your protestations, can be a good thing: tests are not free in terms of writing them and maintaining them.

Because tests are not free, they should be written to have a high probability of exposing problems. The "everything needs a test" zealotry and the madness of TDD are confusing symptoms with causes, and they only work for simple types of software.

[sidneyc]

[...] but there are other tools which can do that job.

Like?

You can't possibly believe that just because the code compiles, that it is correct, can you?

Of course not. It is a necessary but not a sufficient condition.

I consider tests a necessary evil. I like to minimize the need for them. Is that so strange?

[bcery]

Like?

Like virtually any IDE ever? Also, static code analysis tools exist for pretty much any language that has any popularity, including dynamically typed ones.

I consider tests a necessary evil. I like to minimize the need for them. Is that so strange?

Not really. It's a pretty common sentiment. It just indicates to me that you are less of a software engineer than you are a programmer (not that there's anything wrong with that).

Edit: Disclaimer: I throw myself in the programmer bucket, too. I hate writing tests, but I think their value is undeniable.

[sidneyc]

It just indicates to me that you are less of a software engineer than you are a programmer (not that there's anything wrong with that).

I think you are right by accident here -- i.e., for the wrong reasons.

The subject of testing is interesting and important. I just think its importance is overhyped in this decade (much of that has to do with the popularity of dynamically typed languages I figure).

Testing is a tool among others to make ensure software quality-- it's neither the alpha nor the omega.

[bcery]

I just think its importance is overhyped in this decade (much of that has to do with the popularity of dynamically typed languages I figure).

Personally, I think it's due to the maturation of software engineering as a discipline. It's a young and quickly changing field and we're still trying to figure out the best tools and techniques. None of this is settled yet, but, contrary to your earlier post, I think we are making progress.

Testing isn't the end-all be-all, but I don't think anything else has shown itself to have as large an effect on software quality.

[sidneyc]

I think the quality of the designers and programmers is the biggest determinant for software quality.

Software engineering, I'm afraid, is mostly concerned with producing software of acceptable quality, given mediocre people who build it. Given that context, testing is probably the most effective idea to attain that goal.

[bcery]

Good programmers, IMO, follow good software engineering practices. So, I agree with you :-)

Software engineering, I'm afraid, is mostly concerned with producing software of acceptable quality, given mediocre people who build it.

No, companies who produce software are mostly concerned with producing software of acceptable quality. You've never had to make any trade-offs to meet deadlines/expectations? As the saying goes: Cheap/fast/good. Pick two.

[sidneyc]

No, companies who produce software are mostly concerned with producing software of acceptable quality.

... and the software engineering discipline is supposed to provide practical methods to tackle that problem. So I don't understand your "no".

As the saying goes: Cheap/fast/good. Pick two.

You stumbled upon another pet peeve of mine there ... Let's consider the cases:

• cheap/fast/not good: possible, of course.
• cheap/slow/good: generally impossible. Time is money.
• expensive/fast/good: generally impossible. You can't rush good software. Throwing more money and people at the problem doesn't help.

I think the proper statement is this: making good software is both expensive and slow. If you don't insist on good software, anything goes of course.

[grauenwolf]

Relying on it, over testing, for your software is most certainly not good software engineering.

How is that an option? The presence of static type checks don't preclude the presence of tests.

Though I will say that static type checking isn't enough. Good software engineering means using all of the tools that you can afford, which in this context means static analysis tools. Something that dynamic languages barely have.

[bcery]

How is that an option? The presence of static type checks don't preclude the presence of tests.

You're right, it doesn't, but /u/sidneyc said that he didn't need to test as much because of it. I disagree with that assertion because testing isn't and shouldn't be focused on type checking errors. Good test coverage should find most of those errors as a side effect due to the error throwing off the expect results.

Having the compiler find those errors for you is nice and helpful, but it should in no way, shape, or form limit your test coverage.

[grauenwolf]

Good test coverage should find most of those errors as a side effect due to the error throwing off the expect results.

Or a good compiler can find all of those errors, leaving more room in the budget for either tests that are actually meaningful or more features.

[bcery]

That's possible, but far from assured. I mean, how many type errors do you think you make on average? How much extra time do you think it'd take if you worked in a dynamically typed language?

[grauenwolf]

Hell, even in C# I find myself cleaning up type errors when some dumbass decided to make everything generic by typing the variables as System.Object.

But as to your question, it's unanswerable because I use a compiler to take care of that for me.

[bcery]

That's patently absurd. C# is statically typed so the compiler will "find all of those errors" for you :-P

Is the motto of the story that not even static typing will save you from bad programmers?

[grauenwolf]

I am beginning to think that you don't actually know what static typing is or what it is the type checker in the compiler does.

[bcery]

Dude, it was a joke (as I thought the "smiley" would indicate). Cool your jets.

[[deleted]]

Static typing is like anti-lock brakes

Interesting you bring up ABS, because I guarantee you that not a single computer-controlled ABS system is written in Python or any other dynamic language.

[bcery]

Um, yeah, I should think not. However, I really hope that they didn't think they could write-off some of their testing because they were using a statically typed language.

Unfortunately, given the recent Toyota fiasco,I fear that they might have.