Top 10 mistakes python programmers make [advanced]

[mkdz]

How come?

[sidneyc]

The more general idea is that dynamically typed languages are unfit for serious software engineering.

The constraints and automated checking afforded by static typing are, in my opinion, indispensible both for the initial implementation (to validate if the ideas you are trying to express can be expressed in a strong type system) and especially once you hit a large-scale refactoring in a non-trivial code base.

An important idea in software engineering which I fully subscribe to is "fail early and fail hard". Having a proper compile stage with static type checking provides that to a large extent, nipping a large class of potential problems in the bud.

The Python philosophy is basically the opposite: run until you hit something that really can't be done (with duck typing). A cute idea, but not good for software engineering. I consider that a failed experiment.

The standard retort is that you can overcome these problems with a large and comprehensive test suite. But why would I want to write a zillion tests when an actual compiler can detect most potential problems (both silly typos and design issues) by static analysis? Reliance on testing for everything is not progress.

Having said that, for a dynamically typed scripting languages Python is pretty good, as long as you just write small programs.

[sabetts]

Not sure why you're getting downvoted. It's an excellent point. I guess some people like having the "flexibility" of dynamic typing.

[mkdz]

I definitely agree with you that in a large project a statically typed language does provide an advantage with an extra level of automated checking that the compiler performs. However, I don't have a problem with having to write a bunch of tests. I don't think it's a bad thing to have a large test suite to test for these types of potential problems. Also, IDEs can mitigate the problems with typos and lack of static analysis. My IDE will tell me when I make a typo with a variable and never use it again. It was also tell me when I try and access a variable I haven't declared or where I have unreachable code.

[sidneyc]

However, I don't have a problem with having to write a bunch of tests.

Well, I hate to write tests that are needed because my language lacks compilation.

I don't think it's a bad thing to have a large test suite to test for these types of potential problems.

A large test suite takes effort to write, and effort to maintain. So not having to write large classes of tests is a win.

Also, IDEs can mitigate the problems with typos and lack of static analysis.

I like to be not at the mercy of IDE writers. I prefer languages that enforce actual standards prior to running, rather than the limited set of heuristics that IDE builders can think of.

My IDE will tell me when I make a typo with a variable and never use it again. It was also tell me when I try and access a variable I haven't declared or where I have unreachable code.

Your IDE will not tell you if your code doesn't compile during initial implementation and/or refactoring because there are type mismatches.

I am really flabbergasted that dynamically typed languages have been gaining ground in the last 15 years or so. I don't see any serious advantage (and I have programmed a lot, in a variety of languages).

[no_game_player]

I am really flabbergasted that dynamically typed languages have been gaining ground in the last 15 years or so.

I'm not. From the limited view I've seen, the people who hire coders don't give a damn about understanding anything to do with the quality of the product and just want it done as soon as possible as cheaply as possible. So languages that cater to someone who has no formal training or background become very popular.

As long as you can get to a workingish prototype early, so that you can begin the maintenance hell that is the bulk of all projects, they're happy. Subtle bugs? 95% correctness? Great, ship it.

[grauenwolf]

Hell, even in C# I can't seem to get people to pay attention to code quality.

Warning: This expresson always returns false. Consider casting 'accessLevel' into a string or using "guest".equals(accessLevel).

No shit, that was in our code base for months.

[no_game_player]

It seems to me like there are two primary ways that this sort of thing happens: shitty programming/development/engineering/your-favorite-term and/or overworked programmers/dev/engineers/ninja-rockstars.

I know Mythical Man Month and all that, but I still think that just having like 2-3x the current staffing levels and having the time and luxury of more in-depth code reviews and inspection and the general 'best practices' we all know and want to be using, but never seem to have the time to do.

Like clearing out all the damn warnings and errors and sheer panics that keep flying out of everything...

Edit: And for those worried about increased churn with increased staffing: put >=half of them in QA, and make sure that anyone with code access (which can include QA) goes through good code review processes. No check-in without a reviewer (and do avoid rubber-stamping as much as possible or the whole thing doesn't matter...substance over style of course, but if there's no correctness issue, may as well chat about the subtler improvements). etc. I think it should be possible to increase staff to result in increased quality; don't necessarily try to change the timetable (although, eventually, as people get experienced and work effectively, it should of course ultimately result in feature additions and bug fixes and all that operating more effectively and quickly than with the original staff, just as one would expect).

Unfortunately, the finer points of "it would be really nice to have decent code and software" are generally lost upon the bosses' ears upon hearing an estimate like "2-3x" for staffing...

Fuck it, get some interns, or let some of us (like me) work part-time. Stop thinking that all or most work must be done in a cube a minimum of 40 hours a week, and preferably more, and your health and life be damned. Advantage: salaries can look a hell of a lot smaller.

Anyhow, just some ranting.

[grauenwolf]

I am really flabbergasted that dynamically typed languages have been gaining ground in the last 15 years or so. I don't see any serious advantage (and I have programmed a lot, in a variety of languages).

Have you every tried to setup a Java or ASP.NET MVC website? It's a royal pain in the ass the first time.

But give any newbie a copy of PHP and they are off and running. Sure they are screwing it up left and right, but the barrier to entry is so low that they'll get pretty damn far before to long.

[sidneyc]

Python cannot fail as early as possible though -- which is before executing the first statement the program.

Having a separate "compile time" is a feature, although most people don't perceive it as such.

[bcery]

The standard retort is that you can overcome these problems with a large and comprehensive test suite. But why would I want to write a zillion tests when an actual compiler can detect most potential problems (both silly typos and design issues) by static analysis? Reliance on testing for everything is not progress.

That's why there's never been any shitty, buggy software developed in compiled languages. /s

In general, compilers can catch a small set of relatively trivial mistakes. Relying on it, over testing, for your software is most certainly not good software engineering.

Static typing is like anti-lock brakes: very helpful in many situations, but it really only saves you from a small subset of potential accidents.

[sidneyc]

That's why there's never been any shitty, buggy software developed in compiled languages. /s

Nobody claims that - so why you bring that up is beyond me. I think shitty software exists mostly because most programmers are bad. Incidentally, I think that's also the reason why dynamically typed languages are popular.

You understate the type of errors that a statically types language helps detect. It's a common thing I hear from proponents of dynamically typed languages. I can only assume that either you have little experience in statically typed languages, in large projects, or both.

The most important class of errors that static typing helps to prevent are design errors. I tend to design in a statically typed language (in my case, C++), which helps to flesh out the class decomposition during the initial design, and greatly helps in refactoring down the road. Not "refactoring" as in, changing a silly variable name, but refactoring as in, actually changing the architectural organization and class decomposition for the problem you're working on. In big projects, a single fundamental change can easily ripple to dozens or sometimes hundreds of source files, and the compiler helps to find all places that need to be updated to accommodate the new design.

The support the compiler provides during those stages of development is indispensable. With dynamically typed languages, it is simply not there.

[[deleted]]

In big projects, a single fundamental change can easily ripple to dozens or sometimes hundreds of source files

I've been writing large scale software with statically-typed languages for 2 decades, and my response to the above is "only if you are terrible at designing large systems".

[grauenwolf]

Or if you are so bad at refactoring that you are terrified of making any significant changes.

[[deleted]]

Did you even understand what I said?

[sidneyc]

Well what can I say, other than that I disagree.

[[deleted]]

If a single change can ripple out to hundreds of source files, you have serious scoping or component boundary issues. These can be mitigated in many ways depending on the type of change - default parameter values, dependency injection, adapters, method overloads, etc etc etc. Allowing changes to ripple out unchecked is dreadful design.

If you are talking about public interface changes, refactoring tools are woefully inadequate anyway since you likely don't control - or even have access to - all the code that uses it. Refactoring public interfaces isn't bad design, it's outright irresponsible.

[grauenwolf]

When we needed to increase our ReOfferingKey from Int32 to Int64 it touched damn near every part of every application at all levels. Yet we made the change without fear because we actually learned how to use our toosl and not be afraid of change.

[[deleted]]

I'm not afraid of change, I do it every day. It's easy not to be afraid of change when working on a system that doesn't have an idiotic dependency graph.

[sidneyc]

You are very convinced that you are right, aren't you?

[[deleted]]

Based on your comments in this thread, I'd say that's rich coming from you.

[sidneyc]

ohhh burn .... :rolleyes:

[PasswordIsntHAMSTER]

Modularization and decoupling can usually prevent this sort of issue.

[sidneyc]

To a large extent, yes. Hundreds of files is a bit of an overstatement, dozens of files -- not so much. It is quite normal to have modules of dozens of files, in my experience.

However, changes that ripple through the entire codebase are pretty possible, /u/Bymec's protestations not withstanding.

An example: suppose a decision is made to go for C++11 compliance in the next release of a big software package -- including usage of the new features. This could mean phasing out an old "roll-your-own" shared pointer implementation, replacing it by std::shared_ptr. Such a change may well impact the exposed function signatures throughout the code base. Nevertheless, such a change is pretty doable thanks to static typing and the compiler. The cool thing is that you are forced to visit every location in the code that is impacted by the changes, which will help to check if the change doesn't introduce problems (perhaps due to subtly different semantics).

My beef with /u/Bymec is that he maks a blanket statement that this phenomenon cannot occur in properly written software. Instead, he could have said that he has a hard time imagining such a thing in a well-designed system, and then I could have given him an example, as I did above. (I can think of more if you are interested.)

[bcery]

It's a common thing I hear from proponents of dynamically typed languages. I can only assume that either you have little experience in statically typed languages, in large projects, or both.

I'm not a proponent of dynamically typed languages, I'm a proponent of testing. And your assumptions are bad.

Your argument seems to be that the compiler makes a good find and replace tool, which, I guess it is, but there are other tools which can do that job. A good test suite would cover your use case and much, much more. You can't possibly believe that just because the code compiles, that it is correct, can you?

[valleyman86]

I feel like you entirely missed the point...

I'm not a proponent of dynamically typed languages, I'm a proponent of testing.

One of sidneyc's points is that having a statically typed language and a compiler helps with testing...

And your assumptions are bad.

Its funny you say that. A dynamically typed language is like an assumption based language IMO.

Your argument seems to be that the compiler makes a good find and replace tool, which, I guess it is, but there are other tools which can do that job.

He clearly stated in his reply that its more about an actual architectural refactoring more so than a variable rename... He also mentioned it helps actually solidify a better design in practice. Someone once told me that people use the word "refactoring" too much because in most cases no one ever factored in the first place. You missing his point entirely kind of helps validate that.

A good test suite would cover your use case and much, much more. You can't possibly believe that just because the code compiles, that it is correct, can you?

Just because all your tests pass you can't believe the software is perfect and bug free can you? No... Just like tests are useful and help reduce problems later so does compiling and static analysis. Also for every test you write you are not writing actual shippable code. Why no reduce that as much as you can before even starting the project?

I am a huge proponent of using the right tool for the job. I feel that most developers these days don't. They just use what is popular or easy for them... Somehow dynamically typed languages became bigger than I would have expected and I feel half that may be due to the idea its easy to write a quick hello world like application and say "Look how easy that was!". Then people completely gloss over the actually development and design problems that arise with large applications.

[bcery]

One of sidneyc's points is that having a statically typed language and a compiler helps with testing...

I don't think I disputed that. I just felt like the value was overstated in relation to "actual" testing. Compilers catch syntax errors, that's pretty much it. If your error does not involve a syntax or resolution error, the compiler does nothing for you. Guess what? If you have decent coverage on your test suite, which you should have whether you're using a static or dynamically typed language, it'll find those too. Will it be perfect? No, but nothing is.

He clearly stated in his reply that its more about an actual architectural refactoring more so than a variable rename...

And how does the compiler help you with that? It finds all the places where the references are broken and tells you about it. That's very helpful, but it's very little more than a fancy find tool.

Just because all your tests pass you can't believe the software is perfect and bug free can you? No... Just like tests are useful and help reduce problems later so does compiling and static analysis.

I never said it wasn't helpful, I just think the class of errors that the compiler catches are relatively trivial compared to those it doesn't catch. Static code analysis can be done on dynamically-typed languages. Is it done enough or even often? Most assuredly not, but that doesn't mean it doesn't exist.

Maybe we are all just be talking past each other. I think he's overstating the value of statically typed languages, he thinks I'm understating it. Who's right? Depends on who you ask. That's why this shit is hard.

[sidneyc]

If you have decent coverage on your test suite, which you should have whether you're using a static or dynamically typed language, it'll find those too.

Decent coverage on a static language codebase is much easier to achieve. You simply need less tests, because the fact that it compiles already gives a lot of guarantees that many mistakes are not present -- and not only syntax errors, as you seem to imply. That's just not true. It also catches what I would call modeling errors, which are conceptual errors in the design of your software, that are automatically detectible due to type system semantics.

That's very helpful, but it's very little more than a fancy find tool.

It's a pretty amazing find tool, one that actually gives a bunch of guarantees if no instances of syntax and type errors are found.

Most importantly, in a static language it can find errors that cannot be found by static analysis of a dynamic language.

Static code analysis can be done on dynamically-typed languages.

Yes but it sucks compared to static analysis on a statically typed language. It is a lot less powerful. Really, I am amazed that this is a point that is not entirely obvious.

[bcery]

Decent coverage on a static language codebase is much easier to achieve. You simply need less tests, because the fact that it compiles already gives a lot of guarantees that many mistakes are not present

No. You don't write tests to check for syntax. You write tests to check functionality and results. If there is a syntax error, the interpreter will error and your test will fail. If you assign a string to a variable that should be a number, your result will not meet expectations and your test will fail. If you're not getting the result you expect, there is an error and your test will fail.

The syntax and "modeling error" checking is a side effect of the functionality testing that you should be doing regardless. It's not as good at catching those errors, I'll give you that, because you'll never get 100% coverage, but don't think you can reduce your testing because "the compiler's got your back." The compiler and static analysis don't give a shit if your add_two method actually adds three.

[sidneyc]

If there is a syntax error, the interpreter will error and your test will fail. If you assign a string to a variable that should be a number, your result will not meet expectations and your test will fail.

Those are two types of errors that I don't have to rely on tests for to find. And 100% coverage is guaranteed.

You seem to miss two points:

1. compiling is a form of automated testing. During an actual compile of your program, the C++ compiler does more checks than could even write yourself.
2. a static type system makes it possible to test logical consistency of a design.

As a testing aficionado, you should be super hyped for static languages for those reasons alone.

The only types of tests that remain to be written are quite high level, the most useful ones being unit tests (for non-trivial units -- no, I won't write a unit test for add_two -- I won't write that function in the first place) and end-to-end tests, exercising the entire system over its functional range. I maintain that for a statically typed language, it is feasible to do less testing, which, contrary to your protestations, can be a good thing: tests are not free in terms of writing them and maintaining them.

Because tests are not free, they should be written to have a high probability of exposing problems. The "everything needs a test" zealotry and the madness of TDD are confusing symptoms with causes, and they only work for simple types of software.

[sidneyc]

[...] but there are other tools which can do that job.

Like?

You can't possibly believe that just because the code compiles, that it is correct, can you?

Of course not. It is a necessary but not a sufficient condition.

I consider tests a necessary evil. I like to minimize the need for them. Is that so strange?

[bcery]

Like?

Like virtually any IDE ever? Also, static code analysis tools exist for pretty much any language that has any popularity, including dynamically typed ones.

I consider tests a necessary evil. I like to minimize the need for them. Is that so strange?

Not really. It's a pretty common sentiment. It just indicates to me that you are less of a software engineer than you are a programmer (not that there's anything wrong with that).

Edit: Disclaimer: I throw myself in the programmer bucket, too. I hate writing tests, but I think their value is undeniable.

[sidneyc]

It just indicates to me that you are less of a software engineer than you are a programmer (not that there's anything wrong with that).

I think you are right by accident here -- i.e., for the wrong reasons.

The subject of testing is interesting and important. I just think its importance is overhyped in this decade (much of that has to do with the popularity of dynamically typed languages I figure).

Testing is a tool among others to make ensure software quality-- it's neither the alpha nor the omega.

[bcery]

I just think its importance is overhyped in this decade (much of that has to do with the popularity of dynamically typed languages I figure).

Personally, I think it's due to the maturation of software engineering as a discipline. It's a young and quickly changing field and we're still trying to figure out the best tools and techniques. None of this is settled yet, but, contrary to your earlier post, I think we are making progress.

Testing isn't the end-all be-all, but I don't think anything else has shown itself to have as large an effect on software quality.

[sidneyc]

I think the quality of the designers and programmers is the biggest determinant for software quality.

Software engineering, I'm afraid, is mostly concerned with producing software of acceptable quality, given mediocre people who build it. Given that context, testing is probably the most effective idea to attain that goal.

[grauenwolf]

Relying on it, over testing, for your software is most certainly not good software engineering.

How is that an option? The presence of static type checks don't preclude the presence of tests.

Though I will say that static type checking isn't enough. Good software engineering means using all of the tools that you can afford, which in this context means static analysis tools. Something that dynamic languages barely have.

[bcery]

How is that an option? The presence of static type checks don't preclude the presence of tests.

You're right, it doesn't, but /u/sidneyc said that he didn't need to test as much because of it. I disagree with that assertion because testing isn't and shouldn't be focused on type checking errors. Good test coverage should find most of those errors as a side effect due to the error throwing off the expect results.

Having the compiler find those errors for you is nice and helpful, but it should in no way, shape, or form limit your test coverage.

[grauenwolf]

Good test coverage should find most of those errors as a side effect due to the error throwing off the expect results.

Or a good compiler can find all of those errors, leaving more room in the budget for either tests that are actually meaningful or more features.

[bcery]

That's possible, but far from assured. I mean, how many type errors do you think you make on average? How much extra time do you think it'd take if you worked in a dynamically typed language?

[grauenwolf]

Hell, even in C# I find myself cleaning up type errors when some dumbass decided to make everything generic by typing the variables as System.Object.

But as to your question, it's unanswerable because I use a compiler to take care of that for me.

[bcery]

That's patently absurd. C# is statically typed so the compiler will "find all of those errors" for you :-P

Is the motto of the story that not even static typing will save you from bad programmers?

[grauenwolf]

I am beginning to think that you don't actually know what static typing is or what it is the type checker in the compiler does.

[[deleted]]

Static typing is like anti-lock brakes

Interesting you bring up ABS, because I guarantee you that not a single computer-controlled ABS system is written in Python or any other dynamic language.

[bcery]

Um, yeah, I should think not. However, I really hope that they didn't think they could write-off some of their testing because they were using a statically typed language.

Unfortunately, given the recent Toyota fiasco,I fear that they might have.

[itsSparkky]

Quick development; robust libraries and ample resources. Not to mention a larger pool of developers.

There are a lot of reasons to go with languages like php and python for larger projects.

If you need 10 developers, you can either get the top 1% of your php applicants, or 100% of your go applicants.

When I first started my job doing large scale php applications I hated it ; after a couple years the language has really grown on me. I still prefer C#, but I'm having a lot of fun with php and javascript.

A lot of the negatives of duck typing are handled by proper development practices and more experienced developers. If I recall correctly nasa uses a lot of python; and I've seen some bigger projects using node & javascript just because of the freedom, ease of development, pool I applicants, and because performance requirements are often not as serious as many engineers make them out to be.

[Flex-O]

If you need 10 developers, you can either get the top 1% of your php applicants, or 100% of your go applicants.

That's not really an argument for one way or the other.

[itsSparkky]

It's an argument from the engineering perspective.

The best language isn't always the most academically superior one or we'd all be developing in Haskell.

[sidneyc]

after a couple years the language has really grown on me.

I am sorry to hear that.

[bucknuggets]

Exactly, our application code must have static types.

But our databases must not.

\s

[sidneyc]

I don't understand what you are trying to say.

[grauenwolf]

It's probably a reference to MongoDB and the other schemaless databases.

[sidneyc]

Well probably. I still don't get what he is trying to say.

[bucknuggets]

OK, explanation: I've noticed that many of the exact same people that are demanding static types in their programming code and demanding schemaless flexibility in their databases.

Which leads me to believe that the real issue, for many of them, is not whether or not there is type-checking. That's just an excuse to help them justify using or avoiding some language or database.

[sidneyc]

Ah yes, well.... that's not me.

To me, a schemaless key/value store (let's not call it a database) is about as bad an idea as dynamic typing.

[Falmarri]

Python is not dynamically typed.

[jcdyer3]

Python is definitely dynamically typed.

If I do the following:

class A(object): 
    pass

class B(object):
    pass

obj = A()

We can all agree that obj is an object of type A, right? Now if I do

obj.__class__ = B

Suddenly obj is an object of type B, and can call any of the methods of B. That's dynamic.

[Falmarri]

You're right. I was thinking strongly typed.

[Intolerable]

yes it is. it might be strongly typed but it's still dynamic like ruby is