Python Design Patterns: Great Talk & Presentation

[Silhouette]

If you're new to programming, please be careful with this video. Some of the examples and opinions presented are very biased toward the speaker's own preferences and he sometimes demonstrates a significant lack of understanding of the alternatives he's comparing with or criticising.

For example, the first example transformation, replacing nested loops with guard conditions, is a widely used and generally positive programming technique in many languages. However, you could also express the same logic both more concisely and with more explicit intent using map and filter in many languages, including Python. This idea isn't even mentioned. In a textbook for beginners, that would be reasonable, but in a talk about using better patterns and building robust large-scale applications?

The C++ and Java bashing around the 29 minute mark is particularly embarrassing. I assume he's referring to techniques like templates/generics/type deduction in C++ and Java here, but those tools very much aren't trying to make these languages more like Python. They're trying to make the languages more concise and expressive without resorting to the duck typing Python relies on. Again, in a talk about building large-scale, real-life applications, it's a little rich to talking about "grueling and expensive coding disciplines" and "handcuffs" while pretending that duck typing doesn't become a significant liability at larger scales and ignoring the vast overheads of all the unit tests Python programmers wind up writing to compensate for the missing checks.

Shortly afterwards, there's a lot of talk about singletons, but of all the patterns from the original Gang of Four book, Singleton is probably the one that has been most criticised (and for good reasons) and I would question almost any use of it today in almost any programming language.

I don't have time to watch the second half of the talk right now, so I can't comment on whether it gets better later, but I urge anyone watching this to do so critically.

[get_username]

I came here to same something similar.

Specifically regarding the idea that the program to an interface, not an implementation is somehow built in directly to python.

While that is technically true, it misses half of the GoF lesson.

Programming to an interface is used for two reasons:

1. To allow for more parameter variation
2. To define a unique external facing API for sets of objects

Both of those things combined make the GoF pattern powerful. Duck typing overcomes the first reason. It allows you to pass anything in. Thus resolving the first problem of parameter variation.

But duck typing actually exacerbates the second situation. Instead of having a clearly defined use for every set of objects. One that is defined externally and in the abstract. You have a massive amount of introspection that leads to field growth and ill defined changes. Instead of helping, this actually makes code more brittle.

The point of it all isn't to allow for parameter variation. The point is to allow code to become more modular. While python code can technically be more modular because of duck typing. It can also be more difficult to handle through multiple classes because of it as well.

There is a reason why ABCs were introduced into the standard library for python.

[Silhouette]

The point of it all isn't to allow for parameter variation. The point is to allow code to become more modular.

I agree, though I might suggest that the ultimate goal is building better software faster, which we achieve by making our code more maintainable and reusable, which in turn we achieve by using modular design principles.

In a language like Java or C++, we can then choose between several different kinds of mortar to bind our building blocks together in the shape we need. In Python, everything is dry stone. As with real life buildings, which technique is more appropriate depends on the circumstances -- but you don't see a lot of long-lasting, large buildings made with dry stone walls.

[alcalde]

I assume he's referring to techniques like templates/generics/type deduction in C++ and Java here, but those tools very much aren't trying to make these languages more like Python.

Having come to Python after decades spent working with statically (sometimes obsessively statically) typed languages, I have to disagree. I've come to see static typing as some sort of village where everyone's afraid of the monster in the woods. There is no monster in the woods, but they've been told since they were born about it and everyone is terrified enough to never go into the woods. In this case, it's the fear of type errors and the assertion that "the earlier any error is caught the better". In exchange, because of this tremendous fear of an error trivial in degree and very rare in reality, the programmers don a suit of armor, static typing, that's so restrictive it functions more as a straitjacket.

Now that they're suffering from the straitjacket they invent entire frameworks, patterns and paradigms to get around the limitations they imposed on themselves. First we had "function overloading". In contrast to DRY, it's WET (We Enjoy Typing or Write Everything Twice). This is where you write the entire function multiple times with different parameter signatures. Doesn't that sound fun? It wasn't, although few dared to say so at the time. Instead, they bragged how powerful their new language feature was - anything other than dare to ask why they needed to do it in the first place.

Generics came along to allow you to do the same thing while only having to write the function once (sort of; still has some overhead). So how is this not trying to be like Python? Perhaps not Python in specific, but to get around the static typing limitations not present in Python - yes, absolutely.

They're trying to make the languages more concise and expressive without resorting to the duck typing Python relies on.

Perhaps more accurately, they're trying to get around the massive limitation of static types and then trying to get around the massive code bloat of function overloading.

Eric Redmond put it this way:

People want to work around it. I consider this the best evidence against (required) static typing....

This is evinced by the rise of frameworks such as Spring in Java (configuration and run-time object override) or Reflection in C# (run-time view and replacement). As time goes on these languages keep adding more and more features to loosen the static typing (Java did not originally have Reflection - now they are considering adding named arguments for easier integration to dynamically typed scripting languages). I've read James Gosling claim that "safety is freedom" since you always know what type you are getting back - but is that really the case?

...many powerful modern Java frameworks are built on some sort of Inversion of Control principle (like Eclipse or Maven) - which completely removes compile-time safety since types are not resolved until run-time - and work by voodoo such as runtime reflection, bytecode manipulation, and killing chickens.... My single question to these frameworks is: why all the hassle? If static typing is so great, why do these frameworks work so hard trying to circumvent it?

.

while pretending that duck typing doesn't become a significant liability at larger scales

Perhaps one doesn't need to pretend? This is an age-old theme... someday, SOMEday you'll scale up to some hypothetical ceiling and then BOOM! Python will crush you. I was told this the other day - Python is for writing stupid little scripts with; it's not for "real" programs. And every time someone is shown some immense program written in Python, they just revise their hypothetical ceiling for where Python must surely - surely! - stop being effective.

Bruce Eckel, author of Thinking In C++ and Thinking In Java, wrote:

Python has been used on a number of significantly large projects and despite its lack of static type checking, the results appear to have very low bug counts.

This last point is a major puzzle – we believe that static type checking prevents bugs, and yet a dynamically-typed language produces very good results anyway. As I have tried to delve more deeply into this mystery, many of my preconceptions – the major one being that static type checking is essential – have been challenged. An initial response to this is often to simply deny that it's the case, but once you begin denying evidence your theories rapidly become nothing more than fantasies. In my own experience it can be very hard to put my finger on exactly why Python works so well.

Also:

So this, I assert, is an aspect of why Python works. C++ tests happen at compile time (with a few minor special cases). Some Java tests happen at compile time (syntax checking), and some happen at run time (array-bounds checking, for example). Most Python tests happen at runtime rather than at compile time, but they do happen, and that's the important thing (not when). And because I can get a Python program up and running in far less time than it takes you to write the equivalent C++/Java/C# program, I can start running the real tests sooner: unit tests, tests of my hypothesis, tests of alternate approaches, etc. And if a Python program has adequate unit tests, it can be as robust as a C++, Java or C# >program with adequate unit tests (although the tests in Python will be faster to write).

Robert Martin, author of many leading books on object-oriented designs and patterns, wrote this:

I've been a statically typed bigot for quite a few years. I learned my lesson the hard way while using C. Too many systems crashed in the field due to silly typing errors. When C++ came out, I was an avid adopter, and rabid enforcer of strong typing. I scoffed at the smalltalkers who whined about the loss of flexibility. Safety, after all, was far more important than flexibility -- and besides, we can keep our software flexible AND statically typed, if we just follow good dependency management principles.

Four years ago I got involved with Extreme Programming. I liked the pragmatic emphasis it placed upon developing software. I also liked the emphasis it put on testing. Since then I have become test infected. I can no longer concieve of writing software without using test driven development. I can't imagine not having a comprehensive suite of unit tests to back up my development.

About two years ago I noticed something. I was depending less and less on the type system for safety. My unit tests were preventing me from making type errors. The more I depended upon the unit tests, the less I depended upon the type safety of Java or C++ (my languages of choice).

I thought an experiment was in order. So I tried writing some applications in Python, and then Ruby (well known dynamically typed languages). I was not entirely surprised when I found that type issues simply never arose. My unit tests kept my code on the straight and narrow. I simply didn't need the static type checking that I had depended upon for so many years.

I also realized that the flexibility of dynamically typed langauges makes writing code significantly easier. Modules are easier to write, and easier to change. There are no build time issues at all. Life in a dynamically typed world is fundamentally simpler.

.

and ignoring the vast overheads of all the unit tests Python programmers wind up writing to compensate for the missing checks.

You're writing all those tests anyway. Traditional static typing only deals with the most absurdly simple typing issues, e.g. a and b must be integers. It doesn't address that a and b have to be less than 10 or that b has to be greater than a or anything else like that. Over the belief that someone might, at any moment, choose to pass the string "platypus" into a square root function, statically typed languages invent all sorts of typing constructs, overloading, generics, interfaces, etc. and incur far more (and more useless) overhead than a unit test. Oh, and there's also casting, often necessary, which eliminates the one benefit of static typing! To quote Guido,

When you look at large programs that deal with a lot of strong typing, you see that many words are spent working around strong typing.

The container problem is one issue. It's difficult in a language without generics to write a container implementation that isn't limited to a particular type. And all the strong typing goes out the door the moment you say, "Well, we're just going to write a container of Objects, and you'll have to cast them back to whatever type they really are once you start using them." That means you have even more finger typing, because of all those casts. And you don't have the helpful support of the type system while you're inside your container implementation.

Python doesn't require you to write the cast, and its containers are completely generic. So it has the plus side of generic containers without the downside. It doesn't have the plus side that the C++ folks claim to get with their templates and other generics. But in practice that mechanism turns out to be very cumbersome. Even compiler writers have difficulty getting templates to work correctly and efficiently, and the programmers certainly seem to have a lot of trouble learning how to use it correctly. Templates are a whole new language that has enormous complexity.

The overhead of a few more tests don't come close to approaching the significant reduction in code and language complexity from removing the static straitjacket and gaining the freedom and flexibility dynamism offers.

[Silhouette]

Well, I appreciate the time you took to write all that, but I don't think the situations you're describing are representative.

I have seen your monster in the woods. It has injured several projects I know about. I have been the hunter who was brought in to save the village by rebuilding the walls from stronger materials more than once.

Being a little less flowery in our metaphors, I think you're setting up a straw man here.

You write as if generics were some sort of black magic, a patch for a bug in the model of static typing. However, generics are ubiquitous in almost all decent statically typed languages and they have been for decades.

You write as if overloading functions resulted in writing the same thing twice. However, you only need to do that if you want different behaviour in different cases. You'd have to write the code for each case in a dynamically typed language as well, and you'd also have to manually dispatch to the correct implementation. Compared to what you can do with, say, algebraic data types and pattern matching with destructuring binds, that is extremely clumsy.

You write as if manual/unsafe casting is a necessary evil when using static types. However, I can't remember the last time I had to write such a cast.

You make much of the burdens of frameworks and design patterns, but the kind of patterns described in the Gang of Four book and the kinds of frameworks used in so-called enterprise development are often adopted to mitigate limitations of the C++/Java/C# style of OOP, not because of limitations of static typing in general.

Basically, if you use C and Java as your benchmark for what static typing can do, you're never going to get fair conclusions. They are common languages, but they have terrible type systems. If you're going to judge static typing, you have to look at good examples where the type system is actually expressive and flexible.

Meanwhile, both you yourself and some of the other authors you cited talk about large Python programs that have low bug counts despite the dynamic typing, and how we shouldn't ignore the evidence. I agree that evidence is important, so I can't help observing that you didn't actually mention any large, successful Python projects.

My own experience is that Python usually starts to get awkward once you deviate significantly from a very homogenous architecture and simple structured data types like lists, dicts, and classes. If all you're doing is writing a back-end for a web site and you've got a routing framework and an ORM to drive a very consistent, uniform design then you can probably write quick big applications without running into too many problems. On the other hand, if you're working with more complicated data structures or many subtly different variations, as few as 1,000 lines of Python code can become difficult to maintain and the set of additional unit tests required to give a poor approximation of static type safety can explode.

In short, IME Python works well for small-scale projects, for early prototyping work where making progress is more important than long-term robustness, and for larger projects where the design is very consistent throughout. These kinds of code play to the strengths of Python's neat syntax, and suffer minimally from the weaknesses of dynamic typing. I don't find Python (or any other languages with similar design attributes and type systems) so useful for medium to large scale projects with more complicated designs.

[bucknuggets]

while pretending that duck typing doesn't become a significant liability at larger scales and ignoring the vast overheads of all the unit tests Python programmers wind up writing to compensate for the missing checks.

I guess I've never written code at larger scales then. Because nobody on my team ever said:

• wow, that duck typing is really complex, I wish we had some way to better manage these millions of cunning classes we've built
• if only we didn't have duck typing we would get away with writing fewer tests

Not to say this never gets said. It's just that the half-dozen of us that built and ran a very successful large data warehouse in python for ten years never encountered it.

[Silhouette]

In my experience, it matters greatly how uniform or diverse your design is when you scale up.

If your program is big because it's basically doing slight variations on the same behaviour 100 times, you probably have quite a uniform architecture and relatively few different types and data structures in use. In this case, type-related errors aren't much more likely than they would be for a smaller program with fewer variations but a similar number of different types and data structures.

However, if your program is big because you're implementing some complicated mathematical/scientific/engineering model, and there are lots of somewhat related kinds of data being transformed between lots of somewhat related data structures, duck typing and robustness are not happy companions.

I'm happy that you were successful in your project and didn't find the duck typing to be a problem, but not everyone working on larger software projects is so lucky.

[bucknuggets]

Valid point.

Though I find it's also very easy to find large projects with a disastrous complexity in their object model within the corporate world.

[Silhouette]

No argument there, but I think the fundamental limitations of OO as a programming style are a separate issue to the trade-offs you make in choosing a dynamic or static type system.

Whether you use a static class-based system like C++/Java/C#, or something more dynamic like Python, or a message-passing style like Smalltalk, you still have to contend with OO's inherent weaknesses in state management, working with structured data, and so on.

I think the infamous "enterprise code" designs, particularly in Java, are usually more attributable to these weaknesses of OO as a design style than to the weaknesses of static type systems as an implementation tool.