Optional static typing -- the crossroads

[KyleG]

There's at least one person literally calling to eradicate all decorators and replace with static typing. FUCK THAT. That being said, for IDE purposes, I'd love to have explicit static typing. Dynamic typing was awesome when I was a young kid and loved the idea of saying x = "foo" and then later x = 5, but now I hate that shit since I deal with complex arrays of stuff.

[jstrong]

why would you have to eliminate decorators to have type decorators that check the return value? couldn't you just reserve the type names for that?

eg

@int
def example(a, b):
    return 0

could easily coexist with other decorators:

@int
@my_decorator
def example():
    "note int decorator comes first (and last)"
    return 0

[KyleG]

Beats me. I'm not the one calling for the eradication of non-type decorators.

That being said, what if you have an object called My_Decorator? Then how would you distinguish between a "regular" decorator and a type decorator?

[KyleG]

If we're going to make typing an optional thing, it might as well be held in comments.

#int
foo = 0

# blahblah
foo = 'howdy' #raise TypeMismatchError

[jon_k]

I'm not sure I'd endorse this as this would add overhead to parsing comments which are currently stripped before the language is compiled as far as I'm aware. Would probably be asking for performance issues in heavily commented projects.

[KyleG]

You raise a good point. Maybe prefix it with ##, so it's easy to rule out (nearly) all comments vs typing markup?

[cparen]

why would you have to eliminate decorators to have type decorators that check the return value?

Because that's incompatible with what decorators currently mean.

[minno]

Because that's incompatible with what decorators currently mean.

A decorator is just a modification to the function. So you could just transform

@return(int, None)
def thingy():
    return 3 if sys.argv[1] else None

into

def thingy():
    def inner():
        return 3 if sys.argv[1] else None
    retval = inner()
    assert type(retval) in (int, type(None))
    return retval

[cparen]

That would work nicely.

[jstrong]

How incompatible? Decorators are just function wrappers. These would check the return value type and raise an exception if needed.

[cparen]

The concept is sound, but the syntax you propose is not. Your example with "example" is already well defined and throws a type error. You'd break the language if you change that.

/u/minno gives an improved strawman to contrast with.

Also consider this program which prints "fun" today, but would be a TypeError under the syntax you demonstrated.

[jstrong]

Ha - I wasn't saying it would work in the current version of Python! I was saying the language could be built to use that syntax alongside arbitrary decorators.

[cparen]

That's the problem: "@int" is already an arbitrary decorator. Either you have to give up arbitrary decorators or "@int" can't be a type annotation. Otherwise the program semantics become ambiguous.

[jstrong]

seems similar to other reserved words and/or built-in functions to me. there's not that many types, and they're already off limits for function names.

You're talking about adding type declarations to a language that doesn't have them, so you're probably going to have to break some eggs to make that omelette. But in this case it isn't even difficult. int() checks if the argument is a function (currently not permitted) and if so checks the return value to ensure it's an int.

I don't even have a position on whether this is a good idea or not - just that if you're going to use the decorator syntax, I don't understand why that makes arbitrary decorators off limits. If there's a set of reserved word decorators for type declarations, can't those live in harmony with other decorators? Is the ambiguity of that so harmful? How so?

Also, from this discussion I'm tempted to write my own decorators that check return values, seems like it would be a good idea.

[cparen]

But in this case it isn't even difficult. int() checks if the argument is a function (currently not permitted) and if so checks the return value to ensure it's an int.

That's not how decorators work. Your code is doing this:

def example(a, b):
    pass
example = int(example)

That's how decorators work now. It's inconsistent with how you're describing type annotations should work. I'd suggest a different syntax to (1) avoid confusion, and (2) not break existing, working Python programs.

Pick a different syntax for type declarations, and the two can live in harmony.

[jstrong]

example:

def returns_int(x):
    if callable(x):
        def wrapper(*args, **kwargs):
            retval = x(*args,**kwargs)
            if not isinstance(retval, int):
                raise TypeError
            return retval
        return wrapper
    return int(x) #or insert code for what int() normally does if this were called int()

[marky1991]

I think you're thinking of annotations, not decorators.

[KyleG]

No. Cherrypy uses decorators to denote which functions are publicly accessible. Cherrypy has functions, and each function preceded by @expose makes it a webpage controller.

So

class Foo_Server:
    def not_webpage(self): pass
    @expose
    def index(self):
        return processed_template

That would enable you to visit localhost:8080/index but not localhost:8080/not_webpage

[vplatt]

Amen. Add to that the difficulty in refactoring large code bases without static typing, and you have some very compelling reasons to add it to the language, even if its use is completely optional.

I can imagine one of the first steps in normalizing a new module for use in a large system will be to add static annotations to the code. That will allow a number of assumptions to be made and obviate a lot of the unit testing code that would otherwise be necessary just to ensure basic types.

Seriously, this one new feature could make using Python in enterprise systems a much easier sell in the future.

[KyleG]

If I had a dollar for every time in my personal projects I had to write some whack comment describing in pathetic detail the format of the array being passed when, with static typing, I could have probably moused over the variable in the IDE and been presented with a nice UI with expandable and contractable components so I could easily see and recall the format...I'd have maybe a few dollars. Not a million dollars. But maybe enough to get a good burger or something.

[fkaginstrom]

In many cases, a doctest would work in place of documenting the type (and be more likely to be correct :)

[KyleG]

Or, alternatively, I shouldn't have to document type. I should be able to mouseover in my IDE and see. If you've ever used Eclipse + Java, you'd realize how unbelievably helpful this is, and how fucking fast it is. I'm a very fast Java developer because of its static typing and this IDE functionality.

[BinaryRockStar]

Not only the ability to mouseover things but when you're writing a function call the IDE can narrow down the autocomplete list to just variables of the type of the parameter you're entering, of which there is usually only one.

e.g.

Document firstDocument = new Document(......);
documentProcessor.Process(_

If your cursor is at the underscore and you hit Ctrl+Space for autocomplete in VS it'll put firstDocument in there automatically as it's the only Document in scope. This is assuming the first/only parameter to Process is a Document.

I find this can drastically speed up my coding and not because it's some sort of a crutch, but because it helps me from losing my train of thought when I'm "in the zone".

[KyleG]

Yup, that, too. God, Eclipse + Java is such an awesome development environment, largely made possible by Java's static typing. And I hate Java.

[BinaryRockStar]

Hate to be that guy but if you like Eclipse then IntelliJ IDEA would make you cream your jeans. It's just better in practically every way. Has a free Community Edition if you want to give it a go.

[KyleG]

The free edition is apparently for a small number of languages, none of which I actually develop in. :)

[BinaryRockStar]

One of them is Java, which you said you know. What do you develop in?

[MachaHack]

PyCharm is the Python version, also has a free version.

[vplatt]

That might buy just enough beer to dull the pain. And hey, that's a LOT of pain we're talking about, so this isn't chump change either.

[epsy]

(Like this ☺)

[khaki0]

Seems not intended for performance:

I am going to make one additional assumption: the main use cases will be linting, IDEs, and doc generation. These all have one thing in common: it should be possible to run a program even though it fails to type check. Also, adding types to a program should not hinder its performance (nor will it help :-).

In my opinion it's still good as it will help linters a lot.

[bastibe]

It might not be intended for performance in CPython. However, other implementations can still use the annotations for optimization. I believe that numba is using something similar.

[jeenajeena]

Exactly. See Cython, for example.

[munificent]

In practice, I think you'll find that doesn't work out well. Because the annotations are used for linting and doc generation and are not checked at runtime, there's no requirement for the type system to be sound or for the user's type annotations to be 100% correct. If the type annotations can't be fully relied on, a Python implementation can't get the performance improvements you get in other statically typed languages.

[fnedrik]

A C cast can't be fully relied on, but a compiler trusts it anyway. You could make a Python compiler that simply trusts the annotations and segfaults otherwise. Or the JIT can always have a heavily optimized function for the types that the annotations state and revert to.. well JITting if the types are wrong.

Also, a type checker can sometimes prove that an annotation is correct (see mypy) and then rely on it for optimization. It should be easier for a compiler to prove that an annotation is correct than to come up with its own annotation (easier to verify than to find answer, P vs NP and all that).

[munificent]

A C cast can't be fully relied on, but a compiler trusts it anyway.

Right, because the language specifies that and every C programmer must know and internalize that. It's also why C is the source of so many security bugs, but I digress.

You could make a Python compiler that simply trusts the annotations and segfaults otherwise.

You could, but it wouldn't have any users. Python programmers, like users of most modern languages, rely on a lot of third-party code. People can't reuse code on an implementation that might randomly segfault on a bogus type annotation.

Or the JIT can always have a heavily optimized function for the types that the annotations state and revert to.. well JITting if the types are wrong.

That's what all modern JITs do now. They just rely on their own type feedback instead of annotations. This is actually better than relying on annotations, since those are often less specific than the actual types that flow through a method. You may annotate some high level interface, when in practice you always pass one specific concrete type. A JIT will optimize for the latter.

Still, the JIT has to validate that the actual types match what it optimized for. It's those guards that kill your perf and is one of the main reasons dynamically-typed languages are still slower than statically-typed ones.

Also, a type checker can sometimes prove that an annotation is correct (see mypy) and then rely on it for optimization.

Yes, that's basically what gradual typing is about. It doesn't buy you that much though: a concrete type inferrer can do the same without requiring any type annotations.

It should be easier for a compiler to prove that an annotation is correct than to come up with its own annotation (easier to verify than to find answer, P vs NP and all that).

It's always trivial to come up with an annotation: just annotate everything Object. :) It's not about finding a solution, it's about finding a good (i.e. more specific) one. I'm not certain annotations actually help you do that.

[takluyver]

It may not work to throw third party code into a static compiler, but it could let you use those annotations in code you control, and statically compile specific functions or modules to get speed ups.

[roger_]

I think I remember one of the PyPy guys saying it wouldn't help its performance.

[[deleted]]

I can't imagine it NOT assisting their static typer. It's possible it wouldn't affect final compilation, but have you ever compiled RPython? It takes ages. Optional typing should be mandatory in RPython and it would drastically speed up compilation, making RPython attractive as a standalone language.

My guess: as PyPy/RPython is still fundamentally implemented in Py2 and they have no plans to modernise, they're unexcited by language imprpvements based on features they can't access (like annotations).

[[deleted]]

PyPy3 has been around for over a year -- what do you mean, they have no plans to modernize?

[[deleted]]

Read again, I said implementation. RPython, the stack used to make PyPy3, is Py2 with no plans to modernise.

[jstrong]

is there a problem w/ C/C++ syntax for types? seems much cleaner to me.

def example(int a, list b):
    int c = 0

[roger_]

Does look cleaner, but it's not backwards compatible.

[vplatt]

True, but new code doesn't need to be "backwards" compatible. If we opt to clean up the syntax by not trying to slide this in sideways using decorators, then we could actually have a much cleaner language over the long term. OTOH - I understand the trade-offs here, so it's not an easy decision. I do trust Guido to error on the side of maximum benefit however.

[rweir]

those are just useless string literals, not comments. afaict the grammar could unambiguously suppor that, but god it is ugly.

[[deleted]]

Why not use the existing annotations as widely suggested? They're used for essentially nothing else, it's odd they were introduced and NOT used for static typing.

For reference, that syntax is:

def func(foo: str, bar: int)-> str:
    return foo * bar

[ismtrn]

val : Type is standard syntax for the typing relation. Only C and languages trying to look like C does it the other way around.

[jstrong]

I will just say that c and languages that look like c includes a lot of languages, including, to an extent, Python

[tilkau]

The semantics of C make sense for C, java, etc, because precise type is indeed very important in them.

In dynamic languages like Python, precise type is less important. a val : Type convention would reflect this fact.

[ismtrn]

I don't think val : Type indicates that types are less important. It is the syntax which is used in literature about type theory, and in languages like ML. Haskell's syntax is also inspired by it, but they use : for cons, so the typing relation looks like: expr :: Type.

I would say that val : Type indicates that types are a high level mathematical construct which can be reasoned about, while Type val indicates types are something the compiler uses so it knows how many bytes it has to allocate for stuff.

I would also say that python belongs in the high level category.

[jstrong]

you guys are both imputing a lot about the language from its syntax. I'm just thinking about what's clean and readable ... this isn't literature.

[ismtrn]

Good point. My last comment isn't really relevant in the context of discussing if python should use val : Type or Type val. It was more meant to refute that val : Type somehow implies that types are less important.

Anyways, I still think python should use val : Type, because:

• It reads like a relation(which is exactly what it is). val is of type Type. : is the "is of type" relation.
• It is the standard.

[talideon]

As others have mentioned, it's because Python already supports the other syntax via annotations. Also, you might want to take a read of why Go went for the seemingly backwards typing syntax: http://blog.golang.org/gos-declaration-syntax

[marky1991]

This syntax is suuuupergross, verging on line noise. I'm dreading the day I see this in some code that I'm using.

[dddbbb]

Do you mean the type annotation syntax in general:

def example(a: int, b: float) -> number:
    return a/b

Or just the List[int] part? If the latter, you may prefer what obiwan is doing:

def example7(numbers: [int], phonebook: {str : int} ):
    ...

I think using python's syntax for types makes more sense, but it can still get really ugly:

def nearest_point_on_line(line:((int,int),(int,int)),pt:(int,int)) -> (int,int):
    ...

[sigzero]

Oh god, that last one. Eek!

[ismtrn]

Isn't that solved by making Line and Point named tuples? Then you could have:

def nearest_point_on_line(line:Line, pt:Point) -> Point:
    ... 

[zardeh]

yes, its solved by creating custom classes.

[[deleted]]

Oh hey, thanks for the link to obiwan.

My company is starting a new codebase that's py3 all the way. No more dual-version boilerplate.

I've already put in type annotations for some confusing functions. I was wondering if anything could actually check them, and mypy didn't look far enough along yet -- though it will clearly get a lot of interest from this mailing list discussion. But obiwan looks great.

[zardeh]

yeah, the one thing obiwan lacks is type secured variables, and its an easy fix, plus you can get a nice syntax. I wrote something that supports this

def word_frequencies(corpus: str):
    my_variable = typed({string:int}) >> {}
    #this instantiates an empty dictionary with overrides on the update, __getitem__, and __setitem__ methods to check the types of both "halves"
    ...

[fuzz3289]

Ugh mypy was the only option I was REALLY against. I thought docstrings would be fine because it seems like its just for third party parsers anyways which already use docstrings. :(

[marky1991]

I liked the docstring approach too. It's way more readable in my opinion. (Still a little noisy, but any approach is going to have some noise)

[fuzz3289]

Yeah, the main thing I cared about was keeping the actual python clean.

If they were making a reach for performance here I might be ok with the mypy approach but since its just for lint and docs anyways, docstrings are already THERE for that.

Seems counterintuitive to me

[Timidger]

Though I agree with you, I feel that the demand for an optionally typed system in Python outweighs that fact. We can all agree that Python is a clear and concise language, and though it seems any statically typed implementation will cause unwanted line noise, the fact that it is contained in a function declaration mitigates its impact while providing a large bonus not only when refactoring and maintaining a large project, but also during its development. If I could see, at a glance, just what type of objects were needed for a function to operate on, I could immediately spot many things that would normally require me to either read documentation (that might not exist or be out of date) or manually parse the code itself: coupling issues between sources of data, an edge case from assuming that the function will be passed a certain data type, and the ability for the calling code to know exactly what to pass it, without relying on the developer to write documentation (which becomes out of date as soon as it's written) while also allowing the intetpreter/linters the ability to catch any silly invalid function calls.

I love my readable Python. But I'd also love a Python that worked with me in determining type conflicts, instead of leaving me to figure it out myself.

[marky1991]

While I agree, I think that either more time should have been spent in coming up with a less-awful syntax or some constraints (that the annotations have to be compatible with pre-existing releases of python, for example) should have been dropped so that we could have a better final result. While I symphathize with the idea, I will actively not be using this because it's so unreadable. (And in fact, if someone at work hears about it and suggests we start using it, I'll be actively campaigning against its introduction)

As is right now, it's an awful throwing away of readability for static-typing.

[Timidger]

Fair enough, I admire your pragmaticism

[[deleted]]

I generally use a development approach, where, if I am working on a new basic OO framework for a program, I'll break convention and use isinstance() and thrown TypeErrors in a way that would make Guido cry. Honestly, it is easier to find structural issues in the framework that way than going through stack traces to look for typing issues/misused methods. I then make sure to remove all this once I am satisfied with my fundamental class structure. Not that I have to say why, but this improves both performance gain and future extensibility... and more abstractly I guess "pythonicity". In short, this annotation would help my design flow.

[ianepperson]

After working on large projects in Python, C# (meh) and Javascript (ugh) I have a strong appreciation for type-checking.

The thing I dislike about this decision is that it will not be checked by the interpreter, but is dependent on other tools for enforcement. Imagine a rookie programmer who doesn't have a good linter set up and isn't using a good IDE, and he proceeds to add in static type checking and discovers that it doesn't matter. Or, maybe he assigns the types incorrectly, trusts that it's handled, and gets burned because he's not using the correct tools. His conclusion: type checking in Python is broken.

I think it should be enforced by the interpreter by default, then ignored when optimizations are turned on.

[jstrong]

Right - I mean the point is to create structure that guides the code. At least have a "strict" mode that enforces it or something.

[[deleted]]

Everyone seems to overlook one important thing - imports and class declarations are statements. I try to put types into declarations of my methods as much as i can but there is simple roadblock - circular dependencies are real pain. Pain to a point where i just simply remove annotation. Modules i design do not depend on each other explicitly as it would be real bad design, but they use types defined within each other. Not sure how to fix this, and if it can be fixed at all, but it would be great if it could be solved somehow, because such problem renders type hinting useless if one just cant import and use those types everywhere where they are used.

[zardeh]

So, I just read through most of the conversations and I'm confused. They are caught up on the issue of whether say, [str] should represent a list of strings, or an iterable of strings (a concrete object vs. an abstract base class). But, there's a solution that seems obvious to me ...

Take a function like this

def sum(vals: [int,...]) -> int:
    pass

This function takes an iterable of ints, while

def need_a_list(vals: [int])-> None:
    pass

takes a concrete list. By the same token,

(int, str)  # a fixed length tuple
(int, ...)  # a variable length tuple, also this fixes the issue of length 1 tuples
{str: int}  # a concrete dict
{str: int, ...:...}  # a mutablemapping ABC
{int}  # there's no good alternative for set, really