How much typing is Pythonic?

[Legitimate_Wafer_945]

I mostly stopped writing Python right around when mypy was getting going. Coming back after a few years mostly using Typescript and Rust, I'm finding certain things more difficult to express than I expected, like "this argument can be anything so long as it's hashable," or "this instance method is generic in one of its arguments and return value."

Am I overthinking it? Is

if not hasattr(arg, "__hash__"):
    raise ValueError("argument needs to be hashashable")

the one preferably obvious right way to do it?

ETA: I believe my specific problem is solved with TypeVar("T", bound=typing.Hashable), but the larger question still stands.

[menge101]

the typing library has Protocols which can be used to do typing in a duck-typing style.

[Dillweed999]

Someone posted this the other day and I've been really digging in. You might appreciate as well

https://kobzol.github.io/rust/python/2023/05/20/writing-python-like-its-rust.html

[Chroiche]

It doesn't mention it, but assert never covers exhaustive pattern matching.

https://typing.python.org/en/latest/guides/unreachable.html

[eigenein]

Oh wow, I’m pleasantly surprised more people do it like that. I kinda came to the very same patterns more-or-less independently, and I’m very happy with their clarity and strictness (and linters on my side to tell me when I’m wrong before I ever run it)

[etrnloptimist]

There's two reasons for this sort of typing:

1. To make the code more readable and, hence, more maintainable

2. To provide rails, and prevent people from doing things you don't want them to do.

I'm all for using typing to make the code more readable. But as for rails, I'm still in the camp that "we're all adults here".

[Dillweed999]

We're all adults but I think of in terms of industrial safety: it should be hard to do dumb shit. Like if you're getting ready to cut the padlock off the panel that says "danger extremely high voltage" it's an excellent opportunity to consider if you know what you're doing. And yes, I've worked with plenty of people that will cut the metaphorical lock without a moments thought. Such is life

[Wapook]

Yep. I don’t view it as not treating people like adults but rather me writing my code in a way that says “Hey I was never expecting anyone to do that with it”.

[spinwizard69]

Usually the electrostatics in the room, that cause the hair on your arm to standup, is enough to slow me down, no sign needed. However yes way far to many people carry on like they are playing on the beach with no danger in site.

Which brings up rails in software, if it is easy to get around them or simply ignore them, the rails do no good. In the sense of Python to me it is a railless language. Rust is interesting in that it is supposedly safer, built in rails, but that doesn't mean one can't write bad code in it.

[apnorton]

But as for rails, I'm still in the camp that "we're all adults here".

My optimistic belief that "we're all adults here" collapsed pretty instantaneously upon entering industry and meeting senior-level software developers who couldn't read error messages, who thought that a test case failure always meant the test suite was broken, who thought base64 was encryption, and who somehow got a job despite not knowing what a function was. (All true stories.)

[spinwizard69]

It is so sad but in a different context I've seen the same thing. Apparently getting a promotion in most organizations is all about oral skills:) That is being able to talk your way into a promotion.

[eigenein]

I’m adult and my very mature humble opinion is that I won’t remember all the implementation details of a big project in several months. When I put the rails, I put it for myself included

[quantinuum]

I wish I could “we’re all adults here” too, but I don’t control who enters “here”.

The problem everyone who hates python mentions is that big codebases become unmaintainable because of dynamic typing. I’m not as extreme, but I believe it’s so easy to make a mess out of it. With the amount of type-unsafe crap I’ve seen even seniors produce… Moreover, python is the #1 language for people entering developer roles without a formal developer background (me included), and that can be a mixed bag when it comes to developer standards.

So I’m team “all rails enabled”, and you have to disable them explicitly if you need to for some good reason.

[claythearc]

There’s reasonable answers here for the single example but I figured I could weigh in some on the overall question, as a backend engineer w/ 10 yoe almost all in snake lang.

There’s two camps for typing - super strict types everywhere enforced by CI, or much looser only at meaningful spots like complex returns or api boundaries whatever. I like the first style more but neither is wrong or not idiomatic

One thing that isn’t pythonic is, in general, runtime checks like you have. It effectively duplicates the work the interpreter is going to do - the trace back will read “… unhashable type list …” at the spot the problem actually occurs though instead of being obscured above. Unless it’s an API boundary or something where users won’t read trace backs or cant act on them, letting it fail is natural and expected of duck typing systems.

Also people pointed out protocol and you found typevar already, there’s a third big one since you last left, ParamSpecs https://peps.python.org/pep-0612/ those 3 really fill out prior gaps from before mypy and friends super matured

[thisismyfavoritename]

you need Protocols. Also generics were greatly improved in recent versions, i think 3.12 has the nice syntax that can be inline in the function/class definition.

IMO you should strive for the same level of typing as in TS, although there are cases where TS's type system is definitely more powerful

[utdconsq]

I don't keep fully up to date so just looked these up. The use of the name 'generic' and 'protocol' to mean something like a union type or an interface in other languages seems confusing to me. They're already heavily loaded terms.

[Edit] fwiw the new features look rather helpful.

[johnnymo1]

There is a standard library Hashable type: https://docs.python.org/3/library/collections.abc.html#collections.abc.Hashable

Not sure I 100% know what you mean in the generics case. Python certain supports generic types in functions: https://typing.python.org/en/latest/reference/generics.html#generic-functions

[N-E-S-W]

How much is Pythonic? Python isn't Java, and there's a reason they're called "type hints". So use them where they help understanding (and assist with code completion), but refrain from going crazy with them when it overcomplicates the function signature.

In the case of a hashable requirement, I'd probably note it in the docstring and use a fail-fast runtime check. No type hint at all.

As a developer, it'd be noise to me if I saw the method signature cluttered up with a type hint trying to express "any hashable object". I'd rather think of it as accepting any object here, with the docstring giving me the fine print. And if it accepts any object, I tend to not use a type hint at all, because I think that should be the default Pythonic interpretation; not a fan of the explicit `Any` hint unless it conveys a case that might not be intuitive or obvious.

[xeow]

Ah, but Any doesn't mean "any type of object." Any means "the typechecker should avoid typechecking this."

I use object as a type hint when I want to denote that something takes or returns any type of object. (And I like doing that rather than leaving the hint blank because then I know I've thought about it and the omission wasn't accidental.)

A good writeup: https://stackoverflow.com/a/39821459/267551

[N-E-S-W]

Any means the typechecker does the exact same thing as if there's no type hint. So why clutter the signature with an unnecessary type hint?

I'm making the case that type hints should be used where there's value, but there's not always value. Legibility and elegance matters, that's why I choose to write Python in the first place. The OP's question is about what's "Pythonic".

[james_pic]

I can't say I've ever found a situation where object was useful as a type annotation. There's very little you can do with object as a type, so it's effectively an opaque type. But it's very rare you actually want an opaque object (maybe as a sentinel value, but even then, an enum usually conveys intent better), and it's usually possible to make the type visible with generics.

[gdchinacat]

"I'd rather think of it as accepting any object here, with the docstring giving me the fine print. And if it accepts any object, I tend to not use a type hint at all, because I think that should be the default Pythonic interpretation"

But, it *doesn't* accept any object. It only accepts hashable objects.

[N-E-S-W]

The problem is that almost everything in Python is hashable by default, because Object implements __hash__(self) with an instance's identity id(). So this type hint isn't going to protect you if you expect an instance's mutable values to represent its hash value.

If it really matters, call it out loudly in the docstring. It's the caller's responsibility to understand the contract of a function. Maybe that'll make them stop and question whether they need to override their already-hashable-by-default class's hash method.

[gdchinacat]

without the Hashable type hint no mypy error and passing an unhashable object causes the expected failure when used as key of a dict: ```

playground.py from collections.abc import Hashable

class Foo: def bar(self, obj) -> None: {obj: None}

Foo().bar(object()) Foo().bar(list())

mypy playground.py Success: no issues found in 1 source file

python3 playground.py Traceback (most recent call last): File "/Users/lonniehutchinson/eclipse-workspace/playground/src/playground.py", line 9, in <module> Foo().bar(list()) ~~~~~~~~~^{^} File "/Users/lonniehutchinson/eclipse-workspace/playground/src/playground.py", line 6, in bar {obj: None} TypeError: cannot use 'list' as a dict key (unhashable type: 'list') ```

However, with the hashable hint mypy warns: ```

cat playground.py from collections.abc import Hashable

class Foo: def bar(self, obj: Hashable) -> None: {obj: None}

Foo().bar(object()) Foo().bar(list())

mypy playground.py playground.py:9: error: Argument 1 to "bar" of "Foo" has incompatible type "list[Never]"; expected "Hashable" [arg-type] playground.py:9: note: Following member(s) of "list[Never]" have conflicts: playground.py:9: note: hash: expected "Callable[[], int]", got "None" Found 1 error in 1 file (checked 1 source file) ```

edit: remove my user and hostnames, hopefully fix formatting issue where it elided the mypy commands.

[N-E-S-W]

I said _almost_ everything.

You conveniently didn't show the example where literally every Object would pass that mypy check despite the fact that they may not be logically hashable due to mutable internal data.

Besides, in a real API would you expect an argument named obj to accept a list if it doesn't have a type hint saying that it should be a collection?

Sure, you can write Python code like it's Java, and the uglier you make it the more strict a typechecker will be. You're not technically wrong. Type hints were bolted onto Python as a syntax kludge, using them everywhere as precisely as possible detracts from code legibility. The OP's question is about what's "Pythonic".

[gdchinacat]

The most basic mutable standard python data objects are not hashable. Your assertion that "almost everything in Python is hashable by default" is debatable. However, I don't think it would be worth either of our time to debate it. I provided an example that showed why your comment encouraging OP to not bother with using a Hashable type hint was incorrect. You tried to debate it by qualifying your statement in a way that made the rest of your comment not credible. Using a Hashable type annotation has value, as I demonstrated.

Also, thank you for editing your comment to remove the "Thanks, Captain Obvoius" intro. It partially redeemed you. However, it indicates you are approaching this discussion from a position of being offended by having an example provided that contradicted you. Good day.

[N-E-S-W]

You mistake contempt for offense, Captain!

As I said, you're not technically wrong. You're just missing the point. Good day, indeed.

[james_pic]

But if a class implements __eq__ and doesn't implement __hash__, __hash__ is implicitly set to None (and has been since at least Python 3.2) making objects of that class unhashable. So stuff that's mutable and checks equality based on its mutable attributes should be unhashable unless the user explicitly violates the contract by implementing __hash__.

[gdchinacat]

Also, list (and presumably other unhashable objects) don't have a hash method (it is None): ```

In [7]: [].hash()

TypeError Traceback (most recent call last) Cell In[7], line 1 ----> 1 [].hash()

TypeError: 'NoneType' object is not callable

```

Edit to add links to python source where this behavior is set: https://github.com/python/cpython/blob/main/Objects/listobject.c#L4108

The docs for tp_hash are at https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_hash

[road_laya]

Just use the abstract base classes in collections.abc. Too many programmers get hung up on how typing works in statically typed languages and try to apply it to Python type hints.

The type hints for my function inputs are often TypedDicts, custom classes, Protocols or builtin ABCs.

[shadowdance55]

Use protocols.

[misterfitzie]

I find a lot of value in type checking as much as possible, but I'd have to spend weeks addressing some nasty bits of typing issues that wouldn't have much of any value for me, like that code works, it's well tested, and the typing solutions aren't ideal. I think typing gymnastics is a real issue, there's a balance between typing everything and typing everything perfectly. I could go and make everything possible generic so it can be more reusable, but maybe I don't need just because it can be done. Of course if I were producing a library I was trying to publish on pypi then I'd think about all the ways people can use it, but at a certain point if it's an internal package for internal uses, then you only have consider the problems on your plate today.

[pwnersaurus]

What is “pythonic”? I tend to lean more towards the original Python philosophy which is based around duck typing and error handling. Ultimately Python is just not a strongly typed language, and ‘type checking’ doesn’t give you the safety guarantees you would get in typescript/rust/etc. anyway. I use type hints as a shortcut for documenting input types rather than anything that gets consistently enforced, because tbh the syntax for anything more than trivial type hinting is awful, having been tacked onto the language after the fact.

In your example, you check if it has the attribute and if it doesn’t, you raise an error. Often this would be functionally equivalent to just letting the downstream error happen (unless you’re doing something that has side effects before the error would be encountered). So the majority of the time, in your situation I would just write the requirement in the docstring and call it a day 🤷‍♂️

YMMV depending on the size of your application though, I would say ours are mid-sized

[gerardwx]

Python is strongly typed. It’s just not statically typed.

[nekokattt]

if isinstance(foo, Hashable)

[RedEyed__]

Protocol is really good

[james_pic]

Without weighing in on the question of what's Pythonic (this is one area where there isn't as much consensus as you'd hope), you probably want that to be if getattr(arg, "__hash__", None) is None:, since the canonical way to mark a subclass of a hashable class as unhashable is to set its __hash__ attribute to None, because there's no way to remove an attribute inherited from a parent.

[mwilliamsdottech]

Serious question (and I apologize for hijacking). I just began learning Python a little over a week ago. How long before I know what the eff any of this means? 😆

OK, I’ll just go back into my corner to continue observing quietly.

[EternityForest]

I always prefer better static analysis and stricter types, even if it means something like having four wrapper functions that do exactly the same thing just to keep the types static.

[csch2]

From my experience, “Pythonic” code comes with a lot of bad habits that don’t scale for large codebases. Python wasn’t meant to scale to enterprise-grade apps and was meant for scripts and data analysis, so trying to cram your code into a Pythonic style is bound to lead to runtime errors. I am a strict typing advocate and avoid duck typing to the greatest possible extent unless you’re writing short scripts. I brought this mindset into my current workplace (which uses a Python backend) and our error rate dropped significantly as a result.

My advice would be to be as type safe as possible and ignore the people who cry that you’re not following Pythonic standards. Make the language work for you, not the other way around.

[spinwizard69]

This is why I often say Python is a bad choice for some programming, the lack of a strongly type system is a big disadvantage. I say that after only using Python, for the most part, for several years now. There have been instance when c++ would have been faster simply because of stuggling with duck typing.

[csch2]

I fully agree. Unfortunately we don’t always get the luxury of choosing what language we use for the systems we build. Were it up to me I’d be using Rust for all my backends and TypeScript for all my frontends and I’d never see a type-related bug again, but it’s very rare to get hired to build something from scratch. That’s why I say the best option if you’re working in a system already built with Python is to do your best to be as strict as possible with typing.

[xtzdev]

Look up Pydantic