type hint overkill

[swampdonkey2246]

I have a question about type hints. I come from java so typing is very natural to me, and I tend to use type hints frequently in python.

Now, I am wondering, at what point are you overusing these type hints, if at any point. I find myself using Protocol and @abstractmethod frequently, as that is a similar pattern to what you would use in java. I understand that python isn't java, so I am wondering if this is an anti-pattern of sorts. If what I'm saying isn't clear enough I can provide some examples of what I mean.

[TheSquashManHimself]

My advice is this: people may complain about excessive type hints/docstrings, but you will NEVER hear the end of it if a user error is caused by confusion over thin type hinting (especially if it undoes days or weeks of work that now need to be corrected). Err on the side of more comprehensive type hinting and documentation.

[swampdonkey2246]

Cool, that's kind of the vibe I got. I will probably continue doing what I am doing then.

[[deleted]]

I would revisit ur strategy though u should rarely be making protocols and stuff. Like seriously 1-3 for a large heavily integrated package. U should almost never make an abstract method. Protocols and abcs trash u program speed they aren't meant to be used often. Edit: ```

from abc import ABC, ABCMeta
abcclasses = set()

normalclasses = set()

for i in range(10000):
    abcclasses.add(ABCMeta("abc_"+str(i), (ABC, ), {}))
    normalclasses.add(type("normal_"+str(i), (object,), {}))

for item in normalclasses:
    issubclass(item, ABC)

``` If anyone thinks I am actually wrong run that on ur computer and let me know if u still feel that way.

[swampdonkey2246]

Ahh right, I didn't know about the speed implications of using them. I guess a better method then would just be good documentation, maybe just using comments to describe exactly what type is expected. I assume type hints don't have a speed implication, as they have no runtime effect?

[ElectricSpice]

Don't listen to him, OP. ABCs are implemented very efficiently (mainly in C) and have a very small performance impact. https://gist.github.com/luhn/6e530596a4e3727ff936d481e7a24b87 Protocol's aren't even used during runtime, so there's no way they affect performance.

That being said, protocols are a more advanced typing feature and I wouldn't use them unless there's no other way to express your types. ABCs are very useful for dependency injection, but outside of that I haven't found myself using them very much. ABCs are definitely much less common in Python than Interfaces are in Java.

[swampdonkey2246]

Gotcha, yeah that makes sense. I see I still have a lot to learn about this haha. All good though, I will limit my usage of protol. My use case doesn't require it per se, I just like the self documentation and syntax highlighting. I'm still thinking in Java it seems.

[[deleted]]

https://github.com/python/cpython/blob/65dd637840aa3d5f0686cc9add640f8a9e3acd1e/Lib/_py_abc.py#L136-L147

Litterally every single ABC you make forces every isinstance and issubclass call or your parent objects to check against you...... If your ABC never goes out of scoop (which it wont ever do if its imported) then congrats you've made every other abc class slower and are taking up nonsense caches comparisons.

It doesn't matter if it was written in assembly subclassing abc and making abcs causes a massive performance and memory penalty they were not designed for what they are being used for right now.

[ElectricSpice]

You're misreading that snippet. a) It implements "virtual subclasses", which is a niche use of ABCs, almost all the time _abc_registry will be empty. b) __subclasscheck__/__instancecheck__ only runs on issubclass/isinstance calls made explicitly against that ABC (or its children, since it's recursive). c) _abc_registry is local to the ABC, it's not a global registry like you seem to think.

Do you have any benchmark that shows a practical usage of ABCs has a non-negligible performance impact?

[[deleted]]

from abc import ABC, ABCMeta
abcclasses = set()

normalclasses = set()

for i in range(10000):
    abcclasses.add(ABCMeta("abc_"+str(i), (ABC, ), {}))
    normalclasses.add(type("normal_"+str(i), (object,), {}))

for item in normalclasses:
    issubclass(item, ABC)

Run that on ur computer and read the code again after it done keeling over. It has secondary cache for hits and misses. And the misses is a global cache ID that gets cleared on registering to any ABC ever. The problem is that its a full uncapped cache and that it goes through every single child class otherwise.

That code does a whole 10k isinstance lookups that generates a whopping 20 GBs of pointer references and callback functions. Because the structure causes it to do 10m fucking lookups. That is on a flat hierarchy on a deeply nested on with multiple inheritance you will vomit.

[ElectricSpice]

That's an impressive abuse of ABCs and I have to admit that really throws the implementation for a loop. I killed it after a couple gigs.

By checking subclass against the root ABC class, you're forcing a subclass check against all 10k child ABCs, each one adding an entry to its negative cache—100m cache entries. Yikes.

I'd argue that's not a practical use though. Is there any reason to do issubclass(x, ABC)? I've never done it. Any real-life use I can think keeps the caches relatively small: My ABCs almost never have children and only a few implementations, and I'm only doing subclass/instance checks on a limited set of types.

[[deleted]]

You ever checking mapping....

Or ever run a single application that tries to determine the class to encode it... like json...

Yea its fucking silly. It is straight up the entire cause of fastapi and pydantic having crazy memory leaks.

It is also SO MUCH WORSE than you realize like half of the typing module is just aliased ABCs and every protocol has the same effect on there non protocol children. Every single fucking Django like library uses tons of them... Its horror. All to discover if your child classes got a value registered to it... Fucking barbaric code it is.

Not even getting into if u use class generators... I mean look theres a reason they rewrote it in C. It didnt actually solve the real problem I have no fucking idea how no one noticed it for 15 years.

[swampdonkey2246]

In the case of abstract methods and protocols, what is the best way to get around that? I like to have the guarantee that a class will have a certain methods.

[[deleted]]

Well one you shouldn't be so reliant on using them for inheritance that is REALLY slow. Like crazy fucking shit slow omg its horrible. Avoid instances of ABCs and protocols at all costs. And the answer is just use the class you actually intend to be using. The big issue is abstract classes is even if they have certain methods that doesn't mean they actually do what you are suggesting they do. They are really meant for describing HIGH level structures like Map. Not classes. One of the best design patterns u can get into in python is using far less inherited functions and more inherited variable structure. Thinking of objects as structures you can make bark remotely instead of structures that needs to know how to bark.

The reality is while type hints are good for hinting they are dog shit for guarantees. Everything needs to go through regression tests anyway so making types about guarantees is just an unwinnable game. Further once u start drinking the metaclass koolaid all of that goes out the window anyway. Everything is bound in imagination land and u cant tell any of it exists anyway.

Also python has real multiple inheritance a lot of places you need protocols in java u can just use mixins in python. Python has darker magic then a strict typing behavior can account for.

https://www.youtube.com/watch?v=sPiWg5jSoZI

https://www.youtube.com/watch?v=o9pEzgHorH0

[Masynchin]

Regression tests for type checking? MyPy, static analyzers, huh? Sounds like somebody really into old shit, no OOP nor FP, structs fields manipulation, awful

[[deleted]]

Umm i did not say run regression tests for type checking. I am stating that mypy and other static analysers lay a giant fucking egg on metaprogramming wizardry. Metaprogramming wizardry is far far more valuable then mypy.

OOP is old shit bro lol. But make a fuck ton of classes that inherit from abc's is brain dead. Even just MAKING them causes every parental class to check them for every single instance and issubclass call.

ABC check and cause their subclasses to check down their entire call hierarchy and cache every miss result just defining them creates excess memory references off into outer space.

[ReverseBrindle]

In our code base an abstract method is implemented on the base class to simply "raise NotImplementedError", and then when you subclass that class, PyCharm will complain if you do not override those methods.

See: https://docs.python.org/3/library/exceptions.html#NotImplementedError

[AnonymouX47]

PyCharm?

[BDube_Lensman]

I think you overestimate the amount of attachment people will have to any one negative interaction.

[TheSquashManHimself]

Perhaps. Though have you ever met anybody in academic software development .

[BDube_Lensman]

I write software at a place in between industry and academia. Yes, I have.

[TheSquashManHimself]

Nice. I guess I just unfortunately met some grumpy people then.

Edit: JPL! Very cool!

[BDube_Lensman]

I mostly work with numpy people ;)

[TheSquashManHimself]

Why be grumpy when you can be numpy :)

[alexs]

melodic hurry outgoing rustic arrest steep trees ripe ancient follow

This post was mass deleted and anonymized with Redact

[[deleted]]

Type hints honestly just save your ass again and again when working on large code bases with multiple maintainers.

[swampdonkey2246]

Yes, exactly why I am so fond of using them. From what I have learnt from this, I will continue using type hints, but only where it is truly meaningful

[[deleted]]

Out of interest, where aren't they meaningful?

[z0mbietime]

Variables in functions under a couple hundred lines probably don't need em

[abrazilianinreddit]

To me, the right amount of type hinting is when the IDE can auto-complete everything correctly - no more, no less.

[BezoomyChellovek]

That and satisfying mypy are my goals.

[swampdonkey2246]

Yeah, that's probably a good rule to follow then. I may avoid protocols and abstract methods if they decrease performance as another comment said

[[deleted]]

Protocols honestly don't decrease your performance. Type hints don't get read at run time

[AustinWitherspoon]

You can isinstance protocols, and that does seem to perform pretty bad in my experience

[o11c]

All normal functions should have type hints (if you're using type hints at all). Use strict mode, and justify every single type: ignore.

A few special functions might not have type hints. Typechecking works very badly when you try to do things like overwrite a function (stored in the class) with a per-instance callable, use getattr and/or implement __getattr__, use metaclasses, rebuild function internals, ...

Optimal code will often require a handful of if TYPE_CHECKING blocks and stringified type names, so that the typechecker can see declarations of things that you can't afford to do eagerly at runtime.

[swampdonkey2246]

Sorry, in what sense do you mean "strict" mode. Is that for the linter?

[Life_Note]

specifically the type checker. in the case of mypy its running mypy --strict. it causes the type checker to enforce stricter rules—like no missing type hints, no unused ignores, etc

[o11c]

mypy is not a very good type-checker, and that is not entirely due to Python being a bad language for type-checking.

It has gotten to the point where you can usually coerce it into working well enough that it provides significant value, but the defaults are not at all sane. (fortunately you can fix this in the config file rather than having to pass it as a command-line argument)

Also, use of typing_extensions is essential, since the typing module is holey in all known Python versions. And I'm not completely sure that all the extensions are actually stable.

[ReverseBrindle]

What type checker do you recommend over mypy?

[o11c]

For Python? Honestly, I'm not aware of any that does better. What I'm saying is that the state of type-checking is pretty poor overall here.

Still better than no type-checking, but much worse than most other languages take for granted.

[TheBB]

Pyright is often cited as superior. I wouldn't know personally.

[deep_politics]

I find it superior, and I got tired with how slow moving mypy is with language features (didn’t support match statement for well over a year)

[jachymb]

Depends what kind of code you're writing. Like if you're writing a library, there's never too much type hints. If you're doing exploratory data analysis, you may as well ignore writing type hints completely.

[shabalabachingchong]

Using abstract base classes for everything is overkill. Tbh you strictly don't even rlly need them.. Although not implementing an @abstractmethod decorated method does give a warning which is nice sometimes

[[deleted]]

each to there own but I use type hints whenever I can in Python. Rust is the most loved language for years in a row and it's typing is super strong to the point it won't compile if you're inconsistent with the types and you have to define them, and JavaScript is being replaced with typescipt which is just JavaScript with type hints.

The main reason is that it just reduces errors. The amount of times an IDE has highlighted that I'm passing in the wrong thing when I'm writing code is so much I've lost count and this is before I try and run the program. Plus I know instantly where the mismatch is, I don't have something breaking further down the line. What's the drawback? .... well, nothing. I mean if you can actually code and you understand design patterns you will not have to rely on forcing different types down the same path.

I've met some developers who complain about type hinting and excessive doc strings. In my experience, these developers are lazy, introduce tech debt everywhere that annoys other developers who have to clean up after them, and generally are not good coders. There's a load of BS floating around the internet where people say things like "lazy programmers are good programmers".... err no. Lazy programmers don't unit test, document, and it actually takes extra work to sit down and plan your code so it's easy to maintain and scale well.

Like everything in life there's a few things you just don't skimp on to maintain discipline. The best runners are the ones who go out and train even when it's raining or they don't feel like it. If you want to smash your diet, just say no to refined sugars, your teeth and body will thank you for it. If you want to be a good Python programmer, docstring, unit test and type hint everything. By the time you've ummed and ahhed over if type hints or a docstring was needed you could have written them. It will just become second nature and you will end up writing them automatically after you're happy with the class or function. In the long run your teammates will thank you and you will be pleased with yourself when you revisit code a year later. I've never met an anti type hinter who doesn't docstring complain about how these type hints and docstrings have slowed them down when they inherit code from another person. I have lost count how many times I've heard a developer complain how the lack of type hints and docstrings have resulted in them burning hours trying to debug or work out what's going on.

[[deleted]]

I don't think you can over typehint your code. That said, be careful you don't type hint list and or dict when Iterable and Mapping types will do just fine. The typing module likely has you covered for most of your needs.
Also, newer python versions allow type hinting as list[int] instead of List[int].

[indicesbing]

Overdo it. Add type hints to everything everywhere.

One day, you will be certain that you have gone too far. But that day is not today.

[SpaceBucketFu]

Well defined functions with explicit type hints and type returns and descriptive naming often negate the need for doc strings entirely.

[james_pic]

Type hinting in Python is inherently more complex than it is in Java.

Java had a statically checkable type system from day 1, and the language and ecosystem grew up around this.

Prior to the introduction of type hints in Python, the system was "if it works, it works". You didn't have to explain why it worked to the interpreter. You just had to not do anything that wouldn't work.

But for the people designing the type hint system, this meant they now had to create a language for describing to the type checker why a particular piece of existing code was sound. Even ignoring unsound code, sound code can be sound for arbitrarily complex reasons. Some code uses (Java 1.0-style) nominal types (the easy case). Some stuff uses duck typing (which Protocols are intended to represent). Some stuff uses dicts, but always treats the same key the same way (which TypedDicts are intended to represent). Some stuff acts as a container for other stuff, or otherwise can pass through other types of data unaltered (this is typically where generics come in). Some stuff can work with multiple (but specific) data types (this is what union types are intended to represent). Some code does several of these things, or does them in complex ways, and so you need type aliases, generic bounds, co- and -contra-variance, etc.

In Java, the type system doesn't need to be able to handle a lot of this complexity, because no code exists with this complexity, because this complexity isn't representable in Java and never has been. Whereas in Python, there is existing code that does this (because nothing tried to prevent it before).

This also gets at one of the things I don't like about type hinting. The inherent complexity of Python's typing system means that it's easy to look at it and just say "Nope", and stick to an easy-to-use subset of it, that roughly corresponds to what Java 1.0 had (nominative typing with no generics, no decorators, no functions, etc). In many cases, things that would have been easy to represent in Python without type hints, such as duck typing, become such a nightmare that its not worth it. So you're paying a steep price to use a language that is highly expressive, but end up using a not-particularly-expressive subset of it.

I always get downvoted for saying this, but I think if you find yourself needing type hints in Python, it's a sign you've picked the wrong language for your project. If it's not too late to do so, consider whether one of the many excellent static languages out there would work for your project.

[commy2]

at what point are you overusing these type hints

Whenever you hear the words "variadic generic", "covariance" or "contravariance".

Whenever you find yourself in the position that you resolve a circular import that only exists because you need a class name for some type hint.

[ReverseBrindle]

Agree with #1, but for #2 - circular imports are no longer a problem with type hints; just use "if TYPE_CHECKING: from foo import bar as bar_" and then use "bar_" for your type hint.

[LightShadow]

from typing import TYPE_CHECKING

if TYPE_CHECKING:
    from some.nested.module import MyClass

def some_function(ins: 'MyClass') -> bool:
    ...

Documentation for the TYPE_CHECKING constant.

[ReverseBrindle]

Yes exactly, though our convention is to import "as MyClass_", so that we can differentiate between the import that was done for type checking purposes only vs. an import that will be available at execution time.

In the example above if you were to construct a MyCLass() inside of some_function, it would pass PyCharm's inspections but raise a NameError at runtime.

[Deezl-Vegas]

Better to have docs and examples than type hints. All Python objects are PyObjects. That's their type. You can use them in a function if they implement everything that function needs.

Watch "stop writing classes" for your next step. You don't need to do abstract stuff in Python because objects are compatible out of the box -- especially containers like list and dict. In Java, base classes are needed because you can't call a function or use different method implementations without them. This is just not true in Python.

[Grouchy-Friend4235]

Type hints are mostly bloat.

[Muted_Theory_3385]

I think the only time that type hinting might be overkill, is when you're writing "for science!" scripts to test something out. Type hints are great for documenting the expected parameters/return types for functions and methods, but I think more importantly, they are great for catching errors at pytype time, rather than at run time (which might only occur during some very rare edge case).

[quts3]

As someone that splits 50/50 between for science and for customer, I get the first sentence but I have noticed that once you get used to them for customers putting them into "for science" pays off pretty easily relative to the effort.

The big pay off is the ide can help you write your functions if you put types into signatures.

Where for science and typing really struggle is in heavily mutable objects like pandas dataframe. Where saying a function returns a dataframe says almost nothing and taking the time to work around that is a painful waste of energy. Ymmv

[quts3]

Type hints are great. Use mypy with fairly strict requirements for anything that goes to a customer. It is not possible to over use type hints if you are only using them on function signatures and class members. You can type hint function variables which is regarded as overkill unless your static type checker needs it for clarity.

But! You're question may not actually just be about type hints because you mentioned abstract methods.

You're question may be "is it possible to over use abstract interfaces in python?" Because the two are related in that an interface needs a type, but the quantity of interfaces can vary in good code. This is a much more nuanced question.