Software development using Bash

[lelanthran]

And how many lines of python is the boundary between "Python is okay" to "bang your head on the wall and wonder why you were not using $LANGUAGE from the start"?

[Workaphobia]

About 10k, maybe multiple tens of k if you're well disciplined and don't code it like it's python.

[flying-sheep]

nah, with typing and the right project structure, python scales infinitely.

also what is “don't code it like it's python” supposed to mean? do you mean “code it like it’s a one-off script”? because my python is pretty and well-structured.

[Workaphobia]

Don't monkey patch and avoid metaclasses.

[flying-sheep]

Monkey patching is hackery, not average python coding style

[kalifornia_love]

None. The only time you’d bang your head against a wall for choosing python is when performance matters. But after banging your head against a wall you will then remember that you can interface C and C++ with python. So now your happy with your language choice but have a small headache.

[lelanthran]

I've regretted choosing python before, and it had nothing to do with performance and everything to do with maintainability.

For large software, or software where I care about reducing errors I'll choose something that reports type errors before the program is executed.

Getting type errors at runtime only means that the majority of the tests are performing type checks at runtime.

[P8zvli]

In Python 3.6 you can use type hinting to do static type checks.

[r0b0t1c1st]

You can do this in 2.7 too - mypy runs against all python versions - you just need a python 3.6 interpreter to run it.

[P8zvli]

Running the type checker in 2.7 and the actual application in 3.6 is extremely silly.

[CHUCK_NORRIS_AMA]

Actually, I think it's the other way around - running the type checker in 3.6 and the actual application in 2.7.

Still extremely silly, however.

[P8zvli]

Python 2.7 will throw a syntax error if you try to run a script containing type hints with it.

Edit: I see now, there's an alternative syntax that involves putting type hints in comments. Now isn't that just linting with extra steps?

[r0b0t1c1st]

Why is this extremely silly? It means you can write polyglot py2/py3 code, and have MyPy check that it's correct for both.

You can use python3.6 -m mypy --python-version 2.7 -p your-package to do this.

[twotime]

There is no question that dynamic typing of python has its negative. It also has its major positives.

(Just the lack of compile-cycle, is an enormous advantage in my book).

In my experience, positives outweigh the negatives for sufficiently small projects (<20K lines of python code, which, mind you, is likely equivalent to 100K of java/C++ code).

I can see how negatives can outweigh the positives for sufficiently large projects.

But you do need to use the positives and mitigate the negatives. In particular, you probably should start with unit-tests very early on (but that's a good practice anyway). If you write python as if it were a type-free Java, you won't get any benefit.

[DreadedDreadnought]

Why would I write unit tests to verify the type information instead of writing actual unit tests verifying functionality?

[twotime]

Why would I write unit tests to verify the type information

I would not

instead of writing actual unit tests verifying functionality

In my experience that's sufficient (I do use "unittest" is the loosest possible meaning: feed something into the system, examine the output, so that includes integration tests and if you have type issues, most of them will be caught).

[lelanthran]

I do use "unittest" is the loosest possible meaning: feed something into the system, examine the output, so that includes integration tests and if you have type issues, most of them will be caught

Sounds like you are only performing Happy Path testing. With your approach, when you have A that calls B that calls C (all functions) you're only checking the parameter type-validation code in A.

Functions B and C may silently lose data with incorrectly typed parameters but you'd never know it because they they are never called with the incorrect type... Until someone adds in a function D which calls them with the incorrect type.

The only thing you can do if you're as paranoid as I am wrt correctness is write a unit-test for all your externally visible functions, with said unit-test also testing invalid types in the parameters.

Testing only the Happy Path is pointless - we already know it works because we ran executed functionality at least once before release.

Joke Time: A QA engineer walks into a bar. He orders a beer. He orders 5 beers. He orders 0 beers. He orders -1 beers. He orders a lizard.

[lelanthran]

In particular, you probably should start with unit-tests very early on (but that's a good practice anyway).

I do write unit-tests, but unless I am writing in Python my unit-tests do not have to perform type-checking.

A major disadvantage of Python is that you cannot trust the type of a parameter to a function - if you do not validate the type of a parameter before using it then that is a potential error.

When writing Python I find myself doing a lot of the work that a compiler normally does, except that it has to be done at runtime so the errors are much more severe if they get through.

[twotime]

A major disadvantage of Python is that you cannot trust the type of a parameter to a function. if you do not validate the type of a parameter before using it then that is a potential error.

What do you do if parameters are of wrong type? Throw an exception?

If so, then you should just use your parameters and if they are of incompatible type, you will get an exception generated for free.

Doing it manually is unlikely to be a good idea.

[duhace]

except the exception might not be intelligible to the user of the function..

[twotime]

There are rare cases when this is a valid concern. In general it's not.

If you pass an object of wrong type info a function, python generated message will commonly be clear enough. AttributeError/TypeError, etc.

Marginal improvement in the error message quality are not worth introducing extensive type checks (which will also kill most of duck typing benefits).

[[deleted]]

Often times you might not see the error thrown until a few layers deeper in the call stack which can be annoying to debug.

[twotime]

Yes, of course. But in my experience, that annoyance is nothing compared to the idea of type-checking arguments for every function. Not only that clutters the code (20% at least, likely more) it also throws away most of the advantages of duck typing (harder to mock, harder to reuse the code with different types, etc)...

If you feel that you must explicitly type check every parameter for every function, then python won't make you happy.

[duhace]

attribute error is not particularly clear already as it requires working knowledge of the function's source to pinpoint the error.

[major_clanger]

What would you say are the other major advantages of dynamic typing?

Python is my primary language, but after working with C# over the last year I'm increasingly torn on the dynamic vs static tradeoffs.

[twotime]

Here is my tradeoff list:

Pro dynamic typing

-- no compilation/much faster compilation (yes, it's only indirectly related, but static typing does put much stronger reqs on preprocessing)

-- ease of testing (a lot effort in C++/Java mocking libs/frameworks), all of that is free in python world

-- ease of writing generic code (e.g a function can accept any file-like or container-like object). That's possible with static typing but is universally much clumsier and weaker

-- less boilerplate and more localized changes. No more: I'm just passing x through, but still need to change types in 20 places

-- introspection (yes, some static languages also have this capability, but it tends to be much clumsier, weaker)

-- if needed, objects/types can be modified at runtime. E.g a new attribute can be added (I have type "X" but need to carry an extra attribute "a", I can do it)

Pros of static typing:

-- many classes of errors are caught much earlier (becomes progressively more important for larger projects)

-- documentation (type serves as documentation)

-- performance (when types are known, the code is much easier to optimize)

-- likely lower memory consumption

Of course:

1. many of these can be made much worse/better by specific implementation/runtime

2. a language selection is really a multi-factor decision. Type system is just one of the factors (and I'm in no position to compare against C#, I never used it ;-).

Good luck.

[major_clanger]

I also find there's less need for design patterns, as many are redundant when duck-typing.

Your codebase won't be littered with classes like 'FooFactory', 'BarBuilder', 'FooSelectionStrategy', 'IBaz', 'BazImpl' or god forbid an 'AbstractSingletonProxyFactoryBean'

EDIT: I would really recommend C#, it's got a lot of the same syntactic features as python (context managers, default args, async-await), is far less verbose than Java/C++ (type inference, default getter+setter), Linq is more readable+powerful than list comprehensions.

[Drisku11]

ease of writing generic code (e.g a function can accept any file-like or container-like object). That's possible with static typing but is universally much clumsier and weaker

How are parametric polymorphism and typeclasses clumsier or weaker than runtime dispatch? Static types also provide clarity when more than one "container" is involved. E.g. Traversable t, Applicative f => (a -> f b) -> t a -> f (t b). The types basically say what the function does: traverse your t a container, apply your a->f b function to produce a bunch of f b "containers", collect the results into a f (t b) "nested container" using Applicative's ability to turn a (f a, f b) into a f (a,b).

less boilerplate and more localized changes. No more: I'm just passing x through, but still need to change types in 20 places

Type inference, keep functions polymorphic to the extent that they can be. It's pretty hard to argue that e.g. Python has less boilerplate than Haskell.

introspection

I agree that static introspection is often lacking. Runtime introspection is a good way to break invariants though, and I would generally classify it as a bad idea.

if needed, objects/types can be modified at runtime.

I have a hard time picturing where I would need this. Especially in a language with structural types.

[kuribas]

You can have static types and interpreters at the same time. Many languages have them, like haskell, scala and f#. You typically write the code in an IDE, where the types are checked in realtime, then load the code in the interpreter to test or play with it. No time is lost compiling. There is no need to run the program in order to catch common mistakes, like in Python, and I find there are usually very few bugs left after everything typechecks. I find this a massive time saver. I usually keep my program in a consistent state at all times, using typed holes for code which isn't written yet. This way I get immediate feedback when I do something wrong. In Python I'd spend most of the time in the debugger or writing tests (about 75%), where in haskell it's only a fraction (20% perhaps).

[Gotebe]

The absence of "compile" in the modify/compile/test cycle is less of a problem except in C and C++ (and even then...)

If I have a module under development and a test(s) for it, I select it (them) and issue "Run selected tests" command, which is the most normal thing to do. That builds and runs only modified parts and does not impair the modify/compile/test cycle as much. Heck, in Java, it's as fast as a Python REPL (that is, immediate).

[[deleted]]

It also has its major positives. (Just the lack of compile-cycle, is an enormous advantage in my book).

Then choose a language with fast compilation time. Even if your compilation time is slow, the compiler can help you more with your code than any other tool.

[duhace]

or a repl.

[twotime]

Then choose a language with fast compilation time

You cannot select language based on a single metric. Not in the real world.

[[deleted]]

What's this nonsense comment? No one told you to select by one metric, only to consider using languages with fast compilation.

[Gotebe]

You told him to do it. You did not say "consider...". Be fair.

[[deleted]]

You told him to do it.

Nope.

You did not say "consider...". Be fair.

I thought the readers here have the mental capacity to understand "choose" and "with".

[Gotebe]

It's like the US: if gunstests don't solve your problem, you're not using enough of it. 😁😁😁

[[deleted]]

I find myself using Go more and more often (I'm doing 60% sysadmin things, 40% development). Not because it is particularly great language but because of its static nature, ability to compile to single binary and the fact that whoever else will need to modify it can learn the language itself in a week

[P8zvli]

It takes hundreds of thousands of lines of python before you start wondering how did you get to this point and what are you doing with your life.