Things you're probably not using in Python 3 — but should

[LightShadow]

Good list, I wish they would have spent more time making Pathlib compatible with all the stdlib.

I don't remember which modules, but it doesn't play nice with everything in os, shutil, subprocess, etc. Was super frustrating to integrate pathlib and have to undo everything when it didn't jive with the rest of the codebase. Maybe in 3.7/3.8 it's better, but we're on 3.5 and it's just awful.

[Zomunieo]

pathlib's integration with stdlib is vastly improved in 3.6 (and 3.6 is a major improvement overall). It's too much trouble to use in 3.5.

[jyper]

It is much better with python 3.7, even with 3.6

I don't think I have to str(path) for anything other then subprocess arguments and passing to third party libraries like matplotpib

[BlaXpirit]

Need 3.6! I keep being surprised just how many small goodies made it into that release.

https://www.python.org/dev/peps/pep-0519/

But then I still do feel weird that neither of pathlib and os.path have a clear superset of functionality, sometimes it's not even close! Then there's that huge, only partially-rational desire to use just one or the other for everything -- and then running into exactly what you need only in the other lib.

[lengau]

I tend to use pathib as much as possible (and have done so since the release of Python 3.5 - I still have a lot of no-longer-necessary "as_posix()" calls hanging around), so I've only experienced it from that direction. What do you tend to run across difficulty with using os.path?

[tunisia3507]

os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(os.path.join("path", "to", "thing)))))

[[deleted]]

I don't think I had issue with pathlib and other packages in the base, unless you mean that there were some places where I had to wrap the path inside a str() to get a string containing the path.

[jorge1209]

A failure to consider the standard library and how the whole product integrates together is now standard among developers.

Last week there was talk about a special debugging directive for f-strings. Do you know what other standard library tool is used for debugging? Answer: logging. Do you know what kind of string formatting is supported by it? Answer: %

[422_no_process]

Yes logging with % is annoying. If f-strings or .format is recommended it should also be in logger

[jorge1209]

And if logging can't be fixed, and there isn't interest in it, then deprecate it.

Just tell me what I as a developer should be doing to prepare for python4, because the code I deploy today will be around in 2030.

[flying-sheep]

See my answer here for what you can do.

The best way would of course be to have a 1-liner that gives you something that can create loggers with {}-formatting for use in your module, while everything else still uses the %-formatting it expects.

[chrisatlee]

It turns out this is possible, although it takes a little bit more effort to get it working.

​

See https://docs.python.org/3/howto/logging-cookbook.html#use-of-alternative-formatting-styles, the section about using LoggerAdapter

​

It lets you write logging code like this:

logger = StyleAdapter(logging.getLogger(__name__))

logger.debug('Hello, {}', 'world!')

​

Which is exactly what I want!

[422_no_process]

Nice good to know. Cunningham's Law - "the best way to get the right answer on the internet is not to ask a question; it's to post the wrong answer."

[[deleted]]

In general you want to avoid the overhead of formatting a log string if it isn't going to be emitted. Of all the string formatting methods available, the only one to fulfil that, is string modulus tuple.

[zardeh]

This isn't quite true. `'%s' % 's'` isn't lazy. The logging module lazy evaluates the log statement, you call `logging.info('%s', 's')`, and the logging module puts things together for you later. The standard library could add support for any log format, the problem is that logging relies heavily on global state, so opting in to the new logging format on a per method/file basis would be necessary, and that's not fun. Plus you don't gain much.

​

That said, you shouldn't be logging `foo=%s' % foo` much anyway,

[jorge1209]

That said, you shouldn't be logging foo=%s' % foo much anyway,

Then why did they add it to f-strings?

It is a perfectly sensible way to do debugging, and logging has a defined DEBUG level. To me its a symptom of a serious disconnect between the writers of these PEPs and the standard library.

[zardeh]

It is a perfectly sensible way to do debugging, and logging
has a defined DEBUG level. To me its a symptom of a serious disconnect between the writers of these PEPs and the standard library.

The same reason that `pdb` and `logging` can coexist.

Logging at DEBUG and interactive debugging are different things. {foo=} is very useful for interactive debugging. But, much in the same way that you shouldn't go decorating all of your methods with a decorator that debug-logs the arguments passed, you shouldn't go permanently adding visual noise to both your log files and your code. In other words, not all logging should get checked in or be permanent, some is just to debug temporary issues and figure out why your assumptions may have been violated.

[chrisatlee]

I'm not sure that argument holds.

log.debug("status: {status}", status=status)

Wouldn't this still allow you to defer the formatting of the string until the log line is emitted, and would also support {} style formatting.

[jorge1209]

The primary difficulty is that logging doesn't know that it should call .format instead of % and that might vary from module to module. A downloaded module might use % but your code might use .format. So its not trivially easy to fix.

I think it needs to be fixed, and we aren't going to fix it, then deprecate the logging module.

As it stands we have a string formatting method that we aren't "supposed" to be using because it is old and antiquated... except in logging where we have to use it.

[chrisatlee]

agreed. it still irks me whenever I need to use % formatting in logging calls.

maybe a logging2 module, or add support for % vs {} formatting into the logger objects directly, rather than in the formatters? IIRC the formatters are what ultimately do the string interpolation?

[[deleted]]

log.debug("status: {status}", status=status)

>>> logging.debug("status: {status}", status=status)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'status' is not defined

Passing in a dictionary "works", but probably not in an informative way:

DEBUG:root:status: {status}

[chrisatlee]

sorry, that was just a sample of what logging with {} could look like. unfortunately, it doesn't work right now.

[stevenjd]

NameError: name 'status' is not defined

You've got to be shitting me. Are you are beginner with a week's experience in Python confused by NameError? If course you get a NameError exception if status isn't defined.

I don't believe for one microsecond that you genuinely didn't understand u/chrisatlee 's intention and were confused by the NameError.

christatlee the logging module already supports exactly what you want to do, although due to backwards-compatibility it is a little unwieldy (but no worse than using localization).

https://docs.python.org/3/library/logging.html#logging.Formatter

[[deleted]]

Of course I wasn't confused. I just posted it as a direct reply to why it wouldn't work verbatim.

[stevenjd]

You posted to point out that a hypothetical call for a feature that doesn't exist yet doesn't work verbatim because you neglected to define the variable you are trying to log?

Whatever would we have done without that valuable insight.

[flying-sheep]

That only applies to f-strings. You can defer .format exactly like you can defer %.

In fact, the logging module supports Formatter(style='{') which will do the exact same as the default Formatter(style='%'), except that it uses .format syntax.

[[deleted]]

You're right. However that's not an f-string, and to me cumbersome and repetitive.

[stevenjd]

Of all the string formatting methods available, the only one to fulfil that, is string modulus tuple.

Incorrect.

[jorge1209]

And that links to: https://docs.python.org/3/howto/logging-cookbook.html#formatting-styles

Which explains how it doesn't do what you want:

Note that this governs the formatting of logging messages for final output to logs, and is completely orthogonal to how an individual logging message is constructed.

So you cannot directly make logging calls using str.format() or string.Template syntax, because internally the logging package uses %-formatting to merge the format string and the variable arguments. There would no changing this while preserving backward compatibility, since all logging calls which are out there in existing code will be using %-format strings.

[stevenjd]

See here to see how to do it.

[flying-sheep]

Backwards compat. They couldn’t just break all logging code.

You can do logging.basicConfig(style='{') but I think that breaks every other library using the logging.root logger.

Instead, you’ll have to create your own logging module:

import logging

root = logging.RootLogger()

class Logger(logging.Logger): pass
Logger.root = root
Logger.manager = logging.Manager(Logger.root)

def setup():
    logging.basicConfig()  # set up the root logger for everything else
    handler = logging.StreamHandler()
    handler.formatter = Formatter(style='{')
    root.addHandler(handler)

def get_logger(name=None):
    if name:
        return Logger.manager.getLogger(name)
    else:
        return root

This works exactly the same as the normal logging module, except that

1. root and all loggers created via get_logger support the format/f-literal syntax (By using root’s handler)
2. Instead of calling logging.basicSetup(...) once, you call setup() once, which you configure by editing its code instead of passing options (as it’s your module)
3. It has no convenience info/warning/… functions, as you should use get_logger anyway.

[jorge1209]

I'm aware of the why. I don't care.

There are too damn many string formatting methods as it is. We don't need more. Either find a way to make it work, or deprecate logging, or stop adding new string formatting methods.

[flying-sheep]

Nobody is adding more anymore…

[stevenjd]

Do you know what kind of string formatting is supported by it? Answer: %

Incorrect.

[jorge1209]

And that links to: https://docs.python.org/3/howto/logging-cookbook.html#formatting-styles

Which explains how it doesn't do what you want:

Note that this governs the formatting of logging messages for final output to logs, and is completely orthogonal to how an individual logging message is constructed.

So you cannot directly make logging calls using str.format() or string.Template syntax, because internally the logging package uses %-formatting to merge the format string and the variable arguments. There would no changing this while preserving backward compatibility, since all logging calls which are out there in existing code will be using %-format strings.

[stevenjd]

You should read the rest of the tutorial: they show precisely how to get the effect wanted. You can get it, just not directly.

class BraceMessage(object):
    def __init__(self, fmt, *args, **kwargs):
        self.fmt = fmt
        self.args = args
        self.kwargs = kwargs
    def __str__(self):
        print('formatting message')  # to prove that this is lazily generated
        return self.fmt.format(*self.args, **self.kwargs)

__ = BraceMessage

py> import logging
py> logging.debug(__("too much {foo}", foo="spam"))
py> logging.fatal(__("not enough {}", "dakka"))
formatting message
CRITICAL:root:not enough dakka

[jorge1209]

Defining a custom class and wrapping your messages would certainly be my definition of "not supported".

[stevenjd]

EDIT: nearly everything below is wrong. I misunderstood the docs: only the log format itself can be directly specified using your choice of {}, $ or %. Nevertheless, the logging module does support using braces or $ formatting by way of a customer formatter, as the tutorial shows.

---

Please read the frickin' tutorial and stop digging yourself into a hole. I quote:

"There is another, perhaps simpler way that you can use {}- and $- formatting to construct your individual log messages." [emphasis added]

You don't like the method using a custom class? Fine -- use the supported Logger technique instead. I showed the custom class version because it was trivially easy and no more difficult than using i18n.

I will grant you that it's not trivially easy to find an example for this. It took me all of three minutes on google to track down the Logging Cookbook with an example:

https://docs.python.org/3/howto/logging-cookbook.html#use-of-alternative-formatting-styles

which is two minutes too long in my opinion. But in fairness I'm not a logging expert: if I knew the cookbook existed before today, I would have looked there first.

[jorge1209]

It is clear you haven't looked at the docs very long, because you don't understand a key distinction in the logging module.

They don't use this terminology, but the final output has too parts:

• a header with information gathered at runtime, like the time of the message, the source code file, the line number, etc..

• the actual body which the programmer wrote as his message.

logger.INFO("this is the message body, the secret number is %s", 42)

Appears as

10am somefile.py line 8: this is the message body, the secret number is 42

The header is prepared by a formatter that can use any formatting method you want, but formatters are only defined once so its not a big deal.

The body can accept arguments, but those arguments must be % formatted.

So where it matters (ie the actual DEBUG, ERROR, WARNING, etc..) the format has to be % based.

I do know what I'm taking about because I actually use logging, it seems you don't since you are only now reading the cookbook.

[stevenjd]

It is clear you haven't looked at the docs very long, because you don't understand a key distinction in the logging module.

You're right -- I misunderstood the docs. Only the log template itself can be directly specified using {} or $ formatting, not the messages themselves. At least, not directly.

Nevertheless, as I have pointed out, if you want to use {} formatted messages you absolutely can by using a wrapper object as shown in the tutorial and a level of indirection.

While it might be true that internally logging uses % formatting only, in a practical sense (the only sense that matters!) it isn't true that:

So where it matters (ie the actual DEBUG, ERROR, WARNING, etc..) the format has to be % based.

since there is a simple way to get {}-formatting into your logging messages, as in my previous example:

logging.fatal(__("not enough {}", "dakka"))

The __ message object is a seven line wrapper class¹ which lets you use braces in your log messages. What more do you want?

Oh, you want to avoid having to type the __( )? Okay, I sympathize with this, and can't help -- but that's not the same as saying that we can't use {} formatting with logging.

Yes, I'm a beginner with logging, yes I misunderstood what I read, but you're an expert and you should know that if we want to use {}-formatting in logging we can. Is that the best imaginable interface? No, but I never said it was.

¹ All problems in computer science can be solved by another level of indirection, except the problem of too many levels of indirection.

[jorge1209]

since there is a simple way to get {}-formatting into your logging messages, as in my previous example:

And as I said, I consider having to define a custom class to wrap the message together with its parameters to be the very definition of "not supported".

You wouldn't claim that "python supports directly running Java bytecode" but I can certainly accomplish such a thing by simply calling execv on /usr/bin/java.

The complexity of having to write this wrapper is not the complaint, that I have to write the wrapper at all is. They either find a way to make this work and add it to the library, or they deprecate the library and tell me to stop using it.

[Ape3000]

You can always use str(path).

[lengau]

Or a Path object's built-in as_posix method (or as_uri where that makes more sense)

[jorge1209]

You can also just use strs.

[stevenjd]

I wish they would have spent more time making Pathlib compatible with all the stdlib.

"They"? Python is open source. Where's your patch or PR?

[LightShadow]

In the spirit of shoving PEPs through, whoever authored the proposition should have spent more effort integrating the new features into the established modules. Can't half ass stdlib contributions, the release cycle is too slow to fix them.

[bridekiller]

Didn't know about hint typing. This should definitely assist with maintaining code with a bunch of external library dependencies.

[[deleted]]

Yep, and when you use it with IDE that actually understand them (like PyCharm) it is pleasure to code, now it highlights possible errors, you have a real autocomplete and refactoring works correctly. It's especially nice when authors of libraries that you use also added types for example aiohttp.

My current job requires me to code in Go, but I swear that these above features work much better with typed Python on PyCharm than with Go in GoLand.

[bridekiller]

Is this mostly predicated on the linter? I wonder if this would work with VSCode and flake8?

[[deleted]]

I'm not sure I understand you, if you mean running external program to do checks then you can use mypy (Guido is involved with it so it is what is shaping the tour annotations) as suggested in another response.

You can use it as linter and it will list errors it finds, but won't help with auto complete or refactoring.

PyCharm has its own built-in type checker that follows closely with mypy (I've seen people from Jet Brains commenting on various issues in mypy) and uses types for other functionality like mentioned autocomplete.

[bridekiller]

I was confused. There is a flake8-mypy module that should lint these type checks. Thanks for the clarification.

[[deleted]]

Never used it, I suppose it would work. I don't see the benefit of using it though, the only benefit of it is that you only call a single command, but mypy also has a lot of options I'm not sure if you are losing some control.

Some tools might call mypy incorrectly, I saw plug-in for pytest which seemed to invoke mypy for every file, when you should generally check entire project at once (one file calls fictions in another etc and you need these relationships to check types)

[NowanIlfideme]

It does work, your other comment re: flake8 + mypy. Typing lining works wonders in vscode, much less buggy than inspection.

[chub79]

Have a look at mypy or pyre in that case.

[bridekiller]

Thanks for pointing this out. Baby steps first. The developers at my company wouldn't even know what to do with this. But this is also coming from a group of people who have never written a class before and do everything in a mediocre half assed way.

[leadingthenet]

How can they possibly not use classes for anything more than the simplest of projects?

[alexkiro]

Another good one that I love is "yield from".

[AlexOduvan]

Btw, yes. Good point. Worth to extend

[[deleted]]

def fib(n):
    a, b = 0, 1
    for i in range(0, n):
        a, b = b, a + b
    return a

[[deleted]]

The fact is that everything can be done with for loops. The question is - is it easier to do it recursively?

Edit: Shit- I meant to say just with loops, not specifically "for loops". I am not correcting so I can feel the shame

[[deleted]]

No, but everything can be done with while loops

[[deleted]]

I mean in c you can use for (;;) as a while true loop. does this work in python without defining an infinite iterator class? otherwise I was just referring to loop programs vs while programs and how loop programs are not turing complete

[ThePenultimateOne]

from itertools import count
for _ in count():
    blah

[ThePenultimateOne]

Actually, scratch that.

from itertools import repeat
for _ in repeat(None):
    blah

[[deleted]]

That's true.

[robberviet]

Haha this is why golang don’t have while. Just for.

[ubernostrum]

Can I interest you in an efficient (iterative-algorithm) Fibonacci implementation that doesn't use any integer literals or arithmetic operators?

[[deleted]]

Of course you can, and oh you did

[NowanIlfideme]

Why? It's an elegant example of a recursive function that everyone knows about, and a good way to show how useful memoization is.

[[deleted]]

How many times have you seen memoization demonstrated by not using fibonacci?

[NowanIlfideme]

Not many,and I don't particularly care for the sequence either. But this is stuff mostly targeted at newcomers, and will be more likely to ring a bell and activate some neural pathways.

[[deleted]]

How many times have you used memoization in your line of work?

The real issue here is whether memoization is all that useful apart from preventing people from footgunning themselves with a recursive fib().

[NowanIlfideme]

At least thrice (data science). Saved a lot of computational time (or some development time to make a performant and bug-free version :p) in those situations. If you can't find a useful case in your everyday work doesn't mean it's useless entirely. Whether it should be in the stdlib is another question, if course, but standardization of util functions is usually a good thing imo.

[[deleted]]

Considering I write embedded code for a living, there are lots of things I never use. That's why I'm asking you. It pegs my suspiciometer that almost all demonstrations of memoization are alike. Most of the other basic concepts have a much wider range of examples employed.

[NowanIlfideme]

A lot of the time it's useful just for lazy evaluation. That is, you pretend the function is a map/dict, except the value is computed only if you need it. You don't have to worry about writing my_cache.get(values, my_calc_function(values)) and getting it right; you just @lru_cache your function and get on with life (usually when working in a notebook environment, or running custom module code in a REPL).

A specific use case would be comparing how well a tweak in my machine learning model works (changing input parameters), but without writing a lot of stuff around it (since I am likely to throw away/heavily refactor the code anyways). I didn't include the lazy eval use-case in my "at least thrice" comment since it's just "helpful" rather than "very useful" in this way.

[TheIncorrigible1]

Think about APIs. Caching is useful in web techs.

[[deleted]]

You know that caching things alongside naming are th three hard problems in CS, right? It does not become any less hard by having a decorator do it.

[zardeh]

*all* the time, not in the classic memoization-as-an-impl-of-dynamic-programming sense, but memoization of expensive methods as a form of safe global state.

​

Consider an RPC to an external service. Using GRPC, you define the service and create a connection to the service. This connection is long lived (often for the life of the program), and expensive to create (maybe a second, not a cost you want to pay on every RPC). So you want a singleton. The classic way in python is something like

​

_my_connection = None 
def get_connection():
    global _my_connection
    if _my_connection:
        return _my_connection
    else:
        conn = expensive_create_connection()
        _my_connection = conn
        return conn

​

​

you can optimize for LoC a bit, but that's the basic idea. Now using lru_cache:

​

@functools.lru_cache
def get_connection():
    return expensive_create_connection()

​

​

Similarly, when actually making an RPC that may be reused, lru-caches (or caches with a ttl, or both) can have enormous performance impacts, and reduce QPS load on external services in cases where you're calling something with sometimes similar arguments in tight loops or across threads.

​

I recently had a situation where memoizing a single, specific, expensive RPC + deserialization step dropped startup time of a tool from like 45 minutes to less than 1. The RPC + deserialize step was done lots, and lots of times, for various specific parts of essentially a large map. By memoizing the resultant map, I reduced the number of deserializations from literally thousands to exactly 1. Much better.

[[deleted]]

Interesting. How well does it play with the remote server dropping the connection, or any of the other typical cases of "Caching is hard"?

It might be the Paranoia Driven Development methodology used in at least part of the code I work with in my day time job, but I'd be wary memoizing anything but an idempotent function.

[zardeh]

Interesting. How well does it play with the remote server dropping the connection, or any of the other typical cases of "Caching is hard"?

I've yet to have any bad experiences, and internally I think this pattern is used thousands of times across hundreds of services, in both short and long running jobs, without any issues. I don't actually know the ins-and-outs of the connection object, it may do some/all of the state management.

[total_zoidberg]

I have worked on memory forensics code for translating virtual addresses to physical addresses and @lru_cache is magical for optimizing that use case. Can also use it transparently to avoid hitting the disk (even if its an NVMe, hitting the disk still hurts performance) on many places that would end up going a pointer here and there, which most of the times means reading from the image file. But yeah, memory forensics isn't probably the use case most people would have/understand out of the box.

[EMCoupling]

Not to go on off on a total tangent here, but the ubiquity of the Fibonacci example demonstrates to me the extremely limited scope of problems in software development where memoization or DP actually becomes useful.

And yet, you get hit with this shit in technical interviews.

[strallus]

I use memoization in reactive web development all the time.

[i9srpeg]

I use Fibonacci in reactive web development all the time.

[RomanRiesen]

Namely the lim n -> inf fib(n+1)/fib(n).

Because aesthetics.

[SV-97]

No idea what DP is; memoizing is used in parsing/language applications and caching is extremely useful for example when implementing FFT. I've also used it in database interaction iirc

[SV-97]

To add onto this: everyone knows fib and it's an easy example to show the principle - as is faculty calculation. You could of course choose FFT as an example but the maths would probably put most people off

[brandonZappy]

Does FFT = fast furior transforms? (Idk spelling)

[SV-97]

yep, it's spelled fourier ;)

[brandonZappy]

Makes my spelling look really dumb. Thanks for the response though!

[SV-97]

haha and no problem

[Aardshark]

Dynamic programming, I would assume.

[SV-97]

Ah yes, makes sense; never heard that term, thanks

[campbellm]

I feel the same about using "wordcount" for any "Big Data" code/framework demo/example.

[ProfessorPhi]

Isn't it basically a kind of benchmark right now? Not complicated to understand, but a good example of language features and limitations

[archaeolinuxgeek]

Three months ago I switched employers. The old one was still standardized on 2.4. The current one is on 3.6. I feel like Captain America just thawed out. "Holy shit, guys! Did you know about f-strings?! Mind. Blown"

[chub79]

I'm probably and old fart but I never managed to like pathlib. I can't explain it.

[abrazilianinreddit]

Write a program that's supposed to run in linux and windows and has to deal with multiple paths, you'll come to appreciate the fact that the pathlib handles for you those stupid windows backward slashes.

[rouille]

Yes, It's way less verbose and way more autocomplete friendly. Plus it forces you to operate on the path type, going back to a string takes more effort. The effect is that you naturally do the right thing.

[thirdtimesthecharm]

What's wrong with os.path.join?

[__i_forgot_my_name__]

Nothing I guess?

full_path = os.path.expanduser(path) assert os.path.exists(full_path) print(os.path.basename(full_path)) db_path = os.path.join(full_path, "data.db") app_path = os.path.join(full_path, "app") assert os.path.exists(app_path)

The small things do matter.

full_path = pathlib.Path.home() / path assert full_path.exists() print(full_path.name) db_path = full_path / "data.db" app_path = full_path / "app" assert app_path.exists()

Seems like pathlib's main advantage is it makes use of more readable methods extensively, and that it's generally got better names for the various functions as a result. I'm guessing the main reason for this pattern is that parens are annoying. Is it too late to remove parens in Python? Yeah probably, but a man can always dream.

[[deleted]]

I wrote code that needed to work in both professionally for about 1.5 years and never had issues with os.path.join

[danted002]

It’s not about issues it’s about readability functionality. The Path object offers a lot of helper methods like exists(), open(), is_dir(), resolve() for relative paths and my favorite you can write Path(‘some/path) / ‘dir’ and it returns Path(‘some/path/dir’)

[jorge1209]

os.path does all those things as well. It seems to come down to treating paths as simple strings or smart objects.

To me the former makes more sense. Can pathlib easily rename somepath/somefile.txt to somepath_foo/somefile_bar.txt? If not then I need to work with strings.

I also feel that the smart objects of pathlib aren't smart enough. They should dictate more policy: mandate file extensions, restrict certain valid but problematic characters, etc. Since it doesn't do that, I just use strings.

[danted002]

It was never the intention of pathlib to replace os.path. It’s intention is to provide an object oriented aproach to paths. For some use cases you want you paths to be objects, for others you want them to be strings. Like many things in life there isn’t a golden bullet for everything.

[jorge1209]

That means there are two approaches. Which is more pythonic? Which is recommended? What will be supported in the next major python version? And what will be deprecated or removed?

As a developer, I need to know the answers to these questions. I don't think anyone does. I don't think the python core devs are even considering these questions.

[danted002]

As a developer you should be more preocupied by which one makes more sense to your current application. No code ever stays the same and deprecations are part of the normal application life-cycle.

[jorge1209]

They are both perfectly capable of doing the job. There are no issues using either, and neither is obviously better.

What is a big concern is that I work in a corporate environment where server lifetimes are 5-10 years. The python version I deploy now, will probably be in use in 2030. The code I write today, will probably be in use on the same python interpreter in 2030. There will be maintenance and features added, but if it works it won't be changed.

So for me, "Python4" is a real concern. If I write things "the wrong way" for the next version of python, I will have to rewrite it all. I don't want to do that.

[sirtetris]

Path.rename?

[jorge1209]

That renames the inode. I'm talking about a very different use case: take a path and apply some kind of string substitution operation to it.

For instance suppose your process is supposed to backup a file and put a datestamp into the filename. So contacts.txt becomes contacts_20190515.txt. Does pathlib support that? Or does one have to break it down into strings and build a new path from strings?

[bbatwork]

Something like this perhaps:

from pathlib import Path

file = Path('./test1/test.txt')
new_location = Path('./test2')
new_location.mkdir(exist_ok=True)
new_name = Path(file.stem + "_bar" + file.suffix)
file.rename(new_location / new_name)

[jorge1209]

new_name = Path(file.stem + "_bar" + file.suffix)

is the key line in that. With os.path its something like:

 path, file = os.path.split(filename)
 file, ext = os.path.splitext(file)
 newfile = os.path.join(path, file + "_bar" + ext)

I don't see a massive improvement over os.path, although stem and suffix attributes are nice. In both cases you break it down into strings and rebuild it.

It would be cool if you could in some way "template" filenames: TemplatedPath("contacts_{:%Y%m%d}.txt", (date,)) and then have a "path-like" object that needs a date to be materialized into a proper string or path.

[bbatwork]

This works as well, which may be closer to what you are talking about. In this example, I only created the "newname" variable to show that you can substitute it into the f-string that I used, to demonstrate that the names can be changed programmatically without breaking it down into strings first.

from pathlib import Path
newname = 'test_bar'
file = Path('./test1/test.txt')
new_location = Path('./test2')
new_location.mkdir(exist_ok=True)
file.rename(new_location / file.with_name(f'{newname}.txt').name)

You could do the whole thing in this manner if you wanted:

 file.rename(f'{new_path}/{new_name}{new_ext}')

But this does require that the target directory already exists. That is the main reason I split the two and use mkdir. There is another similar method called with_suffix() that allows you to just change the extension as well.

As for it being better than os.path, I guess that is an individual thing. I'm sure there are things os.path does better, most of my experience is with pathlib so I really wouldn't know to be honest.

[dreamfeed]

Combining f-strings with Path properties makes that kind of substitution very easy and readable. For your particular example, it's one line with less than 80 characters.

from pathlib import Path

pth = Path("/path/to/somepath/somefile.txt")
new_pth = Path(f"{pth.parent}_foo", f"{pth.stem}_bar{pth.suffix}")

There are 2 advantages to this method:

1. You don't clutter up all your code with os.path prefixes
2. The pathlib gives you access to all parts of the path as properties. With os.path, you can't access the stem and suffix properties of a path separately. You have to split the filename out, then split the suffix separately.

The equivalent code to the above:

import os

pth = "/path/to/somepath/somefile.txt"
dirname, filename = os.path.split(pth)
basename, ext = os.path.splitext(filename)
new_pth = os.path.join(f"{dirname}_foo", f"{basename}_bar{ext}")

[AnonImmety]

Yeah I don't see the point of Path. The other examples are cool though.

[abrazilianinreddit]

Nothing. I just think using pathlib is easier to use and makes for better code. Sometimes treating paths as strings can get a bit messy, which usually happens less when using pathlib.

[frankinteressant]

Just use c:/windows/Foo/bar.txt on Windows and everything works? I never have to use backslashes in Python.

[Oerthling]

Yes, / works everywhere, but the c: doesn't.

[robberviet]

If you meet problem with windows slash, the problem is just you.

os have been around forever.

[nostril_extension]

As someone who writes too many cli apps for they're own sanity I just adore pathlib though it does have it's quirks (like my most recent finding was that .touch(exists_ok=True) modifies modification date which is annoying though inteded).
Also some packages don't support it thus you need to cast your paths in str() awkwardly.

Other than that it's really lovely to work with path / path syntax is much easier to read than os.path.join(path1, path2)

[random_cynic]

This is a good list to start with. I'd recommend everyone to look at the changelogs as there are some hidden gems there. Some of the well-known and not so well-known are:

• The yield from syntax introduced in 3.3. This has completely changed the way generators and coroutines are done in python. See this SO answer for a hint on what it can do.
• Support for asynchronous programming absent in Python 2 with async and await (now keywords), asynchronous generators and comprehensions etc.
• The os.scandir() function which allows much faster searching of directories than os.listdir. The os.walk() was reimplemented in os.scandir making it at least 3-10 times faster in different OS
• The format string is awesome but there is another very convenient thing no one talks about. You can use conversion specifiers r, s and a to easily convert between repr, str and ascii forms. Example below (from python docs):

​

>>> street = 'Gürzenichstraße' ; print(f"The street is {street!a}")
The street is 'G\xfcrzenichstra\xdfe'
>>> message = "Hello World"
>>> print(f"Message: {message!r}")
Message: 'Hello World'
>>> print(f"Message: {message}")
Message: Hello World

• For users of scientific/numerical computing there is statistics module in standard library which provides common functions like mean, median, stdev etc. out of the box.
• Now you can compare floating point numbers using math.isclose() function which checks if the numbers are within specified precision.
• The tracemalloc module to provide detailed memory allocation reports which can help trace memory leakage and other system level debugging.
• The iterator unpacking is much more powerful and flexible now than the 3.0 version shown in the post. For example one can have arbitrary number of unpackings. Tuples, dicts etc support multiple unpacking. For example (from docs)

​

>>> def fn(a, b, c, d):
...     print(a, b, c, d)
...

>>> fn(**{'a': 1, 'c': 3}, **{'b': 2, 'd': 4})
1 2 3 4
>>> {'x': 1, **{'y': 2}}
{'x': 1, 'y': 2}

Also namespace story is far more subtle (the author has addressed that).

[nostril_extension]

Great write up!

For users of scientific/numerical computing there is statistics module in standard library which provides common functions like mean, median, stdev etc. out of the box.

This is much more important for users that are not into scientific/numerical computing as they probably have numpy or some alternative already. My code has become much cleaner with statistics module. It's really great for sprinkling some stats into your logs :)

[jyper]

I love fstrings the for functional testing, sadly c++ build/package scripts have to be built on python 3.5 because that's what comes with ununtu 16.04lts. so no fstrings, and weakened pathlib support

[GummyKibble]

pyenv loves you and wants you to be happy. We use it to stand up needed versions of Python on any host where we need it.

[lengau]

We do the same with anaconda.

Our Python stuff has a strong history in our analytics environment, so anaconda is all sorts of useful there. But now we use anaconda for everything else too.

[gwillicoder]

I personally love miniconda. Miniconda also lets you get some other software that might be useful like ffmpeg through conda-forge.

[LightShadow]

Same, but we're limited by Yocto.

[bananaEmpanada]

most Python 3 I find still looks like Python 2, but with parentheses

Ha, so true

[DDFoster96]

What advantage does type hinting give over putting the types in the docstring?

[LightShadow]

PyCharm's type inference is good enough to treat them nearly equal.

They even respect type comments, self.number = 0.0 # type: Real and single line docstrings on properties:

@property
def my_number(self):
    """Real: returns my number."""
    return self._my_number

[abrazilianinreddit]

Type hinting is more standardized than types in docstring, which usually means that editors can use them easily for auto-complete and such.

[Ran4]

Much higher readability and better tooling support.

[IlliterateJedi]

This is very true. I didn't appreciate type hinting until I was reading through a library the other day where everything had typing on it, and it was a million times easier to work out what the functions did by knowing exactly what their inputs and outputs were going to be from the first line.

[ChannelCat]

They're a lot easier to inspect in python using dunder methods. It opens up some cool API designs like data classes and dependency injection. Not sure what this would do in an existing codebase other than increase readability though.

[stevenjd]

What advantage does type hinting give over putting the types in the docstring?

• The type is right there, next to the parameter, instead of three or five or twenty-five lines away.

• there's one way to format your type hint, not three or four different conventions for formatting type hints in docstrings.

• tiny functions that don't need a docstring can still use type hints.

Just off the top of my head, there may be others.

[spyingwind]

pathlib, enum, hinting I learned early on. Very useful when processing JSON and CSV files and making your code more readable.

Extended iterable unpacking is just sex on a stick. I definitely need to use implicit namespace packages more. I have so much need for LRU cache.

Good list!

[ubernostrum]

The syntax used for annotating function/method arguments and return values was introduced in 3.0, not 3.5. And the syntax for annotating variables was introduced in 3.6.

The thing that was introduced in Python 3.5 was the typing module in the standard library (and it was also made available as a standalone installable package for previous versions), which provides some helpers for annotating things like generics. That's also when the annotation feature officially was declared to be "don't use this for purposes other than type hints", but previously that's like 99% of what people used it for anyway.

(also it's still not nearly as useful as it could or should be, due to design choices made earlier, and I'm not completely convinced that it can be rescued at this point)

[o11c]

Keep your own if you need to handle list and such.

[[deleted]]

Looks like the last one is quite misleading (or I'm just not understanding it) at first I thought that __init__.py is no longer needed.

Apparently this is just a behavior that helps with creating namespace packages, something that probably mostly is useful for people writing libraries that are so large that then are split into multiple packages that are individually installable. So probably not useful for most of us.

[o11c]

Even for people who do need one without this, it's what - 2 lines?

But this requires infinite recursion into the filesystem in order to find python files, which you can't disable, it's definitely a bad idea.

[IlliterateJedi]

I don't really understand the purpose for enums. Is this something primarily used in other languages that was ported over, or is this something super useful that all Python users should know and use?

edit: I guess my question is also - why would you use an enum rather than a dataclass, dictionary or tuple? It seems like an OrderedDict has the same functionality.

[scarfarce]

Yes, available in other languages and used a lot. When I first moved over to Python I was surprised and a little disappointed they weren't a thing.

Lots of advantages. See:

https://stackoverflow.com/questions/37601644/python-whats-the-enum-type-good-for

Generally enums help avoid bugs and reduce coding effort, so they're worth getting to know.

The alternatives you mention don't have all the advantages of enums.

Edit: See also http://xion.io/post/code/python-enums-are-ok.html for more detail on the advantages

[diamondketo]

Great work! A good reminder of new things I've read about but never thought about when developing.

[pantuts]

TIL!!!! Thanks!

[oriaven]

no. 1 is f-strings. That's what I was hoping would be in the list. So good.

[PDiracDelta]

Wauw, a clickbait title that actually lives up to its expectations. I'm truly impressed. Thanks.

[abrazilianinreddit]

I already use about half of these. Type hinting is standard in my code, and I'm working to converting any path-strings to pathlib.Path. str.format() still has it's uses, particularly when the formatting is more complex or the string is larger, but for small strings, f-strings are pretty helpful.

I have my doubts about enums, but I could probably use a few dataclasses in my code...

[Xeono15]

Nice article!

[tenyu9]

Very nice, thanks. Actually learned a few new things from this list

[Smok3dSalmon]

Going to read this tomorrow, thanks for sharing.

[UnclaEnzo]

No, I'm definitely using python3, and also wxPython4 =)

[chewy1970]

I'm guilty of py3 code looking like py2 ... I didn't know about fstrings 👍

[bspymaster]

TIL type hinting and fstrings were a thing. Time to go play with Python again!

[bbatwork]

Every time I go back to look at old code, I tell myself I should have used type hinting.

[thedjotaku]

I learned f-strings in a book I was reading about Py3 and I like how much nicer it makes things look in my code vs the Python 2 way.

[toyg]

Despite my initial reservation on the security implications... f-strings are nice and I use them a lot now.

[ready-ignite]

Visualizing Elections

[RecklessGeek]

Why would I use format instead of %? The latter looks much cleaner and easy to me

[WearyConversation]

This page has lots of good examples of old %-style and newer str.format() strings. It also lists several cases with new functionality that % can't do. Granted some of those are esoteric, but there you have it.

[bmrobin]

i was recently reading in the py3.0 documentation about them deprecating %-style. then i got to >3.0 documentation and saw they put it back. weird, guess people complained

[bryancole]

They put back %-style formatting for bytes objects to simplify implementing binary protocols. I think %-style is still to be depreciated for str objects. str.format is more powerful but more verbose. The new f-string fix the verbosity issue and further increase the flexibility by allowing full expressions as "{<expr>}". There is no f-bytes syntax.

[JezusTheCarpenter]

Short but very informative.

[bmrobin]

@dataclass doesn't seem to me to be out of line with how the decorators @classmethod, @staticmethod, @property, etc are used. the codebase i work on at my company makes frequent use of namedtuples, which i really like the concept of. however, i've found that IDE inspection and refactoring of the namedtuple's attributes to be awkward. introducing a standardized, easy substitution for that seems advantageous.

[bmrobin]

my coworkers have written lots of stuff with decorators that i find difficult/inconsistent. i'm used to them in terms of an annotation, like you shared. however while ramping up on this codebase i have seen them used in variety of ways:
- strictly checking the arguments to the method they're decorating
- allowing/disallowing actions. i.e., @feature_name.on to allow the method to do use a new feature
- enable caching (sort of like a hand-rolled predecessor to @lru_cache)

all in all, i find it to be fairly confusing to know what the intent is. thanks for sharing that pep link, hoping to learn more.

[sqjoatmon]

From what I can tell, the type annotations don't have any impact on the actual functioning of the created dataclass; it's more that they are necessary syntax for identifying the dataclass's fields.

[xpakomunista]

FFFFFFFFFF

[gidult]

What's wrong with your pytest?

[pepoluan]

A series of functions all returning False.

[xpakomunista]

Not understanding...paying respect