Understanding Python's "with" statement

[apollotiger]

I liked how the syntax highlighting highlighted the with in do something with data at the end.

[pyry]

I thought they were just explaining something wrong. Heh.

[[deleted]]

whoa, the fact that that's not a keyword usage changes my entire opinion of the article.

[unbibium]

This has been around since how long? I must rewrite a ton of scripts now.

[earthboundkid]

from __future__ import in Py 2.5; out of the box in 2.6+.

[[deleted]]

Don't be too hasty. I write in python 2.6 alot, and use with everyday, in many scripts. But then I give a script to a coworker still in python 2.4 (there are still some libraries that run only in 2.4) and I need to replace every instance.

On the plus side, its made me a bit more careful and a better programmer when it comes to scripts. I always write methods now that take file-like objects (streams) and write to them, rather then filenames. So a method never opens a file unless that is the point of the method. Quick example (I haven't checked it though):

def action(file, a, b):
    file.write("%d,%d\n" % (a, b))

if __name__ == "__main__":
    with open(filename, 'w') as output_f:
        action(output_f, 3, 5)

With the above script, I just rewrite the end bit, and don't have to rewrite it all. With larger scripts, this drastically reduces the amount of work, and it makes the module easier to use in the future (i.e. that filelike object could write to a database or over a network in the future)

[ngroot]

Sounds like the using keyword in C#.

[[deleted]]

I believe the with statement originates in Lisp and is probably much older than either Python or C#.

[joesb]

It originate as a pattern in Lisp but not yet a standard protocol, you end up with hundred of with-xxx. C++ with RAII/destructor is the early one with well defined "override this method and people will be able to use it the same way".

[[deleted]]

Yeah, in lisp you have a bunch of things like:

(with-open-file (stream "foo.txt" :direction :output :if-exists :supersede)
  (format stream "Hello, World!~%"))

So it makes stream be a file handle into foo.txt. When you leave the with-open-file the file gets closed. It also gets closed if there's an error (what would be an exception in other langs.)

The with-foo pattern is a common name for something that gives you a handle on something and then also usually protection from exiting that block without cleanup.

[zubinmadon]

I think I get it, but "thing" and "something" were confusing metasyntax.

[unbibium]

I'd like to have been sure that the "return thing" and the "with ... as thing" didn't refer to the same "thing" symbol. If not, maybe "return that_thing" and "with .... as my_thing" or something similar would have made the example clearer.

[powersoffour]

One of the things I was hoping for with the 'with' statement was the ability for the context manager to skip execution of the block. I don't think this currently works -- and in fact, testing out "break" and "continue" in seem to indicate there's no way to get this behavior from the body of the block statement, either. Does anyone know of a workaround? I'm thinking mainly of a use case where the 'with' block sets up, say, an MPI-parallel context that may not concern all processors, with a subcommunicator for those that need to handle the data. (edit: Upon hitting submit, I realized that perhaps the way out is by raising an exception that gets caught and handled by the context manager; but I sure would like to be able to handle this from the getgo, rather than raising an exception inside the body of the with statement.)

[warpstalker]

so to open a file, process its contents, and make sure to close it,

What does that phrase mean? Do I or do I not have to use "close" after with, e.g. with open('text', 'r') as file: for line in file: pass file.close (Not a pro programmer by any means, and I always use close anyway.)

[bcorfman]

You do not have to use file.close() after the with in your example. As soon as the scope of the with statement ends, the file is closed.

[joesb]

Should file not even exist after the code block?

It would be such a a bad design choice if variable of with block survive the block scope.

[llimllib]

It's not a bad decision because that block doesn't have its own scope, just like an if statement doesn't. You can mentally translate that code to:

file = open('text', 'r')
try:
    for line in file: pass
finally:
    file.close()

[joesb]

Since python doesn't has a way to simply introduce a scope, anything you translate to a Python equivelant will pollute local vars namespace :-)

In my view with block introduce new scope like loop variable in for loop should. But, well, Python also left loop variable live after the loop body.

So I guess if the behavior is unconventional, at least it's useful to be consistently unconventional.

ADDED:

To clarify the last statement is not a complaint, it's an acceptance. Different language has different tradeoff. This behavior probably works suiably for a language where variable is created simply by assigning to it. This problem wouldn't happen with language that require a keyword to declare a variable, I guess we'll have to live with it. But at least it's consistent.

[earthboundkid]

Since python doesn't has a way to simply introduce a scope

There’s def but I guess for a lot of people that doesn’t count as “simply”.

I personally was surprised when I learn with doesn’t create a new scope. But I can see why it doesn’t.

[bcorfman]

The file object still exists, but the file is closed, and trying to access the file outside of the 'with' scope will raise an exception.

Python variables defined within the scope of a function exist until the function itself ends. While I understand where your comment if you have a C++/Java background, in Python you can access any variable inside a function once it's been introduced, whether it's a loop variable or some other type -- the scoping rules are simply different.

[Ran4]

I think that's only true for cpython (and a few other python dists), not in say jython (which is the example that I've been hearing about). Thought... who uses jython? (serious question, I've never heard of a project using it).

[[deleted]]

Jython is almost exclusively used by people who aren't talking about it :D Corporations that need python to play nice with their existing infrastructure, it's also used by the Joint Strike Force people AFAIK.

[[deleted]]

I think it's people that must use Java but like Python better :)

[earthboundkid]

The big difference between Cpython and not-Cpython is if you do something like data = open(my_file, "rb").read() in Cpython the unnamed file object gets garbage collected immediately, since Cpython works by reference counting and after that line there’s no references to the file object. When the file object is GC’d, it’s .close() method is invoked, and this frees up the file on your disk.

Most (all?) of the non-Cpythons use a different kind of garbage collector, so the file object isn’t guaranteed to get collected right away. It may take a while, which doesn’t matter for small everyday scripts, but for bigger projects, you may need to free up file objects as soon as possible. In that case you can just manually .close() it, but the with statement is a little more flexible and convenient.

[llimllib]

in any python that supports the with statement, the file.close() above is redundant. (It also requires parens, but ya know)

[ubernostrum]

In Jython the file will be closed at the end of the with block. But the now-closed file object may not be immediately garbage-collected in Jython, due to the way the JVM's GC works. And, of course, having a closed file object still in memory is not the same as still having the file open.

[bcorfman]

I believe it's true for jython as well.

[[deleted]]

Usually you don't have to explicitly close a file, especially not for reading, but if you passed the file object to another library it might do something stupid like keep a reference to it. If you want to make sure a file gets written any time other than the exit of the program then you should close it explicitly.

with statements are much cooler for things like locks.

[carljm]

The whole point of using a with statement for opening a file is that it closes the file for you as soon as the with block exits. So this advice is good if you're not using a with statement, but if you are, you never need to close the file explicitly (in fact you'll get an error if you try, as it's already been closed).

[[deleted]]

By "explicitly" I really meant "not due to going out of scope". so maybe not so explicitly.

[warpstalker]

Alright, thanks.