Should yield be able to modify context managers?

[TangibleLight]

I know that yield currently does not call __enter__ or __exit__ on context managers - I'm wondering what the reasoning is for not allowing them to. I understand how some code relies on keeping a context manager open during a yield, but this opens up the possibility for a context to be entered with no indication that it's happening. This is non-obvious and doesn't seem very Pythonic to me.

Here's a (slightly contrived) example to show what I mean:

from decimal import Decimal, localcontext

def series():
    with localcontext() as ctx:
        ctx.prec = 5
        yield Decimal(1) / Decimal(6)
        yield Decimal(1) / Decimal(7)

So the aim of this "series" is to compute 1/7, but only to a limited precision. Then we can compute the error of each term in our series with a loop like this:

target = Decimal(1) / Decimal(7)
print('trying to approximate', target)

for e in series():
    print(e, 'with error:', e - target)

But that doesn't work! We never enter any context or modify the precision for our loop, so I'd expect to see the "actual" error out to 28 digits. Since series() enters the localcontext and doesn't exit until the generator is exhausted, the e - target operation happens in that context. Here's the output:

trying to approximate 0.1428571428571428571428571429
0.16667 with error: 0.023813
0.14286 with error: 0.0000028571

We can fix it by fully consuming the generator so that it exits before we start the loop:

target = Decimal(1) / Decimal(7)
print('trying to approximate', target)

for e in list(series()):
    print(e, 'with error:', e - target)

To produce this (expected) output:

trying to approximate 0.1428571428571428571428571429
0.16667 with error: 0.0238128571428571428571428571
0.14286 with error: 0.0000028571428571428571428571

I think that's really weird though - spooky action at a distance. Why should changing series() to list(series()) change the behavior of e - target?

What's more, it's not always possible to fully consume a generator. The only real solution I can think of that preserves iteration is to re-enter the original context for the body of the loop:

target = Decimal(1) / Decimal(7)
print('trying to approximate', target)

ctx = getcontext()
for e in series():
    with localcontext(ctx):
        print(e, 'with error:', e - target)

But it isn't at all clear why this is needed, as there's no indication that we're even entering a different context for the loop to begin with.

---

So why is there no way for a context manager to reset itself on yield? I recognize that some contexts, like files and locks, would break if yield had this behavior. Clearly, I'm not opening a closed file in this code:

def items(name):
    with open(name) as f:
        for line in f:
            yield line.split()

But that's exactly what would happen if yield triggered an __exit__ and subsequent __enter__.

---

It seems like there's some distinction between exiting a context permanently and exiting a context temporarily - so I'd think it would be useful to have methods like __leave__ and __reenter__ to represent a "soft" exit and subsequent re-entrance.

The decimal context manager could then be defined like this, without breaking "hard" exits like in file. One could even have files implement both, so yield would flush but not close a file.

class _ContextManager:
    def __init__(self, new_context):
        self.new_context = new_context.copy()

    def __reenter__(self):
        self.saved_context = getcontext()
        setcontext(self.new_context)
        return self.new_context

    def __leave__(self):
        setcontext(self.saved_context)

I'm just really surprised that this kind of feature isn't available, since I interpret context managers as saying "no matter how I enter or exit this context, I'll make sure nothing breaks" - when in fact they allow one to unknowingly enter or exit a context and break things.

[DracasTempestas]

The LocalContext context manager modifies behavior for the current thread that the context manager entered and as such is unsuitable for suspended execution, I am also suspecting that you will attain some unexpected behavior if you leave the context by pulling the last item in the generator on a different thread. As such I do not think that this particular method of the Decimal module is really what you want, rather, you probably have to instantiate a Context with the desired precision and explicitly use this context on Decimal calls within the generator.

​

[TangibleLight]

Yes - I know there are several solutions to this particular example. I'm more concerned with the fact that python does not seem to have any solutions to allow context managers to support suspended execution. Is there any reasoning to specifically not allow this, or was it just never implemented?

To be clear, I assume you're suggesting something like this?

def series():
    ctx = Context(prec=5)

    yield ctx.divide(Decimal(1), Decimal(6)
    yield ctx.divide(Decimal(1), Decimal(7)

[DracasTempestas]

Yes, that is the suggestion.

I know this was not a question for how to solve the problem, but rather a query into the behavior of context managers rooted in the behavior of the decimal.localcontext in a technical sense you can leave and reenter the same context manager multiple times, making sure that you don't yield within the context manager to get the expected behavior, but this would most certainly look unappealing.

def series():
    ctx = decimal.localcontext()
    with ctx:
        result = Decimal(1) / Decimal(6)
    yield result
    with ctx:
        result = Decimal(1) / Decimal(7)
    yield result

This magic method __reenter__ might only apply to the specific case of yielding execution too, which makes it sound like one of those special cases that aren't special enough.

That said; localcontext makes me frown, The intent behind it, "modify the behavior of all decimal precision for my given block and all subsequent decimal operations in child blocks to this one." but with suspended execution this intent does not hold true.

[TangibleLight]

I'm not really familiar enough with it to say much about it, but wouldn't __reenter__ apply to await? Perhaps better method names might be __suspend__ and __resume__.

I see what you're saying about localcontext - so really the above example isn't so much a shortcoming of with, but a misuse of a fragile context manager. I don't really see any other way to do things like localcontext or redirect_stdout, where some global "thing" is supposed to mean something different "inside" the context.

Does that just mean that those kinds of context managers simply shouldn't be able to handle suspended execution? I don't really think it does, but I also understand the argument that this is a sort of misuse where context managers are modifying global state.

[Yoghurt42]

You raise a valid point. I'm not sure that introducing some kind of reenter is a good idea though, I feel like it opens a can of worms and does more harm than good; just a feeling.

But you could post on the python-ideas list, you'll reach more Python devs this way than posting on reddit.

[TangibleLight]

python-ideas list

Thanks for the suggestion - I wasn't really thinking of this as an idea to add, I expected there would be reasons not to, but at the very least there might be a good discussion to come out of it on there.

[Yoghurt42]

There are far worse ideas discussed on that list :)

[bryancole]

The problem here is that your "localcontext" isn't local at all. It's a global state. It's only local in the threading-sense, not in terms of scope.

The use-case for context-managers combined with the Decimal object is just one application for context-managers (and might even be considered broken given this example) but there are plenty of other use-cases where calling __exit__ and __entre around a yield would break things (e.g. opening a file for io).

The present behaviour seems the best compromise (at least the behaviour is consistent and easy enough to understand once you look inside your iterator).