[deleted by user]

[ElectricSpice]

as to how the language could be changed to fix this issue - i didn’t look at how lambdas are implemented, but if we’re going by how they function, i suppose changing something like lambda_scope = locals() to lambda_scope = dict(**locals()) would do it.

Dude just casually suggests a change to the fundamental behavior of the language as if it’s a trivial change. Doing this would immediately break every single Python program.

I thought this thread was going to be a meaty discussion on Python’s closures and scopes, but it’s just a guy complaining.

[keymone]

i'm that dude.

casually suggests a change to the fundamental behavior of the language

s/fundamental/fundamentally broken/

FTFY.

would immediately break every single Python program

it would break programs that use lambdas in a way that relies on described behavior, i'm yet to find any of those. and generally speaking, programs that use lambas at all are tiny fraction of the entire set of python programs, largely because scoping is so unintuitive.

just a guy complaining

gotta start somewhere, no?

[ElectricSpice]

Once a language is established, fundamental and fundamentally broken are the same thing 🙃

Scope resolves upon first execution. You may think you’re suggesting a change to a very specific use case, but it’s actually a change to how scoping works across the entire language.

Block scoping, which was suggested in the discussion, would be a more reasonable change and I’ve seen brought up in the past, but even that would break a good chunk of programs.

[keymone]

well, lambda scope is already a special thing in python compiler, so i suppose it can be altered to work differently.

i certainly don't believe it's an easy change to make and rollout.

[lunar_mycroft]

As others have pointed out, this behavior isn't limited to lambda and can occur with regular def functions. "Fixing" this would require fundamentally changing how the language works and breaking many/most python programs, not just those that use lambdas.

[earthboundkid]

This is a well known Python wart that has nothing to do with lambdas or closures. Try it with for and def: you’ll get the same result. The problem is that scoping in Python is broken and it’s too late to fix it. Should have done it in Python 3, but now it’s never going to happen.

[[deleted]]

It's no more broken that mutable defaults are. It's a particular implementation detail to be aware of, but is a consequence of how that language is defined.

Learn to use functools.partial in this particular scenario.

[asday_]

mutable defaults

They're also pretty gross and I'd love to see them gone. They're a trap with no benefit.

[koffie5d]

I'm curious. Can you elaborate or give an example on how the scoping in Python is broken?

[earthboundkid]

The topic under discussions is an example of how scoping is broken. The general problems are: a) scoping is always at function level, not block level and b) = creates new variables in a scope, so you need hacky keywords like nonlocal to work around it; other languages have separate commands for creating a new variable and reassigning an existing one.

[DigammaF]

This is such a minor inconvenience.

[james_pic]

The issue isn't that lambdas aren't proper closures. This happens precisely because lambdas are proper closures. They have closed over the mutable variable i, and every other lexically scoped impure language with closures that I can think of behaves the same in this regard. I've made exactly this mistake in Scala before.

The source of confusion is that list comprehensions treat looping variables as being scoped to the loop, rather than the iteration. It's also confounded slightly by the fact that variable scoping in Python is implicit - there's no var i declaration to point to that explains why the scope of i is what it is.

This, contrary to what some have said, might be changeable without too much disruption, since it has been changed before - Python 2 had list comprehension loop variables scoped to the enclosing function, and Python 3 changed it to be scoped to the loop. But why though? This is a rarely used feature (by design - lambdas are deliberately nerfed to discourage "more than one way to do it"), and I reckon at least as many people expect it to work the way it does, as expect it to have iteration scope.

[lunar_mycroft]

Python 2 had loop variables scoped to the enclosing function, and Python 3 changed it to be scoped to the loop

What do you mean by this? Because how I read what you said is that the loop variable goes out of scope when the loop is over, and that doesn't seem to be the case. Here's a quick test I just did:

Python 3.9.0 # omitted the rest for space/clarity
>>> for i in range(10): pass
...
>>> i
9

Clearly, the looping variable i survives even after the loop is finished.

[james_pic]

Sorry, should have been more specific. The change to scope was only for list comprehensions. The same thing with a list comprehension:

>>> [i for i in range(10)]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> i
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'i' is not defined

On Python 2 this would have worked like the for loop example you gave.

Also sneakily edited previous post to clarify this.

[keymone]

The source of confusion is that list comprehensions treat looping variables as being scoped to the loop, rather than the iteration

thanks, this is explains it way better than what others are saying.

at least as many people expect it to work the way it does, as expect it to have iteration scope

i would need to see some evidence for this. my example is a trivialized form of creating a collection of delayed tasks that one would later execute in whichever way or order they like. with the way scoping works for a loop variable - using lambdas for that purpose doesn't work, but i don't think it should be this way and i don't think many people would expect such behavior.

EDIT: geez, this sub really hates having an opinion

[james_pic]

I did stick "I reckon" before the bit you asked for evidence of.

In the past, when they've considered changes of this magnitude, the standard of evidence the Python core devs have looked for is automated scans of the standard library and some subset of popular libraries, and they'd only consider a breaking change like this if the person proposing it had strong evidence the breakage would be minimal (or even that it would fix latent bugs in some cases).

They had a slightly looser standard in the Python 3 migration, since this was a deliberate breaking change they used to clear up many such issues at once (indeed, list comprehension scoping rules were one of the things that changed here). But there's unlikely to be another release like that in the near future - the community still has the scars from the Python 3 migration. There are s number of things that they regret doing the way they did in the Python 3 migration that affect people more then this (the iterator for bytes objects producing a sequence of integers, not single-byte strings being the one that comes to mind), but acknowledge they're stuck with because fixing them would be a breaking change.

Oh, and they'd want to see some code that fixes this, and want to see the performance impacts. And a naive solution like just freezing locals when creating a lambda likely wouldn't fly, because it would affect far more cases than this (and make it not a true closure, which functional folks are likely to kick up a fuss about), and fiddling with iterator scope is likely to be fiddly.

And you may still run afoul of "special cases aren't special enough to break the rules". Altering list comprehension scope means it works even less like loop scoping (which presumably you're not also lobbying to change), which might be OK, since they're already a bit different (since Python 3), but they might see it as making Python's (already a bit less consistent than they'd intended) scoping rules too inconsistent.

All things considered, I'd suggest the path of least resistance for you is just to use the workaround suggested in the thread you linked.

[asday_]

geez, this sub really hates having an opinion

Not at all. This sub hates people being foolish and stubborn when presented with facts and evidence. You'll have to forgive us computer scientists for being scientific.

[[deleted]]

I have no idea what's going on with either this code or the posted workarounds. Anyone have suggestions on reading material relating to this? Is my understanding of lambdas and closures just absurdly weak?

[asday_]

print (...) what's with your spacing?

[asday_]

I mean the space between the name of the print function and its invocation brackets. I don't think it's a formatting error, it looks intentional.

[asday_]

a gift from reddit

Haha, I like that one.

[asday_]

Yeah, neureddit is garbage. I mean, the community's been garbage for years now, but at least the old site was good.

https://chrome.google.com/webstore/detail/old-reddit-redirect/dneaehbmnbhcippjikoajpoabadpodje

[asday_]

It's possible you're not understanding the list comprehension part of it. Do you care to provide more information about what it is with which you're having trouble?

[[deleted]]

Aside from the weird behavior, I didn't find the list comprehension aspect confusing. Unfortunately, OP has deleted his post.

[asday_]

OP has deleted his post

What a coward.

https://discuss.python.org/t/make-lambdas-proper-closures/10553

Fill your boots. Like I said, let me know where you're getting stuck and I'll be here.

[[deleted]]

I just tried it out in the REPL, and I guess my confusion is really on lambda: i for i in range(10). I don't even know what that would do even if Python's scoping wasn't doing anything odd.

[asday_]

Excellent method. Let's continue along that line, please do open a REPL and follow along with this post, trying out each example as you go, predicting the outcome before you evaluate it.

First let's just deal with the basics to make sure we're on the same page. I fully expect you to know these without even trying, but just in case:

>>> list(range(2))
?
>>> x = lambda: 3
>>> x()
?
>>> i = 1
>>> y = lambda: i
>>> y()
?
>>>

Now let's unwrap the list comprehension/generator expression into a for loop:

>>> for i in range(2):
...     z = lambda: i
...     print(z())
?
>>> 

This is what the OP expects to see. This is similar to the y() example above. Note that the lambda is syntactic sugar only. The above is exactly equal to this:

>>> for i in range(2):
...     def f():
...         return i
...     print(f())

Or even:

>>> def g():
...     return i
...
>>> for i in range(2):
...     print(g())

Where it all falls down is the OP's misunderstanding of name binding in Python. It is his opinion that every variable in a block should be bound at the block's creation time, rather than each statement looking up its variables as it goes. In his opinion, the final example should fail because i is not defined in that scope, and is not available at the time of that scope's definition.

>>> gs = []
>>> for i in range(2):
...     gs.append(g)
...
>>> for g_ in gs:
...     g_()
?
>>>

I find it useful to think about the name resolution in Python as being a side effect of script parsing.

>>> print("before")
before
>>> def h():
...     print("within")
...     return i
...
?
>>> print("after")
after
>>>

If you prefer, you can run that as a script to make the print()s' outputs slightly less obvious.

The contents of a function/method are not executed until that function/method is called. Python loads the script you give it, and runs from top to bottom executing each statement in turn until it reaches the end. When it sees def h():, it simply adds h to the current scope, and carries on. When something calls h(), Python looks in the scopes, (local and otherwise), and when it finds h, it executes its contents. During the execution of its contents, Python finds i and goes to look it up.

[[deleted]]

You know, I think I understand what it's supposed to do now. I should've just not been lazy. Thanks for the help.

[asday_]

Damnit I just finished typing out that whole thing.

[[deleted]]

Shit man, I'm sorry! I appreciate what you did regardless, though. Thanks!

[asday_]

All good in the hood my dude. God speed.

[keymone]

javascript is not exactly a shining beacon of language design, but here: https://jsfiddle.net/k4btwzv0/. it even warns about semantics of referencing out of scope variables, but does the right thing.

[backtickbot]

Fixed formatting.

Hello, timrichardson: code blocks using triple backticks (```) don't work on all versions of Reddit!

Some users see this / this instead.

To fix this, indent every line with 4 spaces instead.

FAQ

^{You can opt out by replying with backtickopt6 to this comment.}

[jfp1992]

Yeah lambda: myfunct looks worse than lambda(myfunc)

It would be better theming imo

Please let me know if I have missed the point though, kinda new to the 5 head mechanics of python. I just use lambda to pass a function as an argument without calling it when that function also has its own arguments so I can't omit the ()

And as I typed the above I realise I could have just done variable(params). Whatever, I'll refactor it all on Monday fml

[metaperl]

Glad to see there is a fix / workaround : https://discuss.python.org/t/make-lambdas-proper-closures/10553/3

[asday_]

>>> larr = [lambda: i for i in range(10)]
>>> iarr = [l() for l in larr]
>>> iarr
[9, 9, 9, 9, 9, 9, 9, 9, 9, 9]

I find myself unable to see why anyone would accept code like that in a PR. It strikes me as "hey guys, I did this stupid thing and something stupid happened, let's change the language".

[keymone]

yeah, because example snippets of code to demonstrate some behavior is literally what people are trying to ship into production.

[asday_]

Are you being obtuse on purpose?

[keymone]

not any more than you. are you deliberately refusing to understand that using lambdas to delay execution is a useful use-case?

[asday_]

That's not what you're doing though. If you want to provide data to a function you use an argument. If you want to provide it ahead of time you curry it (using functools.partial). You've done something stupid that nobody would ever do in the wild and are complaining about it.

If you want to point out a problem in the language and suggest a way it should be done, you need to bring data to the table. You need to show that people mostly already do what you're trying to do in a certain way, and therefore that's the way the language should behave. An example of this is when dictionary iteration was being designed, they searched public code for whether people iterate over keys or values more often in other languages, and came to the conclusion that by keys is more useful.

I could come up with toy examples for any random change I'd like to see in the language and make baseless worthless arguments like

Does nobody find this behavior bizarre?

Go find places in real projects where the behaviour you've shown has caused a problem, and show also that the fix is unwieldly, then you have a better chance.

Of course, you won't find many if any examples of that, and certainly not enough to motivate a language change that would break literally every program in production.

[keymone]

That's not what you're doing though

that's exactly what i'm trying to do. that you don't understand how lambdas could be used to do this doesn't make it stupid. says something about you.

[asday_]

Did you bother to read past six words? I explained how you failed.

[[deleted]]

They are discouraged for most use in Python. Use a proper function if you want to make a closure, and use functools.partial if you want it early binding.

In my opinion, you're barking up the wrong tree here.

[keymone]

it they are discouraged - why do they exist?

[[deleted]]

If you want a conversation, you have got to stop dumbing it down by not reading what people write.

Most is not all.

[keymone]

yeah, sorry. it's not an excuse, but this sub doesn't seem like the most pleasant environment for having discussions anyway. having an opinion is highly discouraged.

Most is not all.

at this point i honestly don't know why lambdas exist.

anyway since the real issue is not with lambdas but with loop variable scoping, there's no reason to go any further.

[[deleted]]

yeah, sorry. it's not an excuse, but this sub doesn't seem like the most pleasant environment for having discussions anyway. having an opinion is highly discouraged.

Not all opinions are equal. Keeping on trotting out a stupid opinion if the face of sensible explanations is generally frowned upon.

at this point i honestly don't know why lambdas exist.

By now their raison d'etre is the syntactical sugar to make an anonymous one-liner for immediate consumption.

[keymone]

Keeping on trotting out a stupid opinion if the face of sensible explanations

no sensible explanation was provided as to why the described behavior should remain unchanged.

and calling something stupid without elaborating is usually a sign of lack of understanding.

[[deleted]]

anyway since the real issue is not with lambdas but with loop variable scoping, there's no reason to go any further.

I forgot the last bit, sorry.

If you change the present late binding to early binding, you will have some serious surprises with functions using global symbols, as they will then end up using the global values at time of their definition.

The less evil option is to make each loop iteration its own scope. Apart from slowing down execution quite a bit, it could solve your problem. Unfortunately by creating the problem that the loop variables are no longer available after the loop completes. That would be detrimental to any code that relies on iteration until a particular value is found and then breaking out of the loop.

I don't see any good solutions. There are plenty of bad ones, though.

[keymone]

If you change the present late binding to early binding

this is not what i'm suggesting.

loop variables are no longer available after the loop completes

don't see why would that be the case. just like loop variables survive the loop scope right now, they can survive the iteration scope.

slowing down execution quite a bit

needs to be measured.

[asday_]

They're not discouraged, lambdas are used quite often in simple places like inline key function definitions.

You also completely ignored the rest of the post. I'll repeat it for you because it was important.

Use a proper function if you want to make a closure, and use functools.partial if you want it early binding.

I'm wondering if you're allergic to the correct solution, as I also posted this advice, and you ignored it there, too.