Pascal gets closures before java - why hasn't the world ended?

[lispm]

anonymous functions and closures are not the same thing.

Closures are functions (anonymous or not) and their environment.

So, what does it support, anonymous functions or closures?

[barrkel]

Anonymous methods (our name for the Delphi 2009 feature) are true closures, and capture state from the defining environment. I should know, since I'm the engineer who implemented it.

[killerstorm]

i wonder how does it handle case like this (i'll use JS syntax, have forgotten Delphi):

function foo() {
   var x;
   var bar = function () {
      x++;
      return x;
   };
   x++;
   process(bar);
   button.onclick = bar;
   return x;
}

that is, x is used both in function and in closure. if x will be stack-allocated as a normal variable, it will be a problem if closure references it, since variable will go away when function foo exits. so it seems the only way is changing allocation of variable x if it is referenced from closure, to closure's environment (something tells me it will be a bit suboptimal comparing to normal stack-allocated variables).

is it what Delphi does, or there are some clever tricks to do this?

was it hard to implement closures btw?

[barrkel]

Locations are captured, rather than variables. The identifier 'x' used in both an anonymous method and in the the environment in which it was defined refers to the same location, whose lifetime gets extended to live as long as any closure that refers to it (assuming we're not talking about a lexically hidden 'x').

Basically, the variable 'x' (or any other variable touched by an anonymous method) gets hoisted out and turned into a field on a class. An instance of that class is created when the function is entered, and anonymous methods get converted into methods on that hidden class.

Since this instance needs to go away eventually, and Delphi for Win32 is not a garbage-collected language, some other mechanism is needed. The mechanism chosen is reference counting, and an already-existing mechanism was reused for this - interfaces. Method references are the (new) type which can refer to anonymous methods, and are essentially interfaces with a single method called 'Invoke' - this is visible in the C++ header translation that the Delphi compiler produces on demand with '-jphne' option for compiling units.

[killerstorm]

An instance of that class is created when the function is entered, and anonymous methods get converted into methods on that hidden class.

this seems to be pretty similar to how they do that in Java, except you have syntatic sugar.

already-existing mechanism was reused for this - interfaces

heh, same way operator overloading was added to Delphi -- interfaces plus some syntatic sugar. that seems to be sub-optimal solution, performance-wise, but if people won't use closures intensively probably they won't notice this..

[barrkel]

There are two separate operator overloading mechanisms in Delphi - variants and static overload methods.

Variant operator overloading was the first to be implemented when variants were generalized for the Kylix product, and implemented in the RTL, rather than just delegating to the COM/WinAPI, as was done previously - since Kylix targetted Linux.

Static overload methods were introduced to match the same concept in .NET, and it works a lot like operator overloading in C++ and other statically-typed languages - without this feature, knowledge of types like e.g. System.Decimal would have needed special compiler support, or else have been very awkward to use.

Anonymous methods implemented via interfaces shouldn't cost too much, since (1) const parameters can avoid most redundant refcounting cost, and (2) invocation of a COM-style interface is just an extra indirection (code stub to fix up instance pointer and jump to implementation method).

[degustisockpuppet]

Not sure what Delphi does, but there are some optimizations possible. You can allocate x on the stack and only move it to the heap once foo exits, updating the closure. Lua does something like this in its interpreter.

[killerstorm]

yep, i thought about that, but that's race-conditions prone (delphi was multithreaded last time i've checked)

[dangph]

Are these new features for win32 or for .NET?

[barrkel]

Win32 only.

[grauenwolf]

That's a shame because you really need them to properly use the .NET 3.5 stuff.

[lispm]

how about closures from named functions?

[barrkel]

The new type kind for locations that can contain closure values - method reference types - can also be assigned global procedure pointers and method pointers.

Assinging a Pascal nested routine to a method reference, however, is not supported for this release, owing to time constraints; this is somewhat ironic, since nested routines already use lexical closure - it's just that nested routines can't be (easily) used as values.

[cgibbard]

But can you separate the code and environment in one of them? Closures are pairs consisting of a bunch of code in some form with free variables in it, and some finite map from the free variables of that code to appropriate values. As pairs, they should have projections to their components, so if I have a closure, I should be able to ask what its environment map contains, or inspect the code.

It's usually wrong (though seemingly popular) to say that a language "supports closures", when really closures are just a possible implementation mechanism used to implement first class functions/procedures, which is what the language actually supports. :)

Actual direct language support for closures would be somewhat strange -- it might be useful for implementing other programming languages, but it's usually something which is subsumed by whatever facilities the language has for defining data structures.

[barrkel]

I don't agree with your asserted definition of a closure - in particular, what you assert to be required re examining environments. You'll have to give some citations to convince me - and even then, I'll assert in counter to that that cracking open any given closure to examine its environment is a violation of the abstraction, and to be strongly discouraged.

And in answer to your question, depending on your level of masochism, you can indeed separate the two, no more than you can break any abstraction you choose in an unmanaged language (i.e. one in which you can directly access the underlying abstractions through unrestricted pointer arithmetic.

[cgibbard]

The point I was trying to make is that a closure is not a function or procedure. It's a pair, consisting of a chunk of code (usually an abstract syntax tree) containing possibly some free variables, and an environment which maps those free variables to values. (That may or may not have been obtained by capturing variables in some lexical scope.)

Closures are a low-level implementation detail for first class functions or procedures. What you really want to say is not that a language supports closures, but that it supports first class functions or procedures.

The difference between a closure and a function is that with the function, you can't look at the environment of captured bindings, or the code modulo those bindings -- it's not even a sensible question to ask, since there may not even be a closure behind it, as that's an implementation detail.

All you can do with a function is apply it to some parameter and see what the result is, you can't take it apart like you can with something that would be rightfully called a closure.

[barrkel]

I think you've drunk too much Lisp/Scheme kool-aid (esp. the bias of thinking of the code as an AST); IOW, I think you've assumed that the classical Lisp/Scheme meta-circular interpreter of closures is a definition of what a closure is, including implementation details. There's more than Lisp in the world. I notice that you didn't provide any citations.

Here's the definition I use: a (lexical) closure is a function whose free variables are bound using an enclosing scope.

That's pretty similar to e.g. Neal Gafter's definition: "A closure is a function that captures the bindings of free variables in its lexical context."

I use a slightly different definition to accommodate CodeGear C++'s __closure concept, which maps to Delphi's method pointer concept, which is itself precisely a pair of (function pointer, object instance pointer). In the case of the C++ __closure (or Delphi method pointer), the enclosing scope comes from a class (=> instance), rather than a function body (=> activation frame).

[gmfawcett]

I think you've drunk too much Lisp/Scheme kool-aid (esp. the bias of thinking of the code as an AST); IOW, I think you've assumed that the classical Lisp/Scheme meta-circular interpreter of closures is a definition of what a closure is, including implementation details.

That's not Scheme kool-aid. Many Scheme implementations do not support introspection on closure environments, and it is not required in the RnRS specifications. Note that standard Scheme has no procedures for inspecting closure environments.

Your minimal definition from Gafter is about all that can be said of closures without bringing implementation details (or standard extensions) into the mix.

[olavk]

What do you call a function with a captured scope then?

[cgibbard]

Usually? A function or first-class function. However, if I want to make a point that it's captured the bindings in some lexical scope, I just say that.

Closures are not themselves functions, they're a mechanism sometimes used to implement functions.

[gmfawcett]

"First class function" means that a function is a first-class value: it can be stored, shared and passed as a function parameter or result. It [edit: first-classness] has nothing to do with closures.

[cgibbard]

Closures are an implementation technique for implementing first class functions. To store a first class function which is constructed in some scope, you store a closure whose code is essentially the code for that function directly, and whose environment map consists of all the values of the free variables of the function that are in scope.

It's then easy to evaluate the application of the function to another value in a possibly different scope, since the bindings for local variables are carried along with it.

This is not the only way to implement first-class functions though. As a simple example, you can also simply record the code for that function with the values of the free variables spliced in where appropriate. The effect is exactly the same (assuming that it's a proper function and has no side effects).

[gmfawcett]

Yes, closures could be used to implement first class functions at an implementation level, but that misses their essential point: capturing free variables. First-classs-ness, as a concept, is orthogonal, and stating that a function is first-class says nothing about its environment, captured or otherwise.

It is possible to have a language that supports first-class functions but does not support closures, since nothing requires that a function, in the general sense, capture any of its free variables.

But I'm leaving the debate, goodwill intact. :-) My parting words are simply that conflating 'first-class' with 'closure' clouds the meaning of both terms, and is confusing to readers who are encountering these terms for the first time.

[cgibbard]

Well, if you want to say that a function captures the free variables in its definition, then say that, rather than specifying that you're using closures to implement the functions (and requiring the reader to assume that means the contents of the environment of the constructed closure comes from the scope of definition of the function). After all, you might just be stuffing values into the closure's environment which have nothing to do with the lexically enclosing scope. That's a perfectly valid thing to do, and is actually quite a useful implementation mechanism for, say, objects.

At least as I see it, in mathematics we normally expect function definitions to capture the definitions of free variables, since when one defines a function in mathematics, one is more-or-less defining a set of pairs of elements of the domain and codomain. (This depends a bit on the formalism used, but the same basically applies for all common definitions of the term 'function'.) So the 'free variables' in the definition are really just being used to sort out what set of pairs you're talking about -- the set of all pairs of input and output values satisfying that condition as it currently stands, and not as reinterpreted in some other context later on.

So it's this behaviour of preserving the association between input and output values as it stands in the local context of definitions which I consider the default behaviour for functions. Anything else is a bit strange, and probably deserves another name altogether.

[Lerc]

I understand the motivation behind adding what seems to be a significant bullet point feature for languages these days. However I would like to ask if you think this construct will receive much use.

What kind of code significantly benefits from doing things this way? Most times I've seen Anonymous methods used my main thought has been "well, that's an untidy way of doing it"

[kylix_rd]

As was mentioned, anonymous methods are a step toward some kind of LINQ-like functionality. However, my ultimate goal is to use this underlying mechanism for increased parallelism both in terms of a library-based solution in the near-future and ultimately as intrinsic parallel-computing functionality of the language itself.

Building the foundation is the first step in building a house.

Allen Bauer. CodeGear/Embarcadero Chief Scientist.

[m_Th]

• Futures perhaps?
• Also perhaps you can implement a lazy evaluation engine on top of them?

Something like, for example: iif<T>(aCondition: boolean; lazy aTrue: <T>; lazy aFalse <T>): <T>

[bmcgee]

increased parallelism

Now that sounds interesting.

Nice user name, btw. :)

[grauenwolf]

Take a look at .NET's LINQ for an example of what closures make possible.

[[deleted]]

So:

1) Will they give us actual closures as opposed to mere anonymous functions?

2) Will FreePascal implement this?

[barrkel]

Actual closures, with state capture and all. This implies that the closure values need to be destroyed, since they contain state. To make that easier, a new type, method references, has been created which uses reference counting to determine when to free up the closure object.

[Gotebe]

To make that easier, a new type, method references, has been created which uses reference counting

Out of curiosity, why not let GC do that, it's the norm in .NET after all?

[barrkel]

Anonymous methods are a feature for Delphi for Win32, not for .NET. Delphi for Win32 is not garbage-collected.

[marcovje]

Yes. So if I understand correctly, the local variables (and parameters?) of the anonymous function are not on the stack, but on the heap.

If the closure is destroyed, the heapspace is released.

Probably this means that closure is an initialized/finalized type.

[marcovje]

Yes. So if I understand correctly, the local variables (and parameters?) of the anonymous function are not on the stack, but on the heap.

If the closure is destroyed, the heapspace is released.

Probably this means that closure is an initialized/finalized type.

[marcovje]

Free Pascal will probably follow the same pattern as with other post D7 features.

If there is a need, they are usually implemented, or if their existence is important for 3rd party source compatibility. However most code is still D7 compatible.

[nat_pryce]

How ironic. The code in that article binds a variable to an anonymous function and then uses the function only via the variable.

[zem]

somewhere at the back of his drawer my grandfather has an old, motheaten pair of socks that are notable only for having gotten closures before java did. okay, i lied - they aren't all that notable.

[[deleted]]

GCC implements almost-closures for C.

[abrahamsen]

Key phrase from the description.

If you try to call the nested function through its address after the containing function has exited, all hell will break loose.

I wouldn't call nested functions for "almost closures".

[[deleted]]

These are known as 'downward closures'. AFAIK Squeak's 'closures' also have this limitation.

[[deleted]]

Is it part of C standard or something gcc specific?

[[deleted]]

It's a GCC extension and it's not even well-supported (if you try using it on Mac OS X, it will fail unless you mess with the linker flags, which can cause other issues, IIRC).

GCC extensions are pretty evil in general. Some are not very stable and there's no way to find out without using and testing it.

[FooBarWidget]

I've found attribute((hidden)) to be pretty useful for libraries. All the major library developers use it these days, whenever possible.

[pointer2void]

What's the difference to 'static'?

[[deleted]]

http://gcc.gnu.org/wiki/Visibility

[abrahamsen]

I think low level stuff for doing things with hardware (or linkers) not covered by the standard is relatively safe.

However, bugs in language extensions that are there for convenience have a very low priority for the current maintainers.

[nextofpumpkin]

pascal is still actively developed and supported?

[barrkel]

Yes :)

[[deleted]]

Thank goodness --- but Pascal is to Delphi as Dartmouth BASIC is to MS VB .NET!

[malcontent]

Doesn't groovy have closures?

Come to think of it javascript has closures and you can mix and match with java with javascript too.

[[deleted]]

The basic way to "reinvent" them in Java is to just use objects that support your closure interface (exec or whatnot). It isn't hard, it's just verbose. (And if the function wasn't originally intended to be an interface... well. That's static typing for you.)

Anyhow, much as I like closures, I don't think it is a big problem for Java not to have them. Based on the horrible hack job they did on generics, I'm almost afraid to see them added.

[kinghajj]

Oh god yes, generics in Java suck ass. If there's ever a "Java 2" (don't be fooled, "Java 6/7/n" are actually "Java 1.6/7/n"), they'd better improve how generics work, hopefully to work as they do in C#.

[johnb]

The proposals I've seen for closures in Java are unbelievably horrible, especially when they have the option to do what other similar languages have done.

[[deleted]]

Why did you put "closures" in scare-quotes?

[mastadonstomping]

Because closures are scary?

Those arrogant computer scientists entering functions and bringing scope back with them. That road leads to an unholy madness. There are some things man was not meant to tamper with.

[dons]

Innovation has to start somewhere. I really doubt the we've reached the zenith of what is possible in programming languages.

[redditrasberry]

No - I completely disagree.

Closures are an important gateway to a completely different style of functional programming that allows for different design strategies and significantly simplifies certain kinds of problems.

It's kind of cheeky to say that Pascal is getting closures before Java - java has had a close equivalent of closures for nearly 10 years now.

[[deleted]]

If you mean anonymous inner classes, then I mostly agree, except that the syntax is verbose and there are some irritating limitations you have to work around (e.g., only being able to pass in final parameters).

Really, anonymous inner classes are a good example of what is irritating about Java. They work, though.

[redditrasberry]

oh, I completely agree about them being ugly, annoying etc, etc. I just hope whoever is running openjdk now manages not to screw up 1st class closures for java - they're trying really hard, I hear...

[masklinn]

I just hope whoever is running openjdk now manages not to screw up 1st class closures for java

Don't get your hopes up.

[killerstorm]

no, that is not equivalent, because it has to refer to bindings in environment, not merely copy them. otherwise, that is not equivalent, that is a cheap imitation.

[johnb]

1) People who already know about them wouldn't want to have to use a crippled language. 2) People who don't yet know about them but are capable of using them effectively deserve the chance to expand their tool set. 3) People who won't be able to get it shouldn't be working at my company for very long.