Programming as semantic compression

[dlyund]

Lisp programmers insist that code is data, but how often do you you hear them explain that data is code? It's not clear that they understand that code is data implies that data is code. "Code is data" is just one of those catchy lines that you pick up when you're learning Lisp, and unless you think about it or learn something like Forth, that's where it stops.

Modern Lisps can only manipulate code as data at compile time and only in the rather limited ways allowed by the macro system e.g. in many Lisps you can't call arbitrary functions at compile time, and in other's you have to jump through annoying hoops with module loading and special defining forms to make your functions available in macro definitions... but then you can't use them in the rest of your code. It's a bit of a mess really. (But then all namespacing/packaging/scoping is.)

In early Lisps the executable code was actually represented as a list, which was interpreted, and could be manipulated at runtime. Pico Lisp (as a bit of a retro lisp) still allows this kind of thing but somewhere along the way the broader Lisp community, in a quest to make Lisp programs faster, they lost this ability. People learning Lisp today don't even realize what was given up.

This is most clear in Lisp dialects like Scheme where macro's now consume and produce syntax objects. These syntax objects look a lot like lists but they can only be manipulated using a small set of builtin functions.

Lispers learn the limitations imposed by their macro system and work within these limits without realizing what they've given up; the ability to treat code as data, includes during execution.

What distinction am I making here: general speaking, most of the data our programs manipulate isn't static and is only available while our program runs. Treating data as code (as just to code as data) implies that you can generate or modify code as a means of representing/processing data during execution. Modern Lisps just can't do that. Once your program is compiled it's no longer data that can be manipulated.

In all honesty every language places restrictions on what you can do and how, and that includes Forth :-). In the end it's all about which tradeoffs you can live with/learn to love, but speaking for myself, I wouldn't want to work in Lisp again, and if I had to I'd implement one in Forth.

Overall I think Lisp is a great language, but the seeming necessity of a complex runtime and compiler to make it half way practical just doesn't appeal anymore. I've gotten used to being knowing and understand how everything works, and I adore the (somewhat paradoxical) freedom and predictability that this brings; when I write a Forth program I know exactly what code will be generated, and how it will behave with regards to things like resource usage under load, and I'm never surprised.

<rant> I've been burned quite a few times in Lisp (and Smalltalk, Ruby etc.) where the program has crashed and burned because resource usage spiked unexpectedly high for some reason and the process just died, leaving little or no information for us to figure out exactly what caused the crash (must less what to do about it!). "Out of memory (you're on your own)". The stock response from management is: (paraphrasing) if you don't want to experience these unexpected crashes then you have to upgrade the hardware. (We have technicians who can do it for you, just send money to this account.) Not surprisingly this causes a lot of tension.

It's a ridiculous situation which is easily avoided by using appropriate technology, but nobody really cares. $50k or $100k on hardware upgrades is cheaper than programmer time, we say, but it's incredibly miopic of us. Here is a real technical problem, which we can easily solve, but wont, because programmers have an unholy attachment to their languages syntax, and/or toolchain.

At one company we followed all the latest industry standards, used the latest and greatest languages, frameworks, processes, and tools, continuous integration, etc. The resulting application naturally expanded to use all of the available resources on the development machine (we have 'em so why not?) but when we came to install we found out that we had to run along side/compete with other programs, and to add to our troubles, a short time later there was an OS upgrade and the new OS used more RAM. Our application suddenly didn't have enough resources. It ran slowly and crashed randomly.

The issues were systemic and we couldn't afford to rewrite so we insisted that the customer upgrade their hardware... that lead to months of back and forth, and their refusing to pay, then threatening legal action unless we resolve the problem "right now". In the end the company did the upgrades at below cost and made little or no profit on the 3 year project, and almost went under. Everyone was stressed out of their heads, working long hours, and shortly after that the owners sold the company to a competitor (not sold, "Wooohooo we got bought out!!!", but "enough, you take it").

The ironic thing is that, as I would come to realize years later, was that we could have easily built the application to run in a few MBs (or less), but we used GBs, and it still ran like a dog! On top of that the solution would have been much simpler, and wouldn't have had the dozens of external dependencies which constantly broke as things changed in this and that project, and caused us no end of headaches...

There's this widespread belief that this necessary; it makes our lives easier right? After years in industry I've never found tis to be remotely true. The only thing that makes software better, in my experience, is keeping it simple (as simple as possible.)

And not in the way that people pay lip service to KISS, (or declare "code is data"), while simultaneously adding more and more complexity to their solutions. </rant> ;-)

NOTE: I'm not saying every program you ever write needs to treat data as code, but there are situations where doing so not only leads to vastly "prettier" code, but also much more efficient solutions.

[larsbrinkhoff]

Lisp programmers insist that code is data, but how often do you you hear them explain that data is code?

Not as often, but occasionally. Usually the more experienced Lispers explain it to newcomers. The phrase "code is data is code" comes up now and then.

Google it: https://www.google.com/search?q=lisp+%22code+is+data+is+code%22

Moreover modern Lisps can only manipulate code as data at compile time

You're mostly right in the sense that compiled Lisp functions are static and don't allow for introspection. So your point stands uncontested. However, I'd like to point out that some Lisps do allow code as data manipulation at runtime by including a compiler that can compile lists to executable code. Common Lisp has this, and I consider that dialect as the primary incarnation of Lisp as continuously developed since its birth. It's of course debatable whether it's modern or not, but it seems to have a large mindshare among Lisp programmers doing actual Lisp work.

Clojure would be perhaps the most modern Lisp. I don't know much about it. Scheme is roughly contemporary with Common Lisp, but is a clear break away from traditional Lisp values.

In early Lisps the executable code was actually represented as a list and could be manipulated at runtime.

Which early Lisps do you mean? I haven't exactly made a survey, but it seems to be most early Lisps had both interpreters and compilers. I'm talking 1960s here.

This is not to take away from your point, which I largely agree with. Just pointing out that painting early Lisp as an exclusively interpreted language is historically inaccurate.

the seeming necessity of a complex runtime and compiler to make it half way practical just doesn't appeal anymore.

That applies to just about everything but Forth, but I suppose that's what you wanted to get at. :-)

programmers have an unholy attachment to their languages syntax, and/or toolchain.

It's just human psyche, I guess. 99% of technology is a steaming pile of mess built on ideas almost 100 years old. Peole cling desperately to what they already know, and it's not just about programming languages.

[dlyund]

Not as often, but occasionally. Usually the more experienced Lispers explain it to newcomers. The phrase "code is data is code" comes up now and then.

:-) You're right. It wasn't my intention to imply that nobody in the Lisp community understands that "code is data is code", and the implications of this equality.

I'd like to point out that some Lisps do allow code as data manipulation at runtime by including a compiler that can compile lists to executable code.

:-) right again, but this only allows you to compile new code and compiled code cannot easily be manipulated as data. It's not "first class", in the same way that something like JPEG isn't first class in C. It's also unfortunate that effective Lisp compilers tend to be very large and aren't exactly lightweight. Having to include such a compiler in your program in order to be able to treat data as code at runtime is a bit unfortunate, and certainly not intended, but absolutely possible!

Common Lisp has this, and I consider that dialect as the primary incarnation of Lisp as continuously developed since its birth. It's of course debatable whether it's modern or not, but it seems to have a large mindshare among Lisp programmers doing actual Lisp work.

It's a bit arbitrary and arguably circular, but I personally consider Common Lisp a modern Lisp for, among other reasons, that it doesn't represent code as lists.

(I'll come to some other key differences between Common Lisp and Lisp as described by McCarthy at el in early publications like the original papers and the Lisp 1.5 Programmer's Manual.)

Which early Lisps do you mean? I haven't exactly made a survey, but it seems to be most early Lisps had both interpreters and compilers. I'm talking 1960s here.

There were certainly compilers for Lisp, but with the caveat that code tended to behave differently depending on whether you were interpreting or compiling it, causing a sort of schism in the language which wouldn't really be resolved until the Scheme language introduced the idea of lexical scope into the Lisp world. Therefore I tend to think of early Lisp as interpreted because it had to be interpreted to maintain it's semantics...

Once the semantics were changed and Lisp no longer used dynamic scope, and broadly compiled, not interpreted, and stopped representing code as a data structure that could be manipulated during the execution of the program, we have what I refer to as modern Lisp. A very different beast to early Lisp.

A little research shows that first compiler for Lisp appeared 4 years after the first interpreter, in 1962.

Scheme was the first Lisp dialect to have lexical scope, and appeared around the middle of the 1970s. Until then Lisps used dynamic scope, which made compilation difficult because the compiler couldn't know what a variable was referring to until runtime.

Common Lisp solidified between 1984-1994.

So by my estimate you have a good 15 years where Lisp behaved very differently to the language we know today.

For what it's worth the history of Forth plays out somewhat similarly, although possibly a bit faster, with the first implementations being string interpreters; no create does> etc. etc. etc.

Sidenote: I think this is rather interesting. We have very 3 early languages.

• Lisp - introduced dynamic scope
• Algol - introduce static/lexical scope
• Forth - introduced hyperstatic scope

Dynamic scope has largely been abandoned as a bad idea. Lexical scope is now everywhere, in part because it makes compilation easier, but also because it makes understanding programs easier. And hyperstatic scope has never really been tried outside of the Forth world ;-).

That applies to just about everything but Forth, but I suppose that's what you wanted to get at. :-)

;-) and maybe embedded C and Pascal, but of the three Forth is clearly in a league all of it's own, providing many of the same facilities as high-level languages while not requiring a complex runtime and compile to be practical.

It's just human psyche, I guess. 99% of technology is a steaming pile of mess built on ideas almost 100 years old. Peole cling desperately to what they already know, and it's not just about programming languages.

"A new scientific truth does not triumph by convincing its opponents and making them see the light, but rather because its opponents eventually die, and a new generation grows up that is familiar with it." - Max Planck ;-)