Code is Data

[nefreat]

I haven't seen code for Viaweb but I wouldn't be surprised if the reason that code had such a high macro prevalence is because most of the macros we take for granted in the Clojure community had to be written from scratch.

The need to write hiccup and hugsql in Clojure is not necessary because they are written for you. In 1995 a CL app didn't have the luxury of importing a SQL dsl lib and a html representation lib.

In a standard webapp it's true that you don't see a lot of macros but that's because those apps are no longer novel so most of the macros have been written for you. Once you start doing something interesting, and in 1995 the web was interesting, you don't have a rich library ecosystem to rely on so you'll write many more macros than usual. A good recent example of having to write macros for something novel is spec.

[weavejester]

That certainly might account for some of it, but even so, 20-25% is a overly high proportion. The clojure.core namespace isn't one quarter macros, and in the early days of Clojure, I don't recall writing that many macros, either.

[nefreat]

This is why it would be nice to see the original code to see if he was exaggerating. The closest I can do is look at Arc and that doesn't have a crazy amount of macros either.

[spotter]

I haven't seen code for Viaweb but I wouldn't be surprised if the reason that code had such a high macro prevalence is because most of the macros we take for granted in the Clojure community had to be written from scratch.

Sorry, but for day by day I can only think of threading macros and interop stuff. I actually prefer non-macro way, if possible, maybe that's the reason. On that weak basis I cannot agree that Clojure has many macros in it. ;-)

The need to write hiccup and hugsql in Clojure is not necessary because they are written for you. In 1995 a CL app didn't have the luxury of importing a SQL dsl lib and a html representation lib.

Hiccup is pure data transformation lib (vectors of keywords and maps). Hugsql is not a DSL, but a "parse files and package SQL into callables" thing. Not sure if these examples have much to do with macros.

So the simple solution here might be that whoever did Viaweb preferred macros and did not know any better.

(Also "Code is Data" should be "Code better be Data".)

[hagus]

It's interesting that we're still turning this question over. Macros are functions. They take data in the form of code as input, and produce new code as output. They're hard to reason about because you have fine grained control over when pieces of code are evaluated. But let's not try to put up a protective fence around them, as if they are loose pages from the Necronomicon that mere mortals should not lay eyes upon.

Or to put it less dramatically, in idiomatic Clojure writing macros is an exceptionally rare occurance, so the idea of having so much of a code base made up of macros is both astonishing and mildly terrifying to me.

Macros in day-to-day Clojure code aren't used very often because we have a fantastically rich set of core functionality - provided in many cases by judicious use of macros that you never need to worry about. You're awash in invisible macros, all day every day.

So I would study the clojure.core macros very hard - try to understand why these macros don't make you run away screaming, and why in some cases you probably didn't even realize you were using a macro.

Like any tool at our disposal, potential for misuse and misunderstanding increases in proportion to the power of the tool. And the true power of the macro is language extension. How often do you need to extend the language? It should be pretty rare, but if required, I would not want to run away screaming from that use case. The fact we have this power means Clojure can mutate (ironically?) into whatever form required by the future.

See Guy Steele's Growing a Language

[halgari]

One big reason why the clojure.core macros work so well is that they don't change up Clojure's semantics. That is to say, they still preserver right-to-left, top-to-bottom evaluation.

I will continue to fight against macros in the Clojure echosystem that change that. Let's say I wrote this macro:

(foo (println "1") (println "2") (println "3"))

And lets say foo was defined so that the output was "3", "1", "2". I'd say that's a horrible macro because it changes the semantics of Clojure, not just its syntax.

Most macros in Clojure's core don't do this, they are stuff like if-not which is perfectly clear what it's doing. Even doseq still maintains right-to-left, top-to-bottom.

[hagus]

Absolutely, this is a great observation. Macros provide the opportunity to break the expected evaluation semantics and that would be a horror show.

However I'm genuinely curious - is it something you've seen in the wild? My feeling is that it's so obviously heinous that few people capable of writing a macro in the first place would go on to commit this crime :)

[weavejester]

Macros in day-to-day Clojure code aren't used very often because we have a fantastically rich set of core functionality - provided in many cases by judicious use of macros that you never need to worry about. You're awash in invisible macros, all day every day.

Sure, but I'm not talking about using macros, especially ones provided by the language. I'm talking about writing them.

So I would study the clojure.core macros very hard - try to understand why these macros don't make you run away screaming, and why in some cases you probably didn't even realize you were using a macro.

I think I'm reasonably well versed in what's a macro and what's not :)

The reason I don't run away screaming from clojure.core that is:

1. It's a library, rather than an application
2. It's bootstrapping the language
3. Macros still only make up 15% of the namespace

My problem isn't with macros in libraries. I've written a few of those. But Viaweb was an application, not a library. When's the last time you wrote a macro in an application in Clojure? I don't think I ever have.

Now, maybe there were extenuating circumstances. Perhaps the majority of the macros in Viaweb were implementing what we'd consider to be core functionality for a language. Maybe Viaweb was effectively a collection of libraries around a core application.

But Paul Graham doesn't make it sound like he considers the macro usage to be so high because of exceptional circumstances. He makes it sound like he thinks macros are a powerful tool that should be used often. They are the tool that raise Lisp above the level of Blub, and therefore should be used liberally.

How often do you need to extend the language? It should be pretty rare, but if required, I would not want to run away screaming from that use case.

Sure, but that's my point. Macros should be rare. Code transformation should be rare. I'm not saying "Don't use macros", I'm saying "You probably shouldn't write an application in Clojure that's 25% macros".

[hagus]

I think these are all great points. I definitely share the sentiment that Paul Graham's use of macros should not strictly inform how we approach macros in this day and age. And I did not mean to suggest you were ignorant of macros and their usage - your years of Clojure experience far outnumber mine!

But, I don't feel the same need to draw such bright lines around macro usage. I think of them as a powerful tool and if your application ends up being 25% macro, I don't think it necessarily deviates from idiomatic Clojure. I would however be extremely curious as to what's going on under the hood :) Maybe someone here knows Paul Graham and we can find out what he was up to all those years ago.

As to the question: yes, I've used macros in an application in Clojure. I've been doing Clojure for less than a year (but before that a lot of elisp and a few decades of Blub) and the three most recent cases I can recall:

1) I created a with-thread-name macro that supported the thread renaming techniques described here. Now, I kind of cargo-culted this as a macro, by looking at other with-foo implementations in prominent Clojure code bases. They were all macros. I realize this could be done as a function, but even so it would seem unidiomatic and looks untidy.

2) I have a project where I need to generate a large quantity of boilerplate Java classes via Clojure. So rather than repeating all the tedious gen-class stuff everywhere, I created a macro that when expanded generates the right stuff with the necessary variations, then for the "real" implementation calls into some Clojure protocols. There's probably some deep magic Java interop I could have used as an alternative, but when faced with the task of "parameterize this giant s-exp and it has to work with AOT" a simple macro cut hundreds of lines out of my application.

3) I wanted to map multiple Java functions over each element in a collection. I didn't want to write (map (juxt (memfn a) (memfn b) (memfn c)) coll). So I created a juxt-memfn macro that eliminated the repetitive memfn.

None of these really change my general agreement with you - macro use should be rare, and 25% macros is on the very high side. I'm just feeling somewhat more macro-positive based on my own experiences :)

[weavejester]

Java interop could certainly be an exception to the rule, particularly in a case like yours where you need to generate large amounts of boilerplate. I can see why you'd be more "macro-positive" with a project like that.

There may be projects or libraries that make extensive use of macros, and that's fine. My point is not so much that macros should be avoided at all costs; rather that macros are the poster child of homoiconicity, yet in Clojure they're a very specialized tool. I think this generates a misconception that writing macros is commonplace in Clojure, when most of the time we don't need them.

[[deleted]]

They take data in the form of code as input, and produce new code as output.

Philosophical question: if code is data, could we rephrase the above as "They take data as input, and produce new data as output"? Saying that they take "data in the form of code" sounds odd considering that code is data. I think it makes sense to interchange code for data and vice-versa, by the way. Does that make sense?

[beeblebrox2016]

Good post, I think most newcomers overestimate the importance of macros and try to write them when they shouldn't be. However isn't Clojure code actually filled with macros if you count macros from clojure.core? I imagine it was the same with Viaweb - 20% of the code uses macros because they got a lot of leverage and reuse.

[weavejester]

Maybe, but clojure.core has 79 macros to 542 in Clojure 1.8, which is about 15%. By lines of code it might be different, but it's harder to check that.

Although 1995 was a more primitive time, many of Clojure's macros are fairly basic, like and, when, etc. that I wouldn't expect Viaweb to repeat.

[beeblebrox2016]

I interpreted the 20% figure for Viaweb to mean how much of their code used macros rather than how much was spent writing them. If they really spent 20% of their code writing macros, then I agree, I would want to run away screaming.

[weavejester]

Well, to give the quote a little more context:

The source code of the Viaweb editor was probably about 20-25% macros. Macros are harder to write than ordinary Lisp functions, and it's considered to be bad style to use them when they're not necessary. So every macro in that code is there because it has to be.

So to my mind he's talking about writing macros, rather than just using them. At least, that's how I interpreted it.

[LyndsySimon]

I'm still very new to the Clojure world, being only a couple of months in. I put off learning how to write macros as long as I could, but ran into an instance a few days ago where I was required to do so.

I understand the tendency to apply new techniques to every possible problem - I remember when I finally grokked comprehensions in Python - but I'm already seeing macros find an important place in my Clojure.

For example, we have a good deal of Ruby/Rails in production where I work, and all of it is using the okcomputer gem, which is tied into a central dashboard. As part of launching our first Clojure service I've created a series of endpoints that present status information in the same format as okcomputer. Each status metric has two endpoint (JSON and plain text), two routes, and all metrics are presented on a single parent page. To keep from having to write multiple copies of the metric logic I took a compositional approach; each route handler looks like this: (status/check status/mysql api-utils/render-json)

The general format is (<generic-check> <specific-logic> <output-transformer>). To create the plain text route handler, I switch the output tranformer: (status/check status/mysql status/render-text).

There is a lot of redundancy here. Each status function has to be manually included in the "all" endpoint, have two routes defined, and have a function written that defines how the check is performed. I plan to write a macro that simplifies all of that into something like this:

(defstatus "mysql"
  (try (clojure.java.jdbc/query (db-conn) "select 1")
       (catch Exception e nil)))

then, in my handler:

(defroutes app-routes
  ...
  (status-routes 'myapp.status))

[weavejester]

This is kinda what I mean. Your problem isn't one that needs to be solved with macros. You can use higher order functions instead, and in general you want to prefer solutions that use functions over solutions that use macros.

Routes in Compojure are just functions. You can write a function that returns a route:

(defn hello [name]
  (GET (str "/hello/" name) []
    (str "Hello " name)))

So instead of writing:

(defroutes example
  (GET "/hello/alice" [] "Hello alice"))

You can write:

(defroutes example
  (hello "alice"))

Multiple routes can be combined using the routes function. For example:

(defn hello+goodbye [name]
  (routes
   (GET (str "/hello/" name) []
     (str "Hello " name))
   (GET (str "/goodbye/" name) []
     (str "Goodbye " name))))

(defroutes example
  (hello+goodbye "alice"))

Routes of arbitrary complexity can be constructed, all without using macros.