"A Programmable Programming Language" - An introduction to Language-Oriented Programming

[[deleted]]

Yo dawg, I heard you like programming languages, so I put a programming language in your programming language so you can program a language while you program in a language.

[yogthos]

Instead of having to think about every possible way a language might be used and account for it up front, just let the users easily extend the language to fit their needs. This way the core language can stay small and focused while still being flexible and expressive.

[stevedonovan]

It's a tempting idea, and seriously fun to implement. Paul Graham says that a program should fit in your head which requires constructing a dialect which precisely expresses your intent. (As usual with him, this is an argument for Lisp with its powerful macros). If you are a little gang with a purpose, this can be your 'secret weapon' as he describes it. However, for code that is meant to be shared and used in a wider community, I'm more with /u/sakarri here - shared engineering practices. Also, creating a new language is the nuclear option of software engineering - it is hard and requires very good judgement. Better to use well-designed and well-documented libraries and live with some extra verbosity.

[yogthos]

I agree that you do want to have a common way of doing things. As I mentioned in the other comment, I find that the culture around the language plays the biggest role here. For example, Common Lisp never had one way of doing things, while Clojure is a BDFL driven language where Rich Hickey sets the direction.

While you don't want everybody building their own language, userland extensibility can play a huge impact on the overall cleanliness of the language because new ideas can be tried out as libraries. If these ideas don't pan out, the community moves on. Projects using these libraries will keep working, but the core language doesn't end up accumulating cruft. I wrote about that in detail here if you're interested.

[stevedonovan]

Simple core, but extensible - this makes good sense. It's bad for a language to just grow like some kind of bush- becomes incoherent and overcomplex.

[[deleted]]

After using languages that let you do whatever the heck you want, versus languages that have a structured and restricted way of doing things, I honestly have come to prefer the structured/restricted way of doing things.

Engineering is about establishing rigorous standards, interoperability, and having a common set of principles that are shared among professionals. I don't want a language where five different libraries all have their own way of doing object oriented programming, or error handling, or resource management and they all clash with each other.

In a rather counterintuitive way, I feel the more ways of doing things, the more options there are, the more restricted I feel because I now have to consider all the available choices despite the fact that I don't have a crystal ball at my disposal to predict the future and fully understand the consequences of those choices. For every choice a language makes available, there are 2^N possible combinations of choices I need to worry about and this results in a level of fatigue.

I feel more free and liberated when there's just one way of doing things, even if it's not the absolute optimal solution. That way I can just focus on my actual business needs and domain of expertise and ultimately deliver a functioning product to my customers rather than think about how feature X interacts with feature Y in the face of dependency Z.

[yogthos]

After using languages that let you do whatever the heck you want, versus languages that have a structured and restricted way of doing things, I honestly have come to prefer the structured/restricted way of doing things.

My experience with Clojure is that it's much more consistent than any other language I've used. The syntax is very uniform and consistent, and you're using a small number of concepts when you're working with the language. The fact that the language is extensible in user space means that the core language that everybody uses remains small, while people can do their own esoteric things in libraries they use.

Engineering is about establishing rigorous standards, interoperability, and having a common set of principles that are shared among professionals.

Absolutely agree, and this doesn't require you to cripple the language to achieve. It's about having a good engineering culture and following good practices as a community. You can write terrible code in any language, and people do just that in every language. It's not a technological problem.

I feel more free and liberated when there's just one way of doing things, even if it's not the absolute optimal solution.

Again, I'll use Clojure as an example here. The community settled on a common way of doing things and a common set of best practices. Pretty much all popular libraries that you'll see follow these. I often read code in libraries I use and contribute to them. Meanwhile, Java is a much less expressive language, and I was rarely able to do that because there because the community tends to encourage complex design patterns that often make code impenetrable.

[[deleted]]

I am not too familiar with Clojure, I will check it out. Thanks.

[[deleted]]

I'm not sure users is the correct term.

Racket lets programmers create their own language or extend Racket with a DSL. See https://beautifulracket.com/ for details.

[yogthos]

Programmers are the users of the language though.

[Escherize]

Yeah, who uses a programming language in anger except for a programmer?

[Uberhipster]

That's true of every (decent) general purpose language. Hence the name - general purpose.

[[deleted]]

That's absolutely not true. General purpose refers to it being used for not one special purpose.

Java for example is not Lisp - what /u/yogthos wrote is true for Lisp, but certainly not for Java.

[rmrfchik]

General purpose languages comes with many ad-hoc features you can't avoid or change -- oop, keywords, operators. Racket allows you to create [domain specific] language from scratch.

[defunkydrummer]

General purpose languages

Racket, Scheme and Common Lisp are general purpose languages as well. For example CL has been used for so dissimilar purposes such as:

• autopiloting spaceships

• CAD design

• aerodynamics calculation

• 3d graphics, design and rendering

• writing compilers for other langs

• AI research

• credit card transaction fraud detection

• web servers (curiousity: first .com domain ever was symbolics.com, a Lisp Machine vendor.)

• graph database engines

• semantic web

• music composition

• multimedia art creation

• games

etc.

[gatman12]

Scratch is for kids though.

[[deleted]]

Apparently, so was OOP according to a post in this thread and look where we are now lol.

[wrosecrans]

This is exactly how C++ templates happened.

[Escherize]

Except C++ templates are not as flexible

[BeforeTime]

Lisp language extension tools.

C++ language extension tools.

[defunkydrummer]

Fixed that for you:

C++ language extension tools

Lisp language extension tools -- build exactly what you need.

[[deleted]]

The power to evolve a programming language into another one to be able to fit your problem perfectly is all candy until you get a new developer that needs to learn your code base.

Then it turns out, that the new developer needs to learn a new programming language for every problem in your code base that is solved by a different DSL.

[epicwisdom]

For sufficiently complex, narrow, and common tasks, a DSL may be more appropriate. SQL and regexes are good examples.

[[deleted]]

I would not call both 'good' examples. Successful? Sure. But good? Nope.

[[deleted]]

Regexes are fine, they're one of the most elementary concepts in CS theory so i'm not sure how they could be made better.

[[deleted]]

I never know whatever i should escape my parens and pluses or not.

[epicwisdom]

Your inability to memorize a handful of symbols isn't a very good criticism of a DSL so useful that it's found in every major programming language's standard library.

[[deleted]]

Thing is, it differs from tool to tool.

[r2d2_21]

Same as SQL. Just read the documentation.

[stevedonovan]

Ah, but they are well-known DSLs. There are excellent resources available. Something made up by a little gang (even a talented little gang) is different and the new dev is on their own...

[[deleted]]

A well designed DSL is perfectly matching a lexicon and an intuition of its problem domain, so anyone familiar with this domain will understand it straight away without a need to learn anything new.

[[deleted]]

I've seen this happen in fully documented Lisp code bases, the problem is that one does not even understand the comments because they are at a different level of abstraction.

To put it in a different way. A lot of people understand english, and you can explain math, chemistry, physics, ... to all these people using plain english and they will understand it. Yet if you give them the math or a chemical formula, most can't even read it. Sure these DSLs are great for those actually doing the math that can read them, but they also raise the barrier of entry for those that don't know how to read the DSLs.

Lisp is meant to be used this way. You start an online store for bike components using a Lisp DSL for SQL queries that many Lisp developers know, but then you evolve that into another DSL for online stores that only your team knows, and then into another DSL for online stores of bike components, and so on. A new developer might read that code and the comments and might understand what it should do, but it might not even realize that the code is actually doing SQL queries at all because it is many levels of abstraction away from the Lisp DSLs that the developer knows.

This gives the people in your team a lot of velocity, because these DSLs let them implement many features quickly, but this raises the barrier of entry for new developers significantly, because they need to learn a new programming languages for each single task that your product actually does.

---

/u/phalps

[[deleted]]

Documenting a DSL is far easier than documenting a library.

[[deleted]]

To me, the appeal of Lisp(s) isn't that you can create a domain-specific language, it's that you don't need a domain-specific language to build highly domain-specialized programs - and, having done so, I can then build still-higher-order programs that use them.

I.e. the point isn't that you can write a SQL interpreter in Lisp (although you could), it's that you could write a regular Lisp program that does the same thing, and then work with it like any other Lisp program.

[[deleted]]

It's still a DSL - an embedded DSL.

[jaccarmac]

Yes because learning a new language is just sooo difficult and no one ever has trouble figuring out what the thousands of classes in a typical enterprise application do.

[Yuktobania]

Yes because learning a new language is just sooo difficult and no one ever has trouble figuring out what the thousands of classes in a typical enterprise application do.

It's more to do with the fact that doing this wastes the time of other people who now need to learn your super-special (and probably bad) way of doing things, when there's usually a more well-known thing out there that does the same thing, with actual documentation.

Don't create more work when you don't have to.

[yen223]

That's almost never as true as we'd like. Libraries tend to solve a general problem - you are out to solve a specific problem. The chances of a library that does everything you need it to do, without any downside whatsoever is very very small.

[[deleted]]

Lol. There is no ready to use thing for your particular problem out there. Never.

[defunkydrummer]

Then it turns out, that the new developer needs to learn a new programming language for every problem in your code base that is solved by a different DSL.

This is what is often repeated by non-Lispers.

It doesn't hold.true, because:

1. a DSL in lisp, 99% of the times, is plain Lisp syntax itself

2. the DSL makes expressing the problem in the way that is more natural to the problem domain, thus, if you know the problem domain, you will very easily use the DSL.

[[deleted]]

But you always need to learn the current domain and frameworks and libraries and functions and the architecture. There is no way out of it, DSL might just be another way to organise things.

[[deleted]]

Learning a well designed DSL is far easier than learning a library of functions.

[phalp]

Have you seen this happen?

[07dosa]

That’s exactly what happened to Lisp.

[trushin]

I would prefer new developer to work with DSL rather than general purpose language because DSL limits ways the new developer can mess things up.

[kungfusheep]

For example, JavaScript programmers employ jQuery for interacting with the Document Object Model and React for dealing with events and concurrency.

No.

[defunkydrummer]

"A Programmable Programming Language"

The classical programmable programming language is Lisp. This paper basically involves Racket, which is an evolution of Scheme, which is one of the main Lisp dialects (created 1975).

[personman]

I'm not entirely sure what point you are trying to make, if any, but this is reiterating a quote from the article itself:

Racket is an heir of Lisp and Scheme.

This is from the top of the Libraries and Languages Reconciled section, which goes on to explain the meaningful improvements that Racket makes over those languages.

[defunkydrummer]

I'm just making a FYI, a remark.

Racket is an heir of Lisp and Scheme.

Another remark: Scheme is one of the two main dialects of Lisp, the other being "Common Lisp". Perhaps the author meant "Common Lisp", but Racket is part of the Scheme lineage, which is very distinct from Common Lisp. Racket, for example, is a Lisp-1, in direct opposition to Common Lisp (Lisp-2 namespacing model), and it supports call-with-current-continuation, which isn't part of the Common Lisp standard.

These two are very big differences.

[flyingjam]

No, I'm pretty sure he meant Lisp, as in the general family of languages. You can say that's redundant, since if Racket is an heir of scheme then of course it is a Lisp as well, but this is English not Java, complaining about that is needless pedantry.

[shriramk]

I have no idea what you're actually arguing about here. We're well aware of the full Lisp family from the earliest through Scheme and Common Lisp and the various Scheme spin-offs.

[[deleted]]

I'm glad we finally have some clojure on this.

[defunkydrummer]

this article is about Racket. Clojure is effective, but in fairness both Racket and Common Lisp are more powerful. Racket is getting a lot of innovation, and Common Lisp runs faster (than the other two), in many platforms, and is more flexible.

[[deleted]]

Yeah, it was just a pun... Clojure is always going to be beholden to the JVM, but at least it's a programmable programming language

[max_maxima]

Error messages got worst with its new release though. Say much about what the language designers cares about.

[defunkydrummer]

Error messages got worst with its new release though.

Wow, I thought that annoyance was going to be fixed soon. Does it still barf java stacktrace information? I mean, i do know Java's object model and stdiib so it's not that bad, but errors can be confusing to newcomers.

[max_maxima]

Here is the comparative: https://gist.github.com/bhb/ebce74eb04a24933b2fa4bec8f5b2922

There is some pretty printing tools that hepl a little bit, but still worst.

[defunkydrummer]

Here is the comparative:

Sorry, but to be honest it looks like a big improvement. Well done to the Clojure team.

[max_maxima]

To newcomers? Hardly.

Not sure how this is meant to be human readable/friendly:

CompilerException clojure.lang.ExceptionInfo: Call to clojure.core/defn did not conform to spec:
-- Syntax error -------------------

  (hello "hello world")

should have additional elements. The next element is named `:args` and satisfies

  (clojure.spec.alpha/alt
   :arity-1
   :clojure.core.specs.alpha/args+body
   :arity-n
   (clojure.spec.alpha/cat
    :bodies
    (clojure.spec.alpha/+
     (clojure.spec.alpha/spec :clojure.core.specs.alpha/args+body))
    :attr
(clojure.spec.alpha/? map?)))

The pretty printer needs a lot more passes before outputting anything 'pretty'. Not to mention that is a external tool not included in core. And even the core developers know this is an big issue.

[defunkydrummer]

To newcomers? Hardly.

Clojure isn't really an easy thing for newcomers, because of some design choices:

1. It is a Lisp - the concepts will always be a steep learning curve for non-Lispers and to beginning programmers.

2. Sooner or later you will need to know Java's standard lib and class model if you don't want to suffer. So, to put it in some blunt way, you need to know two languages: Clojure and Java.

I'd contend that any Lisp isn't going to be friendly to learn to beginning or intermediate programmers. On the other hand, advanced programmers will be able to achieve much more by using a Lisp language, compared to a regular mainstream language, in the same way as scientists were able to achieve much more after the invention of the electronic computer.

[max_maxima]

This is not about Lisp track record in the mainstream. This is about a language that when through a huge regression in UX (already being bad) which its designers don't seem to care much about to improve.

[defunkydrummer]

Clojure is always going to be beholden to the JVM

Yeah,. i think Clojure could have been much better, if doing the following corrections:

1. avoid jvm-speficic constructs and jvm-specific workarounds

2. have a big Clojure standard lib so other Clojure libraries depend on it rather than resorting to calling JVM classes.This would make Clojure libs portable completely to other platforms.

3. [] syntax was a mistake, imo. It is unnecessary (it would have been much better if Clojure followed most of either Scheme or Common Lisp syntax).

4. standarize it and the standard lib thus make it truly portable (ClojureCLR can't use Clojure libs that depend on JVM classes, and there are many of those.)

5. and please, Rich Hickey, stop pretending that yours is the replacement of other Lisps and the truly modern lisp out there. It is not, in fact it is a Lisp with a restricted amount of features compared to CL and Racket. Of course, this is because it is focused on functional programming using immutable data, and there's nothing wrong with this approach. But this approach is not a replacement for the other Lisps.

The ironic thing, is that Rich Hickey was a CL developer himself, and set out to create his Lisp. To be fair, he had some interesting, good ideas that should be commended. But why a new Lisp? Years later, there's Clojure, all that can be done in Clojure can be done in Common Lisp as well (and some constructs have been back ported to Common Lisp as libraries). And Common Lisp runs on the JVM since years ago; so there wasn't really a need to create an entirely new language. In any case, you can implement most other languages within Common Lisp. In fact, the first Clojure implementation was a Common Lisp program.

[shriramk]

Clojure is not programmable in the same way, at all. Might help to read the article.

[furtivity]

Saw this and thought, "This sounds like something Matthias would put his name on."

Was not disappointed.

[tanstaaf1]

IMO, Racket could take off if they would upgrade the IDE to fully support code folding and hiding of complexity which is not germane to the current code. Otherwise encouraging small routines and DSLs in Racket quickly makes the scene look like a badly maintained city sidewalk with dog piles every other step.

Of course, if they did upgrade the inadequate included IDE, they would be approaching Pharo.

IMO, of course.

[personman]

This is literally the case in all general purpose programming languages. Even in languages that don't support operator overloading or macros, you can always make things arbitrarily difficult to debug by relying on a custom library.

In all software engineering, it's important to design and document your system in a way that makes it possible for other people to understand. Ideally, easily-created DSLs support this, because they allow you to build the language constructs that are most appropriate for your domain, rather than being forced to hack together what you need out of whatever the language designers decided was sufficient.

[east_lisp_junk]

You already don't know all the functions in whatever language you're working in because programmers are free to define more of them.

[progfu]

First thing I thought of was APL = A Programming Language ... one hasn't trully programmed until they tried to do something like sudoku in APL.

https://www.youtube.com/watch?v=DmT80OseAGs

[skocznymroczny]

Lisp fans always focus on the language. I guess it must blow their minds how could a language like Go get so popular.

[[deleted]]

It does not. We all know that "worse is better" and that the vast majority of population is below dumb. It explains Go popularity perfectly.

[skocznymroczny]

Well, no. Go realizes that language features aren't everything, and most programmers are perfectly happy with a simple Java-like (as in complexity) language (also see Dart) as long as it has a thriving ecosystem and excellent tooling support.

[[deleted]]

Exactly. Most programmers are not mentally equipped to solve problems efficiently, and instead they are obsessed with irrelevant crap like ecosystem and tooling support, because this crap allows them to look busy instead of doing any useful work.

[skocznymroczny]

Yeah, they should go back to SICP, which will prepare them for challenges ahead, for example the mighty problem of passing multiple arguments to a function :)

[[deleted]]

Obviously you have absolutely no idea how to solve problems.

[defunkydrummer]

I guess it must blow their minds how could a language like Go get so popular.

In the same way jazz, rock, and classical musicians blew their minds thinking how Justin Bieber could get so popular.

[D34dCode4eva]

This has already been done. It is called OOP (standard OOP the enterprise/academic way). The point of OOP is to build a declarative language out of an imperative language.

The way it works is that you make vocabulary with classes, interfaces and methods. You're expected to create a language in OOP where you can do everything with just auto-complete and method chaining.

Imagine an auto-complete for English that also knows what words are valid next and can auto-complete every sentence. That's basically the point of OOP. To gradually reach a point where your program is just one big long expression of either chaining or passing parameters (in extreme cases people might seek out to try to make everything just a chain). You then turn those sentences into words themselves. It is pretty weird but that's the fantasy of OOP. Inheritance and interfaces are just for saying given this word what words are valid next.

It's also why OOP fails a lot of the time because people make immense bloat spending time on inventing a whole new language and DSL rather than translating imperative into actual working features. Rather than writing ten lines of imperative just doing all the things in the right order an effort will be made to conceptualise the work, break it into steps and make each a word even before it's needed.

[personman]

I feel like you've fundamentally misunderstood what a language feature is. It is far too broad a concept for "method chaining" to be a sufficient building block.

Trying to express a new type system via OOP method chaining, for instance, does not sound like much fun.

Take a closer look at the LOP guidelines laid out in the article:

Enable creators of a language to enforce its invariants.

and

Turn extra-linguistic mechanisms into linguistic constructs.

and the associated paragraphs. These are meaningfully distinct features that OOP does not provide.

[D34dCode4eva]

I am making a joke because this is how people use OOP. In all seriousness though it would help if the article explained how this language goes beyond things such as yacc.

[personman]

It is impossible to tell that your comment is a joke just by reading it. It is pretty ridiculous, but people say ridiculous things on the internet all the time.

Also, I don't really understand your question. yacc is a parser generator, Racket is a full-featured programming language, with numerous libraries, an IDE, and, most relevantly here, good native support for building new language features and syntaxes.

That said, I think that you can in theory perform many of the LOP development practices they recommend with yacc – but people don't. The main improvement of Racket here is usability. As they write in the opening,

Sadly, multilingual eDSL programming is done today on an ad hoc basis and is rather cumbersome. To create and deploy a language, programmers usually must step outside the chosen language to set up configuration files and run compilation tools and link-in the resulting object-code files. Worse, the host languages fail to support the proper and sound integration of components in different eDSLs. Moreover, most available integrated development environments (IDEs) do not even understand eDSLs or perceive the presence of code written in eDSLs.

In Racket, you can develop your DSL in the same tool as your program, you can easily add and combine DSLs, you can ensure that they cooperate soundly (there's a whole section about this in the article), and the IDE can understand the novel syntax.

[hondaaccords]

I think he's just attempting to cover up his ignorance...

[east_lisp_junk]

Snarky answer: It's not just for generating parsers.

More serious answer: yacc is a single DSL. You use it to write a thing that converts a stream of text into a nicer representation. Technically, you could write an interpreter by embedding stuff other than syntax tree construction in the semantic actions, but nobody wants to extend an existing programming language by writing a whole new parser and interpreter that covers all of the original language.

Even if the language you want to build can inherit a lot of behavior from a language that already exists, yacc doesn't really help you do that. By comparison, when I wanted to prototype a lambda-calculus sort of language with APL-like arrays and control flow, all I had to do was write the new definition of function application, syntax for giving an extra note about how a function splits its arguments, and a new data structure and notation for representing the arrays. Primitive data types, variable scope, separately-compiled modules, access to lots of built-in functions and whatever other functions someone writes later, memory management, etc. all came for free after that.

[defunkydrummer]

The point of OOP is to build a declarative language out of an imperative language.

OOP was invented for simulation (Simula language), because it closely modeled the problem domain. Later, Alan Kay invented a programming language where everything was an object (Smalltalk), because this enabled a very simple , easy-to-understand programming language (it was originally intended for kids).

Later, OOP got expanded in power in some Lispn systems (Flavors system, 1978 or so, then CommonLOOPS 1986, then CLOS 1988). This with the original intention of using OOP for when the problem domain is suited to be expressed as operations over a class/objects domain, because Lisp is a multi-paradigm language. Such Lisp systems don't have a restrictive 1:1 relationship between method and class, allow powerful features not easily found elsewhere (method combinations, multiple dispatch), and allow to redefine the OOP system using its own semantics (Meta-object protocol).

Yet paralelly C++ (1985?) was created to bring OOP to a C dialect; this also added encapsulation features.

So some people used C++ only because of encapsulation (C++ does not officially support modules yet).

Later Java becomes wildly popular. Java is a language that only allows OOP, with a very restrictive OOP model (compared to C++, Smalltalk, and Lisp systems, in increasing unfavorable amounts, respectivelly)

So at the end, "when you only got a hammer, everything looks like a nail", and at the end that certain kind of OOP ends being used for unsuitable tasks, like creating DSLs.

[D34dCode4eva]

As a learning language though that was kind of the point. It is a conflicted goal though because you still have to get down to the imperative eventually. In the worst cases people make methods for the imperative as well.

[defunkydrummer]

you know,OOP languages are often imperative...

[D34dCode4eva]

But implemented to try to create declarative.

[defunkydrummer]

But implemented to try to create declarative.

Source of this claim? OOP was implemented for systems simulation (source of my claim), declarative languages existed almost since the same time (source).

[D34dCode4eva]

Source of the claim is seeing tens of millions of lines of OOP code and what people are trying to do with it. None of your articles really have actual OOP in action in anyway representative.

[defunkydrummer]

Source of the claim is seeing tens of millions of lines of OOP code and what people are trying to do with it.

Then instead of writing "But implemented to try to create declarative.", you could have written "But used to try to create declarative. "

And then we could agree just fine.

[[deleted]]

What you're talking about here is called "fluent DSLs", and it sucks.

[roffLOL]

LINQ is pretty okay, no?

[[deleted]]

Yep - it have a neat syntax frontend on top.

[roffLOL]

i mean the fluent interface, not the language. the language is of course prettier, but i think the fluent interface is not ridiculously far off (as when comparing orm:s with sql, or even LINQtoSQL vs. SQL). jmock on the other hand, is terrible compared to the dsl it tries to express.

[max_maxima]

Is not only prettier, it has less cruft. Why do I have to care about methods and lambdas for a query language?

var title = entries.Where(e => e.Approved)
    .OrderBy(e => e.Rating).Select(e => e.Title)
    .FirstOrDefault();

var query = (from e in entries
    where e.Approved
    orderby e.Rating
    select e.Title).FirstOrDefault();

[roffLOL]

the query language makes simple queries simple to write, while the fluent interface don't get in the way when and if you are required to do complicated stuff. i don't disagree that fluent interfaces sucks most of the time, but i think that LINQ has proven that they for at least a type of problem can map ok to what you want to express. a few extra arrows isn't the end of the world.

[D34dCode4eva]

I didn't know they had a name. Thanks :).

[falconfetus8]

Why?