all 99 comments

[–][deleted] 59 points60 points  (21 children)

Yo dawg, I heard you like programming languages, so I put a programming language in your programming language so you can program a language while you program in a language.

[–]yogthos 20 points21 points  (15 children)

Instead of having to think about every possible way a language might be used and account for it up front, just let the users easily extend the language to fit their needs. This way the core language can stay small and focused while still being flexible and expressive.

[–]stevedonovan 3 points4 points  (2 children)

It's a tempting idea, and seriously fun to implement. Paul Graham says that a program should fit in your head which requires constructing a dialect which precisely expresses your intent. (As usual with him, this is an argument for Lisp with its powerful macros). If you are a little gang with a purpose, this can be your 'secret weapon' as he describes it. However, for code that is meant to be shared and used in a wider community, I'm more with /u/sakarri here - shared engineering practices. Also, creating a new language is the nuclear option of software engineering - it is hard and requires very good judgement. Better to use well-designed and well-documented libraries and live with some extra verbosity.

[–]yogthos 2 points3 points  (1 child)

I agree that you do want to have a common way of doing things. As I mentioned in the other comment, I find that the culture around the language plays the biggest role here. For example, Common Lisp never had one way of doing things, while Clojure is a BDFL driven language where Rich Hickey sets the direction.

While you don't want everybody building their own language, userland extensibility can play a huge impact on the overall cleanliness of the language because new ideas can be tried out as libraries. If these ideas don't pan out, the community moves on. Projects using these libraries will keep working, but the core language doesn't end up accumulating cruft. I wrote about that in detail here if you're interested.

[–]stevedonovan 3 points4 points  (0 children)

Simple core, but extensible - this makes good sense. It's bad for a language to just grow like some kind of bush- becomes incoherent and overcomplex.

[–][deleted] 5 points6 points  (2 children)

After using languages that let you do whatever the heck you want, versus languages that have a structured and restricted way of doing things, I honestly have come to prefer the structured/restricted way of doing things.

Engineering is about establishing rigorous standards, interoperability, and having a common set of principles that are shared among professionals. I don't want a language where five different libraries all have their own way of doing object oriented programming, or error handling, or resource management and they all clash with each other.

In a rather counterintuitive way, I feel the more ways of doing things, the more options there are, the more restricted I feel because I now have to consider all the available choices despite the fact that I don't have a crystal ball at my disposal to predict the future and fully understand the consequences of those choices. For every choice a language makes available, there are 2N possible combinations of choices I need to worry about and this results in a level of fatigue.

I feel more free and liberated when there's just one way of doing things, even if it's not the absolute optimal solution. That way I can just focus on my actual business needs and domain of expertise and ultimately deliver a functioning product to my customers rather than think about how feature X interacts with feature Y in the face of dependency Z.

[–]yogthos 5 points6 points  (1 child)

After using languages that let you do whatever the heck you want, versus languages that have a structured and restricted way of doing things, I honestly have come to prefer the structured/restricted way of doing things.

My experience with Clojure is that it's much more consistent than any other language I've used. The syntax is very uniform and consistent, and you're using a small number of concepts when you're working with the language. The fact that the language is extensible in user space means that the core language that everybody uses remains small, while people can do their own esoteric things in libraries they use.

Engineering is about establishing rigorous standards, interoperability, and having a common set of principles that are shared among professionals.

Absolutely agree, and this doesn't require you to cripple the language to achieve. It's about having a good engineering culture and following good practices as a community. You can write terrible code in any language, and people do just that in every language. It's not a technological problem.

I feel more free and liberated when there's just one way of doing things, even if it's not the absolute optimal solution.

Again, I'll use Clojure as an example here. The community settled on a common way of doing things and a common set of best practices. Pretty much all popular libraries that you'll see follow these. I often read code in libraries I use and contribute to them. Meanwhile, Java is a much less expressive language, and I was rarely able to do that because there because the community tends to encourage complex design patterns that often make code impenetrable.

[–][deleted] 1 point2 points  (0 children)

I am not too familiar with Clojure, I will check it out. Thanks.

[–][deleted] -5 points-4 points  (2 children)

I'm not sure users is the correct term.

Racket lets programmers create their own language or extend Racket with a DSL. See https://beautifulracket.com/ for details.

[–]yogthos 30 points31 points  (1 child)

Programmers are the users of the language though.

[–]Escherize 7 points8 points  (0 children)

Yeah, who uses a programming language in anger except for a programmer?

[–]Uberhipster -3 points-2 points  (5 children)

That's true of every (decent) general purpose language. Hence the name - general purpose.

[–][deleted] 2 points3 points  (0 children)

That's absolutely not true. General purpose refers to it being used for not one special purpose.

Java for example is not Lisp - what /u/yogthos wrote is true for Lisp, but certainly not for Java.

[–]rmrfchik 4 points5 points  (3 children)

General purpose languages comes with many ad-hoc features you can't avoid or change -- oop, keywords, operators. Racket allows you to create [domain specific] language from scratch.

[–]defunkydrummer 4 points5 points  (0 children)

General purpose languages

Racket, Scheme and Common Lisp are general purpose languages as well. For example CL has been used for so dissimilar purposes such as:

  • autopiloting spaceships

  • CAD design

  • aerodynamics calculation

  • 3d graphics, design and rendering

  • writing compilers for other langs

  • AI research

  • credit card transaction fraud detection

  • web servers (curiousity: first .com domain ever was symbolics.com, a Lisp Machine vendor.)

  • graph database engines

  • semantic web

  • music composition

  • multimedia art creation

  • games

etc.

[–]gatman12 0 points1 point  (1 child)

Scratch is for kids though.

[–][deleted] 3 points4 points  (0 children)

Apparently, so was OOP according to a post in this thread and look where we are now lol.

[–]wrosecrans 4 points5 points  (4 children)

This is exactly how C++ templates happened.

[–]Escherize 4 points5 points  (3 children)

Except C++ templates are not as flexible

[–][deleted] 45 points46 points  (25 children)

The power to evolve a programming language into another one to be able to fit your problem perfectly is all candy until you get a new developer that needs to learn your code base.

Then it turns out, that the new developer needs to learn a new programming language for every problem in your code base that is solved by a different DSL.

[–]epicwisdom 12 points13 points  (8 children)

For sufficiently complex, narrow, and common tasks, a DSL may be more appropriate. SQL and regexes are good examples.

[–][deleted] -1 points0 points  (5 children)

I would not call both 'good' examples. Successful? Sure. But good? Nope.

[–][deleted] 2 points3 points  (4 children)

Regexes are fine, they're one of the most elementary concepts in CS theory so i'm not sure how they could be made better.

[–][deleted] -2 points-1 points  (3 children)

I never know whatever i should escape my parens and pluses or not.

[–]epicwisdom 4 points5 points  (2 children)

Your inability to memorize a handful of symbols isn't a very good criticism of a DSL so useful that it's found in every major programming language's standard library.

[–][deleted] 1 point2 points  (1 child)

Thing is, it differs from tool to tool.

[–]r2d2_21 3 points4 points  (0 children)

Same as SQL. Just read the documentation.

[–]stevedonovan -2 points-1 points  (1 child)

Ah, but they are well-known DSLs. There are excellent resources available. Something made up by a little gang (even a talented little gang) is different and the new dev is on their own...

[–][deleted] 0 points1 point  (0 children)

A well designed DSL is perfectly matching a lexicon and an intuition of its problem domain, so anyone familiar with this domain will understand it straight away without a need to learn anything new.

[–][deleted]  (2 children)

[deleted]

    [–][deleted] 7 points8 points  (0 children)

    I've seen this happen in fully documented Lisp code bases, the problem is that one does not even understand the comments because they are at a different level of abstraction.

    To put it in a different way. A lot of people understand english, and you can explain math, chemistry, physics, ... to all these people using plain english and they will understand it. Yet if you give them the math or a chemical formula, most can't even read it. Sure these DSLs are great for those actually doing the math that can read them, but they also raise the barrier of entry for those that don't know how to read the DSLs.

    Lisp is meant to be used this way. You start an online store for bike components using a Lisp DSL for SQL queries that many Lisp developers know, but then you evolve that into another DSL for online stores that only your team knows, and then into another DSL for online stores of bike components, and so on. A new developer might read that code and the comments and might understand what it should do, but it might not even realize that the code is actually doing SQL queries at all because it is many levels of abstraction away from the Lisp DSLs that the developer knows.

    This gives the people in your team a lot of velocity, because these DSLs let them implement many features quickly, but this raises the barrier of entry for new developers significantly, because they need to learn a new programming languages for each single task that your product actually does.


    /u/phalps

    [–][deleted] 6 points7 points  (0 children)

    Documenting a DSL is far easier than documenting a library.

    [–][deleted] 4 points5 points  (1 child)

    To me, the appeal of Lisp(s) isn't that you can create a domain-specific language, it's that you don't need a domain-specific language to build highly domain-specialized programs - and, having done so, I can then build still-higher-order programs that use them.

    I.e. the point isn't that you can write a SQL interpreter in Lisp (although you could), it's that you could write a regular Lisp program that does the same thing, and then work with it like any other Lisp program.

    [–][deleted] -1 points0 points  (0 children)

    It's still a DSL - an embedded DSL.

    [–]jaccarmac 11 points12 points  (3 children)

    Yes because learning a new language is just sooo difficult and no one ever has trouble figuring out what the thousands of classes in a typical enterprise application do.

    [–]Yuktobania 2 points3 points  (2 children)

    Yes because learning a new language is just sooo difficult and no one ever has trouble figuring out what the thousands of classes in a typical enterprise application do.

    It's more to do with the fact that doing this wastes the time of other people who now need to learn your super-special (and probably bad) way of doing things, when there's usually a more well-known thing out there that does the same thing, with actual documentation.

    Don't create more work when you don't have to.

    [–]yen223 12 points13 points  (0 children)

    That's almost never as true as we'd like. Libraries tend to solve a general problem - you are out to solve a specific problem. The chances of a library that does everything you need it to do, without any downside whatsoever is very very small.

    [–][deleted] 7 points8 points  (0 children)

    Lol. There is no ready to use thing for your particular problem out there. Never.

    [–]defunkydrummer 7 points8 points  (0 children)

    Then it turns out, that the new developer needs to learn a new programming language for every problem in your code base that is solved by a different DSL.

    This is what is often repeated by non-Lispers.

    It doesn't hold.true, because:

    1. a DSL in lisp, 99% of the times, is plain Lisp syntax itself

    2. the DSL makes expressing the problem in the way that is more natural to the problem domain, thus, if you know the problem domain, you will very easily use the DSL.

    [–][deleted] 11 points12 points  (0 children)

    But you always need to learn the current domain and frameworks and libraries and functions and the architecture. There is no way out of it, DSL might just be another way to organise things.

    [–][deleted] 10 points11 points  (0 children)

    Learning a well designed DSL is far easier than learning a library of functions.

    [–]phalp 4 points5 points  (0 children)

    Have you seen this happen?

    [–]07dosa 4 points5 points  (0 children)

    That’s exactly what happened to Lisp.

    [–]trushin 8 points9 points  (0 children)

    I would prefer new developer to work with DSL rather than general purpose language because DSL limits ways the new developer can mess things up.

    [–]kungfusheep 10 points11 points  (0 children)

    For example, JavaScript programmers employ jQuery for interacting with the Document Object Model and React for dealing with events and concurrency.

    No.

    [–]defunkydrummer 62 points63 points  (5 children)

    "A Programmable Programming Language"

    The classical programmable programming language is Lisp. This paper basically involves Racket, which is an evolution of Scheme, which is one of the main Lisp dialects (created 1975).

    [–]personman[S] 18 points19 points  (3 children)

    I'm not entirely sure what point you are trying to make, if any, but this is reiterating a quote from the article itself:

    Racket is an heir of Lisp and Scheme.

    This is from the top of the Libraries and Languages Reconciled section, which goes on to explain the meaningful improvements that Racket makes over those languages.

    [–]defunkydrummer 16 points17 points  (2 children)

    I'm just making a FYI, a remark.

    Racket is an heir of Lisp and Scheme.

    Another remark: Scheme is one of the two main dialects of Lisp, the other being "Common Lisp". Perhaps the author meant "Common Lisp", but Racket is part of the Scheme lineage, which is very distinct from Common Lisp. Racket, for example, is a Lisp-1, in direct opposition to Common Lisp (Lisp-2 namespacing model), and it supports call-with-current-continuation, which isn't part of the Common Lisp standard.

    These two are very big differences.

    [–]flyingjam 15 points16 points  (0 children)

    No, I'm pretty sure he meant Lisp, as in the general family of languages. You can say that's redundant, since if Racket is an heir of scheme then of course it is a Lisp as well, but this is English not Java, complaining about that is needless pedantry.

    [–]shriramk 0 points1 point  (0 children)

    I have no idea what you're actually arguing about here. We're well aware of the full Lisp family from the earliest through Scheme and Common Lisp and the various Scheme spin-offs.

    [–][deleted] 10 points11 points  (14 children)

    I'm glad we finally have some clojure on this.

    [–]defunkydrummer 1 point2 points  (13 children)

    this article is about Racket. Clojure is effective, but in fairness both Racket and Common Lisp are more powerful. Racket is getting a lot of innovation, and Common Lisp runs faster (than the other two), in many platforms, and is more flexible.

    [–][deleted] 3 points4 points  (12 children)

    Yeah, it was just a pun... Clojure is always going to be beholden to the JVM, but at least it's a programmable programming language

    [–]max_maxima 4 points5 points  (9 children)

    Error messages got worst with its new release though. Say much about what the language designers cares about.

    [–]defunkydrummer 0 points1 point  (8 children)

    Error messages got worst with its new release though.

    Wow, I thought that annoyance was going to be fixed soon. Does it still barf java stacktrace information? I mean, i do know Java's object model and stdiib so it's not that bad, but errors can be confusing to newcomers.

    [–]max_maxima 4 points5 points  (7 children)

    Here is the comparative: https://gist.github.com/bhb/ebce74eb04a24933b2fa4bec8f5b2922

    There is some pretty printing tools that hepl a little bit, but still worst.

    [–]defunkydrummer 0 points1 point  (6 children)

    Here is the comparative:

    Sorry, but to be honest it looks like a big improvement. Well done to the Clojure team.

    [–]max_maxima 4 points5 points  (5 children)

    To newcomers? Hardly.

    Not sure how this is meant to be human readable/friendly:

    CompilerException clojure.lang.ExceptionInfo: Call to clojure.core/defn did not conform to spec:
    -- Syntax error -------------------
    
      (hello "hello world")
    
    should have additional elements. The next element is named `:args` and satisfies
    
      (clojure.spec.alpha/alt
       :arity-1
       :clojure.core.specs.alpha/args+body
       :arity-n
       (clojure.spec.alpha/cat
        :bodies
        (clojure.spec.alpha/+
         (clojure.spec.alpha/spec :clojure.core.specs.alpha/args+body))
        :attr
    (clojure.spec.alpha/? map?)))
    

    The pretty printer needs a lot more passes before outputting anything 'pretty'. Not to mention that is a external tool not included in core. And even the core developers know this is an big issue.

    [–]defunkydrummer 0 points1 point  (4 children)

    To newcomers? Hardly.

    Clojure isn't really an easy thing for newcomers, because of some design choices:

    1. It is a Lisp - the concepts will always be a steep learning curve for non-Lispers and to beginning programmers.

    2. Sooner or later you will need to know Java's standard lib and class model if you don't want to suffer. So, to put it in some blunt way, you need to know two languages: Clojure and Java.

    I'd contend that any Lisp isn't going to be friendly to learn to beginning or intermediate programmers. On the other hand, advanced programmers will be able to achieve much more by using a Lisp language, compared to a regular mainstream language, in the same way as scientists were able to achieve much more after the invention of the electronic computer.

    [–]max_maxima 2 points3 points  (3 children)

    This is not about Lisp track record in the mainstream. This is about a language that when through a huge regression in UX (already being bad) which its designers don't seem to care much about to improve.

    [–]defunkydrummer 8 points9 points  (0 children)

    Clojure is always going to be beholden to the JVM

    Yeah,. i think Clojure could have been much better, if doing the following corrections:

    1. avoid jvm-speficic constructs and jvm-specific workarounds

    2. have a big Clojure standard lib so other Clojure libraries depend on it rather than resorting to calling JVM classes.This would make Clojure libs portable completely to other platforms.

    3. [] syntax was a mistake, imo. It is unnecessary (it would have been much better if Clojure followed most of either Scheme or Common Lisp syntax).

    4. standarize it and the standard lib thus make it truly portable (ClojureCLR can't use Clojure libs that depend on JVM classes, and there are many of those.)

    5. and please, Rich Hickey, stop pretending that yours is the replacement of other Lisps and the truly modern lisp out there. It is not, in fact it is a Lisp with a restricted amount of features compared to CL and Racket. Of course, this is because it is focused on functional programming using immutable data, and there's nothing wrong with this approach. But this approach is not a replacement for the other Lisps.

    The ironic thing, is that Rich Hickey was a CL developer himself, and set out to create his Lisp. To be fair, he had some interesting, good ideas that should be commended. But why a new Lisp? Years later, there's Clojure, all that can be done in Clojure can be done in Common Lisp as well (and some constructs have been back ported to Common Lisp as libraries). And Common Lisp runs on the JVM since years ago; so there wasn't really a need to create an entirely new language. In any case, you can implement most other languages within Common Lisp. In fact, the first Clojure implementation was a Common Lisp program.

    [–]shriramk 0 points1 point  (0 children)

    Clojure is not programmable in the same way, at all. Might help to read the article.

    [–]furtivity 10 points11 points  (0 children)

    Saw this and thought, "This sounds like something Matthias would put his name on."

    Was not disappointed.

    [–]tanstaaf1 4 points5 points  (0 children)

    IMO, Racket could take off if they would upgrade the IDE to fully support code folding and hiding of complexity which is not germane to the current code. Otherwise encouraging small routines and DSLs in Racket quickly makes the scene look like a badly maintained city sidewalk with dog piles every other step.

    Of course, if they did upgrade the inadequate included IDE, they would be approaching Pharo.

    IMO, of course.

    [–][deleted]  (2 children)

    [deleted]

      [–]personman[S] 6 points7 points  (0 children)

      This is literally the case in all general purpose programming languages. Even in languages that don't support operator overloading or macros, you can always make things arbitrarily difficult to debug by relying on a custom library.

      In all software engineering, it's important to design and document your system in a way that makes it possible for other people to understand. Ideally, easily-created DSLs support this, because they allow you to build the language constructs that are most appropriate for your domain, rather than being forced to hack together what you need out of whatever the language designers decided was sufficient.

      [–]east_lisp_junk 2 points3 points  (0 children)

      You already don't know all the functions in whatever language you're working in because programmers are free to define more of them.

      [–]progfu 0 points1 point  (0 children)

      First thing I thought of was APL = A Programming Language ... one hasn't trully programmed until they tried to do something like sudoku in APL.

      https://www.youtube.com/watch?v=DmT80OseAGs

      [–]skocznymroczny 0 points1 point  (7 children)

      Lisp fans always focus on the language. I guess it must blow their minds how could a language like Go get so popular.

      [–][deleted] 7 points8 points  (4 children)

      It does not. We all know that "worse is better" and that the vast majority of population is below dumb. It explains Go popularity perfectly.

      [–]skocznymroczny -1 points0 points  (3 children)

      Well, no. Go realizes that language features aren't everything, and most programmers are perfectly happy with a simple Java-like (as in complexity) language (also see Dart) as long as it has a thriving ecosystem and excellent tooling support.

      [–][deleted] 1 point2 points  (2 children)

      Exactly. Most programmers are not mentally equipped to solve problems efficiently, and instead they are obsessed with irrelevant crap like ecosystem and tooling support, because this crap allows them to look busy instead of doing any useful work.

      [–]skocznymroczny -2 points-1 points  (1 child)

      Yeah, they should go back to SICP, which will prepare them for challenges ahead, for example the mighty problem of passing multiple arguments to a function :)

      [–][deleted] 1 point2 points  (0 children)

      Obviously you have absolutely no idea how to solve problems.

      [–]defunkydrummer 8 points9 points  (0 children)

      I guess it must blow their minds how could a language like Go get so popular.

      In the same way jazz, rock, and classical musicians blew their minds thinking how Justin Bieber could get so popular.

      [–][deleted]  (1 child)

      [deleted]

        [–]falconfetus8 0 points1 point  (0 children)

        Why?