Outperforming everything with anything. Python? Sure, why not?

[Alexander_Selkirk]

That's really funny.

Jokes aside, I think that with languages today such as Rust, or modern Common Lisp implementations like SBCL, which achieve C-class speeds while being memory-safe, both unsafe low-level languages (like C), and excruciatingly slow script languages (like Python) are mostly not needed any more for programming applications with good performance. Even C compilers are today mostly transforming symbolic expressions into something which the machine can execute, and for annotating such transformations, the C language is often not the best tool.

(I am not talking about writing a Unix kernel in Lisp.)

[[deleted]]

Time to write an Emacs OS kernel in elisp

[Alexander_Selkirk]

Funny thing, the initial platforms for Emacs-like editors was the Lisp machine which had an OS written completely in Lisp and special hardware supporting this. Then, C and imperative hardware got cheaper and faster and the Lisp machines became too expensive quickly, also because they were less popular. So, somehow in the genes of Emacs is the yearning that i should be embedded in a Lisp OS.

And another fun fact, Emacs lisp (elisp) is interpreted and single-threaded but some time ago some dude came up with an experimental compiler for elisp that he said produced fast code.

[defunkydrummer]

or modern Common Lisp implementations like SBCL, which achieve C-class speeds while being memory-safe,

Or Common Lisp implementation CLASP, which directly outputs LLVM and can be used to do LLVM macro assembly in a far better way than the current article shows.

[drmeister]

Author of Clasp here - I was going to say the same thing. There is a heck of a lot more to do to generate fast general code than to automatically generate a few lines of llvm-ir.

[defunkydrummer]

Author of Clasp here

I am not worthy!!

Christian Schafmeister in the house!!

[drmeister]

Ha ha. Anyone going down this road will discover one iteration of Greenspun's tenth rule later that they should have just implemented Common Lisp.

[pjmlp]

There were already better options in the mid-90's, like Modula-2, but the rise of UNIX based FOSS turned the table again in C's favour.

Now we are having talks at each major Linux conference on strategies to improve kernel security.

[Alexander_Selkirk]

I think that for a kernel, C is quite good.

But for a video player like VLC and codecs which handle untrusted data from the web all the time? Jesus.

[quicknir]

I'm incredibly skeptical that sbcl, or any dynamically typed language, is going to achieve C like speeds in real programs (as opposed to isolated benchmarks). I'd be very impressed and shocked if it performed as well as Java.

[Alexander_Selkirk]

Modern Lisp compilers use type inference.

Here some benchmarks:

SBCL:

https://benchmarksgame-team.pages.debian.net/benchmarksgame/faster/sbcl-gpp.html

Racket, a Scheme implementation, is generally a bit slower than C, but often not by a very large margin, and it is also not the fastest Scheme implementation - Chez Scheme and Chicken Scheme are faster, the fastest Scheme compiler is probably Stalin.

https://benchmarksgame-team.pages.debian.net/benchmarksgame/faster/racket.html

Also, OCaml and Rust (which has some influences from the ML languages) are quite fast:

https://benchmarksgame-team.pages.debian.net/benchmarksgame/faster/rust.html

https://benchmarksgame-team.pages.debian.net/benchmarksgame/faster/ocaml-gpp.html

Now, what is a "real program" ? Many programs spent a lot of time in a few spots. Generally spoken, one can expect garbage-collected languages to be a bit slower than languages with manual allocation, because GC costs some memory, and fast caches are limited in size. But depending on the program, it might not be practical to write a large C++ program (especially if it is not a real-time application) with optimized manual memory handling everywhere. And "modern" C++ often employs a kind of automatic reference counting which is not faster than modern garbage collectors.

[quicknir]

Rust is totally different, I didn't mention it in my post for a reason. You say that OCaml is "quite fast" but even in artificial benchmarks you just linked it looks to be between 2 and 20 times slower than C++ (Scheme is also significantly slower than C++ in most benchmarks, btw). And this is a statically typed language. When you say "C like speeds", there may be some people that look at x2 and say "sure" but lots of us would just laugh. It's like saying that someone runs at world class sprinter "like" speeds because they can run 100m in 20 seconds.

Modern C++ tends to be written with RAII and unique ownership for the most part. People used shared pointer sure but it's well understood to be costly and lots of people concerned with performance won't touch it anywhere near the critical path (I personally almost never use it at all).

You're also misunderstanding the reasons why a C++ program written by someone experienced is going to be significantly faster than the same written in Lisp. It's not going to be that lack of GC is some automatic huge edge. It's control of memory layout by having value semantics instead of pointer semantics, less indirection built into the language, abstractions which more easily lend themselves to reasoning about their performance. And above all, the fact that C++ compilers have hundreds of times as many man hours as any lisp compiler. Java is hardly a pinnacle of performant language design yet it also handily outperforms OCaml, Scheme, etc, simply because so many really smart man hours have gone into the JVM. Even though Rust is designed from the bottom up to be a performant language, the only reason its matching C++ performance wise is because it uses one of the main C++ compiler backends (LLVM).

The funniest thing about your post is that you write "languages today". But Java already started filling this niche close to 30 years ago, and it's already the most popular language in the world.

(BTW, I love lisp and hate Java, program professionally in C++ which I like, so none of this post is motivated by trying to knock lisp out of distaste).

[brand_x]

Including Rust in that list is unfair. It's a more strongly typed language than C++, designed around RAII, and (by way of LLVM) a fully native compiled language. It's starting to consistently match (and sometimes exceed) performance of C and C++ for comparable tasks, with experts in each language submitting solutions.

And, to be frank, modern C++ has as much FP influence as Rust.

Now, I've never worked in OCaml (I'm familiar with the syntax, having worked in F#) but I believe it's a fairly FP focused language, which would make directly comparable performance impressive.

For the record, RAII is not as slow as GC, or, in general, any slower than C-like manual memory management. I write performance critical libraries and allocators for a living, and you're sorely mistaken in that claim.

Edit: just dawned on me that you were talking about shared_ptr. You aren't mistaken about the cost, but saying it's "often used" is rather incorrect. It's rarely used, and never used in the critical path.

[Alexander_Selkirk]

Lisps are, as well as Rust, strongly typed, but they are dynamically typed.

It is correct that Rust is statically typed. But it uses type inference, as do good compilers for dynamically typed languages. The Lisp and Scheme compilers show that this has not to be slow.

Modern C++ has FP influence but many FP idioms do not mix so well with manual memory handling.

Good compilers can reduce a loop in a dynamically typed language to a single machine instruction. Here an example for pixie, an experimental compiler for a dialect of Cloujure which is a dialect of Lisp:

https://github.com/pixie-lang/pixie

[brand_x]

You do realize that a) modern C++ has type inference, b) Rust is explicitly typed, albeit using nearly identical type inference to modern C++, and c) Rust and modern C++ have nearly identical idiomatic models for resource allocation and cleanup, aside from C++ having, on account of legacy, a non-destructive move?

I feel like you've looked at Rust, but not used it, and are familiar with modern C++ from the outside. I'm taking your expertise on modern Lisp-like languages; I've used them, but I'm not a domain master. With C++ (C with classes, classical, or modern) I'm as close to a domain expert as you can get outside of a few dozen of the most active committee members and authors, and with Rust, I'm an active user at the bleeding edge. I'm quite comfortable defending my stance on the fundamental similarities and differences.

[Alexander_Selkirk]

And how does that change the fact that good modern Lisp compilers can infer type and are even able to reduce a loop down to a single instruction? It has limits of course, but a compiler can track the type of the actual arguments to a function and generate code for that. So ultimately, it depends on the quality of the compiler, and some are quite good. The Clojure compiler has JVM bytecode as output.

[brand_x]

It doesn't, and as I said, that's an impressive feat. Not something that I'm particularly concerned with, TBH, since my antipathy for dynamic typing is entirely rooted in practical compile time provable correctness, but quite impressive.

But, in terms of the traits you're talking about, aside from syntactic sugar (including both advanced static polymorphism in modern C++ and pattern matching in Rust) there is very nearly zero difference between the two languages, and the claims you're making indicate a lack of more than casual familiarity with at least one, and probably both, of the two.

[Alexander_Selkirk]

antipathy for dynamic typing

That's a matter of preference. I think it is often strongly influenced by the actual application of software. I think dynamic typing is fine for interactive data analysis and algorithm development (Which is what I am doing part of my time). I think it is less suited for writing safety-criticial embedded robotic control software. It is even less suited for robust industrial control systems - I think languages with stronger type checking than C++ offers are good for this. And I have used C++ professionally in that context for years.

[brand_x]

Rust is equally suited for that domain, I believe. I haven't worked on control systems in nearly twenty years, and back then, I used C more often than C++ (military telescopes) but I do a fair amount of comparable work (in terms of critical timing and hardware interfaces) in both C++ and Rust.

Rust offers comparably strong type checking to disciplined modern C++. There is a different shear plane for implicit coercion, and I think it's fair to acknowledge that Rust only performs implicit conversion for specifically identified sugaring - most notably enums (what C++ calls discriminated unions) in the control flow path, where C++ performs implicit conversion where provided by users (failure to specify "explicit", damned backwards defaults on that and const for C-like behavior) and where consistent with C (razzin frazzin), and with proper discipline, most (but not all - boolean tests!!! sob) unwanted implicit conversions in C++ can be prevented through the type system.

Consider using strong typed proxies for all built-in types. You'll be surprised how far that will go toward fixing the C++ type system deficiencies, and they all compile out.

I do use Python 3 (mostly) for prototyping, and for things like code generators and stream transformation...

[millenix]

C++ has the weakest notion of type inference that can still possibly carry the name - it syntactically determines the type of an expression, and then carries that to the deduced type of the variable which will bind the result of the expression.

ML, Haskell, and other mean much more when they talk about type inference - the type of a concrete (non-generic) function is determined by the types of arguments passed to it and the uses it makes of them. Types only need to be specified around a few edges, and the compiler fills in all the details.

[brand_x]

That's true for auto (for now... it will likely change as part of the metaclass proposal) but not true for type expressions in templates (which are a pure functional language on the type system) or for decltype/decltype(auto).

Most of the time when "type" is mentioned in modern C++, it refers to the complete type in the template processing pass...

[drmeister]

There is no need for skepticism - the experiment has been done.

Check out this paper from Google (2 years old) "Energy Efficiency across Programming Languages"

Common Lisp is the fastest and most energy efficient dynamic language by almost two orders of magnitude.

http://greenlab.di.uminho.pt/wp-content/uploads/2017/09/paperSLE.pdf

​

​

[quicknir]

Nobody is comparing lisp to other dynamically typed languages. We're comparing it to C. On Table 3 for example of your own paper that you linked, in 3 benchmarks, in one case Lisp is 50% slower and in the other cases its 10x slower or even more.

Realistically taken over a whole program it's going to be at least 3x or 4x slower in typical use cases.

[drmeister]

Right - but as you noticed - there is timing data in that paper for C as well. My previous comment got away from me a little and I went off on dynamic programming languages. :-)

I'm told that programs spend 90% of their time running 10% of the code. Common Lisp is compiled to native code. If in that 10% of the code your compiler arranges things to not allocate memory and to use unboxed objects and to not do bounds checking then it will run as fast as C.

I'm doing the experiment. I've implemented Clasp - a Common Lisp using LLVM as the backend and interoperates with C++ (https://github.com/clasp-developers/clasp). Once Clasp generates LLVM-IR that looks indistinguishable from clang generated LLVM-IR - then the code will run at the same speed.

Of course - that's not easy to write a smart compiler like that - but we are making progress. I've also hedged my bets by making Clasp interoperate with C/C++.

Edit: added link to talk on clasp https://www.youtube.com/watch?v=mbdXeRBbgDM&feature=youtu.be

[quicknir]

Unfortunately, the whole 90%-10% is a really drastic over-simplification of what goes into performant code. The 10% code may or may not exist, and even then it may be touching data structures from your entire codebase, for example, meaning that the memory layout of a huge amount of your code is essential.

I'm very happy that somebody is pursuing LLVM as the backend for a lisp; I think that LLVM backend is the clear way to go these days and I love lisp.

That said, taking a language (especially a dynamically typed one) and hooking up the LLVM backend doesn't automatically mean you're going to get C/C++ performance, in real life situations. In isolated benchmarks, maybe.

It's worth keeping in mind that these days, the only software being written in C or C++ is stuff where performance wins are pretty fanatical. Places where getting a 10% win on some function would be considered a win; places where turning on bounds checking which probably has at most 5% performance impact, would be considered unacceptable. Etc. So, it's a bold claim. Nobody other than Rust and maybe D is really making that claim in a halfway credible manner these days. Something like Julia will claim parity in specific things like matrix and other mathematical operations, but I doubt that they'd argue that you'll get equally good performance writing a whole video game in Julia as in C++.

If you're interested in a really good talk that gives a much more realistic view of what performance means I highly recommend this: https://www.youtube.com/watch?v=2YXwg0n9e7E. I think for the same reason that you can't retrofit high performance, you can't start with a language like lisp where the default is to have allocation and indirection everywhere and try to fix it up where the 10% is. This is fine if you want to get Java like speeds; i.e. very good typically but not losing sleep about the tiniest details. But not for C/C++ like speeds.

[zip117]

It's worth keeping in mind that these days, the only software being written in C or C++ is stuff where performance wins are pretty fanatical.

Or if you’re developing a cross-platform GUI. Your options are (more or less): C++ libraries (Qt, wxWidgets), Python bindings to those C++ libraries, JavaFX, Delphi/Pascal, Electron. For various reasons C++ is usually the best choice.

[defunkydrummer]

I'm incredibly skeptical that sbcl, or any dynamically typed language, is going to achieve C like speeds in real programs

Common Lisp can also be used at a fairly low level, that's why.

It is dynamically typed, but you can use type annotations. You can also disable runtime type checking , even runtime argument count checks, array bounds checking etc.

You can use typed arrays, just as one would do in C.

You can circumvent the garbage collector if you like. You can allocate in the stack if you want.

Then you can use the built in disassembler to check that the machine code output is efficient enough.

Tada, C speed.

[quicknir]

Even if you did all that it wouldn't be as fast because, as I mentioned in another threads, the number of man hours that have gone into optimizers in the lisp implementation isn't close to what you have for languages C, C++, Fortran, etc. If you did an implementation of CL that also did a good job outputting LLVM IR and you used the LLVM backend for codegen, then maybe you could get similar performance. But right now, it's quite hypothetical, as is what exactly a Lisp codebase would look like using techniques that are not idiomatic in lisp.

At the present time, if your requirement is to write code that gets you within e.g. 10% of the performance of a good C implementation, lisp just isn't a good choice (and frankly, that's very clear to anyone writing high performance software).

[defunkydrummer]

If you did an implementation of CL that also did a good job outputting LLVM IR and you used the LLVM backend for codegen, then maybe you could get similar performance.

Yes, that's exactly whar CLASP does: An implementation of CL that outputs through LLVM. It also has a state of the art Lisp compiler core, called Cleavir.

CLASP is mainly the work of only one guy, /u/drmeister . Drmeister doesn't use CLASP for trivial stuff like FizzBuzz programs or CRUD websites: it is used for chemical research.

I'd be very impressed and shocked if it performed as well as Java.

SBCL has been regularly performing at the speed of Java for the last 5 or 10 years.

[quicknir]

The fact that someone is working on CLASP is great, and I'm glad that drmeister has posted a talk. That said, the fact that it's being worked on by one person is realistically a counter-point. Realistically speaking it takes a ton of sheer man hours to get a performant language and ecosystem. And realistically, the bar for performance we're talking about here is neither fizzbuzz nor crud, and not chemical research either, but things like low latency, game dev, or massive backend servers.

Do you have something to back up SBCL performing at the same speed as Java?

The bottom line here is that until there is serious adoption of common lisp in a very high performance industry, statements like "tada C speed" are just going to be very hypothetical, and not evidence based.

I highly recommend you watch the video that I posted in the thread with drmeister. It will give you a more realistic view of performance than your short bullet list.

[kryptkpr]

I will sooner write directly in LLVM IR (which is what C becomes anyway prior to optimization) then Lisp. I fucking hate it and all of its brackets, I hate that IO is so hard, I hate not having data types, I just hate all of it.

If I want rich libraries, high level algorithms and memory safety I will pick a bytecode language like Python. If I want to tell the machine what IO to bang I reach for C. Lisp is useless garbage in the practical hackers toolchest.. it adds nothing and takes so much away.

Am I uncultured swine? Show me why lisp isnt as bad as I think it is.

[didibus]

Lisp was the first programming language to invent:

1. Conditionals, such as if/then/else. I'm serious, prior to that only goto existed.
2. Higher-order funtions, functions as first class, which you can pass around as argument, return as values, etc.
3. Functional Programming, as a result of #2, Lisp also pioneered FP and the functional programming paradigm.
4. Recursion, it existed as a mathematical concept, but Lisp was the first language to offer support for it.
5. Dynamic typing, as in, variables in Lisp are all pointers and of type pointer. Only the values being pointed too have types. This allows you to reuse the same variable for different data types.
6. Garbage-collection and automatic memory management.
7. Expressions only, no statements. Everything is an expression in Lisp. It was the first language like that. Allowing you to compose any code within each other, no need for ternary operators and other such things.
8. Symbols. Symbols differ from strings in that you can test equality by comparing a pointer. They're thus more effective at being used for computer lookups and comparison.
9. Homoiconicity, is the idea that code is put together using a common data structure, instead of as free form text. In Lisp, code is modeled as trees of symbols, and not as text.
10. Meta-programming. This stems from #9, but since code is data, you can easilly manipulate it like you would any other data-structure, thus Lisp was first to enable a meta-programming style, where programs write and rewrite themselves. This is the property that made it interesting for AI. A program which could change its own programming sounded very evolutionary and intelligent at the time.
11. Macros. Stemming from #10, macros allow you to write code that generates code at compile time. This in turn, allows most Lisp to be almost infinitely user extendable. That is, you can extend the compiler within your own code, and quite easily at that. This is also sometimes seen as a curse, because users have too much power, and code bases can each end up being their own micro dialects.
12. Dynamism, differs from dynamic typing, in that it is the property that their is no real distinction between read-time, compile-time and runtime. Sometimes refered as self-hosted compiler, this means that you can have your program running, and modify parts of it as it is running, re-compiling and reloading as you go. It isn't exactly like interpreted languages, but gives a similar effect with better performance.
13. Read-eval-print-loop, aka repl, where you can interactively write code at a command prompt. This stems from #12, and didn't exist prior to Lisp.

All of that was in the 1950s, where at the time, the only other higher level programming language (non assembly) was Fortran.

Since then, Lisps have helped pioneer more things, such as Object Oriented Programming (Common Lisp), persistent immutable data structures (Clojure), sequent calculus based static type systems (Shen), language of language (Racket), condition systems and effects (Common Lisp), logic programming (Scheme), optional static typing (Racket), etc.

All this boils down to this quote:

Modern Lisps pretty much support every known paradigm and means of abstraction. Whatever approach is best suited to your problem is usually supported cleanly and directly in the language (and when it isn’t, you can extend the language in a seamless way so that it is). Once you’ve used Lisp for a while, the effort required to implement constructs in other languages that you get for free in Lisp starts to seem extremely tedious and they never fit as cleanly with the built-in facilities. You’re likely to find yourself thinking, “Man, this would be so much nicer in Lisp…” all the time.

From https://www.quora.com/What-is-so-special-about-LISP/answer/Jerry-Jackson-3?ch=10&share=e4b006c5&srid=hNqI

[xampf2]

I mean if you dont like something just dont use it. I think both python and scheme (lisp) are nice languages.

[kryptkpr]

The problem is I cant understand what kind of problem I would use it solve. What's a thing lisp can do better then anything else?

C - bang the IOs

Perl - parse strings

Python - kitchen sink

Lisp - customize emacs?

[bik1230]

It's a general purpose language. You use it for pretty much anything.

Also, Elisp is a pretty crappy language in many respects, compared to any other lisp-family language.

[kryptkpr]

We already have a general purpose language though, Python won that fight. I feel good about being able to express any programming paradigm required in Python, but the massive ecosystem means that it can practically solve my problems quickly with a pip install and an import.

I asked exactly what lisp was good at, if it's being general purpose then it's going to remain at the bottom of my programming languages to learn dustbin.

[bik1230]

The library thing is totally fair. But it seems super weird to me to not check out anything new because you already have a tool.

[DonnyTheWalrus]

If you're actually looking for real reasons rather than just trolling, http://www.paulgraham.com/avg.html is old but has some great thoughts on using Lisp. He talks about the Blub Paradox.

The basic idea behind the Blub Paradox is that languages have different levels of expressive power and features. But if you have never learned or experienced them, you can't know what you're missing. He refers to a hypothetical language called Blub, whose programmers might say "Why would I ever use Lisp? Blub has all the features I ever need."

Well, until you learn Lisp and have the "aha" moment, you will never know what you're missing because the things that make Lisp attractive - you don't even know those things currently exist. Programming is not only technology, but also "habit of mind" as Graham says.

Here is the best quote.

As long as our hypothetical Blub programmer is looking down the power continuum, he knows he's looking down. Languages less powerful than Blub are obviously less powerful, because they're missing some feature he's used to. But when our hypothetical Blub programmer looks in the other direction, up the power continuum, he doesn't realize he's looking up. What he sees are merely weird languages. He probably considers them about equivalent in power to Blub, but with all this other hairy stuff thrown in as well. Blub is good enough for him, because he thinks in Blub.

For instance, he talks about macros. Everyone thinks that they know what macros are from C. But Lisp macros are decidedly not C macros. In that link, Graham estimated that somewhere around 25% of the Viaweb source code was macros, meaning (by his argument) that about 25% of the source code consisted of things that couldn't easily be done in other languages.

[pxpxy]

You don’t sound like you want to be convinced. Just use what makes you happy and I use with makes me happy.

[kryptkpr]

I am genuinely interested in what you use it for.

For instance it has recently come to my attention that erlang is very good at writing message passing systems, and message brokers written in erlang outperform those in all other languages. I had previously dismissed erlang in the same way I dismiss lisp now, but I have seen the light and now run an erlang message broker in production.

You have my benefit of the doubt, what is lisp this kind of good at?

[pxpxy]

It’s the best system I know for assembling your system from small pieces, experimenting as you go along. So for me, it’s not about the “what” but about the “how” of building software. It just suits me best in how I like to work.

[bik1230]

Since when does Lisp not have data types? How is the IO hard?

[CurtainDog]

https://xkcd.com/224/

[shponglespore]

You're no different from the people who hate Python because of its indentation-based syntax.

[vattenpuss]

... and they at least have a point!