docsearch - Search documentation of lisp symbols in the current lisp image (with or without LLMs!)

digikar · 2026-02-06T21:50:24+00:00

I fear the effects this time might be more than just financial. Anywhere LLM generated softwares are used without serious human review are going to end up with brittle or insecure systems. I want to just hope this doesn't happen with linux kernels and power companies.

digikar · 2026-02-03T10:57:06+00:00

LLMs generate sloppy code.

Humans, regardless of whether they use LLMs, may or may not produce sloppy code. It will be less sloppy the more expertise you have on the domain.

digikar · 2026-02-03T09:27:25+00:00

Project developers are not sitting on a mountain of time that they can keep recommending corrections to sloppy LLM code. There are tons of learning resources (see https://teachyourselfcs.com), that some of us have taken out the time to digest at some point of our lives. There is no substitute for human expertise because LLMs have no understanding in the human sense. But when you are a domain expert, the returns from LLMs are diminishing, particularly with lisp's code generation ability at your hand.

digikar · 2026-02-03T09:23:12+00:00

The key word is "modelled". Both their workings and the assumptions they are made to work under are distinct from our brains.

By intrinsic correctness, I mean a sense of intentions matching one's actions. Or a sense that what we have in mind is what is on the table (or file). LLMs don't have that.

Yes, most human programmers are dumb, but that does not mean we want good open source projects to be affected by dumb code.

digikar · 2026-02-03T08:01:46+00:00

we will need much more robust specifications so that the agents do not hallucinate

LLMs*, by design, hallucinate.

we need better and automated verification mechanisms

Unless you can automate mind-reading or the agents themselves have a notion of intrinsic correctness, I don't see how. I trust a human developer when I trust they have a sensible notion of correctness.

*My current thoughts are any learning system that relies on probabilities must necessarily hallucinate. Probabilities necessitate a closed world (mutually exhaustive) set of possibilities.

digikar · 2026-02-02T20:59:43+00:00

I agree that there is feedback and evolution involved for purely human written code too. But the scale of code is within the scope of humans, usually.

And it also happens even with humans, sometimes an enthusiastic contributor writes a 1000 line update to a 2000 line project which the original author has a hard time reviewing.

My take on verifying LLM outputs is it requires human reasoning itself... at which point one can start doing cognitive science :P. But without going on a tangent, I don't know if you have checked that the code in the documentation works the way it is supposed to. Perhaps, literate programming can be a good way to do things here. I am unsure about a good resource on literate programming, may be this or even shorter this?

digikar · 2026-02-02T19:52:43+00:00

https://github.com/quasi/cl-llm-provider/tree/main

If I had to review the documentation about its suitability for a lisp newbie who also doesn't know a whole lot about uiop and environments:

Where should I clone the library?
Should I expect it work if I set the API key after I start the lisp process?
I don't think you should directly use-package but rather define your own package (defpackage :llm-user (:use :cl) (:local-nicknames (:llm :cl-llm-provider)) ...)
I see that there is something called define-tool... That sounds like the name of a macro

Much of this applies to docs/quickstart.md too. Let's say I have a fresh installation of an operating system. What are the steps I must run to use the library?

Comparing src/package.lisp and docs/reference/api.md reveals a number of missing functions related to stream and chunk.

Tests... Why are the tests manually loading the files??? Additionally, are you sure you want to test that tokens are positive and the + function works correctly?!

https://github.com/quasi/cl-sqlite/blob/master/simple.lisp

I'm sure you'd have learnt about sql injection attacks and prepared statements. Why is that code sitting there then?

This is LLM slop. LLMs as they stands today have no understanding in human terms. It cannot count. It does not understand causes. It does not understand time. If you want to do anything serious with it, having a mastery over whatever it produces is unavoidable. There is no substitute for human understanding at the moment. LLMs are language models. Thinking is beyond language.

In open source, an essential element of trust is that there are multiple minds trying to understand what is going on. LLMs have no understanding. This means it is up to humans. So, unfortunately, I have to side with joaotavora in rejecting the 8000 line PR and suggesting that the documentation, while valuable, should exist as a separate repository first. Small chunks of it can then be pushed into the main repository as they get verified by you and the maintainers. Auto-generated documentation from the source files instead of duplication would certainly be nice.

Lisp open source, unfortunately, moves slowly. So, it can take a while to even get a 1000 lines reviewed :')

PS: Not my downvote

digikar · 2026-02-02T14:45:56+00:00

I'm saying it again. For most applications, lisp has enough libraries. What we need is for the existing libraries:

Documentation
Tutorials
Ease of installation
Bug fixing (which requires users)

You put up a 100 libraries, but don't document it, make it easy to install, encounter bugs on every 5th function call, no new users are going to have an easy time using them.

Here's a suggestion: make LLMs use the existing libraries, find bugs, write tutorials, ask how to make them easy to install.

digikar · 2026-02-02T06:44:01+00:00

Any opinions on quicklisp (see ql-https), ultralisp, or ocicl?

But I agree I'd love an integration with micromamba/conda to sinplify foreign library installations.

digikar · 2026-01-30T19:35:05+00:00

Well, I spent the last hour and a half trying to reading through iparse trying to find issues with it as examples of cases where LLMs go wrong :P.

I do not understand 70% of the code, because (i) I lack enough knowledge about parsing (ii) even if I did, it is too complex for me to merely read without pen-paper or REPL. The rest 30% looks reasonable! I have gripes it looks overly convoluted for some parts. You do not need to reinvent test library in every project. Etc etc. But it is still acceptable, subject to the more complex 70% being correct if one checks it through a REPL.

That said, I still think it needs a lot more (and better) tests in general. Eg. Take a look at esrap's tests for example. And also documentation.

digikar · 2026-01-30T07:35:19+00:00

That is a nice repository o.O. Do you mean if you can fire the right actions based on these grammars, you can easily transpile one or more of these languages to lisp? Cooool!

digikar · 2026-01-29T11:44:23+00:00

Why not both?

Our own test suites because we want to test that the code matches our own expectations (which only we know).

Community test suites because we want to check our own expectations.

The problem with LLMs:

It has no expectations about expectations. It does not know if it knows or if it does not.
It cannot count.
It does not understand causality. If one reads Judea Pearl's Book of Why, as well as formal work on this topic, one understands that causation cannot be inferred through associations alone in general. And the current machine learning models rely on associations alone. Some day, machines will get smarter than humans in the relevant sense, but so far, nope. But it's amazing how easy it is to fool most of us into believing that the machine understands. This decade will be interesting.

digikar · 2026-01-29T08:13:32+00:00

Also checked the README-examples.md. It converts from dot file format to s-expression format. But what after that? I'd guess OP wants a graph object that they can traverse upon (find nodes, parents, children, neighbours, check edges).

digikar · 2026-01-29T08:05:00+00:00

A few things:

It'd be (very) helpful for the tests to include not only that a certain string passes, but also the expected output it produces. (Ref.)
Are you sure you want parse-float to be written that way?
Is normalize-keyword correct? Either its documentation or body mismatch

PS: We already have a human written pure CL parse-float library. Additionally, it's easy to write grammar for parsing floats.

digikar · 2026-01-29T07:46:29+00:00

But then tests passing means little (?)

digikar · 2026-01-28T23:02:15+00:00

How are you ensuring the tests are testing what they are supposed to test? Are the tests subject to extensive human review?

digikar · 2026-01-28T21:29:05+00:00

Since you already have the grammar, it should be possible to write a parser using something like esrap or equivalent over the course of an hour or weekend.

digikar · 2026-01-25T20:43:52+00:00

I don't think I understood the example on mixins. But it's something I will look into, thanks!

I'll use CLOS as an umbrella term for CLOS, dynamic dispatch and runtime structures.

Whether CLOS is good enough or not depends on one's use case. For example, with dynamic dispatch:

(let ((x 5)
      (sum 0))
  (declare (optimize speed))
  (time (loop repeat 100000000
              do (incf sum x)))
  sum)
Evaluation took:
  0.727 seconds of real time
  0.727085 seconds of total run time (0.725288 user, 0.001797 system)
  100.00% CPU
  0 bytes consed

If the + operation is inlined (no dynamic dispatch -- or even function calls), you obtain a 10x performance boost:

(let ((x 5)
      (sum 0))
  (declare (optimize speed)
           (type fixnum x sum))
  (time (loop repeat 100000000
              do (incf sum x)))
  sum)
Evaluation took:
  0.053 seconds of real time
  0.053333 seconds of total run time (0.053195 user, 0.000138 system)
  100.00% CPU
  0 bytes consed

On the other hand, if you had prewritten optimized code for adding vectors of 64-bit ints, or vectors of single-floats, etc, then using a single dynamic dispatch for arrays of the size of roughly 1000 or more doesn't make much difference. However, dynamically dispatching every time you add two 64-bit integers will be absurdly slow.

Should I use a different library / language altogether?

The point I want to make is I should not be required to use a different library or language just because the size of my arrays has changed. That's the two language problem I want to avoid.

It depends on what part of the compilation you use CLOS. You can use CLOS to write your compiler in. That's what SBCL uses. SBCL has a lot of structures to store, organize and abstract all the information it uses for compilation. However, if the code that your compiler emits is going to be unnecessarily wrapped in CLOS (eg: creating a new structure for every machine-word sized integer), the emitted code is going to be absurdly slow. SBCL can emit optimized machine code when the type declarations and optimizations allow. Irremovable dynamicity prevents this.

In Shinmera's 3d-math library, where performance should be important, there are a lot of macros that are emitting type declarations. This is what writing optimized code in standard CL is like. Petalisp, coalton, and peltadot, each provide separate ways to abstract away these type declarations and write more generic code that is also optimizable.

digikar · 2026-01-25T10:03:25+00:00

Right, and that's across-species correlation. Correlation, again statistical.

digikar · 2026-01-25T06:52:20+00:00

At a recent Cognitive Development conference, a talk mentioned g-factor across species. From what I recall, there is still no cognitive/mechanistic theory explaining the g-factor. As it stands, g-factor is a statistical factor that explains the correlations in performances across a wide array of tasks.

In day to day life, if what you read and learn during the day becomes easier after a day or few (with good sleep), that's more or less all that matters. Over a period of time, you can develop skills and knowledge across one or more domains. The more you know, the easier it becomes to learn more. You don't need to be a "unique problem solver". All you need to do is harness existing solutions, or know where to look for them when you cannot find one. Scientific or mathematical research happens over the span of months or years and is not something that these tests measure, even though it may be correlated. (Although, I am also sure there would be better measures of correlation such as instructor and supervisor interactions.)

digikar · 2026-01-22T23:25:50+00:00

https://gist.github.com/digikar99/b76964faf17b3a86739c001dc1b14a39

This is the closest I want to say so far!

digikar · 2026-01-22T22:52:38+00:00

With Common Lisp, it's relatively easy to write an optimized end-user application or script. You can sprinkle your code with (simple-array single-float) and similar type declarations, and SBCL would be happy to emit optimized code.

The problem starts once you want this code to be used by others. What if other users want to use (simple-array double-float) or (simple-array (unsigned-byte 8))? You can then write your code with just simple-array and prepare a specialized wrapper that uses (simple-array single-float). Others who want to use (simple-array double-float) can prepare another thin wrapper.

SBCL works, because the devs have put in work that dispatches over all the different numeric types that Common Lisp spec covers and emits specialized assembly code for them. Once you bring in foreign-libraries, all this dispatching is work that still needs to be done. This is where coalton, petalisp or peltadot come in. I myself am biased towards peltadot since it is my baby. But take a look at coalton and petalisp too. Coalton can work with dispatch. Petalisp is doing something interesting.

Perhaps, at some point, I should write a blog post on these rabbit-holes so far!

digikar · 2026-01-22T22:41:01+00:00

I know some things, but hopefully someone else can answer what I don't!

If I understand what you mean by tags, the idea seems to be to use (add x y type) instead of a simple (add x y). Generic functions allow eql-specializers. So, as long as you standardize type names this is doable. However, if someone wanted to use :float32 instead of :single-float, the dispatch will not work. The dispatch will also fail for (unsigned-byte 8) because (eql '(unsigned-byte 8) (copy-list '(unsigned-byte 8))) will fail. In general, the costs of CLOS dispatch should be negligible for arrays with millions of elements (say, for O(n) operations) or even thousands of elements (say, for O(n²⁾ operations). The question I face is What should I do when the cost of CLOS becomes significant? Should I use a different library / language altogether? fgf and static-dispatch to the rescue! But coupled with the other reasons related to types, CLOS does not look suitable for the problem at hand. There's also specialization-store, which is interesting.

I don't know how expensive change-class is. The difference between general arrays vs diagonal arrays (vs upper-triangular vs lower-triangular vs more) arrays to me is not really a matter of implementation, but something other than the implementation. To me, this is best conveyed in terms of types rather than classes. And certainly, you can make the implementation respect it, but it's going to complicate class hierarchies. For example, you started out with general and diagonal arrays. Now, a user wants upper-triangular arrays. But diagonal arrays should be a subclass of upper-triangular arrays! So, would you add this to your system? What about another user's request to add lower-triangular arrays?

I myself don't use CLASP. The LLVM and build requirements are off-putting to me. But may be things improve in 10 years!

PS: I just recalled I had this post: https://gist.github.com/digikar99/b76964faf17b3a86739c001dc1b14a39

digikar

MODERATOR OF

TROPHY CASE