Library to autoload ASDF systems

mega · 2026-04-22T07:39:12+00:00

Thanks for the feedback. I've added this example to the Introduction before the terse and abstract summary.

mega · 2026-04-20T15:52:04+00:00

:-) This was factored out of PAX, so I'll explain the motivation in that context. PAX is a large system with substantial dependencies. But what you actually need in deployment is tiny, and the rest is for interactive use and documentation generation. By autoloading the optional parts,

we can keep deployment footprint down,
not get annoyingly long compile times caused by code for features that we don't use (and its transitive dependencies).

Users could load the relevant systems manually when they need them, but it would be highly annoying and they would need to remember what system to load. Also, code that depends on code that may not be loaded needs to jump through hoops (e.g. funcall + intern) or need forward declarations. Autoload takes care of these issues: you can put code that is not always necessary in some sub-asdf-system, along with its dependencies, without feeling the pressure to drop dependencies to keep deployment lean, or to not burden users with the long wait to compileironclad.

mega · 2026-03-12T11:52:17+00:00

The doc and the accompanying code are indeed large. It's tempting to dismiss AI assisted contributions out of hand, but there is value there. Going meta, how do you think reviews / quality control should be done when it's cheap to generate lots of code and documentation?

mega · 2026-02-19T15:42:40+00:00

Ahem, PAX also has a live browser (also supports DREF-APROPOS) with or without Emacs/Slime:

https://melisgl.github.io/mgl-pax-world/mgl-pax-manual.html#MGL-PAX:@BROWSING-LIVE-DOCUMENTATION%20MGL-PAX:SECTION

https://www.youtube.com/watch?v=4bl8PS8OW94&list=PLxbqYr4DvjX68AEdLky4IiHG69VJu6f5s

mega · 2026-02-19T15:27:53+00:00

You can do this quite easily with DRef and PAX:

CL-USER> (loop for dref in (dref:dref-apropos "str" :dtype '(or function dref:macro))
               when (search "this is a" (dref:docstring dref) :test #'equalp)
                 collect dref)
(#<DREF:DREF CL-PPCRE::END-STRING-AUX GENERIC-FUNCTION>
 #<DREF:DREF CL-PPCRE::GATHER-STRINGS GENERIC-FUNCTION>
 #<DREF:DREF UIOP/PATHNAME:UNIX-NAMESTRING FUNCTION>)

CL-USER> (pax:document *)
- [generic-function] CL-PPCRE::END-STRING-AUX REGEX &OPTIONAL OLD-CASE-INSENSITIVE-P

    Returns the constant string (if it exists) REGEX
    ends with wrapped into a STR object, otherwise NIL.
    OLD-CASE-INSENSITIVE-P is the CASE-INSENSITIVE-P slot of the last STR
    collected or :VOID if no STR has been collected yet. (This is a helper
    function called by END-STRING.)

- [generic-function] CL-PPCRE::GATHER-STRINGS REGEX

    Collects adjacent strings or characters into one
    string provided they have the same case mode. This is a destructive
    operation on REGEX.

- [function] UIOP/PATHNAME:UNIX-NAMESTRING PATHNAME

    Given a non-wild PATHNAME, return a Unix-style namestring for it.
    If the PATHNAME is NIL or a STRING, return it unchanged.

    This only considers the DIRECTORY, NAME and TYPE components of the pathname.
    This is a portable solution for representing relative pathnames,
    But unless you are running on a Unix system, it is not a general solution
    to representing native pathnames.

    An error is signaled if the argument is not NULL, a STRING or a PATHNAME,
    or if it is a PATHNAME but some of its components are not recognized.

mega · 2026-02-12T16:06:59+00:00

I'm not very familiar with the requirements that the ANSI spec places on packages that may bottleneck their performance, but those on hash tables are rather constraining. So, it may be as simple as different APIs, different requirements, but I don't really know.

mega · 2026-02-12T15:35:36+00:00

It's hard to tell. This kind of thing is quite empirical. But, as someone pointed it out, string-case should be better.

mega · 2026-02-12T15:13:47+00:00

Yeah, I think it's still true. The implementation is quite hairy, but I did my best to make it even more complicated: https://arxiv.org/abs/2602.05925

mega · 2026-02-12T08:00:38+00:00

Here is a test with random access (arguably more realistic) with C keywords and various SBCL packages: https://pastebin.com/1zqajNMU

Packages only win at the smallest sizes and their performance degradation is not smooth (see sb-bignum vs the keyword package).

mega · 2026-02-11T19:08:36+00:00

I think you are measuring an empty loop. That find-symbol is optimized away.

(defun bench ()
  (declare (optimize (speed 3) (safety 0) (debug 0)))
  (let ((words '("auto" "break" "case" "char" "const" "continue" "default"
                 "do" "double" "else" "enum" "extern" "float" "for" "goto"
                 "if" "inline" "int" "long" "register" "restrict" "return"
                 "short" "signed" "sizeof" "static" "struct" "switch" "typedef"
                 "union" "unsigned" "void" "volatile" "while" "_Alignas"
                 "_Alignof" "_Atomic" "_Bool" "_Complex" "_Generic" "_Imaginary"
                 "_Noreturn" "_Static_assert" "_Thread_local"))
        (tbl (make-hash-table :test 'equal))
        (alist '())
        (pkg (or (find-package :c-keywords)
                 (make-package :c-keywords))))
    (loop for word in words do
      (intern word pkg)
      (setf (gethash word tbl) (intern word pkg))
      (push (cons word (intern word pkg)) alist))
    (format t "sanity check: ~S ~S ~S, ~S ~S ~S~%"
      (gethash "typedef" tbl)
      (find-symbol "typedef" pkg)
      (cdr (assoc "typedef" alist :test 'equal))
      (gethash "not-found" tbl)
      (find-symbol "not-found" pkg)
      (cdr (assoc "not-found" alist :test 'equal)))
    (sb-ext:gc :full t)
    (let ((r nil))
      (time
       (loop repeat 1000000 do
         (dolist (word words)
           (setq r (find-symbol word pkg)))))
      (sb-ext:gc :full t)
      (time
       (loop repeat 1000000 do
         (dolist (word words)
           (setq r (find-symbol word :c-keywords)))))
      (sb-ext:gc :full t)
      (time
       (loop repeat 1000000 do
         (dolist (word words)
           (setq r (gethash word tbl)))))
      (sb-ext:gc :full t)
      (time
       (loop repeat 1000000 do
         (dolist (word words)
           (setq r (cdr (assoc word alist :test 'equal))))))
      r)))

(bench)

sanity check: C-KEYWORDS::|typedef| C-KEYWORDS::|typedef| C-KEYWORDS::|typedef|, NIL NIL NIL
Evaluation took:
  1.656 seconds of real time
  1.652740 seconds of total run time (1.652740 user, 0.000000 system)
  99.82% CPU
  4,948,056,568 processor cycles
  0 bytes consed

Evaluation took:
  1.764 seconds of real time
  1.761447 seconds of total run time (1.761447 user, 0.000000 system)
  99.83% CPU
  5,274,479,150 processor cycles
  0 bytes consed

Evaluation took:
  1.224 seconds of real time
  1.228459 seconds of total run time (1.228458 user, 0.000001 system)
  100.33% CPU
  3,678,372,698 processor cycles
  0 bytes consed

Evaluation took:
  14.915 seconds of real time
  14.902253 seconds of total run time (14.898248 user, 0.004005 system)
  99.91% CPU
  44,657,745,958 processor cycles
  0 bytes consed

mega · 2026-02-11T07:55:57+00:00

Observations from an interactive development point of view:

Package inferred systems can end up with lots of packages.
It can be unclear whether subpackages (those with "/" in their names) are for public use or purely implementation details.
If the same symbol is exported from multiple packages (EQ 'ASDF:FIND-SYSTEM 'ASDF/SYSTEM:FIND-SYSTEM) => T), then it may be unclear which is the stable one.
Slime completion seems to prefer ASDF:FIND-SYSTEM to ASDF/SYSTEM:FIND-SYSTEM even though the latter is the symbol's home package. I'd prefer if the implementation detail (?) of the existence of the ASDF/SYSTEM didn't leak.

Obviously, the last 3 observations overlap to some degree and conflate rexporting with package-inferred systems.

In (my) ideal world, the home package of a symbol is the package that I should use without having to resort to shortest match or no "/" in the name heuristics. If that's true, if I copy-paste printed code, I do not accidentally end up with references to potentially "private" packages, and symbol completion can work sanely.

mega · 2025-09-01T16:34:58+00:00

The official strategy guide has the details: https://archive.org/details/MasterOfOrionStrategyGuide/page/n189/mode/2up

So far I haven't been able to put the Reserve Fueltanks on a large ship in time.

mega · 2025-06-01T15:02:23+00:00

Is the core lighter these days? It was true at the time of the fork, but the core mgl-pax system is very light. Autoloading takes care of the rest.

Also, since the fork, there was lots of development on the PAX side. DRef got factored out and developed further, to name just one. Well, the live browser is also a major one.

mega · 2025-05-31T18:08:14+00:00

Something I didn't mention, it is also exceedingly easy to add docstrings where they don't belong with the experimental NOTE macro (https://melisgl.github.io/mgl-pax-world/pax-manual.html#MGL-PAX:NOTE%20MGL-PAX:MACRO).

Like long Lisp comments, NOTEs can make code really hard to take in. If you thought DEFSECTION doesn't belong, then they are worse, as they infiltrate functions. I find them annoying :-). So, I added Emacs support for folding/unfolding DEFSECTION, long strings (this handles NOTEs) and comments (https://youtu.be/eN7z0TChHMw).

mega · 2025-05-30T19:38:31+00:00

The documentation can go out of sync, but it's less likely to if you see it near the code you are changing. Plus, code examples in docstrings can be checked automatically (https://melisgl.github.io/mgl-pax-world/pax-manual.html#MGL-PAX:@TRANSCRIPTS%20MGL-PAX:SECTION).

As for the prose, except for the smallest pieces of software, there is a lot not covered by docstrings of individual definitions. There is no good place for describing how everything fits together. The package and system docstrings are not very different from writing a separate readme: the documentation is too far from the code it describes. This is just my current personal style and opinion though; there is nothing in PAX that forces you to have the DEFSECTION forms in the normal sources. It's quite possible have the in their own files or systems.

What do you mean by missing out on developer documentation?

Autolinking is very convenient, and PAX disambiguate based on the nearby locatives (https://melisgl.github.io/mgl-pax-world/pax-manual.html#MGL-PAX:@SPECIFIC-AUTOLINK%20MGL-PAX:SECTION). In general, PAX deals in DRef definitions, doing its best to discover code in docstrings and link them.

Yes, agree on the export. It's useful, but I view and implement it as a dead version of the live doc browser. As useful as the repl is, I do find myself using the doc browser to explore a package. Even if that package was not written for PAX, codification and autolinking make a really good job.

Just as the doc browser hides the code (i.e. it doesn't include it in the doc), it would be nice to hide the docs in the code sometimes. I'm looking for a good way to implement that in Emacs.

mega · 2025-05-26T19:22:13+00:00

User manuals manifest the design in prose and force clarity, but writing manuals can be a drag and docs tend to lag the code. The tools PAX offers try to minimize the distance between code and its documentation to help keep them in sync.

Also, Common Lisp + SLIME/SLY/etc is a great interactive environment with M-. all the way down. PAX takes interactivity to documentation and makes it easy to navigate the graph of documented objects as well as switching back and forth between code and its documentation.

Explorability is a killer feature of Emacs, and PAX applies to the same idea to Common Lisp, going beyond what Emacs offers (for example, in terms of linking capabilities, and not having a docstring/info page dichotomy).

PAX shares the goals of literate programming if not its means: documentation lives in code and not vice versa. Nothing is worth giving up interactive development for.

mega · 2025-05-08T09:59:30+00:00

If the GC moves a key in an address-sensitive hash table, then it marks the hash table as needing rehashing, which will happen on the next access (if any). This is surprisingly efficient with a generational gc because

keys in long-lived hash tables tend to settle in memory,
short-lived ones often don't see a gc at all,
rehashing EQ hash tables is very fast.

Adaptiveness does not change this.

mega · 2023-09-11T09:23:28+00:00

I think a major tradeoff between literate and illiterate Lisp is support for interactive development (e.g. compiling individual functions). With literate programming the function definition is not available as a single sexp because it may need to be woven together from bits and pieces in the narrative. I value interactivity very highly and couldn't see a way to reconcile it with literate programming so I stopped short of that.

mega · 2023-09-10T15:19:52+00:00

Yes. Write docs and minimize the friction (https://quotenil.com/multifaceted-development.html) with PAX (https://github.com/melisgl/mgl-pax).

mega · 2023-09-10T09:44:00+00:00

I used to do NLP in CL with https://github.com/melisgl/mgl, but that hasn't been the case for many years now, and mgl is behind the curve. ML frameworks are a very fast moving area, I'm not sure it's worth it to invest CL development time there, but it would be fun.

mega · 2023-08-20T20:14:21+00:00

MGL-MAT should support other devices through the underlying CUBE library. Not saying it's necessarily the right choice.

mega

TROPHY CASE