News | Didier Verna's Scientific Website

Clon 1.0b27 is out

Sun, 02 Jul 2023 00:00:00 +0000

A new version of , the Common Lisp / Command-Line Options Nuker, is out.

This release fully qualifies the demo programs with a net.didierverna.clon.demo prefix. This avoids polluting global namespaces and makes Quickref find Clon’s primary system correctly.

Clon 1.0b26 is out

Mon, 27 Jun 2022 00:00:00 +0000

A new version of , the Common Lisp / Command-Line Options Nuker, is out.

This version makes the termio subsystem loadable on its own, which is required for Quicklisp to setup up its meta-data properly (thanks to Jake Connor).

Also in this release, there’s a new error handling option to automatically print the usage string before quitting when a problem arises.

Quickref 4.0b1 is out

Thu, 26 May 2022 00:00:00 +0000

As , I have released the first version compatible with Declt 4.0 (which is now at beta 2) and which is going to keep track of the development over there. Because Declt 4 is currently considered in beta state, so is Quickref 4 for the time being.

There are a number of important changes in this new major version of the library. Not much interesting from the outside is an infrastructure overhaul which improves the parallel support. The index generation code has also been rewritten to benefit from the recent changes in Declt. Slightly more interesting from the outside is an improvement of the self-documenting aspects of the public interface when used interactively, along with more usage correctness checks. Also, the build environment has been upgraded to Debian Bullseye.

But the most critical change, of course, is the name for the 4.x series. In compliance with the general theme (Iron Maiden songs), and because Quickref 4 is meant to closely follow the brave new Declt version, I thought “Brave New World” would be nice. On the other hand, as Declt wanders through its uncharted 4.0 beta territory, Quickref 4 is likely to suffer the consequences, so perhaps “The Aftermath” is more appropriate…

Anyway, the Docker images are up to date, and so is the , currently documenting 2110 Common Lisp libraries.

Declt 4.0b1 is out

Tue, 10 May 2022 00:00:00 +0000

Today, after two years and a half of (irregular) development, and 465 commits, I have released the first beta version of 4, my reference manual generator for Common Lisp Libraries.

I rarely release official beta versions of software, let alone make announcements about them, but this one deserves an exception. Since version 3, Declt has been undergoing a massive, SOC-oriented, overhaul. The intent is to reimplement it into a clean 3-stages pipeline. Stage one, the “assessment” stage is in charge of gathering information. Stage two, the “assembly” stage, creates a documentation template specifying how the information is to be organized. Finally, Stage three, the “typesetting” stage, renders the template in a specific output format. Of course, the purpose is to eventually be able to support different kinds of templates and output formats.

Declt 4.0b1 marks the achievement of Stage one, which is now complete. It also contains a lot of new (and long awaited) features and improvements. The user manual has been updated to provide an overview of the new architecture, along with some details about stage one of the pipeline. From now on, I’m also going to release new versions of to closely follow the evolution of Declt.

Apart from the aforementioned architecture overhaul, which really boils down to internals shaking and shouldn’t affect the end-user much, a lot of new features and improvements are also provided in this release. They are listed below.

Backward Incompatible Changes

For the benefit of Separation of Concerns, but also for other reasons, a number of keyword arguments to the declt function have been renamed. :hyperlinks becomes :locations, :version becomes :library-version, :texi-directory becomes :output-directory, and :texi-name becomes :file-name.

New Features

Default / Standard Values

Declt now has the ability to advertise or hide properties that get default / standard values (e.g. standard method combination, :instance slot allocation, etc.).

Support for Aliases

Declt now recognizes and advertises “aliases”, that is, funcoids which have their fdefinition, macro, or compiler macro function set manually to another, original, definition.

Support for New Programmatic Entities

Typed structures and setf compiler macros are now properly detected and documented with all the specific information.

Proper Support for Uninterened Symbols

Such symbols may be encountered on several occasions such as slot names (trivialib, for example, defines structures with uninterned slot names in order to prevent access to them). Definitions named with uninterned symbols are considered private, and denoted by the ‘’empty set’’ package (∅).

Requirement for SBCL 2.1.2

The following enhancements depend on it.

Short form setf expanders now get correct source information.
Method combinations lambda-lists are now documented.

Domestic vs. Foreign Definitions

The concept of domestic (as opposed to foreign) definition has been extended to include those created in one of the sources files of the library being documented, even if the symbol naming the definition is from a foreign package. If the source file is unknown, then the symbol’s package must be domestic. The general intent is to consider language extensions as domestic, hence, always documented. For example, a new method on a standard generic function (say, initialize-instance) will now always appear in the documentation.

This refinement is accompanied by a new option allowing to include foreign definitions in the generated documentation. Of course, only foreign definitions somehow related to the library being documented will appear in the documentation. Also, those definitions will in general be partial: only the parts relevant to the library being documented will appear. Continuing with the initialize-instance example, new methods normally appear as toplevel, standalone definitions in the documentation (and their parent generic function is simply mentioned). If foreign definitions are included however, there will be a toplevel entry for the generic function, and all methods defined in the library will be documented there (truly foreign methods won’t appear at all).

Introspection Heuristic

Until now, there was a single heuristic for finding domestic definitions, which was to start from domestic packages and scan as many connections between symbols as possible. That heuristic is reasonably fast, but may occasionally miss some domestic definitions. Declt now has the ability to scan all symbols in the Lisp image instead of only the ones from domestic packages. This new option ensures that all domestic definitions are found, at the expense of a much greater computation time.

Supported Licenses

The Microsoft Public License has been added.

Info Installation Category

In other words, the value of Texinfo’s @direntry command is now customizable (instead to being hardwired to “Common Lisp”).

Improvements

Documentation Thinning

Packages reference lists do not point to methods directly anymore (as they can be reached via the generic function’s reference). Also, only slots for which the parent classoid is named from another package are now referenced.
Files reference lists do not point to slots anymore (as they can be reached via the parent classoid’s reference). Also, only methods for which the parent generic function is defined in another file are now referenced.
The readability of long references has been improved. In particular, they don’t advertise the type of the referenced definitions anymore, when there is no ambiguity.
Slot documentation now advertises the slot name’s package only when different from that of the parent classoid.
Non standalone method documentation now advertises the source file only when different from that of the parent generic function.
The rendering of EQL specializers has been inmproved.
The documentation of setf / writer methods doesn’t render the “new value” argument / specializer anymore.

Merging

Generic definitions containing only reader / writer methods are upgraded to specific reader / writer categories, and definition merging is now only attempted on those.

Lambda Lists

Uninformative parts of lambda lists are now filtered out. This includes &whole, &environment, &aux variables, along with options / keyword variables and default values.
Method specializers in lambda lists now link back to their respective class definitions.

Method Combinations

The method combination discovery scheme has been upgraded to benefit from SBCL 1.4.8’s enhancements (themselves following my ). The old code didn’t break, but prevented unused method combinations from being detected.

Bug Fixes and Workarounds

ASDF

Better handling of missing / unloaded components or dependencies (this can happen for instance with feature-dependent conditional inclusion).
Cope with the lack of specification of the license information in ASDF systems by coercing to a string.
Fix several cases of system files documentation duplication. Declt automatically documents .asd files as special cases of Lisp files. However, some systems have the bad (IMHO) habit of mentioning them explicitly as components (e.g. static files). When this happens, Declt silently discards that definition and keeps its own (at the expense of having a slightly incorrect system documentation).

Anchoring

After years of vain attempts at providing human-readable yet unique anchor names (which was only really useful for Info, BTW), I finally got rid of my last bit of idealism. Anchor names now use numerical definition UIDs, and since Texinfo allows me to expand references with some more information than just the anchor, it’s good enough. Besides, it fixes the last remaining rare cases exhibiting the fact that it’s just impossible to have anchors that are both human-readable and unique.

`Setf` Expanders

Fix the computation of short form setf epxander lambda lists (which didn’t correctly handle the presence of optional or rest arguments before.
Handle the potential unavailability of a setf expander’s update function or short form operator. Document it if applicable. Also signal a warning when the expander is domestic.

Clon 1.0b25 is out

Wed, 24 Mar 2021 00:00:00 +0000

Today, I’m releasing the next beta version of , the Common Lisp / Command-Line Options Nuker. The previous official release occurred 6 years ago. Since then, a number of changes had been quietly sleeping in the trunk but never made their way into Quicklisp.

More recently, I have also applied a number of changes that are worth mentioning here.

First of all, a large part of the infrastructure as been updated, following the evolution of the 8 supported compilers, and that of ASDF and CFFI as well. This should normally be transparent to the user though, provided that one uses reasonably recent compiler / ASDF versions (“reasonably” intentionally left undefined).

Other than that…

The constraints on termio support auto-detection had become slightly too restrictive, so they have been relaxed.
The exit function has been deprecated in favor of uiop:quit.
The support for running in scripts rather than in dumped executables has been improved, notably by offering the possibility to provide an alternate program name when argv[0] is not satisfactory.
Clon is now compatible with executables dumped via either ASDF’s program-op operation or its own dump function. The demonstration programs in the distribution have been updated to illustrate both dumping methods.
The documentation on application delivery has been largely rewritten, and has become a full chapter rather than a thin appendix.
There are also a few bug fixes in this release.
Clon now depends on named-readtables.
Several readtable problems have been fixed for CCL, CLISP, and ECL (thanks to Thomas Fitzsimmons).
Clon now compiles its termio support correctly with a C++ based ECL (thanks to Pritam Baral).
One problem in the conversion protocol for path options has been corrected (thanks to Olivier Certner).

Quickref 3.0 "The Alchemist" is out

Thu, 21 Nov 2019 00:00:00 +0000

Following up with the latest I’m also happy to announce the release of 3.0 “The Alchemist”, our reference manual aggregator for Quicklisp libraries. The has been updated yesterday with both the latest Quicklisp version, and the new Declt, resulting in the documentation of 1792 Common Lisp libraries.

Declt 3.0 "Montgomery Scott" is out

Tue, 19 Nov 2019 00:00:00 +0000

This is a new major release of , my reference manual generator for Common Lisp Libraries, although the changes may not be so visible from the outside. The main concern in this release has been to increase the robustness of the output, from three different angles.

Many places from where Declt attempts to extract meaningful information are underspecified (ASDF system slots notably).
The pretty-printing of Lisp items can be difficult, given the liberalism and extensibility of the language.
Finally, several restrictions in the syntax of Texinfo itself (anchor names notably) get in the way. These issues were described in a presented at the TeX Users Group conference, this summer in Palo Alto. This release goes a long way towards fixing them.

The new generated reference manuals look mostly the same as before, but some important things have changed under the hood, notably the names of hyperlinks and cross-references (backward-incompatible, hence the increase in the major version number). The output is now also much more robust with respect to the final output format: the generation of HTML, DVI, Postscript, PDF, and even plain Info should work fine and has been tested on all Quicklisp libraries (in fact, Quickref will also be upgraded in the near future to provide all those formats at once).

Those improvements do come at a cost. Unicode is now required (even for Info readers). To be honest, many Lisp libraries already had that implicit requirement. Also, this release depends on improvements and bug fixes only available in Texinfo 6.7, now a requirement as well. I have to thank Gavin Smith, from the Texinfo team, for his collaboration.

Apart from that, this release also has a number of bug fixes.

Some method specializers were not handled properly.
The manuals were missing documentation for some non-Lisp source files.
There were some glitches in the pretty-printing of unreadable objects.

Quickref 2.0 "Be Quick or Be Dead" is out

Mon, 08 Apr 2019 00:00:00 +0000

Surfing on the energizing wave of ELS 2019, the , I’m happy to announce the release of 2.0 “Be Quick or Be Dead”, our reference manual aggregator for Common Lisp libraries.

The major improvement in this release, justifying an increment of the major version number (and the very appropriate codename), is the introduction of parallel algorithms for building the documentation. I presented this work last week in Genova so I won’t go into the gory details here, but for the brave and impatient, let me just say that using the parallel implementation is just a matter of calling the build function with :parallel t :declt-threads x :makeinfo-threads y (adjust x and y as you see fit, depending on your architecture).

The second featured improvement is the introduction of an author index, in addition to the original one. The author index is still a bit shaky, mostly due to technical problems (calling asdf:find-system almost two thousand times simply doesn’t work) and also to the very creative use that some library authors have of the ASDF author and maintainer slots in the system descriptions. It does, however, a quite decent job for the majority of the authors and their libraries’reference manuals.

Finally, the repository now has a fully functional continuous integration infrastructure, which means that there shouldn’t be anymore lags between new Quicklisp (or Quickref) releases and new versions of the .

Thanks to Antoine Hacquard, Antoine Martin, and Erik Huelsmann for their contribution to this release! A lot of new features are already in the pipe. Currently documenting 1720 libraries, and counting…

Quickref 1.0 "The Trooper" is out

Tue, 07 Aug 2018 00:00:00 +0000

8 months after the , we finally open-sourced , our reference manual aggregator for Quicklisp libraries. The complete source code is now available .

I gave a lightning talk at , in which I claimed I would release the code in “something like two weeks”. That was 4 months ago 😃. I apologize for the delay, especially to Zach who expressed interest in how we did the Docker thing at the time. I wanted to clean up the code before the first public release (I’m a maniac like that), which I did, but it took some time.

Compared to what I announced at ELS, Quickref also has a couple of new features, most importantly the ability to generate documentation for only what’s already installed, rather than for the whole Quicklisp world. This can be very convenient if you just want some local documentation for the things you use daily. Finally, there is also a very rudimentary support for multithreading, which currently doesn’t bring much. But the code has been prepared for going further in that direction; this will be the topic of another internship which will start next September.

Browse no less than 1588 reference manuals right now at , and recreate your own local version with this Docker 2-liner:

docker run --name quickref quickref/quickref
docker cp quickref:/home/quickref/quickref .

Lisp, Jazz, Aikido, 10 years later

Wed, 02 May 2018 00:00:00 +0000

10 years ago, I published a entitled “Lisp, Jazz, Aikido”, barely scratching the surface of what I found to be commonalities between the 3 disciplines. At the time, I had the intuition that those ideas were the tip of a potentially big iceberg, and I ended the blog with the following sentence: “I’d like to write a proper essay about these things when I find the time… someday.”

Well, 10 years later, I did. The essay, which is 50 pages long, has been published in the , and actually received the Reviewers’ Choice Award 2018. I’m not the bragging type, far from it, but I had to mention this because this essay is so personal, and I invested so much in its preparation (more than 300 hours) that I am as deeply touched by the award as I would have been hurt, had it been negatively received…

The live presentation has unfortunately not been recorded, but I took the time to make a screencast afterwards, which is now . Just like the essay, this presentation is not in the typical setting that you’d expect at a scientific conference…

If you’ve got an artistic fiber, if you’re sensitive to the aesthetic dimension in what you do, you may enjoy this work…

Announcing Quickref: a Global Documentation Project for Common Lisp

Wed, 13 Dec 2017 00:00:00 +0000

Today, we deployed the first version of , a new global documentation project for Common Lisp.

The purpose of Quickref is to provide a centralized collection of reference manuals for the whole world. This means around 1500 libraries, for a total of around 3000 ASDF systems. The reference manuals are generated by , which is probably the most complete documentation system for Common Lisp currently available, and delivered in HTML (PDF versions could easily be made available as well).

A lot of things can still be improved, but I’m pretty satisfied with the result so far. 3000 ASDF systems is a hell of a test suite for Declt, and I’m happy to report that it passes on practically all of them. Only a couple of issues remain, not even due to Declt itself, and less than a dozen or so libraries still pose problems (mostly technical difficulties due to foreign dependencies).

Quickref was made by Antoine Martin, as part of an internship with me. Many thanks to him! We still have some cleanup and packaging to do, but we expect to open-source the infrastructure soon. I also want to thank Mark Evenson, Erik Huelsmann and the for hosting the project on common-lisp.net (it was only natural)!

Finally, let me restate this again (and again): reference manuals are not user manuals. They are… reference manuals. Although automatically generated, there are some things you can do, as a library author, to improve the output (this is an area of Declt which I intend to work on in the future). Please refer to the Declt user manual (notably section 3.2 ) for more information.

Declt 2.3 "Robert April" is out

Sat, 02 Dec 2017 00:00:00 +0000

I’m happy to announce the next release of , my reference manual generator for Common Lisp Libraries.

The improvements and bug fixes in this release are the result of running Declt against the whole Quicklisp world (around 3000 ASDF systems for 1500 libraries). See for more information.

New in this release:

Advertise file extensions in references.
Advertise the type of foreign definitions.
More robust display and indexing of, and with, lambda-lists.
Use UTF8 special characters to denote invisble ones.
More robust support for Texinfo brace escaping.
Handle modules sharing the same location.
Ensure output is done with standard IO syntax.
Fix potential duplication of some (non-lisp) files and document all static files.
Fix potential duplication of packages documentation.

Standard IO syntax and the Robustness Principle

Fri, 27 Oct 2017 00:00:00 +0000

Here is a flagrant illustration of the , or rather, of a failure to honor it.

I was investigating a bug in where some floating point numbers were printed with exponent markers (e.g. 0.5f0 instead of just 0.5) in the Texinfo file, which broke the parsing of the file by Perl.

Eventually, I found out a double infringement of the robustness principle. First of all, Declt failed to comply with part 1 of the robustness principle: “be lenient with the others”. The Texinfo file generation routine should have been wrapped into a call to with-standard-io-syntax and it wasn’t. Always do that to be on the safe side.

This failure on my part, however, had the interesting consequence of exhibiting what I consider a serious infringement of part 2 of the robustness principle: “be strict with yourself”. It would have remained unnocited otherwise. The culprit here is not Declt. This time, it’s the library. Problem: the simple fact of loading this library globally changes the value of *read-default-float-format* from single-float (the default) to double-float. This is bad, and it can break your code in all sorts of nasty ways.

Explanation

*read-default-float-format* tells the reader how to read floats when no exponent marker is provided. By default, 0.5 will be read as a single-float. But this variable also influences the printer (out of a concern for printing readably I guess): when printing a float of a different format than the current default, then the appropriate exponent marker is also printed. So here is precisely what happened. Declt had been compiled with some floats (e.g. 0.5) read as single-floats. Later on, those floats were supposed to be printed aesthetically as such. But right after loading common-lisp-stat, the default format changed to double-float and all of a sudden 0.5 started to be printed as 0.5f0.

Consequences

This is bad enough already, but consider that messing with the standard IO syntax globally like this can break your code in all other sorts of even nastier ways. Imagine for instance that common-lisp-stat had been loaded before Declt, and Declt needed to be recompiled. All of a sudden, Declt would be using double floats and the bug would be gone. That is, until the next start of the REPL, after which all floats would be printed like 0.5d0…

So granted, my code wasn’t robust enough. But please, don’t mess with standard IO syntax globally.

Declt 2.2 "Christopher Pike" is out

Mon, 16 Oct 2017 00:00:00 +0000

I’m happy to announce the next release of my reference manual generator for Common Lisp Libraries.

The improvements and bug fixes in this release are the result of running Declt against the whole Quicklisp world (around 3000 ASDF systems for 1500 libraries). See for more information.

New in this release:

Require a UTF-8 environment.
Understand ASDF’s notion of inferred system, and also be more protective against ASDF extensions.
Support for improper lambda lists (e.g. destructuring ones).
Improve contact defaulting code.
Update support for SBCL’s setf expanders introspection.
Accept ASDF system designators.
Various bug fixes in the areas of method combinations, accessor definition merging and setf expanders.

Declt 2.1 "Jonathan Archer" is out

Tue, 28 Feb 2017 00:00:00 +0000

I’m happy to announce the next release of , my reference manual generator for Common Lisp Libraries.

New in this release:

Handle recent change in SBCL’s sb-int:info API.
Handle list of contacts (strings) in ASDF systems author and maintainer slots.
Some backward-incompatible changes in the keyword arguments to the declt function.
More hyperlinks between systems and source files.
More robust system’s packages collection (no more code walking).
More robust handling of unavailable ASDF components.
More robust naming of cross-references.

Declt 2.0.1 "Benjamin Sisko" is out

Wed, 04 Nov 2015 00:00:00 +0000

The next edition of , my reference manual generator for Common Lisp Libraries, is out. This is a bugfix release with one internal change (a special variable was not following the earmuffs convention) and one actual bugfix (the same Texinfo anchor was generated for symbols with the same name but in different packages).

ASDF-FLV 2.0

Mon, 12 Oct 2015 00:00:00 +0000

I’ve just released version 2.0 of , my ASDF extension for supporting file-local variables (a la *package*). The code hasn’t changed, but as for my other libraries, the system and package names are now prefixed with net.didierverna. ASDF-FLV is also available on now.

The reason I’m doing this now is that at least two of my other libraries are going to use it in a mandatory way, either directly or indirectly (and in turn, that’s because no implementation has bothered to implement yet 🥲.

The Return of Segfaults

Wed, 05 Aug 2015 00:00:00 +0000

I was watching the discussion between Gilad Bracha and Matthias Felleisen on gradual typing this afternoon (it’s available on ). This was the last event at the STOP workshop, part of ECOOP 2015 in Prague. I couldn’t attend it because I was somewhere else (Curry On) at the time. The discussion is interesting, but if you go all the way, in the last 10 minutes or so, you will notice that Matthias seems to be completely obsessed with what he calls the “Return of SegFaults”.

Basically, the point is the following. Mathias dislikes the optional types in Common Lisp because it’s opening the door to unsafety. Initially, any dynamic language has a sound type system (all type errors are caught; at run-time, yes, but they are caught). As soon as you introduce an optional type system a la Lisp, the compiler implementors are “pushed” to use the provided type information for optimization, hence weakening the type system and breaking soundness. It’s the “return of segfauts”.

Of course, I agree with that, at least in principle. Yes, Common Lisp’s weak, optional type system is an antiquated thing. However, it seems to me that Matthias is forgetting two important things on the practical level:

By default in many implementations that I know, if not all of them, introducing type declarations doesn’t actually break the soundness of the type system but leads to even more type checking. For example, (defun plus (a b) (+ a b)) works on every possible numerical value, but add (declare (type fixnum a b)) in there and it will suddenly stop working on anything else but integers. It’s only if you require the compiler to optimize for speed at the expense of safety that you effectively weaken your type system.
In practice, the risk of re-introducing segfaults in your application may be mitigated by the interactive aspect of the development (TDD made easier, simultaneous write / compile / run / test / debug phases etc.).

So my impression is that Matthias is largely exaggerating the problem, but I’m not really qualified to tell. That’s why I would like to know from you guys, working with Lisp in the industry, writing large applications, lots of type annotations, and (declaim (optimize (speed 3) (safety 0) (debug 0)) (EDIT: that’s exaggerated of course, I really mean breaking safety for performance reasons): how much of a problem the "return of the segfaults" really is in practice ?

As a side note, this reminds me of the dynamic vs. lexical scoping debate. Many people were and still are strongly opinionated against dynamic scoping by default. Of course, I too, at least in principle. But how dangerous dynamic scoping really is in practice (EDIT: I’m not talking about expressiveness, e.g. closures, here. Only unsafety)? What I can tell you is that in the 15 years I was actively maintaining the XEmacs codebase, I may have run into name clashes due to dynamic scoping… twice.

Declt 2.0 "Kathryn Janeway" is out (important)

Mon, 13 Jul 2015 00:00:00 +0000

A new major version of , my reference manual generator for Common Lisp Libraries, is out.

This release doesn’t contain any change in functionality, yet deserves a major version upgrade since it contains 3 important changes: an infrastructure revamp (along the lines of what endured not so long ago), a license switch from the GNU GPL to a BSD one, and finally a system / package name change. The prefix is now net.didierverna instead of com.dvlsoft. Do I need to apologize for this again? 😀

Declt 1.1 "Jean-Luc Picard" is out

Mon, 29 Jun 2015 00:00:00 +0000

As promised last week, I’ve just released a new version of , my reference manual generator for Common Lisp Libraries. This new version is now able to document again (the documentation of which has been updated on the website).

New in this release:

Declt now properly handles and documents complex system and component dependencies, such as :feature :require and :version statements,
Declt also documents a system’s :if-feature if any.

But the most important addition is the ability to document several ASDF systems in the same reference manual. More precisely, Declt now documents not only the main system but also all its subsystems. A subsystem is defined as a system on which the main one depends on in any way, and which is also part of the same distribution (under the same directory tree). Declt also understands multiple system definitions from the same .asd file.

Clon 1.0b24 is out (important)

Thu, 25 Jun 2015 00:00:00 +0000

I’m happy to announce the release of the next beta version of , the Common Lisp / Command-Line Options Nuker. This release contains few functional changes, but a lot of infrastructure ones, plus very important and backward-incompatible modifications. So if you’re a Clon user, please read on.

First of all, a huge revamp of the library’s infrastructure (package hierarchy, ASDF systems, and Makefiles) has occurred. A large portion of this work is actually not mine, but Faré’s (big thanks to him, ‘cause the level of ASDF expertise required just means that I couldn’t have done that by myself). The purpose here was twofold: first, remove all logic from the ASDF files (so that other system managers could be used; not sure that’s actually useful right now) and second, split the library in two: the core, basic functionality and the non-standard platform-dependent bells and whistles (read: termio support). The result is that Clon now comes with 4 different ASDF systems. A setup system allows you to configure some stuff prior to loading the library, a core system allows you to load only the basic functionality and the regular one loads everything, autodetecting platform-dependent features as before. The fourth system is auxiliary and not to be used by hand. All of this is properly documented. For a code maniac like me, this new infrastructure is much more satisfactory, and I’ve learned a lot about ASDF less known features.

Next, I’ve moved the repository to . Please update your links! It seems that I’ve lost all my former tags in the process, but oh well…Only the Git repo has moved. The main Clon web page still contains the full history of tarballs, the preformatted documentation, and will continue to do so in the future.

Finally (I’ve kept this to myself until the last minute because I’m scared like hell to tell): I’ve changed the systems and packages names… The com.dvlsoft prefix has been replaced with net.didierverna. All other libraries of mine will eventually endure the same surgery. It’s for the best, I apologize for it and I swear I will never ever do that again, ever (fingers crossed behind my back).

So what’s next? Before considering an official 1.0 release, there are two things that I want to do. First, cleanup some remaining “fixmes” and some shaky error handling. Second, provide an even simpler way of using Clon than what the Quick Start chapter in the doc demonstrates. The idea is to just implement a main function with keyword arguments, and those arguments magically become command-line options.

A side-effect of this work is that now chokes on Clon, because some ASDF features that it doesn’t understand are in use. So Declt has a couple of new challenges ahead, and you should expect a new release in the weeks to come.

Declt 1.0 "James T. Kirk" is out

Sat, 24 Aug 2013 00:00:00 +0000

After 15 betas, I’m happy enough with the current state of , my reference manual generator for Common Lisp Libraries, to finally make a 1.0 release.

A lot of things have changed since the previous version, sometimes in a backward-incompatible way. Many more items are documented (including, as you have recently , method combinations). In addition to the reference manual, generated by Declt itself, there is now a real user manual.

Declt is still SBCL-only, requires ASDF 3 and Texinfo 4 but generates code that is compatible with Texinfo 5. Also, beware, I’ve deleted the old repository and moved the project to . Below is a more precise description of what Declt currently does.

Declt (pronounce “dec’let”) is a reference manual generator for Common Lisp libraries. It works by loading an ASDF system and introspecting its contents. The generated documentation contains the description for the system itself and its components (modules and files), the packages defined in that system and the definitions found in those packages.

Exported and internal definitions are listed separately. This allows the reader to have a quick view on the library’s public API. Within each section, definitions are sorted lexicographically.

In addition to ASDF system components and packages, Declt documents the following definitions: constants, special variables, symbol macros, macros, setf expanders, compiler macros, functions (including setf ones), generic functions and methods (including setf ones), method combinations, conditions, structures, classes and types.

The generated documentation includes every possible bit of information that introspecting can provide: documentation strings, lambda lists (including qualifiers and specializers where appropriate), slots (including type, allocation and initialization arguments), definition source file etc.

Every documented item provides a full set of cross-references to related items: ASDF component dependencies, parents and children, classes direct methods, super and subclasses, slot readers and writers, setf expanders access and update functions etc.

Finally, Declt produces exhaustive and multiple-entry indexes for every documented item.

Reference manuals are generated in Texinfo format (compatible, but not requiring Texinfo 5). From there it is possible to produce readable / printable output in info, HTML, PDF, DVI and PostScript with tools such as makeinfo, texi2dvi or texi2pdf.

The Declt reference manual is the primary example of documentation generated by Declt itself.

Lisp Corner Cases: Method Combinations

Fri, 16 Aug 2013 00:00:00 +0000

In the process of writing , I had to deepen my knowledge of some Lisp corner cases, notably in the area of introspection. As you know, Common Lisp has more than 2 namespaces (calling it a “Lisp-2” is somewhat misleading). Sometimes, introspecting a potential symbol definition in one namespace is trivial (compiler-macro-function is one example). The “functional” namespace is heterogeneous but you can still make your way out of macros, regular or generic functions very easily. Distinguishing constants, special variables and symbol macros in the “variables” namespace is more complicated because there is no standard way to access that information, but with the help of some vendor-specific machinery (e.g. sb-int:info), it’s still doable.

At some point, I tackled the case of method combinations, and to my greatest surprise, found out that introspecting a method combination by name is not simple. In fact, it’s not even doable. The reason is that method combinations don’t have a namespace proper. Let me explain why.

You define a new method combination of some NAME with define-method-combination, and you use it with the :method-combination option to defgeneric. But how do you introspect a NAME for a potential method combination definition? Well, you can’t do exactly that. First, there is no standard way to do so. Next, let’s look at what the MOP provides. One would expect something along the lines of find-class…

There is indeed something called find-method-combination, but either I’m missing something, or it is broken by design. The arguments to this function are: a generic function meta-object, a method combination name and some method combination options. So if this function is supposed to be the equivalent of find-class for method combinations, what are the 1st and 3rd arguments for? In fact, this funjction is not supposed to do what its name suggests. According to the AMOP, p.191, the purpose of this function is to “determine the method combination object used by a generic function”. In practice however (at least in SBCL), it’s not doing that either (see below), and anyway, determining the method combination object used by a generic function is better achieved by using the generic-function-method-combination accessor.

So this protocol doesn’t seem to make any sense. In order to understand what to make of all this, I looked at how SBCL handles method combinations (I don’t know what other vendors do) and here is what I found. When you call define-method-combination for some NAME, SBCL creates a new method for find-method-combination, eql-specialized on that NAME. This method is encapsulated in a closure containing the method combination’s definition, ignores its first argument (the generic function meta-object; that’s why I said that it’s not really doing what the AMOP says it does), recreates and returns a new method combination object on the fly every time it is called.

This has several consequences. First, the notion of “method combination named NAME” doesn’t really make sense. Method combinations don’t have a proper namespace. Every time you create a new generic function, the method combination essentially becomes local to that function. Redefining a method combination has no effect on existing generic functions using a method combination of the same NAME. To put it differently, you can end up with several generic functions using different, yet equally named, method combinations.

In Declt, I’m now assuming that programmers behave and only define method combinations once. Which brings us to the second consequence. With that assumption in mind, the “proper” way to introspect a NAME for a method combination definition (at least in SBCL) is to first look for the existence of a find-method-combination method eql-specialized on that NAME, and then call it to retrieve the actual definition.

I haven’t thought about it a lot yet, but it would be interesting to investigate the idea of cleaning this up a bit. I mean, establishing a real global namespace for method combinations, creating an alternative find-method-combination protocol more along the lines of find-class. And then, it could also be interesting to investigate on the potential applications of allowing method combination redefinition to affect live generic functions, just like class redefinition affects live instances…

Nice little trick du jour

Wed, 26 Jun 2013 00:00:00 +0000

This morning, I came up with a nice little trick which made my day. Perhaps this is already well known or even idiomatic, but in case it is not, it goes like this.

Sometimes, I have to read a whole Lisp file containing more than just one toplevel form and get the contents as an unprocessed list of expressions. This happens for instance when loading a DSL program that needs post-treatment.

Until now, I did something like this:

(with-open-file (stream filename)
  (loop :for expr = (read stream nil :eof)
	:if (eq expr :eof)
	  :return exprs
	:else
	  :collect expr :into exprs)))

For some reason, this has always felt clunky to me. I mean, instead of ogling the family jewels of this poor shy file, I should be able to read it as a whole (hint: read-delimited-list) and preserve its intimacy.

And then I realized that the only thing that prevents me from using read-delimited-list is the lack of an ending parenthesis. But as always, Common Lisp immediately comes to the rescue, this time with its rich streams API. The idea is that you can create a string stream that just contains the closing parenthesis, and concatenate this stream at the end of the other one. I can now do something like this instead:

(with-open-file (stream filename)
  (read-delimited-list #\)
	(make-concatenated-stream stream (make-string-input-stream ")"))))

Not only it is shorter, it also looks much more elegant to me.

Declt 1.0b15 is out

Tue, 23 Oct 2012 00:00:00 +0000

I’ve just released a new version of , my reference manual generator for Common Lisp Libraries.

This version underwent a major internals overhaul, required by some of the new features described below:

Packages sections now advertise all definitions instead of just the symbols naming them. They also advertise their use-list and used-by-list, with cross-references.
Conditions, structures and classes now advertise their sub- and super-classes, direct methods, initargs and slots, with cross-references.
Slots documentation include docstring, type, initargs, initforms, readers and writers with cross-references.
Declt now documents symbol macros and compiler macros.
The *link-files* special is gone (M-x all-hail-purely-functional-style).
All ASDF components now advertise their descriptions and long descriptions, if any.
Docstrings are displayed in a more reader-friendly fashion.
Documentation entries for methods are nested within the corresponding generic function entry.

Clon 1.0b23 is out

Wed, 26 Sep 2012 00:00:00 +0000

A new version of , the Common Lisp / Command-Line Options Nuker, is out.

Amongst other things, the following improvements have been made.

Support for ABCL has been updated, now that it provides a full MOP.
A workaround for SBCL’s CC environment variable problem has been implemented. If the variable is not set (required for sb-grovel), Clon now switches to restricted mode instead of aborting.
The dump macro has been extended to accept a &rest argument that will be passed on to the underlying, implementation-specific, dumping facility.
A contrib directory has been added, for storing unapplied patches, suggestions etc.

Declt 1.0b14 is out

Tue, 25 Sep 2012 00:00:00 +0000

I’ve just released a new version of , my reference manual generator for Common Lisp Libraries.

This release containts some improvements based on Sabra Crolleton’s feedback. The most notable improvements are support for two new license types (MIT and LGPL), a new :declt-notice keyword argument that gives you control on the “automatically generated by Declt” notice that appears in the reference manuals, and a bug fix (missing support for empty lists in lambda-lists).

Language wars

Thu, 12 Jul 2012 00:00:00 +0000

Programming languages are tools. Just like hammers. There’s nothing personal about them. They just help you build stuff. Yet, there are many religious wars about programming languages out there. Wars which despite being all about science, are much more related to emotions, beliefs and personal aggressions than about objective arguments. This is funny because do you know of any religious wars about hammers? So what’s the difference?

Here’s a recent personal example. A guy explaining how the static typing of Common Lisp works (type declarations) and what kind of performance-oriented optimization can be achieved with it (C-like, static but weak typing in SBCL for instance). And then, there’s inevitably the troll (whom I “know” knows better) in the audience who goes:

So, yeah, what you’re doing is just C code, and you have to type manually, and it’s ugly. So if it’s just for writing C code, I don’t see the point in using another language.

Hmmm. Let’s see. So, yes indeed, static typing in Common Lisp is ugly. And yes, it’s not even strong typing. And yes, it would be nicer to have run-time hotspot detection and automatic type-dependant optimization like what’s found in some other languages or virtual machines, rather than having to do it by hand. But what does that really tell you? That no language is perfect? Wow, thank you very much, that’s new. For as much as I think that Lisp The Idea™ is perfect, I don’t think anyone ever pretended that Common Lisp (or any other Lisp for that matter) was. But is that a reason for not using it “at all” and sticking to C? Any sane computer scientist knows that the choice of a language doesn’t boil down to only one parameter. Any computer scientist who tells otherwise is a troll.

In spite of all its defects, I still have a gazillon reasons to prefer Common Lisp over C, C++, or any language that I know of currently (and this may very well change in the future). The flexibility of lambda lists, the macros, the MOP or more generally its structural and behavioral reflexivity. Its run-time compilation, debugging, introspection and intersession capabilities. These are just a few examples. Still, I don’t deny anyone the right to prefer another language for whatever purpose and whatever reasons they may feel legitimate.

So, I normally just ignore those purposedly trollesque and completely idiotic remarks. Yet, sometimes like yesterday, I snap. It gets on my nerves and I become all upset and angry. Why? I never get angry when someone tells me that my hammer is a piece of crap (it’s not). I do enough to know how to control my temper, but for some reason, it doesn’t always work when it comes to programming languages. So what is it about them that in spite of all your efforts, you can’t help from getting personally and emotionally involved once in a while?

I think the answer becomes apparent when you consider the artistic aspect in programming. When an artist creates a piece (music, theater, dance, painting, architecture, whatever you like), they expose a very intimate part of themselves through their creation. The art is the artist, and the artist is their art. In doing so, they put themselves in danger. It’s a well know fact that many artists are very fragile, in the sense that they suffer from their creation not being liked. Because a piece of art is intrinsically a piece of the artist himself, when you say “I don’t like this piece of art”, you’re actually saying “I don’t like the artist”. Then, it’s up to artist to handle the fact of not being liked.

And that’s the whole problem, which, also as a , I know all too well. Where does the artistic fiber come from? It’s an urge to express yourself. To express something that you can’t express in any other way. A very deep and perpetual wound of some sort, a feeling of not really belonging. More importantly, it’s a calling. Sometimes, the simple fact of creating is enough to heal you a bit, but more often, you create in order for your creation to be seen or heard. So yes, it’s a cry to the Other. You expect them to answer your call by telling you that they like you (your art, but that is the same). Artists often have this urge to be liked by the others. So when you dislike some artwork, you’re also not liking the artist himself (the part of him that lives in his creation) and you’re actually giving him the exact opposite of what he was looking for. And that hurts.

Back to programming languages. Why do we get all emotional about them, and not about hammers? The answer is in fact quite simple. Look at an architectural masterpiece. Do you see the hammer that was used to build it? Now look at a beautiful (definition left to the reader) program. Do you see the language that was used to write it? That’s the crucial difference. You cannot decouple the language from the program, even once it has been written (the art is not in the executable; it’s in the source code). The language itself will always be here for you to contemplate.

All in all, I think that this is why there will always be language wars. Programming languages are not just tools, actually. They’re not just like hammers. Programming languages are… languages first: means to expressing your inner self. As soon as you care about the code you write, your software becomes artwork, you become an artist, and you start to be personally and emotionally involved. Your software becomes part of you. And contrary to the hammer, your sticky programming language, being intrinsically bound to the artwork, also becomes part of you. That’s when the battle for objectivity is lost. By criticizing the language, the troll also criticizes your artwork, and in doing so, he tells you that he doesn’t like you. That may hurt.

It’s good to consider programming as art. Unfortunately, this also means that there will always be language wars.

Declt 1.0b13 is out

Mon, 04 Jun 2012 00:00:00 +0000

I’ve just released a new version of , my reference manual generator for Common Lisp Libraries.

This release includes some uninteresting internals update, plus an important bug fix: there were two calls to find-method missing an errorp flag set to nil, leading to Declt throwing an error where it shouldn’t have.

Clon 1.0b22 is out

Tue, 22 May 2012 00:00:00 +0000

A new version of , the Common Lisp / Command-Line Options Nuker, is out.

The most important change in this release is the support for LispWorks, which brings the number of supported implementations to 8. One left to go, and I may eventually switch to RC status. Thanks to Martin Simmons for providing a fully functionnal version of LW 6.1. As for CLISP and Allegro, there is an optional dependency on CFFI for LispWorks.

Two backward incompatible changes that may affect you:

Variables renamings: *current-context* has been renamed *context*, and *default-synopsis* has been renamed *synopsis*. This should remain transparent unless you’re using Clon in a somewhat advanced way.
clon:exit has been upgraded to SBCL’s new quitting protocol. If you use this function (or if you want to compile the demo programs), please upgrade to SBCL 1.0.57.

Finally, support for terminal autodetection and stream handling in general has been improved for all implementations.

The Syntax Extension Myth

Mon, 14 May 2012 00:00:00 +0000

Here’s a little Monday Troll.

To my greatest disappointment, I discovered today that it is not possible to replace Lisp parenthesis by, say, curly braces. What a shame. Hell, it’s not even possible to freely mix the two. Very naively, I had expected that

(set-macro-character #\{ (get-macro-character #\())
(set-macro-character #\} (get-macro-character #\)))

would suffice, but no. All implementations that I have tested seem to agree on this, although the error messages may differ. For instance, trying to evaluate {and} gives you an “unmatched close parenthesis error” except for CMU-CL which chooses to ignore it, but then report an end-of-file error. The unmatched close parenthesis, of course, is the closing curly brace. So what is going on here?

When an opening curly brace is read, the original left paren macro function is called. In SBCL for instance, this is sb-impl::read-list, which looks for a hardwired right parenthesis on the stream. Yuck. It doesn’t find one, but it finds my closing brace which triggers the “standalone” right paren behavior (spurious parenthesis alert). In passing, it also surprised me that sb-impl::read-list is not implemented in terms of read-delimited-list.

EDIT: as mentioned in several comments, we could use read-delimited-list to look for a closing curly brace, but even this won’t work completely. The problem is with dotted lists (see Pascal’s comment). SBCL hard-wires #\) in its dotted lists parsing procedures.

So it appears that dispatching macro characters are only shaky. What we miss is a true concept of syntactic categories (Common Lisp character syntax types are close, but not quite there yet). In fact, TeX, with its notion of “catcodes” (category codes), seems to be the only language that gets this right. Ideally, any character with a status of list terminator should do as good as a right paren (the problem is only with closing, not opening).

Instead of hard-wiring the right parenthesis in the Lisp parser, a quick workaround would be to check whether the next character on the stream is a dispatching one, and in such a case, whether its macro function is the one originally associated with the right paren. If so, it should then simply stand as a list terminator. This is actually an interesting idea I think: could the built-in macro functions become equivalent to actual category codes, and could we completely remove hard-wired characters in Lisp parsers?

Anyway, this whole story is a true scandal because it ruined an otherwise cool live demo of mine. So much for syntax extensibility.

Comments

From Pascal Bourguignon

Indeed, the main point is that CL doesn’t export facilities to read dotted-lists, only proper lists. And implementing dotted list reading is not even easy (you cannot just use peek-char and read, you need lower level parsing).

Nonetheless, syntax extensibility can be as easy as:

cl-user> (make-lisp-list-macro-character #\{)
#\}
cl-user> '{a {1 2 3} {} c . d}
(a (1 2 3) nil c . d)

See .

PS: Thanks for that blog entry, I found a bug in my lisp reader 😉

From Eivind Midtgård

This is explained in CLTL2, just the way you found it to be, in the entry for set-syntax-from-char, p.541, 542.

From Greg Pfeil

You can achieve your goal almost as simply, though:

(set-macro-character #\{
  (lambda (stream char) (read-delimited-list #\} stream t)))
(set-macro-character #\} (get-macro-character #\)))

The problem with your original attempt, IMO, is that if CL were defined in a way that allowed that, the result would be to allow (and} and {and) in addition to the expected {and}, which seems very non-intuitive to me.

If you want to (} and {) to work, it’s possible, but it does (and should) require more work to do so.

That said, with Unicode, an implementation might be able to define #( such that it matched the mirrored character in the Unicode DB, and then (set-macro-character #\{ (get-macro-character #\()) could work in the intuitive way, but I don’t know if that would be conforming, and it certainly wouldn’t be portable.

From DanM

I’m sorry you’re disappointed, but think that your expectation is simply not in accord with the CL reader’s implementation model. You should be able to quite easily do what you wanted thus:

(set-macro-character #\{
  (lambda (stream char) (read-delimited-list #\} stream)))
(set-macro-character #\}
  (lambda (stream char)
	(error "an object cannot start an object with #\}")))

The second call serves two purposes: it causes the closing brace to be treated as something other than a constituent character when it appears alone, and (because the optional non-terminating-p argument to set-macro-character is allowed to default to nil) makes it terminate a symbol without becoming part of it.

So you can do what you want easily with no additional infrastructure.

This is quite elegant and requires no further.

From me

@Pascal: exactly. There are 2 other places in SBCL where #\) is hard-wired, and these have to do with reading dotted lists. Thanks for the paste BTW!

@Eivind: thank you for the pointer.

@Greg, Cosman, Dan: yes, you can use read-delimited-list, but this will still not function properly in all cases. See Pascal’s comment (I should have mentionned this in my original post).

From Jerry Boetje

Yes, CLforJava and I are still around and working. When I saw post, I went to see how CLforJava handles your test. Of course, it had a similar result. However, after reading the rest of your text, I remembered that we devised a character syntax type system into the base of the reader. Might be an interesting job for a student…

From Marc Espie

TeX gets it right, except it does not have enough introspection.

It’s darn nigh impossible to get it to regurgitate text and fudge with the catcodes, leading to appendix D and the very obscure \afterassignment hack.

Clon 1.0b21 is out

Mon, 12 Mar 2012 00:00:00 +0000

One year between b19 and b20. 4 days between b20 and b21…

This new version of , the Common Lisp / Command-Line Options Nuker, introduces support for a new compiler, Allegro Common Lisp, in both standard and modern form. Support for dumping is only rudimentary for ACL (although it’s only a marginal feature of the library). The dump macro uses Allegro’s dumplisp mechanism to dump a lisp image which is not directly executable (full application delivery is complicated and only available in the Enterprise edition). Apart from that, the rest should work fine. As in the case of CLISP, Allegro may benefit from the presence of CFFI in order to provide terminal autodetection. This is an optional dependency only.

Clon 1.0b20 is out

Thu, 08 Mar 2012 00:00:00 +0000

I’m happy to announce a new release of , the Common Lisp / Command-Line Options Nuker. In addition to a lot of uninteresting code and infrastructure changes, this new release comes with several important improvements and new features.

At the end-user level

There is a new error handler available via the --clon-error-handler option, called “interactive”. This error handler provides the same restarts as the one called “none” (which actually triggers the Lisp debugger), but in a less frightening way for end-users not knowing about Lisp at all. In particular, the error and restart messages are more readable and you don’t see a Lisp stack anywhere. See the and the for more information.
There is a new option called --clon-lisp-information which, as its name suggests, provides information about the underlying Lisp (implementation type and version). This will in fact be more useful for developers than for end-users, but it’s still and end-user level feature. See the for (not) much more information.

At the user-level

Clon now provides a command-line polling API through the functions cmdline-options-p and cmdline-p. See the for more information.
Support for using Clon interactively, that is, “without” dumping executables has been improved. This is mostly useful for debugging purposes. See the for more information.
Clon now provides a compile-time / run-time unified configuration facility thanks to a variable named cl-user::com.dvlsoft.clon.configuration that is handled before the ASDF system is loaded. Thanks to this, Clon is now able to communicate its own indentation information to Emacs directly (thanks to a process that I’ve previously described ), and also handles portability problems in a smoother way (see below).
One of the available configuration options is called :restricted mode. In this mode, Clon never attempts to communicate with TTYs via IOCTL calls, at the expense of terminal autodetection (size and highlighting). This is implemented by making a termio ASDF module conditionally loaded in the system definition. There are cases where Clon will switch to restricted mode automatically (e.g. when using CLISP compiled without FFI support). However, some other situations are more problematic, for instance when using SBCL under MinGW, in which case the sb-grovel module is available but doesn’t work. In such situations, it is necessary to configure Clon explicitely for restricted mode before loading the system. See the for more information.

CL-RCFiles 2.0 is out

Mon, 28 Nov 2011 00:00:00 +0000

Just a quick note to mention the release of 2.0. This version adds pre-loading initialization files.

This very small Common Lisp library provides a way to add initialization files to ASDF systems. Every time ASDF loads system, one or several corresponding system.lisp files are loaded automatically afterwards. This lets you conditionally plug in additional behavior on a per-system basis without cluttering up any global Common Lisp initialization file.

By default, these initialization files are expected to be found in:

~/share/common-lisp/rc/pre/ for pre-loading initialization,
~/share/common-lisp/rc/post/ for post-loading initialization.

For backward-compatibility, files found directly in the rc/ directory are considered to be post-loading initialization files.

You can modify the rc-files location by changing the values of the global variables *directory*, *pre-directory*, and *post-directory*.

One More Indentation Hack

Wed, 20 Jul 2011 00:00:00 +0000

Here’s yet another indentation hack that I came up with recently.

All the work done by Nikodemus on the Slime indentation contrib is pretty cool, especially the notion of indentation style (though I wish the styles were Custom variables, but that is another story). I tend to use indentation styles for global, maybe collaborative preferences, but on several occasions however, I find that this approach has a couple of drawbacks.

One of them is that the indentation information is far away from the corresponding symbol, in a separate file. If you change a function’s prototype for instance, you may also need to load the file(s) in which the corresponding style(s) is (are) defined and edit them.
The other problem is that if you want to let other people edit your source code and honor your indentation style, you also need to provide them with the style definition, and they need to load it separately.

For those reasons, I tend to think that the indentation style approach is not very well suited for project-specific indentation settings. What I would like is to provide indentation information close to the function definition, and also to have that information automatically available when anyone loads the project into Slime. Here’s a way to do it.

The key to success here is the function swank:eval-in-emacs which, as its name suggests, sends some Emacs Lisp code to your Emacs session for evaluation. This function effectively allows you to trigger some Emacs Lisp computation from a Common Lisp file. Remember that indentation information is stored in the common-lisp-indent-function property of a symbol. The function clindent below does this:

(defun clindent (symbol indent)
  "Set SYMBOL's indentation to INDENT in Emacs.
This function sets SYMBOL's `common-lisp-indent-function' property.
If INDENT is a symbol, use its indentation definition.
Otherwise, INDENT is considered as an indentation definition."
  (when (and (member :swank *features*)
	    (let ((configuration
		       (find-symbol "MY.LIBRARY.CONFIGURATION" :cl-user)))
	      (when (and configuration (boundp configuration))
		    (getf (symbol-value configuration) :swank-eval-in-emacs))))
    (funcall (intern "EVAL-IN-EMACS" :swank)
	  `(put ',symbol 'common-lisp-indent-function
	    ,(if (symbolp indent)
		  `(get ',indent 'common-lisp-indent-function)
		  `',indent))
	     t)))

As explained in the docstring, this function will ask Emacs to put SYMBOL’s common-lisp-indent-function property to a definition, either provided directly, or retrieved from another symbol. For example, if your library defines an econd macro, you may want to call it like this:

(clindent 'econd 'cond)

This function ensures that Swank is actually available before using it (first condition in the and clause). I will explain the other weird bits later on.

The next question is when exactly do we want to call this function? The answer is: pretty much on all occasions. Your code might be loaded from source and interpreted, or it might be compiled. But then, it might be compiled within or outside a Slime environment. In any case, you want your indentation information to be sent to Emacs everytime it’s possible. So obviously, we’re gonna wrap this function in an eval-when form thanks to a macro. This is also a good opportunity to save some quoting.

(defmacro defindent (symbol indent)
  "Set SYMBOL's indentation to INDENT in Emacs.
SYMBOL and INDENT need not be quoted.
See `clindent' for more information."
  `(eval-when (:compile-toplevel :execute :load-toplevel)
     (clindent ',symbol ',indent)))

And now, right on top of your econd definition, you can just say this:

(defindent econd cond)

Now here’s one final step. If your library uses its own readtable, it’s even more convenient to define a reader-macro for indentation information. I choose #i:

(defun i-reader (stream subchar arg)
  "Read an argument list for the DEFINDENT macro."
  (declare (ignore subchar arg))
  (cons 'defindent (read stream)))

(set-dispatch-macro-character #\# #\i #'i-reader *readtable*)

And now, the code will look like this:

#i(econd cond)
(defmacro econd #|...|#)

All right. We still have two weirdos to explain in the clindent function.

First, you noticed that the function’s computation is conditionalized on the existence of a cl-user::my.library.configuration variable, which actually stores a property list of various compiling or loading options for this package. The option we’re interested in is :swank-eval-in-emacs, which must be set to non-nil. Here’s why. The execution of Emacs Lisp code from Swank is (rightfully) considered as a security risk so it is disabled by default. If you want to authorize that, you need to set the (Emacs) variable slime-enable-evaluate-in-emacs to t. Otherwise, calling swank:evaluate-in-emacs is like calling 911. So we have a chicken-and-egg problem here: if we want to avoid an error in clindent, we would need to check the value of this variable, but in order to do that, we would need to evaluate something in Emacs… 🤨

The solution I choose is hence to disable the functionality by default, and document the fact that if people want to use my indentation information, they need to set both the Slime variable and my library-specific option to t before loading the library (possibly setting them back to nil afterwards). They also need to trust that I’m not going to inject anything suspicious into their Emacs session at the same time…

The last bit we need to explain is the final t argument passed to swank:eval-in-emacs. The corresponding parameter is called nowait in the function’s prototype. It has something to do with asynchronous computation, and in fact, I don’t really know what’s going on under the hood, but what I do know is that if you set it to t, Swank doesn’t care about the return value of your form anymore, which is fine because we’re only doing a side effect. On the other hand, if you omit that parameter, Swank will try to interpret the return value in some way, and you will most probably get a serialization error. Indeed, the return value is the indentation definition itself, so for example, (&rest (&whole 2 &rest 1)) doesn’t make (Common Lisp) sense.

Declt 1.0b12 is out

Wed, 29 Jun 2011 00:00:00 +0000

I’ve just released the next version of , my reference manual generator for Common Lisp Libraries.

This release includes some fixes for the Info format target but most notably, support for documenting (generic) writer functions and methods. When it makes sense, the documentations for func and (setf func) are grouped together. Getting closer to an official 1.0 stable version…

Read-Time String Concatenation

Tue, 21 Jun 2011 00:00:00 +0000

Sometimes, I miss the string concatenation capability of C. I mean, the way that you can split a long string into smaller ones and have them automatically concatenated together. The format function has a tilde (~) directive that does something along these lines, but there are two problems:

first, I’m not necessarily facing the problem in format calls,
next, my favorite text editor cannot properly indent the string contents when I press the TAB key.

Here’s an example:


(defclass sample ()
  ((slot :documentation
	 "A very long slot documentation, that doesn't even fit in 80 columns, which is a shame...")))

If that were C code, I could write this:

(defclass sample ()
  ((slot :documentation
	 "A very long slot documentation, "
	 "that doesn't even fit in 80 columns, "
	 "which is a shame...")))

But this is not C code. We can come very close to that however. Here’s a small reader function that will automatically concatenate strings prefixed with a tilde (~) character (in honor of the eponymous format directive):

(defun ~-reader (stream char)
  "Read a series of ~\"string\" to be concatenated together."
  (declare (ignore char))
  (flet ((read-string (&aux (string (read stream t nil t)))
		   (check-type string string "a string")
		   string))
	(apply #'concatenate 'string
	   (read-string)
	   (loop :while (char= (peek-char t stream nil nil t) #\~)
		 :do (read-char stream t nil t)
		 :collect (read-string)))))

Let’s add it to the current readtable:

(set-macro-character #\~ #~-reader)

And now I can write this:

(defclass sample ()
  ((slot :documentation
	 ~"A very long slot documentation, "
	 ~"that doesn't even fit in 80 columns, "
	 ~"which is a shame...")))

Everyday of my life I thank Common Lisp for being so flexible. The next step would be to make cl-indent.el aware that tilde-strings really are the same object, and hence should be vertically aligned in all contexts. I’m not there yet. Jeeze, this indentation hacking will never end… 😃

Declt 1.0b11 is out

Tue, 31 May 2011 00:00:00 +0000

I’ve just released a new version of , my Texinfo reference manual generator for Common Lisp Libraries. This release contains only one bugfix: when trying to create links to source files, Declt now checks whether the files actually exist or not.

Tracking this bug down had the side-effect of exhibiting a misfeature of SBCL’s introspection facility: the copy-<struct> functions (automatically generated by defstruct calls) have their definition source set to target-defstruct.lisp which is an SBCL source file. It would make more sense to set it to the file containing the defstruct call instead, as is already the case for constructors, predicates and accessor functions. Patch sent to the SBCL developers.

Common Lisp indentation in Emacs

Fri, 06 May 2011 00:00:00 +0000

Update: since the original publication of this article, Nikodemus Siivola and I have done some more work on various other aspects of Common Lisp indentation, and I must say that the result is pretty satisfactory. Nikodemus has merged all the changes into Slime, and I have done so for XEmacs. If you’re an XEmacs user, you don’t need to use the slime-indentation contrib to get these improvements. Simply upgrade the “prog-modes” package and load cl-indent.el.

I have just modified to improve the indentation of Common Lisp code. This change involves two things: the support for multiple method qualifiers in a call to defmethod and, much more importantly, a cleaner and more flexible scheme for indenting lambda-lists. The patch has also been submitted to the developers. Below is a more detailed description of what you get with these changes.

Method qualifiers

Until now, only one method qualifier was understood. Below are some examples demonstrating that the one method qualifier and the argument list are indented by 4 characters, and the method’s body only by 2:

(defmethod foo :around ()        (defmethod foo :around
  do-this)                           ()
                                   do-this)

(defmethod foo
    :around ()                   (defmethod foo
  do-this)                           :around
                                     ()
                                   do-this)

Now let’s add a second method qualifier:

(defmethod foo comb :around ()        (defmethod foo
                    do-this)              comb :around ()
                                          do-this)

Woops. Neither is correct. But now, you get this instead:

(defmethod foo comb :around ()        (defmethod foo
  do-this)                                comb :around () 
                                        do-this)

Three more examples to show how confused we were:

(defmethod foo comb :around     (defmethod foo comb     (defmethod foo 
  ()                                :around                 comb
  do-this)                        ()                        :around
                                  do-this)                ()
                                                          do-this)

And how better we just got:

(defmethod foo comb :around     (defmethod foo comb     (defmethod foo
    ()                              :around                 comb
  do-this)                          ()                      :around
                                  do-this)                  ()
                                                          do-this)

Indeed, you can see that everything between the method’s name and its body is now correctly indented by 4.

Lambda-Lists

The next round of changes deals with the formatting of lambda-lists. As such, this will apply everywhere a lambda-list is expected, such as defun, defgeneric, defmethod, etc.

First of all, here are two examples showing that we were not very clever before:

(defun foo (mand1 mand2
            &optional opt1
            opt2
            &rest args
            &key key1 key2
            (key3 val3) &aux aux1
            aux2)
  do-this)

(defun foo (mand1 mand2 &optional opt1
            opt2
            &rest args
            &key key1 key2
            (key3 val3) &aux aux1
            aux2)
  do-this)

Basically, everything was blindly aligned at the beginning of the lambda-list. Here is what you get now:

(defun foo (mand1 mand2
            &optional opt1
              opt2
            &rest args
            &key key1 key2
              (key3 val3) &aux aux1
                            aux2)
  do-this)

(defun foo (mand1 mand2 &optional opt1
                          opt2
            &rest args
            &key key1 key2
              (key3 val3) &aux aux1
                            aux2)
  do-this)

The difference is that keyword parameters are indented with respect to their corresponding keyword. The amount of indentation is provided by a new customizable user option named lisp-lambda-list-keyword-parameter-indentation (oh boy, what a mouthful). If you prefer, you can also have the parameters vertically aligned with each other. Set the new customizable user option named lisp-lambda-list-keyword-parameter-alignment to t and you will get this instead:

(defun foo (mand1 mand2
            &optional opt1
                      opt2
            &rest args
            &key key1 key2
                 (key3 val3) &aux aux1
                                  aux2)
  do-this)

(defun foo (mand1 mand2 &optional opt1
                                  opt2
            &rest args
            &key key1 key2
                 (key3 val3) &aux aux1
                                  aux2)
  do-this)

Finally, just as you could align keyword parameters together, you can also align the keywords together. Set the new customizable user option named lisp-lambda-list-keyword-alignment to t, and you will get this (only the second example differs):

(defun foo (mand1 mand2 &optional opt1
                          opt2
                        &rest args
                        &key key1 key2
                          (key3 val3) &aux aux1
                                        aux2)
  
  do-this)

(defun foo (mand1 mand2 &optional opt1
                                  opt2
                        &rest args
                        &key key1 key2
                             (key3 val3) &aux aux1
                                              aux2)
  
  do-this)

These are in fact my preferred settings, although both alignment options default to nil. Here is a final example demonstrating how you could format a long lambda-list with plenty of arguments:

(defun foo (mand1 mand2
            &optional
              opt1 opt2 opt3
            &key
              key1 key2 key3
            &aux
              aux1 aux2 aux3)
  do-this)

These indentation problems have been a huge peeve of mine for quite a long time. I hope you will find the changes useful!

File-Local Variables

Tue, 26 Apr 2011 00:00:00 +0000

Update: this document is finalized. It can now be officially referred to as “ ”.

In my lightning talk, I announced the development of a new Common Lisp library called XFormat (update: now actually called FoCus), which provides extensible format strings. A first release of this library is imminent. In this talk, I also mentionned the need for what I called “file-local variables”. A file-local variable is a user-defined special variable that would behave as *package* and *readtable* with respect to *load* and *compile-file*.

I have just created and released a very small library called which provides file-local variables through ASDF. In an ideal world, this library would become obsolete because file-local variables are so simple to implement that they should rather be done as an extension to the Common Lisp Standard, and provided by every Lisp vendor. That is the purpose of a new CDR proposal that I also just published on the discussion mailing list.

Declt 1.0b9 is out

Wed, 23 Feb 2011 00:00:00 +0000

The next edition of , the Documentation Extractor from Common Lisp to Texinfo, is out.

In this release:

Bugfix: rendering the documentation for methods with EQL specializers didn’t work.
Feature: licensing and copyrighting the reference manual is now optional. Licenses currently supported are BSD and GPL.
Declt now generates its own reference manual by default.
Some package infrastructure changes that should remain transparent.

Clarification proposal for CLHS 22.3

Fri, 11 Feb 2011 00:00:00 +0000

Update: this document (in version 1.1) has been submitted and accepted as a new item in the Common Document Repository. It can now be officially referred to as “ ”.

There are some fuzzy corner cases in the way the Common Lisp standard describes the syntax and semantics of format directives. I have prepared a first draft of a clarification proposal for the relevant parts of the standard. I’m giving myself some time for comments and feedback before actually submitting this to the .

Clon 1.0b19 is out

Wed, 09 Feb 2011 00:00:00 +0000

, the Common Lisp / Command-Line Options Nuker, version 1.0b19 has just been released. This version comes with a couple of fixes related to error handling and a switch from the GPL to the BSD license.

Which Open Source License for Lisp?

Tue, 08 Feb 2011 00:00:00 +0000

It’s been a long time since I last thought about this. Software licensing is probably the most dreadful computer science topic to me. So out of boredom, laziness, but also out of a feeling of adherence to the ideas behind the FSF, I used to blindly stick the to most of my code. A couple of things made me rethink about that passive attitude recently though, notably the fact that the viral nature of the GPL is not something that I always want.

Even in Lisp, you can make a difference between (more or less) standalone applications and actual language extensions or libraries that other people might want to use in their own code. While I still think that the GPL is okay for my applications (as Pascal Bourguignon would put it: “I want to see your code too!”), I find it too restrictive for my libraries. I sympathize with the FSF concerns, and we probably need something so idealistic to make the open source culture progress, but I can also understand why people wanting to make a living out of their development “need” to close their source. For example, I consider the few proprietary Common Lisp vendors as my friends, and I would be glad to see them use my libraries in their own products.

The other thing with the GPL is that it is long, difficult to read and gives me a feeling of over-complication. Probably just me and my allergy to software licensing issues, but on the other hand, it is notorious that the GPL doesn’t fit very well with Lisp, for the same reasons that it doesn’t fit very well with C++ code constituted mostly of templates. In those environments, the distinction between static or dynamic linking, compile-time or run-time, etc., is blurred to a considerable extend, to the point that it doesn’t make much sense anymore.

Choices

So I was ready to spend, say, 30 minutes looking for a non-viral alternative to the GPL. The first one that comes to mind is of course the , but it looks even more complicated than the GPL itself, and as misfitted to Lisp as its mother license. To the point that some people at created the “Lisp Lesser GNU General Public License”, a.k.a. the LLGPL. This license is in fact constituted with the original LGPL plus a aimed at clarifying or precising some terms used in the license itself, in the context of Lisp. This preamble left me quite sceptical. While it seems to indeed clarify a few points, I think it also adds to the confusion. For instance, it says at some point:

Since Lisp only offers one choice, which is to link the Library into an executable at build time […]

which I think is confusing. What about a Lisp application that dynamically loads an ASDF system. Shouldn’t this be considered as very close to dynamic linking? So no. I want something simpler, widely used and accepted. This leaves me basically with , or style, plus that Daniel Herring suggested on comp.lang.lisp.

The MIT license is probably too liberal for me. It is essentially a big disclaimer but that’s all. Also, some words are frightening. For instance, what does “sublicensing” exactly mean in this context? The Boost one is nice, apart from one thing: it makes acknowledgments optional for binary distribution, and I don’t like that. It seems to me that the least you can do when you’re using a software component written by someone else is to mention it. It will also help people interested by the feature in question to find its source.

So that leaves me with BSD which I decided to use with a bit of rewording and one specific part removed: the “All rights reserved” part. I think it doesn’t make any sense since the purpose of the license itself is to grant you some rights.

Acknowledgments

Even though I find that acknowledgments are important, I don’t want it to be a burden on the shoulders of my “clients”. One simple way to ease their life is to include the complete license as a header in every single file of your library. The BSD license is so short that it fits nicely in headers (actually, the whole license doesn’t take much more space that the GNU GPL header alone). For binary distribution, I also include a LICENSE file in my libraries, so that people can just copy it into their own packages and be compliant.

Texinfo Files

As for Texinfo files, the copyright material needs some adjustment with respect to the original GNU recommendation. Most notably, I don’t need the FSF to approve translations of the copyright notice anymore. I find that the following template fits my needs nicely:

@copying
@quotation
Copyright @copyright{} @value{COPYRIGHT_DATE} Didier Verna

Permission is granted to make and distribute verbatim copies of this
manual provided the copyright notice and this permission notice are
preserved on all copies.

@ignore
Permission is granted to process this file through TeX and print the
results, provided the printed document carries a copying permission
notice identical to this one except for the removal of this paragraph
(this paragraph not being relevant to the printed manual).
@end ignore

Permission is granted to copy and distribute modified versions of this
manual under the conditions for verbatim copying, provided also that the
section entitled ``Copying" is included exactly as in the original.

Permission is granted to copy and distribute translations of this manual
into another language, under the above conditions for modified versions,
except that this permission notice may be translated as well.
@end quotation
@end copying

So that’s it. You can expect most of my Lisp libraries to be released under a BSD style license now.

Clon 1.0b18 is out

Tue, 25 Jan 2011 00:00:00 +0000

, the Common Lisp / Command-Line Options Nuker, version 1.0b18 is now out. Compared to the previous version, the only small bit of change is the fact that the CLISP implementation now depends only optionally on CFFI, whereas that dependency was mandatory before. Doing this is actually quite simple but raised the question of optional ASDF system dependencies, a topic on which there’s quite a bit to be said.

A bit of context first. Clon has a terminal autodetection feature which allows it to automatically highlight its output on a TTY (see --clon-highlight=auto) and also compute the output line width in case the COLUMNS environment variable is not set. This feature requires an IOCTL call which is beyond Lisp. Since one of my goals is to have as few dependencies as possible (zero being the ideal number), I looked at compiler-specific solutions to this problem. The result is that SBCL, CMU-CL, CCL and ECL provide solutions almost out-of-the-box: SBCL has a grovel contributed module, CMU-CL and CCL already have system bindings and ECL has its own FFI interface. The ABCL port doesn’t support terminal autodetection yet, but that’s another (Java) story. The only black sheep in the picture is CLISP which, as far as I can tell, neither has a native binding for IOCTL, nore any built-in grovel facility, hence the need for CFFI.

Previously, my system definition file was like that:

#+clisp (asdf:operate 'asdf:load-op :cffi-grovel)

(asdf:defsystem :com.dvlsoft.clon
  #| ... |#
  :depends-on (#+clisp :cffi)
  :components ((:file "package")
	#+clisp
	(:module "clisp"
	  :depends-on ("package")
	  :components ((cffi-grovel:grovel-file "constants")))
	(module "src"
	  :depends-on ("package" #+clisp "clisp")
	  :components #| ... |#)))

And then later, I had a file with the concerned utility function:

(defun stream-line-width (stream)
  #+clisp (#| CLISP implementation |#)
  #| etc. |#)

After that, I started looking at ways to make the dependency on cffi optional. After all, it makes sense to avoid that dependency at the cost of not being able to autodetect terminals.

One thing I fell on almost by accident is ASDF’s :weakly-depends-on keyword. Here is an interesting about it. It was an accident because that feature is 🙁. People, however, have very mitigated feelings about it, as shown in other thread (syntax and semantics both appear somewhat shaky and underspecified). Besides, the aparent behavior is that if A weakly depends on B and B is not present, then A is operated anyway. So I could probably have my “src” module weakly depend on the “clisp” one, but that doesn’t change the fact the “clisp” module should not be included at all if CFFI is not there.

In the same thread, somebody proposes another kind of dependency called :contigent-on which looks closer to what I would need for the “clisp” module: a module the contingency of which is not present will not be processed. This new kind of dependency doesn’t seem to be implemented yet, however.

So, all of this seems a bit frightening. Too borderline for my taste. Fortunately, my solution is much simpler, although probably not universal.

This first thing to do is try to load cffi-grovel:

(eval-when (:load-toplevel :execute)
  #+clisp
	(handler-case (asdf:operate 'asdf:load-op :cffi-grovel)
	  (asdf:missing-component ()
		(format *error-output* "~
*********************************************************************
* WARNING: ASDF component CFFI-GROVEL not found.
* Clon will be loaded without support for terminal autodetection.
* See section A.1 of the user manual for more information.
*********************************************************************"))))

After that, if loading it was successful, we end up with cffi as a feature, so we can just conditionalize on that:

(asdf:defsystem :com.dvlsoft.clon
  #| ... |#
  :depends-on (#+(and clisp cffi) :cffi)
  :components ((:file "package")
	#+(and clisp cffi)
	(:module "clisp"
	  :depends-on ("package")
	  :serial t
	  :components ((cffi-grovel:grovel-file "constants")))
	(module "src"
	  :depends-on ("package" #+(and clisp cffi) "clisp")
	  :components #| ... |#)))

One last problem remains however: what to do in the source code, for the feature-dependent parts. Conditionalizing an ASDF system may indeed lead to trouble: for instance, what would happen if the function stream-line-width was compiled with CFFI around, and later used in a context where it is not? To be on the safe side, what you really need is to check for the feature dynamically. One possible solution is this:

move all feature-dependent code to the “clisp” module and make that a protocol,
everytime you need to access the feature, dynamically check whether the protocol functions are fbound.

In my specific case, what I did was to implement a CLISP-specific version of stream-line-width, called clisp/stream-line-width and put it in a new file in the “clisp” module, now defined as follows:

	#+(and clisp cffi)
	(:module "clisp"
	  :depends-on ("package")
	  :serial t
	  :components ((cffi-grovel:grovel-file "constants")
				   (:file "util")))

Then, the original function is rewritten like this:

(defun stream-line-width (stream)
  #+clisp (when (fboundp 'clisp/stream-line-width)
			(clisp/stream-line-width stream))
  #| etc. |#)

So now I think I’m on the safe side, and Clon has zero mandatory dependency again…

Clon 1.0b17 is out

Mon, 24 Jan 2011 00:00:00 +0000

I’m happy to announce the release of , the Common Lisp / Command-Line Options Nuker, version 1.0b17. This version notably introduces support for ABCL and expands the documentation in the portability section.

Towards ABCL Standalone Executables

Sat, 22 Jan 2011 00:00:00 +0000

UPDATE: ABCL now supports the POSIX-compliant use of --, so I’ve modified this article to reflect that. I’ve also added a final trick to emulate the existence of a proper argv[0].

Creating standalone executables in Common Lisp is already a portability minefield, as the standard doesn’t specify anything in that regard. The situation is even worse with ABCL because it is a Java-based implementation of Common Lisp, and Java has no notion of standalone executable at all. In this article, we will try to figure out how far we can go in that direction nevertheless.

This work comes from my wish to port to ABCL. In order to make it work, ABCL itself needed some changes, so if you want to try out the code provided below, beware that you need at least revision 13156 from the ABCL Subversion trunk (or version 0.24.0).

Let’s start by writing a great Common Lisp application for ABCL, and call it greet.lisp.


(defun main ()
  (format t "Hello,~{ ~A~}!~%" extensions:*command-line-argument-list*)
  (extensions:exit :status 0))

(main)

As you guessed, extensions:*command-line-argument-list* stores the list of command-line options passed to the program. Now, let’s try it out.

$ abcl --noinform --noinit --nosystem --load greet.lisp John Doe

Hello, John Doe!

So far so good: ABCL-specific options were eaten by ABCL. Not very standalone however. Let’s create a wrapper script around that, and call it greet:

#! /bin/sh

exec abcl --noinform --noinit --nosystem --load greet.lisp ${1+$@}

In fact, we have a problem:

$ ./greet --help me

Parameters:

--help displays this help
--noinform suppresses the printing of version info
--eval <form> evaluates the <form> before initializing REPL
--load <file> loads the file <file> before initializing REPL
--load-system-file <file> loads the system file <file> before initializing REPL
--batch enables batch mode. The --load, --load-system-file and --eval
  parameters are handled, and abcl exits without entering REPL 
--noinit suppresses loading a .abclrc startup file
--nosystem suppresses loading the system startup file

Woops! We were expecting to get Hello, --help me! weren’t we? In other words, our own command-line options may conflict with ABCL’s which is not nice. One thing we can do with a recent ABCL version is to use -- to separate ABCL’s own command-line arguments from ours. Our script becomes:

#! /bin/sh

exec abcl --noinform --noinit --nosystem --load greet.lisp -- ${1+$@}

And we now get this:

$ ./greet --help me

Hello, --help me!

This is better, although we’re still not very standalone: our Lisp file is a separate beast, and the abcl thing is also just a wrapper script (see below). If we want to be more standalone, we need to go the Java way.

The first thing to understand is that there’s actually no such thing as an ABCL executable. ABCL is packed into a JAR (Java archive) file, a zip file, really, and what the wrapper does is simply to call java -jar /path/to/abcl/abcl.jar options…. The JAR file contains a Main class with a main function the purpose of which is to launch a Lisp interpreter. This is done by calling the method Interpreter.createDefaultInstance, which, amongst other things, handles the command-line options. There is also a method called Interpreter.createInstance which does not handle the command-line at all, so it seems that this is the one we could use, provided that we find a way to set extensions:*command-line-argument-list* manually.

The following code does exactly that. Let’s call it Greet.java.

import org.armedbear.lisp.*;

public class Greet
{
  public static void main (final String[] argv)
  {
	Runnable r = new Runnable ()
	{
	  public void run()
	  {
	    try
		{
		  LispObject cmdline = Lisp.NIL;

		  for (String arg : argv) cmdline = new Cons (arg, cmdline);
		  cmdline.nreverse ();
		  Lisp._COMMAND_LINE_ARGUMENT_LIST_.setSymbolValue (cmdline);
		  Interpreter.createInstance ();
		  Load.load ("greet.lisp");
		}
	    catch (ProcessingTerminated e)
		  {
		    System.exit (e.getStatus ());
		  }
		}
	  };

	new Thread (null, r, "interpreter", 4194304L).start();
    }
}

Our Greet class mimics what ABCL’s Main class does, only with some tweaks of our own:

We build the value of extensions:*command-line-argument-list* manually from the actual argv passed to the function main, by consing equivalent Lisp objects and then reversing the list. That way, we completely bypass ABCL’s options processing code.
As mentioned earlier, we use Interpreter.createInstance instead of Interpreter.createDefaultInstance.
We also take the opportunity to load our lisp file directly with the function Load.load.

Finally, the try/catch construct intercepts calls to extensions:exit and effectively leaves the Java environment.

Let’s compile this beast now.

$ javac -cp  /path/to/abcl/abcl.jar Greet.java

The -cp option to javac, the Java compiler, tells it where to find ABCL classes. This command gives us a Greet.class file, but also a second one named Greet$1.class. That’s because our code creates an “anonymous inner class”, whatever that means, in the Java jargon. Now, how do we bypass ABCL’s main function and use or own instead? This is actually very simple. The Java command-line lets you specify (read: override) the name of the class in which to find the main function. So it turns out that we don’t even have to modify ABCL itself for this to work. We can simply go:

$ java -cp  /path/to/abcl/abcl.jar:. Greet John Doe

Hello, John Doe!

and it works. By the way, we can also go:

$ java -cp  /path/to/abcl/abcl.jar:. Greet --help me

Hello, --help me!

and see that it still works. Since ABCL doesn’t process its command-line anymore, there’s no question about -- or whatever. Note that this time, we need to add the current directory to the class path option in order for Java to find the files Greet.class and Greet$1.class. Note also that with our custom interepreter, there’s no more need for all those “standalone” options like --noinit, --noinform etc. Wrapping this in a shell script is left as an exercise, although not a difficult one.

Still, we’re not quite standalone yet. For example, it’s not cool to have these two Greet class files floating around like this. All other class files are in the ABCL JAR file, so we can think of adding ours in too. If we want to do that, we also need to state that the main class in the JAR file should be our Greet class, not ABCL’s Main one. This can be done by modifying the so-called JAR ‘‘manifest’’.

Let’s create a manifest file first and call it Greet.txt.

Main-Class: Greet

It says what it says: our application’s Main class (where the main function is) is Greet. Now let’s make a copy of abcl.jar and add our class file to it:

$ cp /path/to/abcl/abcl.jar greet.jar
$ jar umf Greet.txt greet.jar Greet.class Greet$1.class
Jan 22, 2011 5:50:32 PM java.util.jar.Attributes read
WARNING: Duplicate name in Manifest: Main-Class.
Ensure that the manifest does not have duplicate entries, and
that blank lines separate individual sections in both your
manifest and in the META-INF/MANIFEST.MF entry in the jar file.

The u stands for “update”, the m stands for “manifest” and the f stands for target (JAR) “file”. The manifest and JAR files must appear in the same order as their corresponding options. Don’t worry about the warning. That’s because we’re changing the Main class (ABCL had one already).

Let’s try it out.

$ java -jar  greet.jar John Doe

Hello, John Doe!

Cool. But what about the Lisp file? It’s still floating around. As a matter of fact, the JAR file may contain anything you want, not just class files. ABCL itself has a bunch of lisp files in its JAR file already. So we can put it in there just as well:

$ jar uf greet.jar greet.lisp

The last question is: how to have ABCL load a lisp file that’s in its own JAR? The answer is: dig into the code and figure out that there is a method called Load.loadSystemFile which does just that. ABCL uses it to load its own boot.lisp for instance. We hence need to modify our main function and replace the line that calls Load.load with this one:

Load.loadSystemFile ("/greet.lisp", false, false, false);

Note the leading slash. It is needed because we have stored greet.lisp at the root of the JAR file. Let’s update our JAR file once again:

$ javac -cp /path/to/abcl/abcl.jar Greet.java
$ jar uf greet.jar Greet.class Greet$1.class

And voilà! What you get this time is your application completely packed up in the JAR file. You can move this file around, distribute it, … everything’s inside.

Here’s a final additional trick and I think we’re done. As you know, ABCL stores the command-line argument list in extensions:*command-line-argument-list*. One potential problem is that there’s no notion of a program name (argv[0]) in ABCL (the program is in fact always Java). Since we’re going standalone, however, maybe we would like to have that. Something like, say, a variable named extensions:*argv0*. So here’s a way to do it does this as part of its built-in dump facility): in the Greet class, replace the line

Interpreter.createInstance ();

with this:

Interpreter interpreter = Interpreter.createInstance ();
interpreter.eval ("(defvar extensions::*argv0* \\"greet\\")");
interpreter.eval ("(export 'extensions::*argv0* 'extensions)");

So that’s it I guess. It seems to me that this is as far as you can go in the direction of standalone ABCL executables. Of course, this example is overly simple. We didn’t try to incorporate full ASDF systems in the archive, and stuff. But you get the idea…

Clon 1.0b16 is out

Tue, 30 Nov 2010 00:00:00 +0000

I’m happy to annouce the release of , the Common Lisp / Command-Line Options Nuker, version 1.0b16. Most notable news for this release is the support for CLISP, which comes at the expense on one dependency (over CFFI).

Clon 1.0b15 is out

Thu, 18 Nov 2010 00:00:00 +0000

I’m happy to annouce the release of , the Common Lisp / Command-Line Options Nuker, version 1.0b15. Most notable news for this release is the support for ECL.

Clon 1.0b12 is out

Thu, 11 Nov 2010 00:00:00 +0000

I’m happy to annouce the release of , the Common Lisp / Command-Line Options Nuker, version 1.0b12. Most notable news for this release is the support for CCL.

Clon 1.0b11 is out

Tue, 09 Nov 2010 00:00:00 +0000

I’m happy to annouce the release of , the Common Lisp / Command-Line Options Nuker, version 1.0b11. Most notable news for this release is the support for CMU-CL.

Announcing Declt, a Documentation Generator for Common Lisp Libraries

Tue, 21 Sep 2010 00:00:00 +0000

I’m happy to announce the first public release of , the “Documentation Extractor from Common Lisp to Texinfo”.

Declt (pronounce “dec’let”) is a reference manual generator for Common Lisp libraries. It extracts and formats documentation from ASDF systems, including the system itself and its components, the packages defined in the system and definitions like constants, special variables, macros, functions, generic functions and methods, conditions, structures and classes.

Reference manuals are generated in Texinfo format which can be subsequently converted into info, HTML, DVI, PostScript or PDF. The generated manuals are fully indexed and provide a complete set of cross-references between documentation elements. For instance, files and packages point to the definitions they provide, and those definitions point back to package and file in which they can be found.

Declt requires SBCL to work. If you’re interested in porting it, grep PORTME in the sources. Also, ASDF 2 has not been tested yet.

Announcing Clon, the Common Lisp / Command-Line Options Nuker

Tue, 31 Aug 2010 00:00:00 +0000

I’m happy to announce the first public release of , the Common Lisp / Command-Line Options Nuker, the Common Lisp / Command-Line Options Nuker library for Common Lisp.

Clon is a library for managing command-line options in standalone Common Lisp applications. It provides a unified option syntax with both short and long names, automatic completion of partial names and automatic retrieval/conversion of option arguments from the command-line, associated environment variables, fallback or default values. Clon comes with a set of extensible option types (switches, paths, strings etc.). Clon also provides automatic generation and formatting of help strings, with support for highlighting on TTYs through ISO/IEC 6429 SGR. This formatting is customizable through “themes”.

Clon currently requires SBCL to work. If you want to help porting it to other implementations, please grep PORTME in the sources.

:o( Smilisp :o) A new and revolutionary dialect of Lisp

Wed, 01 Apr 2009 00:00:00 +0000

Today, I just released ☹️SmiLisp🙂, a new dialect of Lisp featuring a novel programming paradigm called “Emotional Programming”. This is truely a revolution in the Lisp world… The logo is not exactly like that, but my blogging software wants to put smileys everwhere…

Get it .

Lisp, Jazz, Aikido

Mon, 08 Sep 2008 00:00:00 +0000

This is something I originally posted on my , but I’m reposting it here because, as you will figure out pretty quickly, it belongs in here as much as in my Jazz website…

What can computer science, music and martial arts possibly have in common? These are like “strange attractors” in my life: no matter how much distance I may have put between them and me in the past, I always ended up coming back to them, and I know this will remain the case in the future.

I can remember pretty well the excitement I felt when I discovered the Lisp language, when I was first introduced to Jazz and improvisation, and when I had my first Aikido practice session. Different things, same emotion. And also the feeling that in some way, I was born to be a lisper, a jazzman and an aikidoka. I just didn’t know until then.

Recently I was talking about my half-scientific / half-musical life with an old teacher of mine, and he asked me if I had suddenly turned completely schizophrenic. He was right! To ask, I mean… So I started thinking about it, and I tried to figure out what these three domains have in common and why they all adhere so well to my own philosophy of life.

But perhaps I should begin with explaining what’s my philosophy of life, then. I guess it’s basically described in three words: Beauty, Fun, Unification.

Beauty lies in being able to evolve comfortably within a set of constraints, limits or rules. Note that this begins with accepting the existence of these constraints, limits or rules in the first place.
Fun, however, lies in breaking those rules at will, knowing how to get rid of them, and then get back to them, a bit like a cat jumping in any kind of direction and yet always falling back on his feet.

There’s a corollary to these two points: real freedom is not to have no limit, but to know your limits so well that you can either evolve at will within them, or break them at will.

Unification means drawing bridges between apparently unrelated fields, starting to figure out what is the common essence of things. By the way, this is precisely what I am doing right now… 😉

Why is Lisp beautiful, fun and unifying ?

There’s beauty in writing code in any language (yes, there’s even beauty in writing shell code). The beauty lies in your ability to adapt your concepts to the constraints of the language you’re using, in other words, to make the best out of it, given its inherent limitations in expressiveness.
However, and this is where the fun lies, Lisp allows you to break the rules of traditional languages because you can adapt the language to your concepts as much as you need to adapt your concepts to the language. Lisp is known (or at least, it should be) as the “programmable programming language”: thanks to the power of its macro system and the customizability of its reader for instance, you can create a completely new language (even with a completely new syntax, see the loop macro for instance) within Lisp and adapt it to your personal needs. This makes Lisp the language of choice for implementing a DSL (Domain Specific Language) for instance.

It is interesting to note that with some experience in Lisp, adapting the language to your needs becomes an integral part of the art of programming; a rule in itself. In this way, what you do is really pushing the limits farther away, making a rule of what was an exception before.

Lisp is also the language of unification. While it is mainly known to be a functional language (pure or impure, by the way), it is also imperative, procedural, object-oriented and even context-oriented if you want it to be. It is also declarative: look at the abundant literature on how easy it is to implement Prolog in Lisp. So Lisp really is any kind of language you want it to be: where a particular programming paradigm exists by construction in another language, it is usually implemented as a mere library in Lisp. Most recent (and fashionable) programming languages today are just re-discovering things that existed in Lisp since its invention in the 60’s.

Why is Jazz beautiful, fun and unifying ?

There’s beauty in playing a song, in any kind of music. The beauty lies in your ability to adapt your personal musical ideas, your way of playing, in other words, your musical personality, to the constraints of the song. People often forget that a song is by definition (or, so to speak, by composition) limited in expressiveness, just as an average programming language is: it has a pitch, a tempo, a rhythmic style, a chord progression; all things actually specified by the score. In traditional music, you are expected to evolve within these limits.
However, and this is where the fun lies, Jazz (specifically improvisation) allows you to break the rules of traditional music by playing “out”, both harmonically and rhythmically. Improvisation is by essence the musical practice that allows you to modify a score in real time: you can change the ambiance, the chords, the rhythm, you can temporarily escape from the song and then get back to it (remember the cat ?), even play “out” (roughly meaning using scales that do not correspond to the underlying chords). During this process, you are actually adapting the song to your musical concepts instead of adapting your musical concepts to the song. This is exactly like adapting Lisp to your programming needs instead of adapting your needs to the language. And when the other musicians follow you on this “song tweaking game”, that’s were the fun really begins!

It is interesting to note that with some experience in jazz, playing “out” becomes an integral part of the art of improvisation; a rule in itself. When Miles Davis started to mix major and minor harmony (for instance using a minor 3rd on a major chord), numerous conservator alligators wanted to burn the heretic alive. Now, all of this is well known, and you can learn actual techniques for chords substitution and atonal improvisation in Jazz schools. Again, in this way, what you do is really pushing the limits farther away, making a rule of what was an exception before.

Jazz is also the music of unification. A very narrow view of it is as a musical style: the ternary “chabada” drums pattern, the walking bass and so on. But Jazz is really not that. It is a philosophy, a way to envision all styles of music. Michel Petrucciani once said “Jazz is a music of thieves”, and he was right ! A Jazz musician is fundamentally curious. He’s interested in everything he can ear, and tries to appropriate all the ideas he’s exposed to by adapting them to his own personality. This is a process that happens in composition as well as in improvisation, but improvisation is the key factor that unifies all musical styles in Jazz. So just like Lisp unifies all programming concepts into a philosophy of programming, Jazz unifies all music styles into a philosophy of music.

Why is Aïkido beautiful, fun and unifying ?

There’s beauty in practicing any martial art (as long that it has not become just sports). The beauty lies in executing the techniques that define your martial art to the perfection. But one has to understand that as long as you are practicing only a set of techniques (which is a very narrow view of martial arts), you are evolving in a very limited environment, however beautiful (and Aikido is aesthetically beautiful).
The fun begins when you start to understand the Budo which is behind martial arts, and especially behind Aikido. The Budo is a philosophy, just as Jazz or Lisp are philosophies. By emphasizing on values such as personal cultivation, self-control and self-awareness, the Budo renders techniques unimportant, or at least secondary, because being able to react in any situation is more important than the way you react to them. Techniques are just tools to reach a greater goal. Aikido masters are so far away beyond technique that you can’t see them anymore in their movements. The rules are broken, the techniques are gone. What’s left is a “purified” state; what’s left is the Ki.

It is interesting to note that in Aikido, breaking the rules, that is, avoiding being enslaved by technique is a constant preoccupation. For example, in Aikido, there is no distinction between beginners and experienced practitioners. Mixing levels in practice is one of the ways of ensuring that you will constantly face new and unexpected situations. As such, breaking the rules has also become a rule in itself. The ultimate rule-breaking exercise in Aikido is probably the Randori. When faced with 2 or 4 adversaries simultaneously, there is no room for rules or techniques, you just have to react. This is exactly like improvising in Jazz: in real time, there is no room for rules or harmony analysis, you just have to play.

Aikido is also the martial art of unification. At least in two ways. On the technical plan, we know that Morihei Ueshiba, the father of Aikido, was a master in several martial arts (both “soft”, like ju-jutsu, and “hard”, like ken-jutsu or jutte-jutsu) when he founded his own. As such, Aikido unifies martial arts in general by incorporating techniques from many different sources. This is also pointed out by the fact that Jigoro Kano, founder of Judo, sent his best students to learn Aikido. On a spiritual plan, O Sensei’s interpretation of the Budo not only encompasses the traditional meaning of the term (notably including personal cultivation) but extends to such notions as love and protection of all things, respect for all lives (see his “revelation” in spring 1925), which are much more universal concerns.

Conclusion

Lisp, Jazz and Aikido are more than kinds of programming, music, or martial arts. They are philosophies of programming, music and martial arts. More than that, they are actually different appearances of the same philosophy of life. There is much more to say about it. There are also links to establish with scientific research in general. But not in a blog… I’d like to write a proper essay about these things when I find the time… someday.

CCL Objective-C Bridge Code Indentation

Tue, 01 Jul 2008 00:00:00 +0000

A whole bunch of CCL Cocoa coding involves calls to Objective-C methods. For instance, the following Objective-C code:

[cell drawWithFrame: frame inView: view];

will translate into this in CLL:

(#/drawWithFrame:inView: cell frame view)

Indeed, Objective-C is designed around the record-based model (methods belong to classes), so the cell object receives the drawWithFrame:inView: message via its Smalltalk descendant syntax. On the other hand, the Lisp translation involves a generic function call with the usual funcall syntax. The dispatch occurs on the first argument (the cell), and the rest is actual arguments to the message.

This layout is obviously not optimal for readability. One could define a bracket reader-macro to simulate Objective-C’s message passing syntax in Lisp, but I prefer to stick to the “Lisp Way”. The first argument should still be considered “special” though.

For readability (especially when the method name is long), I like to put the arguments to the message (not the receiver!) on subsequent lines. However, Emacs’s cl-indent-function will indent this as an ordinary function call, like this:

(#/drawWithFrame:inView: cell
                         frame view

which is awfull. What I want is the following indentation:

(#/drawWithFrame:inView: cell
  frame view

This kind of indentation is normally achieved by putting a common-lisp-indent-function property of 1 on the function symbol. However, you don’t want to do that on all possible Objective-C message by hand (and you don’t know them all in advance anyway). The following advice on common-lisp-indent-function (from the cl-indent package does the trick. It dynamically puts the property on each relevant symbol every time it is subject to indentation. A bit brute force, but it works smoothly.

(defadvice common-lisp-indent-function (before ccl-objc-bridge activate)
  "Improve indentation scheme of the CCL Objective-C bridge.
Currently, this does the following:

- All (#/doThis:withThat: ...) forms are indented as per a
lisp-indent-function property of 1. This effectively treats the first argument
(an object or a class) as special. The indentation you get is:

(#/function arg1        instead of:        (#/function arg1
  arg2 ...)                                            arg2 ...)"
  (let ((containing-form-start (elt state 1)))
    (save-excursion
      (goto-char containing-form-start)
      (forward-char 1)
      (cond ((looking-at "#/\\\\(\\\\w\\\\|:\\\\)*:")
	     ;; We're looking at a (#/doThis:withThat: ...) form. In its holy
	     ;; brokenness, common-lisp-indent-function with the help of
	     ;; parse-partial-sexp will consider that the function name in
	     ;; this form is "/functioncall:". Our trick here is to
	     ;; dynamically put a lisp-indent-function property of 1 on this
	     ;; symbol, so that the subsequent (original) indenting function
	     ;; will handle it.
	     (let* ((beg (progn (forward-char 1) (point)))
                (sym (progn
                       (forward-sexp 1)
                       (intern (downcase (buffer-substring beg (point)))))))
	       (put sym 'common-lisp-indent-function 1)))))))

One last thing and I’ll be happy. I want to indent CCL’s slet and slet* constructs just as let:

(put 'slet 'common-lisp-indent-function 
     '((&whole 4 &rest (&whole 1 1 2)) &body))
(put 'slet* 'common-lisp-indent-function
     '((&whole 4 &rest (&whole 1 1 2)) &body))

And voilà!

News | Didier Verna's Scientific Website

Clon 1.0b27 is out

Clon 1.0b26 is out

Quickref 4.0b1 is out

Declt 4.0b1 is out

Backward Incompatible Changes

New Features

Default / Standard Values

Support for Aliases

Support for New Programmatic Entities

Proper Support for Uninterened Symbols

Requirement for SBCL 2.1.2

Domestic vs. Foreign Definitions

Introspection Heuristic

Supported Licenses

Info Installation Category

Improvements

Documentation Thinning

Merging

Lambda Lists

Method Combinations

Bug Fixes and Workarounds

ASDF

Anchoring

Setf Expanders

Clon 1.0b25 is out

Quickref 3.0 "The Alchemist" is out

Declt 3.0 "Montgomery Scott" is out

Quickref 2.0 "Be Quick or Be Dead" is out

Quickref 1.0 "The Trooper" is out

Lisp, Jazz, Aikido, 10 years later

Announcing Quickref: a Global Documentation Project for Common Lisp

Declt 2.3 "Robert April" is out

Standard IO syntax and the Robustness Principle

Explanation

Consequences

Declt 2.2 "Christopher Pike" is out

Declt 2.1 "Jonathan Archer" is out

Declt 2.0.1 "Benjamin Sisko" is out

ASDF-FLV 2.0

The Return of Segfaults

Declt 2.0 "Kathryn Janeway" is out (important)

Declt 1.1 "Jean-Luc Picard" is out

Clon 1.0b24 is out (important)

Declt 1.0 "James T. Kirk" is out

Lisp Corner Cases: Method Combinations

Nice little trick du jour

Declt 1.0b15 is out

Clon 1.0b23 is out

Declt 1.0b14 is out

Language wars

Declt 1.0b13 is out

Clon 1.0b22 is out

The Syntax Extension Myth

Comments

From Pascal Bourguignon

From Eivind Midtgård

From Greg Pfeil

From DanM

From me

From Jerry Boetje

From Marc Espie

Clon 1.0b21 is out

Clon 1.0b20 is out

At the end-user level

At the user-level

CL-RCFiles 2.0 is out

One More Indentation Hack

Declt 1.0b12 is out

Read-Time String Concatenation

Declt 1.0b11 is out

Common Lisp indentation in Emacs

Method qualifiers

Lambda-Lists

File-Local Variables

Declt 1.0b9 is out

Clarification proposal for CLHS 22.3

Clon 1.0b19 is out

Which Open Source License for Lisp?

Choices

`Setf` Expanders