Create a scatter plot specification comparing horsepower and miles per
(vega:defplothp-mpg`(:title"Horsepower vs. MPG":description"Horsepower vs miles per gallon for various cars":data,vgcars:mark:point:encoding (:x (:field:horsepower:type:quantitative)
1 - Installation
Installing and configuring Lisp-Stat
New to Lisp
If you are a Lisp newbie and want to get started as fast as possible,
then Portacle is your best option. Portacle is a multi-platform IDE
for Common Lisp that includes Emacs, SBCL, Git, Quicklisp, all
configured and ready to use.
We assume an experienced user will have their own Emacs and lisp
implementation and will want to install according to their own tastes
and setup. The repo links you need are below, or you can install with
clpm or quicklisp.
All that is needed is an ANSI Common Lisp implementation. Development
is done with Genera and SBCL. Other platforms should work, but will
not have been tested, nor can we offer support (maintaining & testing
on multiple implementations requires more resources than the project
has available). Note that CCL is not in good health, and there are a
few numerical bugs that remain unfixed. A shame, as we really liked
The easiest way to install Lisp-Stat is via
Quicklisp, a library manager for
Common Lisp. It works with your existing Common Lisp implementation to
download, install, and load any of over 1,500 libraries with a few
Quicklisp is like a package manager in Linux. It can load packages
from the local file system, or download them if required. If you have
quicklisp installed, you can use:
Quicklisp is good at managing the project dependency retrieval, but
most of the time we use ASDF because of its REPL integration. You only
have to use Quicklisp once to get the dependencies, then use ASDF for
You can install additional Lisp-Stat modules in the same way. For example to install the SQLDF module:
Once you have obtained Lisp-Stat via Quicklisp, you can load in one of two ways:
Loading with ASDF
If you are using emacs, you can use the slime
load systems by typing , and then load-system in the mini-buffer.
This is what the Lisp-Stat developers use most often, the shortcuts
are a helpful part of the workflow.
Loading with Quicklisp
To load with Quicklisp:
Quicklisp uses the same ASDF command as above to load Lisp-Stat.
When a new release is announced, you can update via Quicklisp like so:
There are a couple of IDE’s to consider:
Emacs, with the slime
package is the most tested IDE and the one the authors use. If you
are using one of the starter lisp packages mentioned in the getting
started section, this will have
been installed for you. Otherwise, slime/swank is available in
quicklisp and clpm.
This is a very popular IDE, with improving support for Common Lisp.
If you already use this editor, it is worth investigating to see if
the Lisp support is sufficient for you to perform an analysis.
You can install the info manuals into the emacs help system and this
allows searching and browsing from within the editing environment. To
do this, use the
command. As an example, on my MS Windows 10 machine, with MSYS2/emacs
installs the select manual at the top level of the info tree. You
can also install the common lisp hyperspec and browse documentation
for the base Common Lisp system. This really is the best way to use
documentation whilst programming Common Lisp and Lisp-Stat. See the
and “How do I install a piece of Texinfo
for more information on installing help files in emacs.
See getting help for
information on how to access Info documentation as you code. This is
the mechanism used by Lisp-Stat developers because you don’t have to
leave the emacs editor to look up function documentation in a browser.
You can put customisations to your environment in either your
implementation’s init file, or in a personal init file and load it
from the implementation’s init file. For example, I keep my
customisations in #P"~/ls-init.lisp" and load it from SBCL’s init
file ~/.sbclrc in a Lisp-Stat initialisation section like this:
Settings in your personal lisp-stat init file override the system defaults.
Here’s an example ls-init.lisp file that loads some common R data sets:
"Data sets loaded as part of personal Lisp-Stat initialisation.
Available in every session.")
With this init file, you can immediately access the data sets in the
*default-datasets* list defined above, e.g.:
This manual is organised by audience. The overview
and getting started sections are applicable
to all users. Other sections are focused on statistical practitioners,
developers or users new to Common Lisp.
This part of the documentation contains worked examples of statistical
analysis and plotting. It has less explanatory material, and more
worked examples of code than other sections. If you have a common
use-case and want to know how to solve it, look here.
This section contains tutorials, primers and ‘vignettes’. Typically
tutorials contain more explanatory material, whilst primers are
short-form tutorials on a particular system.
The manuals are written at a level somewhere between an API reference
and a core task. They document, with text and examples, the core APIs
of each system. These are useful references for power users,
developers and if you need to go a bit beyond the core tasks.
The reference manuals document the API for each system. These are
typically used by developers building extensions to Lisp-Stat.
Common Lisp and statistical resources, such as books, tutorials and
website. Not specific to Lisp-Stat, but useful for statistical
practitioners learning Lisp.
This section describes how to contribute to Lisp-Stat. There are both
ideas on what to contribute, as well as instructions on how to
contribute. Also note the section on the top right of all the
documentation pages, just below the search box:
If you see a mistake in the documentation, please use the Create documentation issue link to go directly to github and report the
3 - Getting Help
Ways to get help with Lisp-Stat
There are several ways to get help with Lisp-Stat and your statistical
analysis. This section describes way to get help with your data
objects, with Lisp-Stat commands to process them, and with Common
We use the algolia search engine to index
the site. This search engine is specialised to work well with
documentation websites like this one. If you’re looking for something
and can’t find it in the navigation panes, use the search box:
If you’re not quite sure what you’re looking for, you can use the
apropos command. You can do this either from the REPL or emacs.
Here are two examples:
This works even better using emacs/slime. If you use the slime command sequence C-c C-d a, (all the slime documentation commands start with C-c C-d) emacs will ask you for a string. Let’s say you typed in remove-if. Emacs will open a buffer like the one below with all the docs strings for similar functions or variables:
Restart from errors
Common lisp has what is called a condition system, which is somewhat unique. One of the features of the condition system is something call restarts. Basically, one part of the system can signal a condition, and another part of it can handle the condition. One of the ways a signal can handled with is by providing various restarts. Restarts are handled by the debugger, and many users new to Common Lisp tend to shy away from the debugger (this is common to other languages too). In Common Lisp the debugger is both for developers and users.
Well written Lisp programs will provide a good set of restarts for commonly encountered situations. As an example, suppose we are plotting a data set that has a large number of data points. Experience has shown that greater than 50,000 data points can cause browser performance issues, so we’ve added a restart to warn you, seen below:
Here you can see we have options to take all the data, take n (that the user will provide) or take up to the maximum recommended number. Always look at the options offered to you by the debugger and see if any of them will fix the problem for you.
You can use the describe command to print a description of just
about anything in the Lisp environment. Lisp-Stat extends this
functionality to describe data. For example:
MotorTrendCarRoadTests>Documentation:MotorTrendCarRoadTestsDescriptionThedatawasextractedfromthe1974MotorTrendUSmagazine,andcomprisesfuelconsumptionand10aspectsofautomobiledesignandperformancefor32automobiles (1973–74models).NoteHendersonandVelleman (1981) commentinafootnotetoTable1:‘Hocking[originaltranscriber]'snoncrucialcodingoftheMazda'srotaryengineasastraightsix-cylinderengineandthePorsche'sflatengineasaVengine,aswellastheinclusionofthedieselMercedes240D,havebeenretainedtoenabledirectcomparisonstobemadewithpreviousanalyses.’SourceHendersonandVelleman (1981),Buildingmultipleregressionmodelsinteractively.Biometrics,37,391–411.Variables:Variable| Type |Unit| Label
-------- |----| ---- |-----------MODEL| STRING |NIL| NIL
MPG |DOUBLE-FLOAT| M/G |Miles/(US) gallonCYL| INTEGER |NA| Number of cylinders
DISP |DOUBLE-FLOAT| IN3 |Displacement (cu.in.)
HP| INTEGER |HP| Gross horsepower
DRAT |DOUBLE-FLOAT| NA |RearaxleratioWT| DOUBLE-FLOAT |LB| Weight (1000 lbs)
QSEC |DOUBLE-FLOAT| S |1/4miletimeVS| CATEGORICAL |NA| Engine (0=v-shaped, 1=straight)
AM |CATEGORICAL| NA |Transmission (0=automatic,1=manual)
GEAR| CATEGORICAL |NA| Number of forward gears
CARB |CATEGORICAL| NA |Numberofcarburetors
The documentation command can be used to read the documentation of a function or variable. Here’s how to read the documentation for the Lisp-Stat mean function:
"The mean of elements in OBJECT."
You can also view the documentation for variables or data objects:
"If non-nil the system will ask the user for confirmation
before redefining a data frame"
When Lisp prints an interesting object to emacs/slime, it will be
displayed in orange text. This indicates that it is a presentation, a
special kind of object that we can manipulate. For example if you type
the name of a data frame, it will return a presentation object:
Now if you right click on this object you’ll get the presentation menu:
From this menu you can go to the source code of the object, inspect &
change values, describe it (as seen above, but within an emacs
window), and copy it.
an alternative inspector for emacs, with some additional
Slime documentation provides ways to browse documentation from the editor. We saw one example above with apropos. You can also browse variable and function documentation. For example if you have the cursor positioned over a function:
and you type C-c C-d f (describe function at point), you’ll see this
in an emacs window:
Lambda-list: (&KEY (HEAD NIL) (STREAM *STANDARD-OUTPUT*))
Derived type: (FUNCTION (&KEY (:HEAD T) (:STREAM T)) *)
Print all data frames in the current environment in
reverse order of creation, i.e. most recently created first.
If HEAD is not NIL, print the first six rows, similar to the
Source file: s:/src/data-frame/src/defdf.lisp
Often your project will have sample data used for examples
illustrating how to use the system. Such example data goes here, as
would static data files that your system includes, for example post
codes (zip codes). For some projects, we keep the project data here
too. If the data is obtained over the network or a data base, login
credentials and code related to that is kept here. Basically,
anything neccessary to obtain the data should be kept in this
The lisp source code for loading, cleaning and analysing your data.
If you are using the template for a Lisp-Stat add-on package, the
source code for the functionality goes here.
Generated documentation goes here. This could be both API
documentation and user guides and manuals. If an index.html file
appears here, github will automatically display it’s contents at
project.github.io, if you have configured the repository to display
documentation that way.
Load your project
If you’ve cloned the project template into your local Common Lisp
directory, ~/common-lisp/, then you can load it with (ql:quickload :project). Lisp will download and compile the neccessary
dependencies and your project will be loaded. The first thing you’ll
want to do is to configure your project.
Configure your project
First, change the directory and repository name to suit your
environment and make sure git remotes are working properly. Save
yourself some time and get git working before configuring the project
The project.asd file is the Common Lisp system definition file.
Rename this to be the same as your project directory and edit its
contents to reflect the state of your project. To start with, don’t
change any of the file names; just edit the meta data. As you add or
rename source code files in the project you’ll update the file names
here so Common Lisp will know that to compile. This file is analgous
to a makefile in C – it tells lisp how to build your project.
If you need project-wide initialisation settings, you can do this in
the file src/init.lisp. The template sets up a logical path
To use it, you’ll modify the directories and project name for your
project, and then call (setup-project-translations) in one of your
lisp initialisation files (either ls-init.lisp or .sbclrc). By
default, the project data directory will be set to a subdirectory
below the main project directory, and you can access files there with
PROJECT:DATA;mtcars.csv for example. When you configure your
logical pathnames, you’ll replace “PROJECT” with your projects name.
We use logical style pathnames throughout the Lisp-Stat documentation,
even if a code level translation isn’t in place.
The project templates illustrates the basic steps for a simple
The first step is to load data. The PROJECT:SRC;load file shows
creating three data frames, from three different sources: CSV, TSV and
JSON. Use this as a template for loading your own data.
load.lisp also shows some simple cleansing, adding labels, types and
attributes, and transforming (recoding) a variable. You can follow
these examples for your own data sets, with the goal of creating a
data frame from your data.
PROJECT:SRC;analyse shows taking the mean and standard deviation of
the mpg variable of the loaded data set. Your own analysis will, of
course, be different. The examples here are meant to indicate the
purpose. You may have one or more files for your analysis, including
supporting functions, joining data sets, etc.
Plotting can be useful at any stage of the process. It’s inclusion as
the third step isn’t intended to imply a particular importance or
order. The file PROJECT:SRC;plot shows how to plot the information
in the disasters data frame.
Finally, you’ll want to save your data frame after you’ve got it where
you want it to be. You can save project in a ‘native’ format, a lisp
file, that will preserve all your meta data and is editable, or a CSV
file. You should only use a CSV file if you need to use the data in
another system. PROJECT:SRC;save shows how to save your work.