Lisp-Stat Blog

• News About Lisp-Stat
• Getting Started
• 2023 End of Year Summary
• Introduction to the Practice of Statistics
• Statistical Analysis with Lisp-Stat
• New Releases
• Introducing Quick Plot: ggplot-Style Plotting for Lisp-Stat
• 2022 Release

News About Lisp-Stat

Getting Started

It’s never been easy for a developer to get started in Common Lisp. Emacs, though a powerful editor, isn’t considered an IDE by modern standards. Setting up a compiler, quicklisp, slime, swank, and then learning an entirely new programming paradigm has scared off many would-be entrants to the Common Lisp community.

Given the size of the Common Lisp community this is understandable. Making the new user experience smooth and frictionless as possible is hard work. It’s the kind of work that no one volunteers for; it’s the kind of work you have to be paid for.

Still, it’s a pre-requisite for new users, so I’ve created ls-dev-image, a ‘batteries included’ OCI image for Lisp-Stat or just plain Common Lisp development. Assuming you have an OCI (e.g Docker) runtime installed. Here’s how to get started.

Shell

docker run --rm -it --user vscode -w /home/vscode ghcr.io/lisp-stat/ls-dev:latest bash

Now you’re in the container with a bash shell and can configure the machine as you like.

Common Lisp REPL

If you want a Common Lisp REPL:

docker run --rm -it --user vscode -w /home/vscode ghcr.io/lisp-stat/ls-dev:latest ls-repl

and you should see:

Linedit version 0.17.6, smart mode, ESC-h for help.
CL-USER(1):

You could also have gotten here from the shell with the sbcl command. This is a bare common lisp image, with nothing loaded. Now you load lisp-stat from the Quicklisp repositories with:

(ql:quickload :lisp-stat)
... lot's of compilation output ...
[package ls-user]
(:LISP-STAT)
CL-USER(2):

and to start working with Lisp-Stat:

(in-package :ls-user

This REPL has been configured with a few packages to make it easier to work with.

• linedit, provides customizable line-editing. You can use emacs key bindings to edit the REPL commands.
• Acl-repl is a SBCL extension that gives you command history (via up/down arrows) and some short command codes such as :cs for compile-system, :ts for test-system, etc.

Generally you won’t be doing development with the REPL, you’ll be doing it in emacs.

Emacs/Slime

Now you can use Common Lisp with emacs, quicklisp and slime. From the shell you can type ’emacs’ followed by ‘M-x slime’ and start hacking Common Lisp.

Lisp-Stat

The upstream repositories are often out of date. To get the latest and to keep them synced you can use the init.sh script with ls-init.sh --mode experimenter. Experimenter will download the repos so you can try out the source code, but you won’t be able to push your changes or make pull requests. For that you want to be a ‘contributor’: ls-init.sh --mode contributor.

Now when you start ls-repl you should see the lisp-stat REPL prompt, indicating you’re in the LS-USER package:

ls-init.sh --mode experimenter
# ... output from repo checkouts, linking, configuration

ls-repl

lot’s of recompilation because you’re now using local source repos and then:

...
To load "ls-server":
  Load 1 ASDF system:
    ls-server
; Loading "ls-server"
.....

Linedit version 0.17.6, smart mode, ESC-h for help.
LS-USER(1):

From here emacs and M-x slime (from the shell) will load Lisp-Stat.

LS-Server

A ls-server is also configured to start automatically on port 20202. If you open your browser and point it to https://localhost:20202 (or other port, depending on your OCI container configuration) you’ll see the web interface for displaying plots and viewing/editing data-frames.

You can also run this OCI image in GitHub codespaces.

Staying up-to-date

You will want to run docker pull occasionally to get the latest ls-dev-image. This should stabilize in a month or so and after that it won’t need to be updated frequently. To keep lisp-stat source in sync, run ls-init.sh --refresh. If you want help with ls-init.sh, run ls-init.sh --help.

Contributions and bug reports are welcome and encouraged.

2023 End of Year Summary

We started the year by working through the examples from the first chapter of the Introduction to the Practice of Statistics. Whilst this is an excellent test for Lisp-Stat, Chapter 2 presented some major challanges to the plotting system, so the highlight for this year is Plot 2.0. This is a breaking change, however the fix for plots is relatively simple.

The updates for 2023 include:

• IPS Chapter 1 complete
• IPS Chapter 2 plotting
• Sampling data frames
• Adopted cl-gists so we can work with the Vega eco-system on Github
• Added Lisp Linear Algebra, CFFI wrappers for BLAS and LAPACK
• Improved tutorials, examples and documentation
• Plot 2.0
  • Smoothers for non-parametric regression
  • Plot helpers for qq-plots and scatterplots
  • New data sources: named data sets, URLs, embedded
  • Publish plots to Github gists
  • Edit plots in Vega Edit online editor
  • Fully support multi-layer, multi-data set plotting

Introduction to the Practice of Statistics

The examples from the first chapter of the book are now completed. This chapter focuses mostly on graphics, and the examples here supplement those in the plotting examples.

Statistical Analysis with Lisp-Stat

Lisp-Stat is the culmination of many months work to pull together the best-in-class statistical analysis packages available in Common Lisp, under a commercially friendly license, usable ‘out of the box’. It is an environment for statistical computing, conceptually similar to R, that is also suitable for front-line production deployments. It grew out of a desire to have an environment for rapidly prototyping analytical and A.I. solutions, and move directly to production environments with minimal friction. Typically there is a gap between the modeling environment, perhaps R or Python, and the delivery environment. Lisp-Stat closes that gap.

Lisp-Stat is in daily use for statistical analysis and is now considered ready for a broader audience.

Why Lisp?

We had a few requirements when evaluating options. Specifically the system had to:

• Work well in the kind of exploratory environment conducive to analytics and AI
• Be robust enough to be deployed in an enterprise-level production environment
• Be available under a license without source code restrictions

Common Lisp was the only framework that met all these requirements. And, it is not the first time lisp has been used in a statistical setting. XLISP-STAT, our spiritual predecessor, was a contemporary of R in the early days of development. Wikipedia says about it: “XLispStat was historically influential in the field of statistical visualization” and its author, Luke Tierney, was a member of the original R core team.

What does Lisp-Stat do?

Lisp-Stat provides support for vectorized mathematical operations, a super-set of common lisp array operations, and a comprehensive set of statistical methods that are implemented using the latest numerical algorithms. In addition, Common Lisp provides a dynamic programming environment that includes a REPL for interactive evaluation of code, CLOS, an excellent object oriented environment and a meta-object protocol (MOP).

We have also compiled an archive of XLISP-STAT libraries that can be ported with the aid of a compatibility package (XLS). This gives Lisp-Stat a leg up on ecosystem development; there are many useful statistical routines in the archive.

Getting Started

If you are familiar with R, and want to understand how to do similar things with Lisp-Stat, the basic tutorial is the place to begin. If you have a Common Lisp installation and want give Lisp-Stat a quick try, start with Install to Plot in 5 minutes. For newcomers to lisp, see the Installation section for how to get going from ground zero.

What’s in the box?

All the basics for descriptive statistical analysis.

Data Frames

Load the classic R mtcars data set into a data frame:

(data :mtcars)

Plotting

The plotting examples show how to plot the horsepower vs. miles-per-gallon of the similar Vega-Lite data set:

(plot:plot
  (vega:defplot hp-mpg
  `(:title "Horsepower vs. MPG"
    :data (:values ,vgcars)
    :mark :point
	:encoding (:x (:field :horsepower :type :quantitative)
	           :y (:field :miles-per-gallon :type :quantitative)))))

Jupyter notebooks

For reproducible research and sharing.

Examples & Tutorials

The examples show how to create commonly used statistical plots and the analytic examples work through the examples of the book Introduction to the Practice of Statistics using Lisp-Stat.

Example data sets

About 1500 R data sets can be loaded using the rdata package.

Where do I get help?

The community page page describes how the project is structured. Help can be obtained from:

• Lisp-Stat mailing list
• Stack overflow
• Github discussion

Where did it come from?

Several of these libraries are based on the work of Tamas K Papp, who was working as a practicing econometrician using Common Lisp before leaving the community for Julia. One of the design goals for Lisp-Stat was to avoid the Lisp Curse that plagues so many common lisp niches, including statistics. So, rather than reinvent the wheel, we took some existing wheels, greased and polished them, added documentation and filed off the rough edges to make them work smoothly together in an integrated environment. The greenfield work includes special functions and plot.

Haven’t I seen this before?

You may have. In late March 2021 existence of Lisp-Stat was leaked to Hacker News and Reddit. At that time, we were still getting Lisp-Stat ready for general release, and things were not quite where we wanted them to be. If you saw the original leaked version, here are the changes:

• Documentation:
  • Reorganized getting started, tutorials, reference
  • Added plotting and data-frame examples
  • Added “Install to plot in 5 minutes” getting started guide
  • Expanded analytic examples
  • Expanded basic tutorial
  • QA: spelling, grammar, typos, examples
  • Added resources section
  • Added search
• Data Frame
  • Added pretty printing
  • Added markdown printing
  • Added variable shortcuts, you can now refer to a variable by name in all contexts
  • Added head, tail, summary, column-name functions
  • Added define-data-frame macro
  • Miscellaneous bug fixes
• Plot
  • Added box plot
  • Added histogram

New Releases

Introducing Quick Plot: ggplot-Style Plotting for Lisp-Stat

Today we’re releasing quick-plot, a new system for Lisp-Stat that bring ggplot2-style composable plotting to Common Lisp. plot/vega is able to render any Vega-Lite specification (here’s 50 examples), but it can at times be a bit wordy constructing the nested plists by hand.

Overview

Instead of specifying each plot in detail, quick-plot provides a set of helper functions that work similarly to geom in ggplot. Essentially we borrow ggplot’s concept of a plot as a composition of independent layers (which is what Vega-Lite does anyway), each with one concern:

Each function returns a plist fragment. A recursive merge-plists combines them into a single Vega-Lite spec. A mark function never sets axis titles; label never touches encodings; theme never alters mark types. You can add, remove, or swap any layer without affecting the others.

Simple example

Setup environment:

(asdf:load-system :quick-plot)
(vega:load-vega-examples)

(import '(geom:point geom:bar geom:histogram geom:box-plot geom:line))
(import '(gg:label gg:axes gg:coord gg:theme gg:tooltip))
(import '(qplot:qplot))

Basic scatter plot

The simplest scatter plot needs just a name, a data frame, and two field names:

(qplot 'cars-basic vgcars
  `(:title "Horsepower vs. MPG")
   (point :horsepower :miles-per-gallon))

Add labels

Scatter plot with labels:

(qplot 'blog-labels vgcars
  `(:title "Vega Cars")
   (point :horsepower :miles-per-gallon :filled t)
   (label :x "Engine Horsepower" :y "Fuel Efficiency (MPG)"))

Get fancy

Now you can add size of the point to encode some data, along with color:

(qplot 'cars-bubble vgcars
  `(:title "Bubble: Size = Acceleration")
   (point :horsepower :miles-per-gallon
          :color :origin :size :acceleration
          :filled t)
   (label :x "Horsepower" :y "Miles per Gallon"))

Get started

See the quick-plot cookbook for recipes for some common plot types.

2022 Release

Overview

This release is the third major refactoring of Lisp-Stat, with particular emphasis on plotting. I have attempted to make plotting as easy in Lisp-Stat as it is in Vega-Lite, upon which it is based. In addition, much clean-up and polishing of sharp edges has taken place as a result of using Lisp-Stat on real-world projects.

As always, comments and contributions are welcome.

What’s New

Lisp-Stat

• Data loading functions akin to R’s data function. You can now load the Lisp-Stat data sets with this function, e.g. (data :mtcars). The mechanism is extensible for add on packages to use as well.
• Additional cleansed and annotated data sets included.

Plot

• The plotting system has been completely reworked. For example, to create a scatter plot of horsepower vs. miles per gallon:

(asdf:load-system :plot/vega)
(data :vgcars)
(plot:plot
  (vega:defplot hp-mpg
  `(:title "Horsepower vs. MPG"
    :data (:values ,vgcars)
    :mark :point
	:encoding (:x (:field vgcars:horsepower)
	           :y (:field vgcars:miles-per-gallon)))))

• 50+ new plotting examples for commonly used statistical plots, along with a plotting tutorial.

Data-frame

• Paid technical debt, refactored, made ready for future progress.
• Added filter-rows, remove-columns, remove-columns! and rename-column!.
• Improved unit tests.
• Added column types: categorical (factor) and temporal.
• Improved error handling, conditions and restarts.
• Working prototypes for stack, split-apply-combine, sort operations.

Array Operations

Array operations has been rehomed and now lives in the Lisp-Stat github organisation. Significant improvements to documentation over the previous repository.

Documentation

There probably isn’t a part of the documentation that hasn’t been touched. Improvements in all areas, including organisation, expanded topic coverage and quality improvements.

Semantic Versioning

Previously Lisp-Stat used semantic versioning appropiate for binary patch distribution. In this scheme the major number indicates whether or not the lisp image needed to be cold started, and a minor version indicating if a patch could be applied. Since it’s unlikely we’ll have a viable patching system in the open source Common Lisp world anytime soon, we’ve moved to semantic versioning for source releases. Since the semantics of versioning are different, all systems have been reset to 1.0.0. Going forward we’ll use the major.minor.patch scheme.