Tutorials

Setting up testing and coverage for Emacs Lisp on Gitlab CI

Table of Contents

As many projects use GitHub Actions for CI, there seems to be more resources for setting GitHub Actions up for Emacs Lisp, including purcell/setup-emacs and leotaku/elisp-check. There are also many examples to copy from.

On the other hand, I was only able to find one Emacs Lisp project using Gitlab CI: joewreschnig/gitlab-ci-mode.

To learn how to do this, I added automatic testing to tst.el, a tiny library hosted on Gitlab. This is the result.

.gitlab-ci.yml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
default:
  before_script:
    - cask install

test-26.3:
  image: silex/emacs:26.3-ci-cask
  script:
    - cask exec ert-runner

test-27.1:
  image: silex/emacs:27.1-ci-cask
  script:
    - cask exec ert-runner

This tells Gitlab CI to automatically run tests (with ert-runner) under Emacs 26.3 and Emacs 27.1.

Cask

The Cask file:

1
2
3
4
5
6
7
8
(source gnu)
(source melpa)

(package-file "tst.el")

(development
 (depends-on "ert-runner")
 (depends-on "undercover"))

Cask lets you write down your project’s dependencies in a declarative way and install them in one command, instead of having to write a bunch of boilerplate in an ad-hoc init file.

The entire Cask file language is documented in this 700-word page.

Unit tests

ERT

ERT is a unit testing library that has been included since Emacs 24.1. (There’s also another option, Buttercup, that offers more features.)

Tests are written like this:

1
2
3
4
5
6
7
8
9
(require 'ert)
(require 'tst)

(ert-deftest tst-get-test ()
  ;; Empty
  (should (null (tst-get "abc[]")))
  (should (null (tst-get "abc")))
  ;; Not at end of string
  (should (null (tst-get "abc[a b]def"))))
  1. ert-deftest defines a test. tst-get-test is the name of the test.
  2. should makes the test fail if its body is nil.

You can evaluate this then run M-x ert to run the test.

Some articles that introduce ERT better:

ert-runner

Running tests with ERT involves a long command buried in its manual.

1
2
cask install
cask emacs -batch -l ert -l my-tests.el -f ert-run-tests-batch-and-exit

(Using cask emacs so that dependencies are available.)

ert-runner is designed to simplify this, and make running ERT tests less painful.

1
2
cask install
cask exec ert-runner

Coverage

Coverage means how much of your code is covered by unit tests.

Typically (as far as I know) one uses a coverage library for their language to compute it, then upload the results to a coverage tracking service.

The coverage library for Emacs Lisp is undercover.

Coverage services include Coveralls and Codecov (as mentioned in undercover’s README). I rolled a dice and landed on Coveralls, so that’s what I’m using.

Setting up undercover

Install it with Cask:

1
2
(development
  (depends-on "undercover"))

Then require the library and specify a wildcard that matches your source files before you load your package:

1
2
3
4
5
6
7
8
9
(when (require 'undercover nil t)
  (undercover "*.el"))

(require 'ert)
(require 'tst)

(ert-deftest tst-get ()
  ;; Empty
  (should (null (tst-get "abc[]"))))

Undercover will then automatically upload the results to Coveralls if a token has been given (through the COVERALLS_REPO_TOKEN environment variable).

Setting up Coveralls