Setting up testing and coverage for Emacs Lisp on Gitlab CI

As many projects use GitHub Actions for CI, there seems to be more resources for setting GitHub Actions up for Emacs Lisp, including purcell/setup-emacs and leotaku/elisp-check. There are also many examples to copy from.

On the other hand, I was only able to find one Emacs Lisp project using Gitlab CI: joewreschnig/gitlab-ci-mode.

To learn how to do this, I added automatic testing to tst.el, a tiny library hosted on Gitlab. This is the result.

.gitlab-ci.yml

default:
  before_script:
    - cask install

test-26.3:
  image: silex/emacs:26.3-ci-cask
  script:
    - cask exec ert-runner

test-27.1:
  image: silex/emacs:27.1-ci-cask
  script:
    - cask exec ert-runner

This tells Gitlab CI to automatically run tests (with ert-runner) under Emacs 26.3 and Emacs 27.1.

  • test-26.3 and test-27.1 are job names for Gitlab CI.
  • default is applied to every job. Only some properties can be specified here. script isn’t one of them.
  • Alternative: you can also specify a .test rule, then use extends: .test to reduce duplication. This is what joewreschnig/gitlab-ci-mode does.
  • silex/emacs is a comprehensive set of Emacs Docker images. Some of them are great for interactive use, others are designed for use in CI. There’s also the images from Flycheck (flycheck/emacs-cask); those include Cask but not have Git, which undercover needs for reporting coverage.

Cask

The Cask file:

(source gnu)
(source melpa)

(package-file "tst.el")

(development
 (depends-on "ert-runner")
 (depends-on "undercover"))

Cask lets you write down your project’s dependencies in a declarative way and install them in one command, instead of having to write a bunch of boilerplate in an ad-hoc init file.

The entire Cask file language is documented in this 700-word page.

  • cask install installs all dependencies, including development dependencies, in a local folder (my-app/.cask/).
  • cask emacs can then be used to run Emacs with those local dependencies made available.

Unit tests

ERT

ERT is a unit testing library that has been included since Emacs 24.1. (There’s also another option, Buttercup, that offers more features.)

Tests are written like this:

(require 'ert)
(require 'tst)

(ert-deftest tst-get-test ()
  ;; Empty
  (should (null (tst-get "abc[]")))
  (should (null (tst-get "abc")))
  ;; Not at end of string
  (should (null (tst-get "abc[a b]def"))))
  1. ert-deftest defines a test. tst-get-test is the name of the test.
  2. should makes the test fail if its body is nil.

You can evaluate this then run M-x ert to run the test.

Some articles that introduce ERT better:

ert-runner

Running tests with ERT involves a long command buried in its manual.

cask install
cask emacs -batch -l ert -l my-tests.el -f ert-run-tests-batch-and-exit

(Using cask emacs so that dependencies are available.)

ert-runner is designed to simplify this, and make running ERT tests less painful.

cask install
cask exec ert-runner

Coverage

Coverage means how much of your code is covered by unit tests.

Typically (as far as I know) one uses a coverage library for their language to compute it, then upload the results to a coverage tracking service.

The coverage library for Emacs Lisp is undercover.

Coverage services include Coveralls and Codecov (as mentioned in undercover’s README). I rolled a dice and landed on Coveralls, so that’s what I’m using.

Setting up undercover

Install it with Cask:

(development
  (depends-on "undercover"))

Then require the library and specify a wildcard that matches your source files before you load your package:

(when (require 'undercover nil t)
  (undercover "*.el"))

(require 'ert)
(require 'tst)

(ert-deftest tst-get ()
  ;; Empty
  (should (null (tst-get "abc[]"))))

Undercover will then automatically upload the results to Coveralls if a token has been given (through the COVERALLS_REPO_TOKEN environment variable).

Setting up Coveralls

  • Log in with Github, Gitlab, or Bitbucket
  • Maybe connect with the other two services, so that you don’t accidentally create another account if you forget which service you logged in with.
  • Authorize its access
  • Connect your repository
  • Copy the repository token
  • Add a secret environment variable for your repository on Gitlab:

    • Go to your project → settings → CI / CD → Variables → Expand → Add Variable
    • Set Key to COVERALLS_REPO_TOKEN, Value to the repository token you just copied
    • Make sure both Protect Variable and Mask Variable are checked.
  • Maybe add the badge to your README.

/tst-el-readme-20210116.png