avatar Kisaragi Hiu

Setting up testing and coverage for Emacs Lisp on Gitlab CI

emacs-lisp gitlab ci tutorials

As many projects use GitHub Actions for CI, there seems to be more resources for setting GitHub Actions up for Emacs Lisp, including purcell/setup-emacs and leotaku/elisp-check. There are also many examples to copy from.

On the other hand, I was only able to find one Emacs Lisp project using Gitlab CI: joewreschnig/gitlab-ci-mode.

To learn how to do this, I added automatic testing to tst.el, a tiny library hosted on Gitlab. This is the result.

.gitlab-ci.yml

default:
  before_script:
    - cask install

test-26.3:
  image: silex/emacs:26.3-ci-cask
  script:
    - cask exec ert-runner

test-27.1:
  image: silex/emacs:27.1-ci-cask
  script:
    - cask exec ert-runner

This tells Gitlab CI to automatically run tests (with ert-runner) under Emacs 26.3 and Emacs 27.1.

  • test-26.3 and test-27.1 are job names for Gitlab CI.
  • default is applied to every job. Only some properties can be specified here. script isn’t one of them.
  • Alternative: you can also specify a .test rule, then use extends: .test to reduce duplication. This is what joewreschnig/gitlab-ci-mode does.
  • silex/emacs is a comprehensive set of Emacs Docker images. Some of them are great for interactive use, others are designed for use in CI. There’s also the images from Flycheck (flycheck/emacs-cask); those include Cask but not have Git, which undercover needs for reporting coverage.

Cask

The Cask file:

(source gnu)
(source melpa)

(package-file "tst.el")

(development
 (depends-on "ert-runner")
 (depends-on "undercover"))

Cask lets you write down your project’s dependencies in a declarative way and install them in one command, instead of having to write a bunch of boilerplate in an ad-hoc init file.

The entire Cask file language is documented in this 700-word page.

  • cask install installs all dependencies, including development dependencies, in a local folder (my-app/.cask/).
  • cask emacs can then be used to run Emacs with those local dependencies made available.

Unit tests

ERT

ERT is a unit testing library that has been included since Emacs 24.1. (There’s also another option, Buttercup, that offers more features.)

Tests are written like this:

(require 'ert)
(require 'tst)

(ert-deftest tst-get-test ()
  ;; Empty
  (should (null (tst-get "abc[]")))
  (should (null (tst-get "abc")))
  ;; Not at end of string
  (should (null (tst-get "abc[a b]def"))))
  1. ert-deftest defines a test. tst-get-test is the name of the test.
  2. should makes the test fail if its body is nil.

You can evaluate this then run M-x ert to run the test.

Some articles that introduce ERT better:

ert-runner

Running tests with ERT involves a long command buried in its manual.

cask install
cask emacs -batch -l ert -l my-tests.el -f ert-run-tests-batch-and-exit

(Using cask emacs so that dependencies are available.)

ert-runner is designed to simplify this, and make running ERT tests less painful.

cask install
cask exec ert-runner

Coverage

Coverage means how much of your code is covered by unit tests.

Typically (as far as I know) one uses a coverage library for their language to compute it, then upload the results to a coverage tracking service.

The coverage library for Emacs Lisp is undercover.

Coverage services include Coveralls and Codecov (as mentioned in undercover’s README). I rolled a dice and landed on Coveralls, so that’s what I’m using.

Setting up undercover

Install it with Cask:

(development
  (depends-on "undercover"))

Then require the library and specify a wildcard that matches your source files before you load your package:

(when (require 'undercover nil t)
  (undercover "*.el"))

(require 'ert)
(require 'tst)

(ert-deftest tst-get ()
  ;; Empty
  (should (null (tst-get "abc[]"))))

Undercover will then automatically upload the results to Coveralls if a token has been given (through the COVERALLS_REPO_TOKEN environment variable).

Setting up Coveralls

  • Log in with Github, Gitlab, or Bitbucket
  • Maybe connect with the other two services, so that you don’t accidentally create another account if you forget which service you logged in with.
  • Authorize its access
  • Connect your repository
  • Copy the repository token

  • Add a secret environment variable for your repository on Gitlab:

    • Go to your project → settings → CI / CD → Variables → Expand → Add Variable
    • Set Key to COVERALLS_REPO_TOKEN, Value to the repository token you just copied
    • Make sure both Protect Variable and Mask Variable are checked.
  • Maybe add the badge to your README.

/tst-el-readme-20210116.png