# Towards future-safe emacs.d

April 9, 2021
(emacs)

TL;DR This post describes an approach to make byte compiler and various linters happy and useful in your .emacs.d, while maintaining startup performance, the ability to write embedded packages and test them. This is going to be a long post, so grab a bottle of wine, snacks and follow me. In case you lack these things, just take a look at results.

The longest project in my life is environment, it started with Emacs configurations - personal frustration and my biggest time waster investment. I might be masochistic, but I never felt sorry for falling into this trap world. And boy, sometimes it is painful to maintain something in this ever-mutating and dynamic system.

There are various tools to help maintaining Emacs package, all fall into one of four categories - project management tools (e.g. Cask, doublep/eldev, alphapapa/makem.sh), compiler (e.g. built-in byte compiler), linters (e.g. purcell/package-lint, mattiase/relint, gonewest818/elisp-lint, emacs-elsa/Elsa) and test frameworks (e.g. ERT, jorgenschaefer/emacs-buttercup). The tricky part comes when you want to use them for maintaining your own Emacs configurations, as they have different requirements compared to regular Emacs packages. At least in my case, they do.

First of all, I want Emacs to start as quickly as possible (e.g. in less than a second), meaning that I need to use tools like use-package for deferred loading. So I can’t require most of the packages directly except those that are used in the bootstrapping process. This easily makes compiler sad.

Secondly, I have lots of additional functions extending or combining functionality of one or more packages. But I hate to define them inside use-packge macro. Aside from aesthetics, I want to retain functionality of xref-find-definitions. Having definitions outside of use-package means that I will get many false byte compiler warnings and errors. Which is not helpful!

Thirdly, bootstrap process is different as project management tools isolate your package development from your Emacs configurations, which makes sense in general, but doesn’t make sense when you develop the aforementioned configurations.

The closest approach I know about is hlissner/doom-emacs, but even there .emacs.d is ignored by compiler. Let me just quote a docstring from there:

This checker (flycheck) tends to produce a lot of false positives in your .emacs.d and private config, so it is mostly useless there. However, special hacks are employed so that flycheck still does some helpful linting.

But it’s Emacs, right? Everything is possible! So let’s find a way to make byte compiler and linters helpful and enable testing of Emacs configurations.

Before we dive too much into details, let me describe the solution from higher level.

The structure of my .emacs.d looks like this:

.
├── Eldev
├── Makefile
├── early-init.el
├── init.el
├── lisp
│   ├── config-aaa.el
│   ├── config-bbb.el
│   ├── ...
│   ├── config-zzz.el
│   ├── init-elpa.el
│   ├── init-aaa.el
│   ├── init-bbb.el
│   ├── ...
│   ├── init-zzz.el
│   ├── lib-aaa.el
│   ├── lib-bbb.el
│   ├── ...
│   ├── lib-zzz.el
├── templates
│   ├── emacs-lisp-mode
│   │   ├── template_1
│   │   ├── ...
│   │   └── template_n
│       ├── template_1
│       ├── ...
│       └── template_n
├── test
│   ├── lib-aaa-test.el
│   ├── lib-bbb-test.el
│   ├── ...
└── └── lib-zzz-test.el


As you can see, all lisp files are located inside of lisp directory (you should not put them on the same level as init.el file as that directory can’t be part of load-path), and all tests are located inside of test directory.

The following naming convention is used.

• init-xxx is a file lazily initializing xxx feature, it can be a programming language (e.g. init-haskell) or a feature (e.g. init-project)
• this is the only file type describing what packages to install, how to initialize and configure them;
• it is safe to require these files, as they should defer any loading as much as possible;
• lib-xxx is a file containing various utilities depending on packages defined in init-xxx file
• these files are loaded via autoloads, and they should never be required directly;
• these files can safely require any packages defined in init-xxx to help linter and byte compiler;
• in some sense, lib-xxx are packages that are not distributed via MELPA, but rather located in emacs.d folder;
• various extensions around org-mode called vulpea are good examples of lib files:
• config-xxx is a file containing variables and constants required by both init-xxx and lib-xxx files, allowing to avoid circular dependencies;
• as they do not load any packages, it is safe to require this file from any other file;

The only exception from this convention is init-autoloads.el file containing autoloads. Name comes from init.el file.

# Content of early-init

See relevant section in Emacs Help for more information on The Early Init File, introduced in Emacs 27.1. Basically, this file is great for frame customizations. In my case I love to disable as much clutter as possible:

(add-to-list 'default-frame-alist '(tool-bar-lines . 0))
(add-to-list 'default-frame-alist '(vertical-scroll-bars))

This file is totally optional, you can safely omit it in your setup. But if you have any frame customization, putting them in early-init file might speed up your Emacs and fix some visual clutter upon startup.

# Content of init.el

The goal of this file is to require all init-xxx files. The structure is trivial:

1. Add lisp folder to load-path, so we can use require.
2. Adjust garbage collection thresholds, so things run smoother.
3. Load config-path declaring various path constants.
4. Load init-elpa which ‘bootstraps’ your package and configuration management tools.
6. Load all other init-xxx files.
7. Load custom-file, even if you are not using customize interface, you need this to use .dir-locals.el.

## Add lisp directory to load-path

;; Since we might be running in CI or other environments, stick to
;; XDG_CONFIG_HOME value if possible.
(let ((emacs-home (if-let ((xdg (getenv "XDG_CONFIG_HOME")))
(expand-file-name "emacs/" xdg)
user-emacs-directory)))
;; Add Lisp directory to load-path'.
(add-to-list 'load-path (expand-file-name "lisp" emacs-home)))

## Garbage collection thresholds

Garbage collection is a huge contributor to startup time. We temporarily increase this value to prevent garbage collection from running, then reset it to some big number in emacs-startup-hook. I discovered this trick thanks to hlissner/doom-emacs. But it is widely used by many people, for example purcell/emacs.d.

In addition it is a good idea to use emacsmirror/gcmh (aka Garbage Collector Magic Hack) to improve performance of interactive functions.

;; Adjust garbage collection thresholds during startup, and thereafter
(let ((normal-gc-cons-threshold (* 20 1024 1024))
(init-gc-cons-threshold (* 128 1024 1024)))
(setq gc-cons-threshold init-gc-cons-threshold)
(lambda () (setq gc-cons-threshold
normal-gc-cons-threshold))))

## Bootstrap

(require 'config-path)
(require 'init-elpa)

Literally, that’s it. Checkout content of init-elpa to find out how it works.

## Setup custom-file location

Before we load anything, we should setup location of our custom-file, otherwise Emacs customization system will pollute our init.el file.

(setq custom-file (concat path-local-dir "custom.el"))

The constant path-local-dir is defined in config-path:

(defconst path-local-dir
(concat
(file-name-as-directory
(or (getenv "XDG_CACHE_HOME")
(concat path-home-dir ".cache")))
"emacs/")
"The root directory for local Emacs files.

Use this as permanent storage for files that are safe to share
across systems.")

;; load autoloads file
(unless elpa-bootstrap-p
"eru install emacs"))
(load path-autoloads-file nil 'nomessage))

The most important bit here is the last line, which loads file containing autoloads and errors out if it doesn’t exist. We want to load this file before any other modules to make autoloaded functions available there. But of course we can’t load this file during bootstrap process which generates this file.

Now comes the easy part, we just load all init-xxx files that we have.

;; core
(require 'init-env)
(require 'init-kbd)
(require 'init-editor)
;; ...

;; utilities
(require 'init-selection)
(require 'init-project)
(require 'init-vcs)
(require 'init-ide)
(require 'init-vulpea)
(require 'init-vino)
(require 'init-pdf)
;; ...

;; languages
(require 'init-elisp)
(require 'init-sh)
;; ...

While this might sound stupid to manually load files that has clear naming pattern, I still like to do it manually, because it helps byte compiler, it has less footprint on runtime performance, the list is not big and I rarely add new files. Another option would be to generate this list during ‘compilation’, but again, I would love to avoid any unnecessary complications.

## Loading custom-file

And the last thing to do is to load custom-file:

;; I don't use customize' interface, but .dir-locals.el put 'safe'
;; variables into custom-file'. And to be honest, I hate to allow
;; them every time I restart Emacs.
(when (file-exists-p custom-file)
(load custom-file nil 'nomessage))

# Content of init-elpa

Part of our bootstrap process is setting up package management and package configuration tools, which is performed in init-elpa file.

## Bootstrap straight.el

The bootstrap process of raxod502/straight.el is quire simple and well documented in the official repository. Additionally, we want to avoid any modification checks at startup by setting the value of straight-check-for-modifications to nil, so everything runs faster. Also we want to install packages by default in use-package forms. And then everything is straight-forward.

(require 'config-path)

(setq-default
straight-repository-branch "develop"
straight-check-for-modifications nil
straight-use-package-by-default t
straight-base-dir path-packages-dir)

(defvar bootstrap-version)
(let ((bootstrap-file
(expand-file-name "straight/repos/straight.el/bootstrap.el"
path-packages-dir))
(bootstrap-version 5))
(unless (file-exists-p bootstrap-file)
(with-current-buffer
(url-retrieve-synchronously
(concat "https://raw.githubusercontent.com/"
"raxod502/straight.el/"
"develop/install.el")
(goto-char (point-max))
(eval-print-last-sexp)))
(load bootstrap-file nil 'nomessage))

The only bit I am not describing here is how I configure retries for networking operations.

## Setup use-package

Now it’s easy to setup use-package:

(setq-default
(straight-use-package 'use-package)

There are packages (or rather libraries) that should be loaded eagerly because they are used extensively and they do not provide autoloads.

(use-package s)
(use-package dash)

# Content of Eldev

Eldev file defines our project. You can read more about this file in doublep/eldev repository.

## Specify project files

Eldev is quite powerful when it comes to fileset specification, but I find it not working properly with extra directories out of box. Since we can not place our lisp files in the same directory with init.el file, we configure eldev-main-fileset and add lisp folder to loading roots for certain commands.

(setf eldev-project-main-file "init.el"
eldev-main-fileset '("init.el"
"early-init.el"
"lisp/*.el"))

;; path, so we have to put other Emacs Lisp files in directory. Help
;; Eldev commands to locate them.
(eldev-add-loading-roots 'bootstrap "lisp")

## Use MELPA

We are going to use certain 3rd party packages for project management (e.g. testing and linting), so we must tell Eldev where to load them from. This part is a little bit confusing as Eldev will install packages from MELPA and for our configurations we are going to use straight.el. But Eldev isolates these packages in it’s working dir and they will not interfere with our configurations. Ugly, but safe.

;; There are dependencies for testing and linting phases, they should
;; be installed by Eldev from MELPA and GNU ELPA (latter is enabled by
;; default).
(eldev-use-package-archive 'melpa)

## Define bootstrap command

Bootstrapping Emacs is simple, we just need to load init.el file.

(defun elpa-bootstrap ()
"Bootstrap personal configurations."
(setq-default
elpa-bootstrap-p t
(require 'config-path)

;; We want to run this before any build command. This is also needed
;; for flyspell-eldev to be aware of packages installed via
;; straight.el.
(add-hook 'eldev-build-system-hook #'elpa-bootstrap)

We set the value of elpa-bootstrap-p to t, so that autoloads file is not required from init.el (we are going to generate it during bootstrap flow). We also set load-prefer-newer to t so that Emacs prefers newer files instead of byte compiled (again, we are going to compile .el to .elc).

We hook this function into any build command in order to install packages and get proper load-path in all phases.

Upgrade flow is simple and uses straight.el functionality, because we use it to manage packages.

(defun elpa-upgrade ()
"Bootstrap personal configurations."
;; make sure that bootstrap has completed
(elpa-bootstrap)

(straight-fetch-all)
(straight-merge-all)

;; in case we pinned some versions, revert any unneccessary merge
(straight-thaw-versions)

;; rebuild updated packages
(delete-file (concat path-packages-dir "straight/build-cache.el"))
(delete-directory (concat path-packages-dir "straight/build") 'recursive)
(straight-check-all))

(add-hook 'eldev-upgrade-hook #'elpa-upgrade)

Now is the most dirty part - autoloads generation. Eldev provides a plugin for autoloads generation, but unfortunately it works only with root directory, but we need to generate our autoloads for files in lisp directory. So we write our own plugin.

;; We want to generate autoloads file. This line simply loads few
;; helpers.

;; Eldev doesn't traverse extra loading roots, so we have to modify
;; autoloads plugin a little bit. Basically, this modification
;; achieves specific goal - generate autoloads from files located in
;; Lisp directory.
:type           many-to-one
:message        target
"Delete the generated package autoloads files."
:default t)
;; To make sure that update-directory-autoloads' doesn't grab files it shouldn't,
;; override directory-files' temporarily.
:around
(lambda (original directory &rest arguments)
(let ((files (apply original directory arguments)))
(if (file-equal-p directory eldev-project-dir)
(let (filtered)
(dolist (file files)
(when (eldev-any-p (file-equal-p file it) sources)
(push file filtered)))
(nreverse filtered))
files))))
(let ((inhibit-message   t)
(make-backup-files nil)
(pkg-dir (expand-file-name "lisp/" eldev-project-dir)))
;; Always load the generated file.  Maybe there are cases when we don't need that,
;; but most of the time we do.

'eldev-build-system-hook
(lambda ()
(expand-file-name "lisp/init-autoloads.el" eldev-project-dir))))

## Linting configuration

And again, we need to tell Eldev which files to lint.

(defun eldev-lint-find-files-absolute (f &rest args)
"Call F with ARGS and ensure that result is absolute paths."
(seq-map (lambda (p)
(expand-file-name p eldev-project-dir))
(seq-filter (lambda (p)
(apply f args))))

(advice-add 'eldev-lint-find-files :around #'eldev-lint-find-files-absolute)

Then we ask Eldev to use gonewest818/elisp-lint for linting and configure it a little bit.

;; Use elisp-lint by default
(setf eldev-lint-default '(elisp))
(setf elisp-lint-ignored-validators '("byte-compile")))

;; Tell checkdoc not to demand two spaces after a period.
(setq sentence-end-double-space nil)

What I love about gonewest818/elisp-lint is that it combines multiple linters, including purcell/package-lint. While package-lint is a useful linter, it enforces naming convention which I don’t agree with when it comes to Emacs configurations. E.g. it wants every function in lib-vulpea.el to have a prefix lib-vulpea. While in general it makes sense, I want to avoid lib part here. The same goes for init and config stuff. So we intrusively change that rule:

;; In general, package-lint' is useful. But package prefix naming
;; policy is not useful for personal configurations. So we chop
;; lib/init part from the package name.
;;
;; And eval-after-load'. In general it's not a good idea to use it in
;; packages, but these are configurations.
(defun package-lint--package-prefix-cleanup (f &rest args)
"Call F with ARGS and cleanup it's result."
(let ((r (apply f args)))
(replace-regexp-in-string "\$$init\\|lib\\|config\\|compat\$$-?" "" r)))

"Do nothing."))

We also need eval-after-load, so let’s just noop. It makes sense to discourage usage of eval-after-load in packages, but in Emacs configurations it doesn’t make sense.

And the last bit is emacsql. I use emacsql-fix-vector-indentation to format my SQL statements, and I want linter to be happy about it:

;; Teach linter how to properly indent emacsql vectors.
(lambda ()
(require 'emacsql)
(call-interactively #'emacsql-fix-vector-indentation)))

Now that everything is configured, we can use eldev to bootstrap, compile, lint and test our configurations. The first thing we do is autoloads generation, which is as simple as

$eldev build :autoloads Though I prefer to clean autoloads before generating new ones. $ eldev clean autoloads
$eldev build :autoloads This generates lisp/init-autoloads.el file. And in case you were wondering bout its content, then it looks like this: ;;; init-autoloads.el --- automatically extracted autoloads -*- lexical-binding: t -*- ;; ;;; Code: (add-to-list 'load-path (directory-file-name (or (file-name-directory #$) (car load-path))))

;;;### (autoloads nil "config-path" "config-path.el" (0 0 0 0))

(register-definition-prefixes "config-path" '("path-"))

;;;***

;;; ...
;;; ...
;;; ...

;;;### (autoloads nil "lib-buffer" "lib-buffer.el" (0 0 0 0))

Return lines of BUFFER-OR-NAME.

Each line is a string with properties. Trailing newline character
is not present.

\(fn BUFFER-OR-NAME)" nil nil)

Call FN on each line of BUFFER-OR-NAME and return resulting list.

As opposed to buffer-lines-each', this function accumulates
result.

Each line is a string with properties. Trailing newline character
is not present.

\(fn BUFFER-OR-NAME FN)" nil nil)

(function-put 'buffer-lines-map 'lisp-indent-function '1)

;; ...
;; ...
;; ...

;;;***

;;;;;;  (0 0 0 0))

Show main org-agenda' view." t nil)

Show main org-agenda' view." t nil)

(defconst vulpea-agenda-cmd-refile '(tags "REFILE" ((org-agenda-overriding-header "To refile") (org-tags-match-list-sublevels nil))))

(defconst vulpea-agenda-cmd-today '(agenda "" ((org-agenda-span 'day) (org-agenda-skip-deadline-prewarning-if-scheduled t) (org-agenda-sorting-strategy '(habit-down time-up category-keep todo-state-down priority-down)))))

;;; ...
;;; ...
;;; ...

;;;***

;; Local Variables:
;; version-control: never
;; no-byte-compile: t
;; coding: utf-8
;; End:
;;; init-autoloads.el ends here

As you can see, it uses autoload to define a symbol (function or variable) and where to load it from. It also sets up indentation based on decalre from the body of function. And all constants are embedded as is, they are not getting autoloaded.

Please note that eldev commands need to be run with working directory pointing to the directory containing Eldev file, e.g. from $XDG_CONFIG_HOME/emacs or $HOME/.config/emacs.

# Compiling

The second operation in the bootstrap process is byte compilation. It is said that byte compiled lisp executes faster, but there is also an experimental branch for native compilation called gccemacs, which is also available via emacs-plus. Another aspect of byte compilation is… well compilation which produces valuable warnings and errors. In our setup it is very easy to compile all our .el files.

$eldev clean elc$ eldev compile

That’s it.

# Linting

The third step of the bootstrap process is linting. Once everything compiles we just need to check what linter has to say. Just to remind, we are using gonewest818/elisp-lint. As you might already figured, with Eldev this step as trivial as

# Makefile

Since certain operations consist of two steps (e.g. clean followed by build) and I also want to always pass extra arguments to eldev for verbosity and debuggability, I have a Makefile with all available commands.

.PHONY: clean
clean:
eldev clean all

.PHONY: bootstrap
bootstrap:
eldev -C --unstable -a -dtT build :autoloads

eldev -C --unstable -a -dtT upgrade

.PHONY: compile
compile:
eldev clean elc
eldev -C --unstable -a -dtT compile

.PHONY: lint
lint:
eldev -C --unstable -a -dtT lint

.PHONY: test
test:
eldev exec t
eldev -C --unstable -a -dtT test

# org-roam

In addition, I love to build org-roam and vino databases during bootstrap process, so I don’t spend time on this when I use Emacs. For this I have defined the following function in my lib-vulpea file.

;;;###autoload
(defun vulpea-db-build ()
"Update notes database."
(when (file-directory-p vulpea-directory)
(org-roam-db-build-cache)))

Now we can evaluate this function from command line via eldev:

$eldev exec "(vulpea-db-build)" If you are using vino, then vulpea-db-build also triggers vino database update, but since it vino-setup happens in after-init-hook, we need to run it before executing vulpea-db-build. (use-package vino ;; unrelated code :hook ((after-init . vino-setup)) ;; unrelated code ) So we change our eldev command a little bit. $ eldev exec "(progn (run-hooks 'after-init-hook) (vulpea-db-build))"

And we can put that into Makefile.

.PHONY: roam
roam:
eldev exec "(progn (run-hooks 'after-init-hook) (vulpea-db-build))"

# eru

And the last yet optional bit of the whole puzzle is Eru, a script I use to setup and maintain my environment. I have it in my PATH, so I can rely on its might whenever I am. In short, I have the following commands:

$eru install emacs # autoloads, compile, lint, roam$ eru upgrade emacs
$eru test emacs Since Eru is a beast, you might not want to use it, but the core idea here is that you can create an executable that will glue all things together for you. #!/usr/bin/env bash set -e ACTION=$1

emacs_d=$HOME/.config/emacs if [[ -d "$XDG_CONFIG_HOME" ]]; then
emacs_d="$XDG_CONFIG_HOME/emacs" fi function print_usage() { echo "Usage: emacs-eru ACTION Actions: install Install dependencies, compile and lint configurations upgrade Upgrade dependencies test Test configurations " } if [ -z "$ACTION" ]; then
echo "No ACTION is provided"
print_usage
exit 1
fi

case "$ACTION" in install) cd "$emacs_d" && {
make bootstrap compile lint roam
}
;;

cd "$emacs_d" && { make upgrade compile lint } ;; test) cd "$emacs_d" && {
make test
}
;;

*)
echo "Unrecognized ACTION $ACTION" print_usage ;; esac For convenience, this script is available as a GitHub Gist, so you can download it, save in somewhere in your PATH, chmod it and use. $ curl -o ~/.local/bin/emacs-eru https://gist.githubusercontent.com/d12frosted/b150fcaaf2de06b1b29af487ebbbf9c1/raw/6fc70215afce2472e4f289c2c8500fbfc9a3f001/emacs-eru
\$ chmod +x ~/.local/bin/emacs-eru

# What’s next

Tinkering with Emacs, of course! This is an endless effort, constant struggle but most importantly, divine pleasure. On a serious note, I would love to cover most critical parts with tests and integrate emacs-elsa/Elsa into my flow. And I would love to hear from you, how do you approach safety problem of your emacs.d?

Safe travels!