Towards future-safe emacs.d
TL;DR This post describes an approach to make byte compiler and various linters happy and useful in your .emacs.d, while maintaining startup performance, the ability to write embedded packages and test them. This is going to be a long post, so grab a bottle of wine, snacks and follow me. In case you lack these things, just take a look at results.
The longest project in my life is environment, it started with Emacs configurations - personal frustration and my biggest time waster investment. I might be masochistic, but I never felt sorry for falling into this trap world. And boy, sometimes it is painful to maintain something in this ever-mutating and mutable dynamic system.
There are various tools to help maintaining Emacs package, all fall into one of four categories - project management tools (e.g. Cask, doublep/eldev, alphapapa/makem.sh), compiler (e.g. built-in byte compiler), linters (e.g. purcell/package-lint, mattiase/relint, gonewest818/elisp-lint, emacs-elsa/Elsa) and test frameworks (e.g. ERT, jorgenschaefer/emacs-buttercup). The tricky part comes when you want to use them for maintaining your own Emacs configurations, as they have different requirements from regular packages. At least in my case, they do.
First of all, I want Emacs to start as quickly as possible (e.g. in less than a second), meaning that I need to use tools like use-package for deferred loading. Meaning, I can’t require most of the packages directly except those that are used in the bootstrapping process.
Secondly, I have lots of additional functions extending or combining functionality of one or more packages. But I hate to define them inside use-packge macro. Aside from aesthetics, I want to retain functionality of xref-find-definitions. But having definitions outside of use-package means that I will get many false byte compiler warnings and errors. Which is not helpful!
Thirdly, bootstrap process is different as project management tools isolate your package development from your Emacs configurations, which makes sense in general, but doesn’t make sense when you develop the aforementioned configurations.
The closest approach I know about is hlissner/doom-emacs, but even there .emacs.d is ignored. Let me just quote a docstring from there:
This checker (flycheck) tends to produce a lot of false positives in your .emacs.d and private config, so it is mostly useless there. However, special hacks are employed so that flycheck still does some helpful linting.
But it’s Emacs, right? Everything is possible! So let’s find a way to make byte compiler and linters helpful and enable testing of Emacs configurations.
Retries with straight.el
We all know, network is unreliable. I love those moments when I decide to upgrade all my Emacs packages while connected via 3G. And since I am using raxod502/straight.el, the process requires to process each repository one by one. Now imagine my frustration, when due to unstable connection it fails on one of the repositories and I need to start the process again… from the beginning.
So I say, no more, raxod502/straight.el should retry read operations over network! Unfortunately, there is no such option out of box (or at least I could not find it). Luckily, it’s Emacs!
Task management with org-roam Vol. 6: Select a person and view related tasks
In one of the previous articles (Vol 3 to be precise) we talked about automatic setup of filetags, so each of the task in note related to a person is automatically tagged thanks to tag inheritance. Then, in Vol 4 we talked about automatic tagging of tasks whenever a person is mentioned either in the title or the body of some task. This all makes org-agenda matching capabilities really useful for when we want to see the list of all tasks related to specific person.
In this article, we are going to write a small utility function that asks user to select a person and then presents and org-agenda buffer with tasks related to selected person.
Believe me, intro is longer than the content!
Change Log:
[2021-03-02 Tue]: Update naming convention to match personal configurations.[2021-05-10 Mon]: Update post to reflect changes in org-roam v2. Previous version of this article is available on GitHub.
Task management with org-roam Vol. 5: Dynamic and fast agenda
In previous articles (Vol 1 and Vol 2) we talked about moving tasks from regular org-mode files to org-roam notes. This relied upon adding all org-roam files to org-agenda-files, which doesn’t scale well, as when you build an agenda buffer, it needs to traverse each file. Once you have more than 1k notes, things become sluggish.
In my experience, once I reached 1200 note files, org-agenda constantly took more than 50 seconds to build, rendering this tool completely useless. But then I realised that only 3% of those files actually contain any TODO entries, so there is no need to traverse whole org-roam-directory!
In this article we are going to optimise org-agenda back to less than 1 second by dynamically building org-agenda-files list to include only files with TODO entries. All thanks to the power of org-roam and some hooks I am going to describe.
Change Log:
[2021-03-02 Tue]: Update naming convention to match personal configurations.[2021-03-08 Mon]: Gustav shared thatorg-element-maphas an optional parameterfirst-matchthat works likeseq-find, meaning thatvulpea-project-pcan be optimised.[2021-05-10 Mon]: Update post to reflect changes in org-roam v2. Previous version of this article is available on GitHub.
Task management with org-roam Vol. 4: Automatic tagging
In the previous article we covered automatic tagging of notes related to a specific person, and today we are going to cover automatic tagging of an org-mode heading upon insertion of link related to a person. To put it simple, when I mention someone in the task, I would love this task to be automatically tagged with that persons name. As they say, it’s better to see once, than imagine multiple times, so here is a screencast.
Change Log:
[2021-01-24 Sun]: Since some of the functionality mentioned in the original article was merged toorg-roam, all code is updated to reflect the current state of affairs.[2021-03-02 Tue]: Update naming convention to match personal configurations.[2021-05-10 Mon]: Update post to reflect changes in org-roam v2. Previous version of this article is available on GitHub.
Task management with org-roam Vol. 3: FILETAGS
In the previous articles (vol1 and vol2) we walked the path to org-roam and solved the issue with garbage in the category column of agenda. Today we are going to explore meta projects dedicated to specific person, tag inheritance and moving such projects to separate org-roam files. As result, we will have code for automatic tagging based on the title.
Aside from regular meta projects (like personal blog) I also create meta projects for people and locations. This is helpful, because some of the tasks are really related to someone specifically. For example, when I need to return a borrowed book, I just create a task for this.
* Frodo Baggins :@FrodoBaggins:
** TODO Return 'The Lord of the Rings' book
** TODO Farewell party :PROJECT:
It feels like Mr. Frodo is about to live Shire. So we are going to setup a
farewell party for him.
*** TODO Talk to Samwise Gamgee :@SamwiseGamgee:
*** TODO Talk to Meriadoc Brandybuck :@MeriadocBrandybuck:
*** TODO Talk to Peregrin Took :@PeregrinTook:
*** TODO Tie a pair of wool socks
I am not sure where he is going, so a pair of warm wool socks should be good.
At least they can be used to protect bottles of wine during journey. That is in
case Frodo doesn't wear socks. But how could it be? Everyone does!
Change Log:
[2021-01-24 Sun]: Since some of the functionality mentioned in the original article was merged toorg-roam, all code is updated to reflect the current state of affairs.[2021-03-02 Tue]: Update naming convention to match personal configurations.[2021-05-10 Mon]: Update post to reflect changes in org-roam v2. Previous version of this article is available on GitHub.
Task management with org-roam Vol. 2: Categories
In the previous article we set a ground for moving tasks to org-roam, and encountered an issue with visual garbage in the agenda buffer. Namely, org-roam file id as part of the category. In this article, we are going to explore the means to overcome this issue.
Change Log:
[2021-03-02 Tue]: Update category extraction function to useTITLEof the note and enforce length limit. Kudos to Tim Ruffing for the idea.[2021-03-02 Tue]: Update naming convention to match personal configurations.[2021-05-10 Mon]: Update post to reflect changes in org-roam v2. Previous version of this article is available on GitHub.
Task management with org-roam Vol. 1: Path to Roam
org-mode is a simple outliner for note taking and list management. It doesn’t impose any complex features on users. This is something users excel at by themselves. My suggestion is to continue this tradition and discover task management capabilities in org-roam, an org-mode extension for non-hierarchical notes management.
Change Log:
[2021-05-10 Mon]: Update post to reflect changes in org-roam v2. Previous version of this article is available on GitHub.
Org-roam tags
Org-roam is a note-taking tool built on top of Emacs and Org. Essentially, it’s a replica of Roam Research. These tools provide an easy way to create and manage non-hierarchical notes. If you wish to learn more, just take a look at the Org-roam manual or watch Making Connections in your Notes video by Matt Williams. Believe me, Org-roam and Roam Research are game-changers. Or even better, don’t believe me and validate my claim by yourself.
Since I am already addicted, it was only natural to prefer org-roam over some web application. Apart from being developed on top of mature Org ecosystem, Emacs brings many merits and extensibility is one of them. Once org-roam introduced tags system in v1.1.1 I felt the lack of functions to manage them. Adding and removing them by hand is not nice. So in this article I am sharing a snippet that I’ve forged to ease the unbearable lightness of being.
Change Log:
[2020-10-12 Mon]: Functionality described in this post (and similar functionality to manage aliases) is merged to the upstream. Now simply use one of the following functions:org-roam-tag-addorg-roam-tag-deleteorg-roam-alias-addorg-roam-alias-delete
Emacs: moving to beginning of line
In stocks Emacs there is a function move-to-beginning-of-line which is bound to C-a by default. It does exactly what its name implies - moves the cursor to the beginning of line. In case of indented code this is usually not what I want, but instead I would love to move the first visible character of line.
There are packages that does this and even more (for example, mwim), but if for some reason you don’t want to add one more package to your dependencies, or you care only about moving to the beginning of line, then you might grab the code that I share in the details. There is a version that also knows how to work with org mode headers and lists.
Readings Vol. I: precondition encoding
When we write libraries for others (including ourselves) to use we often require some preconditions to be met. Sometimes we just make them implicit (in the form of folklore, passed from developer to developer during post mortem ritual). In other cases we encode the possibility of failure due to unsatisfied preconditions.
safeHead :: [a] -> Maybe a
safeHead [] = None
safehead (a:_) = Just aIs this is the only thing we can do? Definitely no! Is it the best thing we can do? Depends on the situation.
This article is part of the readings series where I take one topic and share links to related articles and papers. This time our focus is precondition encoding.
Predicate composition
They say functional programming has many essences and the composition is one of them. Thanks to the wonderful dot operator, we know how to compose functions like a -> b and b -> c to get a function a -> c. But in some cases functions are not that simple and it becomes tricky to compose them nicely.
valid :: a -> Bool
valid = \a -> check1 a && (check2 a || check3 a)
where check1 = undefined :: a -> Bool
check2 = undefined :: a -> Bool
check3 = undefined :: a -> BoolIt would be lovely to express it in a more declarative way by abstracting away function application and result combination.
valid :: a -> Bool
valid = check1 .&& (check2 .|| check3)
(.&&) :: (a -> Bool) -> (a -> Bool) -> a -> Bool
(.||) :: (a -> Bool) -> (a -> Bool) -> a -> BoolApart from implementing combinators for predicate composition, we want to avoid any runtime penalty from using abstractions. In this article we are going to implement the following functions and investigate how far we can go with abstractions until performance degrades. Or maybe it won’t degrade. Who knows?
emacs-plus: current state
Not everyone knows, but emacs-plus is almost an April Fool’s joke that was late for 1 day. Its initial release was on 2nd April 2016 and the only feature it had compared to the official Emacs formula back in the day was Spacemacs icon (installed by default). Homebrew maintainers didn’t want to increase the complexity of the Emacs formula, so I’ve put that burden on myself, created a tap with one additional icon and after several years info command for emacs-plus prints a wall of options.
I am very happy to see so many users of this feature-inclusive (and sometimes experimental) formula. I am grateful to all of you. I would’ve abandoned this project a year ago if it would not you! And I think it’s fair to (in)formally share the current state of the project and shed some light on my plans.
env-extra v1.0.0.0
It’s hard (though possible) to imagine a script that doesn’t access environment variables. In Haskell ecosystem there is a good built-in module System.Environment that does the job, but it has several drawbacks. First of all, it uses String data type as input and as an output. And secondly, it lives in IO. Both of these drawbacks are not that critical, but calling all these Text.pack, Text.unpack and liftIO in every program drives me nuts. So several years ago (somewhere in 2016) I decided to write a simple library that wraps it for me.
I’ve been using this library for a long time and today I uploaded it to Hackage. While it’s really small, I still think that some of you might find it useful, because it has nice features that original System.Environment doesn’t have. Let’s take a look!
flyspell-correct v0.6
flyspell-correct is a package for distraction-free words correction with flyspell via selected interface. It’s available on GitHub and MELPA.
Recently, there was a major release of flyspell-correct. Version 0.6 introduces new interface (avy-menu), fixes point movements, warnings, autoloads and switch to the lexical binding. Read further for the complete change list.
Emacs: reusing window for helpful buffers
Ironically, I find the helpful package quite helpful. It boosts Emacs help buffer with much more contextual information. If you haven’t tried it out yet, I advice you to do so.
However, by default, it doesn’t play nicely with windows.
Usually when I write some Elisp and I want to read the documentation of some function or variable, I hit C-h f or C-h v respectively and the help buffer is shown in the separate window. Which is convenient in my opinion, because I can see the code and the help.
Sometimes help contains links to other entries that I need to navigate. And when I hit <RET> window containing code shows another help buffer. Which might be good for some people, but I hate this behaviour, because usually I want to see the code that I am editing.
This is also annoying if you set the value of helpful-max-buffers to 1. Help window and the window with code are swapped on every navigation.
The good thing, it’s configurable (as almost everything in Emacs land).
(setq helpful-switch-buffer-function #'+helpful-switch-to-buffer)
(defun +helpful-switch-to-buffer (buffer-or-name)
"Switch to helpful BUFFER-OR-NAME.
The logic is simple, if we are currently in the helpful buffer,
reuse it's window, otherwise create new one."
(if (eq major-mode 'helpful-mode)
(switch-to-buffer buffer-or-name)
(pop-to-buffer buffer-or-name)))Asking for input in Bash
Yet another tale about bash script refactoring. This time the configuration script of mirror-elpa had undergone some changes and I like the result so much that I decided to share it.
Imagine that you want your users to give you some input, for example, repository owner and name. And you want the value so desperately, that if the user doesn’t provide one, you ask the user again. On the other hand, you also want to support default values. So for example, if the user doesn’t provide any value, but there is a default value, you could use that.
And you can’t use anything except for bash.
Revisiting Eru
As you might know, Eru is the supreme deity of Arda. The first things that Eru created where the Ainur. He then bade the Ainur to sing to him. Each Ainu had a particular theme given by Eru. Sure enough, Eru makes the ‘World and All That Is’.
So when I get a new clean system there is nothing yet. And so I call upon the wisdom and power of Eru.sh - the one who creates Ainur and the ‘World and All That Is’.
$ curl https://raw.githubusercontent.com/d12frosted/environment/master/bootstrap/eru.sh | bashI just have to wait patiently, while everything is being downloaded and installed, while all configuration cogs are being placed on the right spot.
Equality of booleans in Emacs
There is a fun story about booleans in Emacs Lisp - there are no booleans in Emacs Lisp. Sort of. Because we have a symbol nil, which means an empty list. You can write it as nil or () - they both stand for the same object, the symbol nil.
(eq nil ()) ; => tSince LISP is all about list processing, empty list is something very false. So false that we don’t have special symbol for false values, as empty list serves this purpose well.
Everything that is not an empty list has a meaning of true. However, there is a symbol t which is the preferred way to represent the truth value true.
So nil and t are considered canonical boolean values. There is a function booleanp that returns t if the argument is a canonical boolean value and nil otherwise.
The fun begins when you need to check if two boolean values are equal. Since non-nil (or not an empty list) can mean many different things (like "Emacs is
the only true editor") you can’t just do regular equality check.
(equal t "Emacs is the only true editor") ; => nilThere are, however, several tricks to get it working. The most obvious solution is to convert value to a canonical boolean value.
High quality GIF from video
When it comes to converting video to GIF, one usually gets a huge file and a questionable quality. Most of the guides suggest to use FFmpeg to do the conversion, but usually, they don’t bother with the quality of the result. As it turns out, folks from FFmpeg made some huge steps in improving the GIF output.
flyspell-correct v0.5
flyspell-correct is a package for distraction-free words correction with flyspell via selected interface. It’s available on GitHub and MELPA.
Recently, there was a major release of flyspell-correct. Version 0.5 introduces rapid mode for fixing multiple words in one run, breaking renames and tests. Read further for the complete change list.
Random bytes generation with OpenSSL
Sometimes you just have to generate some random bytes with high entropy. And OpenSSL can help you with this task.
Structure templates in Org mode
Anyone would agree that typing #+BEGIN_SRC ... #+END_SRC blocks manually feels strange.
In order to make life easier, some people implement helpers to insert these blocks. For example, Chris Done has org-begin-template function in his configs. Other people use yatemplate to create generic or specific templates for blocks.
But what is most fascinating, is that org-mode bundles with templates for some structural elements like these blocks. And instead of copy-pasting solutions into your .emacs file, take a look at Structure Templates.
AppleScript and the GitHub API
AppleScript is a powerful scripting language that allows to directly control scriptable applications and some parts of macOS. The language itself can be a little bit frustrating for those who are not familiar with natural language programming paradigm. For example, with AppleScript, you can tell application
"Preview" to quit or ask someone to print pages 1 thru 5 of document 2.
But my goal is not to promote or demote AppleScript. Instead, I wish to share a little script that I’ve fetched while experimenting with OmniFocus 2. I hate to manually capture issues that I care about, so I started to work on a script that takes URL from an active tab in Safari and sends it to OmniFocus into the right project with useful title and body.
For example, I want d12frosted/flyspell-correct#30 to go into the flyspell-correct project with title and description taken from that issue.
Fish: notify me when you finish
Have you ever been in a situation when you called git fetch, stared at the screen for several seconds and then switched to the browser to read something ‘useful’ while git fetches updates? And in five minutes you’re like ‘Oh wait, I was doing something important, no?’. Rings the bell, doesn’t it?
Git: conditional configurations
More than five years ago Jeff King has added include directive to git config subprogram. This directive allows splitting configuration file across multiple files. A nice feature by itself was boosted in Git 2.13.0 by adding conditional includes - one of my favourite features of this release. And something that makes me sad.
Deep in the black bashic
It’s been a while since my latest blog post. You know how it works, right? Family to care about, work to be done, places to visit, nature to explore, tea to experience, wine to enjoy, WoW to waste your time on… And then you think to yourself: ‘I want to write something to my long forgotten blog. Something that would interest at least someone’. And then you realise that your blog is ‘hosted’ on the cheapest EC2 instance and has terrible loading time. You try to fix it by moving to S3 with CDN, but you still have hakyll as a blog compiler and not everything works as you wish. You fix several minor issues and then you notice the part where you upload your blog using some black bashic, so you go to CodeBuild and CodeDeploy to automate everything and end up configuring Docker. You waste time and money, but comfort yourself saying ‘At least I am learning something new, something I can share with folks later’.
That’s how it works, isn’t it?
Being an org-mode addict
It is funny how I vainly try to escape devilish org-mode. I have several major concerns regarding it as a GTD system and a note taking system, but after all, whatever I try to replace it with, I always return to org-mode. Mostly because it’s simple (at least at the top level) and makes me feel more confident about my thoughts, tasks and the future. But on the other hand, it makes me really angry because it forces me to make some uncomfortable decisions.
Errors in shell scripts
Writing shell scripts might be pretty tedious because of all those failing commands leading to a corrupted state. And it gets even worse when you overpopulate your script with if statements which distract you from solving your initial problem. And fortunately, there are several neat tricks for handling errors in shell scripts that everyone should be familiar with.
Using Spacemacs with several Emacs versions
Since Spacemacs officially supports at least two major versions of Emacs, as a collaborator I have to switch between Emacs versions time to time. And since installed packages don’t usually work between different major version of Emacs, I came up with a hack allowing me to seamlessly switch Emacs versions.
flyspell-correct v0.1
flyspell-correct is a package for distraction-free words correction with flyspell via selected interface. It’s available on GitHub and MELPA.
Not long ago I have released the first version of flyspell-correct. There are several interesting changes compared to initial implementation. First of all, I divided this package into four different packages - core and interfaces. This allowed me to properly setup dependencies and also solved some problems with helm interface. Second important change - I’ve added a function called flyspell-correct-previous-word-generic, which is useful when the misspelled word is not under point, but long behind it. Right now I realise that it doesn’t work as perfect as I want, so I’ll improve it in the next release.
You can find more information about flyspell-correct on GitHub.
Emacs and Composability
It’s not a secret that I enjoy using Emacs. And people often ask me why it’s Emacs and not Vi(m). The top issue for me is the modal editing thing. Somehow I just don’t get it. I understand that some people are able to fully utilize this feature, but I can’t say that about myself. In any case, recently I was reading an article on Irreal about Vim being not that hard to learn compared to Emacs and found a link to another interesting article about composability of Vim.
I totally agree that composability is a great feature that unfortunately wasn’t widely adopted. It’s very powerful in use and great in terms of learnability and consistency. Just as Mike Kozlowski has stated. I don’t want to repeat his article, instead just go ahead and read it.
What I want to do is to defend Emacs (and some other editors). I really think that it was very smart move to place the stake on the ability to extend the editor. And Emacs is a great example of such editor. Just take a look at evil. Whenever I think about it, I am amazed. It doesn’t mean that I want to use it, it just shows that extensibility matters and its limits are very loose.
Let’s return to composability in Emacs. One might argue that Emacs comes with built-in composability, but let’s be honest even if it’s not true - it’s still easy to implement vim-like composability even in Emacs. It’s just a matter of effort.
To follow Mike Kozlowski’s trend, I admit that Emacs is not perfect. Some people don’t like Emacs Lisp, and out-of-box experience is far from being ideal. In order to perfect Emacs The Editor you have to invest a lot of time in writing configurations. You also have to use dozens of third-party packages and also configure them appropriately. But it gives you freedom. And in case you don’t like writing configs, just take a look at these great projects - Purcell’s emacs.d, Prelude and of course, Spacemacs. You have plenty of options. And composability is one of them.
flyspell-correct
flyspell-correct is a package for distraction-free words correction with flyspell via selected interface. It’s available on GitHub and MELPA.
It’s very common to make mistakes while writing notes, comments or whatever. And a desire to have spell checking software is natural. In Emacs, we usually use Flyspell for that purpose. It’s a great piece of software, but sometimes it lacks better interface for correcting words. Especially since there are completion packages like popup.el, helm, ivy etc. There are already available separate packages for correcting words via popup.el (wiki, flyspell-popup) and helm (helm-flyspell). Since recently I have switched from helm to ivy, I’ve found a lack of similar package for ivy, so I decided to write my own.
But available packages are all the same, except the part that calls completion front-end, so my goal was to provide a generic package allowing usage of any interface. Either from the predefined set (ivy, helm, popup or dummy) or any user-defined.
This is an introduction post, for more up-to-date documentation, please refer to README file.
Cabal and executables
Maybe I did something really wrong in my life, because I constantly run into Cabal Hell. That feeling of powerlessness is refreshing, though depressing. And I hate it the most when I just need to install an executable from Hackage, like pandoc.
But hey, we are software engineers after all. So I decided to write a little helper script to avoid world destruction and get desired executable in my $PATH.
Important note. In the last few years Haskell community did a great job in order to fight the Cabal Hell. And thanks to Stack and Nix-style Local Builds my solution is no longer required. I keep it just for historical reference (and to keep my blog relatively busy).
Make the Fish fly
I have always been inspired by the people who use (or at least can use) terminal for any kind of activity that is associated with programming. Sure, every task has its own set of instruments to be solved with. But there are instruments that you can count on in almost any situation. That’s why I spend so much time in the terminal.
For a long time (like a year and a half) I was using zsh beefed with oh-my-zsh. While it was providing me a lot of crucial functionality, I wasn’t very happy about oh-my-zsh, so when someone mentioned fish in comments to Use Haskell for shell scripting thread on Reddit, I decided to give it a try after few minutes of reading how awesome fish is.
In this article I am going to share my thoughts after using fish. But keep in mind that this is not a tutorial to fish (mostly because it already has a good written one, which covers most important aspects of day-to-day fish, but you might also want to read full documentation to get the full grasp).
Exploring new stuff is fun. So even if you are totally happy with your setup, I highly advice you to take a look at fish.
CanonicalPath v0.3.0.0
Recently there was the third major release of CanonicalPath library introducing some good stuff, but unfortunately breaking API (just a tiny little bit).