Task management with org-roam Vol. 5: Dynamic and fast agenda

January 16, 2021
(emacs, org-roam, org-mode)

In previous articles (Vol 1 and Vol 2) we talked about moving tasks from regular org-mode files to org-roam notes. This relied upon adding all org-roam files to org-agenda-files, which doesn’t scale well, as when you build an agenda buffer, it needs to traverse each file. Once you have more than 1k notes, things become sluggish.

In my experience, once I reached 1200 note files, org-agenda constantly took more than 50 seconds to build, rendering this tool completely useless. But then I realised that only 3% of those files actually contain any TODO entries, so there is no need to traverse whole org-roam-directory!

In this article we are going to optimise org-agenda back to less than 1 second by dynamically building org-agenda-files list to include only files with TODO entries. All thanks to the power of org-roam and some hooks I am going to describe.

The core idea is very simple - optimising reads during writes. So every time a file is modified, we check if it contains any TODO entries, and depending on that we either add or remove a Project tag from ROAM_TAGS property. And then, before calling org-agenda, we simply org-roam-db-query for files that have a Project tag.

Marking a Project

In order to mark a note as a Project, we need to check if it contains any TODO entries. One of the way to do it is to use Org Element API, a set of parsing functions.

(defun +org-notes-project-p ()
  "Return non-nil if current buffer has any todo entry.

TODO entries marked as done are ignored, meaning the this
function returns nil if current buffer contains only completed
tasks."
  (seq-find                                 ; (3)
   (lambda (type)
     (eq type 'todo))
   (org-element-map                         ; (2)
       (org-element-parse-buffer 'headline) ; (1)
       'headline
     (lambda (h)
       (org-element-property :todo-type h)))))

This might look a little bit too much, so let me explain the code step by step.

  1. We parse the buffer using org-element-parse-buffer. It returns an abstract syntax tree of the current Org buffer. But sine we care only about headings, we ask it to return only them by passing a GRANULARITY parameter - 'headline. This makes things faster.
  2. Then we extract information about TODO keyword from headline AST, which contains a property we are interested in - :todo-type, which returns the type of TODO keyword according to org-todo-keywords - 'done, 'todo or nil (when keyword is not present).
  3. Now all we have to do is to check if this list contains at least one keyword with 'todo type.

Now we need to use this function to add or to remove Project tag from a note. I think that it should be done in two places - when visiting a note and in before-save-hook. This way you leave no room for missing a file with TODO entries.

(add-hook 'find-file-hook #'+org-notes-project-update-tag)
(add-hook 'before-save-hook #'+org-notes-project-update-tag)

(defun +org-notes-project-update-tag ()
  "Update PROJECT tag in the current buffer."
  (when (and (not (active-minibuffer-window))
             (+org-notes-buffer-p))
    (let* ((file (buffer-file-name (buffer-base-buffer)))
           (all-tags (org-roam--extract-tags file))
           (prop-tags (org-roam--extract-tags-prop file))
           (tags prop-tags))
      (if (+org-notes-project-p)
          (setq tags (cons "Project" tags))
        (setq tags (remove "Project" tags)))
      (unless (eq prop-tags tags)
        (org-roam--set-global-prop
         "ROAM_TAGS"
         (combine-and-quote-strings (seq-uniq tags)))))))

(defun +org-notes-buffer-p ()
  "Return non-nil if the currently visited buffer is a note."
  (and buffer-file-name
       (string-prefix-p
        (expand-file-name (file-name-as-directory org-roam-directory))
        (file-name-directory buffer-file-name))))

That’s it. Now whenever we modify or visit a notes buffer, this code will update the presence of Project tag. See it in action:

Building agenda

In order to dynamically build org-agenda-files, we need to query all files containing Project tag. org-roam uses uses skeeto/emacsql, and provides a convenient function org-roam-db-query to execute SQL statements against org-roam-db-location file.

(defun +org-notes-project-files ()
  "Return a list of note files containing Project tag."
  (seq-map
   #'car
   (org-roam-db-query
    [:select file
     :from tags
     :where (like tags (quote "%\"Project\"%"))])))

This function simply returns a list of files containing Project tag. Sure enough it can be generalised for other needs, but it’s good enough for our simple use case. The query is run against the following scheme:

(tags
 [(file :unique :primary-key)
  (tags)])

Now we can set the list of agenda files:

(setq org-agenda-files (+org-notes-project-files))

But the real question is - when to do it? Some might put it in the init.el file and call it a day, but unless you are restarting Emacs like crazy, I would argue that it’s not the best place to do it. Because we need an up to date list of files exactly when we build agenda.

(defun +agenda-files-update (&rest _)
  "Update the value of `org-agenda-files'."
  (setq org-agenda-files (+org-notes-project-files)))

(advice-add 'org-agenda :before #'+agenda-files-update)

And that’s all. You org-agenda is up to date and fast again!

Migration

So far we covered what to do with notes we edit. But when you have more than 10 notes it becomes tedious to visit each of them and make sure that they have update state of Project tag. Fortunately, this task is easily automated.

(dolist (file (org-roam--list-all-files))
  (message "processing %s" file)
  (with-current-buffer (or (find-buffer-visiting file)
                           (find-file-noselect file))
    (+org-notes-project-update-tag)
    (save-buffer)))

This will visit each of your files and update the presence of Project tag according to presence of TODO entry. Now you are ready to go.

Result

With little amount of emacs-lisp code we dramatically optimized org-agenda loading from \(> 50\) seconds to \(< 1\) second. Effectiveness of this approach depends on amount of files with TODO entries (the more you have, the less effective this approach becomes). One of the drawbacks is small (in my experience, neglectable) performance degradation of note visiting and note saving. Obviously, if a file contains thousands of headings, it affects performance. In defence, I would argue that such files are against the philosophy of org-roam, where you keep lots of small files as opposed to few huge files.

For you convenience, the full code is displayed below. It is also available as GitHub Gist.

(defun +org-notes-project-p ()
  "Return non-nil if current buffer has any todo entry.

TODO entries marked as done are ignored, meaning the this
function returns nil if current buffer contains only completed
tasks."
  (seq-find                                 ; (3)
   (lambda (type)
     (eq type 'todo))
   (org-element-map                         ; (2)
       (org-element-parse-buffer 'headline) ; (1)
       'headline
     (lambda (h)
       (org-element-property :todo-type h)))))

(defun +org-notes-project-update-tag ()
  "Update PROJECT tag in the current buffer."
  (when (and (not (active-minibuffer-window))
             (+org-notes-buffer-p))
    (let* ((file (buffer-file-name (buffer-base-buffer)))
           (all-tags (org-roam--extract-tags file))
           (prop-tags (org-roam--extract-tags-prop file))
           (tags prop-tags))
      (if (+org-notes-project-p)
          (setq tags (cons "Project" tags))
        (setq tags (remove "Project" tags)))
      (unless (eq prop-tags tags)
        (org-roam--set-global-prop
         "ROAM_TAGS"
         (combine-and-quote-strings (seq-uniq tags)))))))

(defun +org-notes-buffer-p ()
  "Return non-nil if the currently visited buffer is a note."
  (and buffer-file-name
       (string-prefix-p
        (expand-file-name (file-name-as-directory org-roam-directory))
        (file-name-directory buffer-file-name))))

(defun +org-notes-project-files ()
  "Return a list of note files containing Project tag."
  (seq-map
   #'car
   (org-roam-db-query
    [:select file
     :from tags
     :where (like tags (quote "%\"Project\"%"))])))

(defun +agenda-files-update (&rest _)
  "Update the value of `org-agenda-files'."
  (setq org-agenda-files (+org-notes-project-files)))

(add-hook 'find-file-hook #'+org-notes-project-update-tag)
(add-hook 'before-save-hook #'+org-notes-project-update-tag)

(advice-add 'org-agenda :before #'+agenda-files-update)

Thank you for your patience.

References