In previous articles (Vol. 1 and Vol. 2) we talked about moving tasks from regular org-mode files to org-roam notes. This relied upon adding all org-roam files to org-agenda-files, which doesn't scale well, as when you build an agenda buffer, it needs to traverse each file. Once you have more than 1k notes, things become sluggish.

In my experience, once I reached 1200 note files, org-agenda constantly took more than 50 seconds to build, rendering this tool completely useless. But then I realised that only 3% of those files actually contain any TODO entries, so there is no need to traverse whole org-roam-directory!

In this article we are going to optimise org-agenda back to less than 1 second by dynamically building org-agenda-files list to include only files with TODO entries. All thanks to the power of org-roam and some hooks I am going to describe.

Change Log:

[2021-03-02 Tue]: Update naming convention to match personal configurations.
[2021-03-08 Mon]: Gustav shared that org-element-map has an optional parameter first-match that works like seq-find, meaning that vulpea-project-p can be optimised.
[2021-05-10 Mon]: Update post to reflect changes in org-roam v2. Previous version of this article is available on GitHub.
[2021-08-19 Thu]: Gustav proposed to modify buffer only when tags have changed. Code was updated accordingly (both in the post and on GitHub Gist).
[2021-09-07 Tue]: rngesus-wept proposed an interesting solution on how to make sure that any extra stuff in org-agenda-files are not wiped out.

Task management with org-roam Vol. 7: Capture – Sharing my org-roam capture workflow, focusing on quick task entry and efficient processing. I've set up a dedicated inbox file per machine to avoid sync issues, and use org-agenda with the REFILE tag to process captured items. For meetings, I've created a smarter capture template that automatically places one-on-one notes in the right person's file - a nice example of using org-roam's query capabilities to streamline capture.
Task management with org-roam Vol. 6: Select a person and view related tasks – Sharing a quick utility I wrote to view all tasks related to a specific person in org-roam. By combining vulpea's selection functions with org-agenda's tag matching, we can easily see everything tagged with a person's name (including their aliases). Just a few lines of code, but it makes a big difference in managing person-related tasks!
Task management with org-roam Vol. 4: Automatic tagging – Sharing how I automated task tagging in org-roam with vulpea. When you mention someone in a task by linking to their note, the task automatically gets tagged with that person's name. I'm using vulpea-insert hooks to handle this cleanly, avoiding the need for advice on org-roam functions. The code includes checks to ensure we only add person tags to actual TODO items.
Task management with org-roam Vol. 3: FILETAGS – Sharing my approach to managing person-related tasks in org-roam notes. I explain how to use filetags to automatically tag all tasks in a person's note (like @FrodoBaggins), maintaining the tag inheritance functionality from regular org-mode while moving to individual roam notes. I've included code to automate the tagging process, making it easier to maintain consistency when adding new people notes.
Task management with org-roam Vol. 2: Categories – Sharing how I improved the agenda view for org-roam notes by fixing category display. Rather than showing file IDs, we can now show meaningful categories based on note titles or explicit CATEGORY properties. I've included a function to automatically extract and format categories, plus options to handle long titles gracefully. The result is a cleaner, more readable agenda that better integrates with org-roam's note-taking approach.
Task management with org-roam Vol. 1: Path to Roam – Sharing my approach to task management in org-roam, showing how to organize tasks and projects across notes while maintaining compatibility with org-mode's agenda features. While org-roam favors smaller files over larger ones, we can still implement a familiar structure of tasks, projects, and meta-projects. The article explains the basic setup, though there are some visual quirks in the agenda buffer that we'll address in a follow-up post.

The core idea is very simple - optimising reads during writes. So every time a file is modified, we check if it contains any TODO entries, and depending on that we either add or remove a project tag from filetags property. And then, before calling org-agenda, we simply org-roam-db-query for files that have a project tag.

Since filetags are inherited by default (see the value of org-use-tag-inheritance), every heading in your file will inherit project tag, which is not desirable. Since tag inheritance is useful in general, my advice is to disable inheritance specifically for project tag by adding it to org-tags-exclude-from-inheritance:

(add-to-list 'org-tags-exclude-from-inheritance "project")

some random image you can find on barberry garden

Marking a Project

In order to mark a note as a project, we need to check if it contains any TODO entries. One of the way to do it is to use Org Element API, a set of parsing functions.

(defun vulpea-project-p ()
  "Return non-nil if current buffer has any todo entry.

TODO entries marked as done are ignored, meaning the this
function returns nil if current buffer contains only completed
tasks."
  (org-element-map                          ; (2)
       (org-element-parse-buffer 'headline) ; (1)
       'headline
     (lambda (h)
       (eq (org-element-property :todo-type h)
           'todo))
     nil 'first-match))                     ; (3)

This might look a little bit too much, so let me explain the code step by step.

We parse the buffer using org-element-parse-buffer. It returns an abstract syntax tree of the current Org buffer. But sine we care only about headings, we ask it to return only them by passing a GRANULARITY parameter - 'headline. This makes things faster.
Then we extract information about TODO keyword from headline AST, which contains a property we are interested in - :todo-type, which returns the type of TODO keyword according to org-todo-keywords - 'done, 'todo or nil (when keyword is not present).
Now all we have to do is to check if the buffer list contains at least one keyword with 'todo type. We could use seq=find on the result of org-element-map, but it turns out that it provides an optional first-match argument that can be used for our needs. Thanks Gustav for pointing that out.

Now we need to use this function to add or to remove project tag from a note. I think that it should be done in two places - when visiting a note and in before-save-hook. This way you leave no room for missing a file with TODO entries. It uses vulpea-buffer-tags-get and vulpea-buffer-tags-add from vulpea library (for now you should use org-roam-v2 branch).

(add-hook 'find-file-hook #'vulpea-project-update-tag)
(add-hook 'before-save-hook #'vulpea-project-update-tag)

(defun vulpea-project-update-tag ()
      "Update PROJECT tag in the current buffer."
      (when (and (not (active-minibuffer-window))
                 (vulpea-buffer-p))
        (save-excursion
          (goto-char (point-min))
          (let* ((tags (vulpea-buffer-tags-get))
                 (original-tags tags))
            (if (vulpea-project-p)
                (setq tags (cons "project" tags))
              (setq tags (remove "project" tags)))

            ;; cleanup duplicates
            (setq tags (seq-uniq tags))

            ;; update tags if changed
            (when (or (seq-difference tags original-tags)
                      (seq-difference original-tags tags))
              (apply #'vulpea-buffer-tags-set tags))))))

(defun vulpea-buffer-p ()
  "Return non-nil if the currently visited buffer is a note."
  (and buffer-file-name
       (string-prefix-p
        (expand-file-name (file-name-as-directory org-roam-directory))
        (file-name-directory buffer-file-name))))

That's it. Now whenever we modify or visit a notes buffer, this code will update the presence of project tag. See it in action:

Building agenda

In order to dynamically build org-agenda-files, we need to query all files containing project tag. org-roam uses uses skeeto/emacsql, and provides a convenient function org-roam-db-query to execute SQL statements against org-roam-db-location file.

(defun vulpea-project-files ()
  "Return a list of note files containing 'project' tag." ;
  (seq-uniq
   (seq-map
    #'car
    (org-roam-db-query
     [:select [nodes:file]
      :from tags
      :left-join nodes
      :on (= tags:node-id nodes:id)
      :where (like tag (quote "%\"project\"%"))]))))

This function simply returns a list of files containing project tag. Sure enough it can be generalised for other needs, but it's good enough for our simple use case. The query is run against the following schemes:

(nodes
 ([(id :not-null :primary-key)
   (file :not-null)
   (level :not-null)
   (pos :not-null)
   todo
   priority
   (scheduled text)
   (deadline text)
   title
   properties
   olp]
  (:foreign-key [file] :references files [file] :on-delete :cascade)))

(tags
 ([(node-id :not-null)
   tag]
  (:foreign-key [node-id] :references nodes [id] :on-delete :cascade)))

Now we can set the list of agenda files:

(setq org-agenda-files (vulpea-project-files))

But the real question is - when to do it? Some might put it in the init.el file and call it a day, but unless you are restarting Emacs like crazy, I would argue that it's not the best place to do it. Because we need an up to date list of files exactly when we build agenda.

(defun vulpea-agenda-files-update (&rest _)
  "Update the value of `org-agenda-files'."
  (setq org-agenda-files (vulpea-project-files)))

(advice-add 'org-agenda :before #'vulpea-agenda-files-update)
(advice-add 'org-todo-list :before #'vulpea-agenda-files-update)

And that's all. You org-agenda is up to date and fast again!

Migration

So far we covered what to do with notes we edit. But when you have more than 10 notes it becomes tedious to visit each of them and make sure that they have update state of Project tag. Fortunately, this task is easily automated.

(dolist (file (org-roam-list-files))
  (message "processing %s" file)
  (with-current-buffer (or (find-buffer-visiting file)
                           (find-file-noselect file))
    (vulpea-project-update-tag)
    (save-buffer)))

This will visit each of your files and update the presence of Project tag according to presence of TODO entry. Now you are ready to go.

Result

With little amount of emacs-lisp code we dramatically optimized org-agenda loading from $> 50$ seconds to $< 1$ second. Effectiveness of this approach depends on amount of files with TODO entries (the more you have, the less effective this approach becomes). One of the drawbacks is small (in my experience, neglectable) performance degradation of note visiting and note saving. Obviously, if a file contains thousands of headings, it affects performance. In defence, I would argue that such files are against the philosophy of org-roam, where you keep lots of small files as opposed to few huge files.

For you convenience, the full code is displayed below. It is also available as GitHub Gist.

(defun vulpea-project-p ()
  "Return non-nil if current buffer has any todo entry.

TODO entries marked as done are ignored, meaning the this
function returns nil if current buffer contains only completed
tasks."
  (seq-find                                 ; (3)
   (lambda (type)
     (eq type 'todo))
   (org-element-map                         ; (2)
       (org-element-parse-buffer 'headline) ; (1)
       'headline
     (lambda (h)
       (org-element-property :todo-type h)))))

(defun vulpea-project-update-tag ()
    "Update PROJECT tag in the current buffer."
    (when (and (not (active-minibuffer-window))
               (vulpea-buffer-p))
      (save-excursion
        (goto-char (point-min))
        (let* ((tags (vulpea-buffer-tags-get))
               (original-tags tags))
          (if (vulpea-project-p)
              (setq tags (cons "project" tags))
            (setq tags (remove "project" tags)))

          ;; cleanup duplicates
          (setq tags (seq-uniq tags))

          ;; update tags if changed
          (when (or (seq-difference tags original-tags)
                    (seq-difference original-tags tags))
            (apply #'vulpea-buffer-tags-set tags))))))

(defun vulpea-buffer-p ()
  "Return non-nil if the currently visited buffer is a note."
  (and buffer-file-name
       (string-prefix-p
        (expand-file-name (file-name-as-directory org-roam-directory))
        (file-name-directory buffer-file-name))))

(defun vulpea-project-files ()
    "Return a list of note files containing 'project' tag." ;
    (seq-uniq
     (seq-map
      #'car
      (org-roam-db-query
       [:select [nodes:file]
        :from tags
        :left-join nodes
        :on (= tags:node-id nodes:id)
        :where (like tag (quote "%\"project\"%"))]))))

(defun vulpea-agenda-files-update (&rest _)
  "Update the value of `org-agenda-files'."
  (setq org-agenda-files (vulpea-project-files)))

(add-hook 'find-file-hook #'vulpea-project-update-tag)
(add-hook 'before-save-hook #'vulpea-project-update-tag)

(advice-add 'org-agenda :before #'vulpea-agenda-files-update)
(advice-add 'org-todo-list :before #'vulpea-agenda-files-update)

Thank you for your patience.

References

Org Element API
skeeto/emacsql
Code from this article is available as GitHub Gist

Task management with org-roam Vol. 5: Dynamic and fast agenda

Related posts

Marking a Project

Building agenda

Migration

Result

References