path: root/content/emacs/notmuch.md

+++
title = "Read email with Notmuch"
author = ["Michał Sapka"]
date = 2023-07-03T23:00:00+02:00
categories = ["emacs"]
draft = false
weight = 3002
abstract = "My email based workflow for GitHub Pull Review Requests"
aliases = ["/2023/notmuch/",  "/cool-emacs/read-email-notmuch"]
[menu]
  [menu.cool-emacs-ways]
    weight = 3002
    identifier = "read-email-with-notmuch"
    name = "Reading and automating email using Notmuch"
+++

Web email interfaces have taken over the world a long time ago.
Except for Outlook users, only a few people even consider using an actual application for it.
This is one of the primary reasons for our low opinion of electronic mail today, since with limited tools, you get limited capabilities.
While there are options for automatic filtering, tagging, or folding with most clients, those capabilities are barebones.

Luckily there are still compelling alternatives.


## Managing email locally {#managing-email-locally}

Back in the day of POP, this was the standard: your computer downloaded new messages from the server, and you had local copies from this point.
Then IMAP came, and after it, Exchange.
Now you have a fast and convenient option to sync all changes with other clients and not bother with hard drive space.
But this also puts us at the mercy of email providers.
If a given service doesn't support tags, folders, or saved searches, you cannot use it.
I tried to reverse this direction and never looked back.
I want all my email to be on my machine, available at all times, and accessible instantly.

Mbsync(1)[^fn:1] is a popular application to populate local ~/Mail folder with remote messages fetched via IMAP.
The config is quite simple. For Gmail[^fn:2]<sup>, </sup>[^fn:3]:

```shell { linenos=true, linenostart=1, anchorlinenos=true, lineanchors=org-coderef--c9c8f8 }
MaildirStore gmail-local
Path ~/Mail/gmail/
Inbox ~/Mail/gmail/INBOX
Subfolders Verbatim

IMAPStore gmail-remote
Host imap.gmail.com
SSLType IMAPS
AuthMechs LOGIN
User <username>
PassCmd <password>

Channel gmail
Far :gmail-remote:
Near :gmail-local:
Create Both
Expunge
Patterns * !"[Gmail]/All Mail" !"[Gmail]/Important" !"[Gmail]/Starred" !"[Gmail]/Bin"
SyncState *
```

Then type `mbsybc -a`, wait for a few (hundred? depending on the mailbox size) minutes, and voila.
Your =

~/Mail/gmail/= is now populated with all your messages. Let's break down the config.

Mbsync(1) assumes two stores exist: local (on your computer) and remote (the IMAP server).
A store is a place where mail exists.
We have then configured in lines [1](#org-coderef--c9c8f8-1)-[4](#org-coderef--c9c8f8-4) and [6](#org-coderef--c9c8f8-6)-[11](#org-coderef--c9c8f8-11).

The remote one is self-explanatory.
One thing to remember: some providers will require you to use an app-specific password and reject auth attempts with the normal one.
Our password in line 11 can be either a string with the password or an arbitrary command (think `cat ~/my-secret-password` or a CLI password manager).

The local store is just a definition of local folders to use.
It can be anywhere, but `~/Mail` is the standard, anxd many mail clients will assume that you store your email there.

On line [13](#org-coderef--c9c8f8-13), we start to define a channel, which is how mbsync(1) works.
One store is `far` (remote), while the other is `near` (on your machine).
The rest of the config defines behavior.
Refer to the manual, but in my case:

-   it will create non-matching mailboxes
-   it will delete messages in a store if a message was deleted on the other
-   it will sync all messages except a few matching the pattern.
-   it will store the synchronization state file in the Mail dir.

One last thing to add is a simple cron rule.
You can force mail fetching manually, but I opted for the automatic option.
Therefore, my crontab(1) has:

```cron
1/5 * * * * killall mbsync &>/dev/null; <msync_dir>/mbsync -a -V 2>&1 > ~/.mbsync_log
```

We will now fetch new messages every 5 minutes.


## Local clients {#local-clients}

Now we need to choose a local email client to use.
There is a lot to choose from! Thunderbird[^fn:4] seems to be the most popular option for GUI users.
For terminal users, we have Mutt(1)[^fn:5], its successor Neomutt(1)[^fn:6], and many, many more.
I used Neomutt(1)[^fn:7] for a while, and it was a pleasure compared to web clients.

However, ever since I started to use Emacs more, I wanted to move my Email inside Emacs.
And to little surprise, we also have a lot to choose from.
By default, Emacs comes with Gnus[^fn:8], a newsgroups reader with an email client capability.
However, the two most popular packages are Mu4e[^fn:9] and notmuch(1)[^fn:10]

The last two are based on fast email indexing and searching but assume different workflows.
Mu4e is based on filtering, while notmuch around tagging.

A friend[^fn:11] recommended Notmuch(1) to me.
I have never tried mu4e as notmuch fully satisfies my needs.


## Setting up notmuch(1) {#setting-up-notmuch--1}

Notmuch(1) is not a client but an indexer.
It indexes and tags existing email and allows one to search messages.
It should be present on most systems package management, so install it.
They run `notmuch`, answer a few questions, and you've got yourself a ready notmuch.

Whenever a new mail arrives, it won't be known to notmuch before indexing.
You can manually run `notmuch new` or make a cron definition for it.

One killer feature of notmuch is its sheer speed.
The name comes from the fact that it can work on gigantic mailboxes very swiftly - oh, you have one million messages? That's not much!

Let's try a simple search:

```shell
$ notmuch search 'from:*@github.com'
```

You can search based on sender, receiver, dates, attachments, contents, titles, etc.
Refer to man pages for `notmuch-search-terms(7)`.
However, to get the most out of notmuch, we need to learn about tags.

Let's add a "gh" tag for messages from Git Hub.

```shell
$ notmuch search 'from:*@github.com'
$ notmuch tag "+gh" -- "from:@github.com"
```

Now, we can search for such messages

```shell
$ notmuch search "tag:gh"
```

We can also join multiple search criteria with "and", "or" and other boolean operators.
We now have a fully working local email reader - however we can not send email.
I will not discuss sending email here as it's a separate subject. Notmuch(1) is not for sending email.

Using CLI for reading email is far from pleasant.
All those commands will come in handy, but first, let's add a user interface.


## Notmuch(1) in Emacs(1) {#notmuch--1--in-emacs--1}

Notmuch(1) can be used with different UIs, like Mutt(1).
However, it comes with an Emacs(1) package, so let's enable it.

```emacs-lisp
(use-package notmuch
  :commands notmuch-hello
  :bind (("C-c m" . notmuch-hello)))
```

The key binding "C-c m" is very popular, but you can use whatever you want.

After running `notmuch-hello`, we are not greeted with a list of messages but with a search interface.
You've got access to saved searches, recent searches, and a list of tags.
This tells us that we are dealing with a completely different beast than webmail, and the user needs to think of new workflows.

The way we are thought of thinking of email is a list of messages.
Some clients can mark messages that are more important, favorite, tag them, move them into folders, etc.
But everywhere I know, the primary interface is just a list - unread email on top, read on the bottom.
The behavior I always expect is to open (or mark as read) all incoming messages and then ignore most of them.
If you spent some time on configuration, maybe you have automatic rules - like moving all newsletters to a "newsletter" folder and removing them from your inbox.
Hey[^hey] is even based on separating all incoming messages into three tiers: important mail, newsletters, and paper trial.
[^hey]: [Hey email website](<https://www.hey.com/>). It's the best email service I know, but it doesn't allow any external clients, and the pricing is unacceptable for me.

But back to notmuch.
Look at saved searches - we will use them later.
Open "unread," and we see a semi-normal list of messages.
Use "n" and "p" to select email, enter to open it, and so on - standard Emacs stuff.
One thing to remember is that by default, reading an email will not mark it as read.
You need to manually remove the tag via `k u` - either one by one, or on all messages in a selected region (C-spc, it's still a buffer, after all).
"Unread" here is just another tag.
We can be much smarter with marking actionable items.


## Automatic Github Pull Review workflow {#automatic-github-pull-review-workflow}

What we've seen before is nothing more than a normal email client with extra steps.
We read emails in Emacs(1), which is great, but we don't get anything extra.
It's time for a real-world example.

I am a software engineer forced to work with GitHub.
One thing I do is to review pull requests.
The primary problem here is knowing that someone wants me to review something.
The review itself is the easy part :-)

I rely solely on email for this information, ostensibly ignoring all nudges on Slack[^slack].
[^slack]: Back when Slack was first sold, it was proposed not only as a chat tool but also as a single place for all information.
We see it now: we connect Slack to everything - GitHub, jira, Data Dog, or pager duty.
The general idea is great, but Slack is a pretty mediocre application.
The only way to manage what you receive is to leave a channel.
But then you lose all other messages sent there, so the price may be high.

First of all, we need to enable email notifications from GitHub.
Remember to mark that you want to get emails about your own actions.

Now, let's think about what we want to achieve.
For me it is "I want to know about all the pull requests I should look at without opening browser[^fn:12]".
This means I want to see all the review requests I was assigned (personally or by being part of a team) that I have not yet reviewed.
And it can achieve the same.
However, let's ignore it for now, as the same model of email-based dashboards can be expanded to many other things.

Luckily, GitHub allows us to get that from email:

1.  When you are first assigned a review, you get a dedicated email
2.  When you approve or reject a PR, you get an email
3.  When someone asks you to re-review an email, you get the same email as it was the first request for this PR.

We now know that we can use Notmuch(1).
There are two ways: we can use `notmuch-hooks(5)` and place a shell script in =~/Mail/.notmuch/hooks/post-new, but it never worked reliably for me.
Instead, I have a cron job that runs a script:

```shell { linenos=true, linenostart=1 }
#!/usr/bin/env sh

notmuch tag +gh -unread -- '(from:notifications@github.com)'

# Mark new review requests
for thread in $(notmuch search --sort=oldest-first --output=threads -- "\"requested your review on\" and tag:gh and -tag:gh-pr-done"); do
    for msg in $(notmuch search --sort=oldest-first --output=messages -- "$thread"); do
      txt=$(notmuch show "$msg")

      (echo "$txt" | grep "requested your review on") && notmuch tag +gh-pr-todo -- "$thread"
      (echo "$txt" | grep "@michalsapka approved this pull request") && notmuch tag -gh-pr-todo -- "$thread"
      (echo "$txt" | grep "@michalsapka requested changes on this pull request") && notmuch tag -gh-pr-todo -- "$thread"
      (echo "$txt" | grep "Merged.*into") && notmuch tag -gh-pr-todo +gh-pr-done -- "$thread"
    done
done
```

Let's break it down:

1.  First, we add a "gh" tag to all notification emails from GitHub and remove the "unread" tag.
    I don't need to be notified about all such emails, but I can still look at the "gh" tag if needed.
2.  Then we search for threads where an email informs me about a review request.
    I limit the search to emails from GitHub via the tag from #1 and those without "gh-pr-done" tag. More on the second one in a moment
3.  Then I search for all messages in such threads.
    I force order as oldest-first to make it possible to reason about.
    In normal PR, all actions happen with a significant delay between them, so this should be enough not to get lost in the timeframe.
    If I ask for a change in review, the re-request will not happen instantly.
    Note that I get the email body as a variable on line 8.
4.  Then comes the meat.
    I will tag the entire thread multiple times based on the body of the message.
    When a request comes, a "gh-pr-todo" is added.
    I need to look at it.
    When I approve or reject a PR, the tag is removed.
    If someone asks for a review, logic from line 10 will be triggered, and the tag will be added again.
    This means that I want to handle all email threads with the "gh-pr-todo" tag.
5.  Lastly, when a PR is merged, I ensure that the "gh-pr-todo" tag is removed, and I add the "gh-pr-done" tag so this thread will not be found in step 2 in the future.

There are other ways to tag, like afew(1)[^fn:13], but keeping it to simple shell script working with notmuch(1) directly gives us the greatest amount of freedom and made it easier for me to tell this story.


## Making it more visible {#making-it-more-visible}

This alone would be a challenge to manage.
An email with a tag would be easily missed.
Notmuch has us covered yet again! My emacs config has a few dedicated lines:

```emacs-lisp { linenos=true, linenostart=1 }
(setq notmuch-search-line-faces
     '(("gh-pr-todo" . ((t :foreground "#f77"))))
     notmuch-saved-searches
     ;; other saved searches omitted
     ( :name "GitHub[reviews]"
       :query "tag:gh-pr-todo"
       :sort-order newest-first))
```

This makes two changes.

1.  Firstly, all messages with "gh-pr-todo" will be shown in red in any email list.
    All red items are actionable since we remove this tag in the workflow.
2.  Secondly, amongst other saved searches, I have one dedicated to PRs.

With those two things, every time I enter `notmuch-hello` screen, I get instant information about the work I need to do.


## Making it extra visible {#making-it-extra-visible}

But we can go one step further.
Prot's[^fn:14] excellent notmuch-indicator[fn: not-ind] allows us to add saved searches to the mode line.
After installing it, the configuration is straightforward:

```emacs-lisp
(notmuch-indicator-mode 1)
(setq notmuch-indicator-refresh-count 60
      notmuch-indicator-hide-empty-counters t
      notmuch-indicator-args
      '((:terms "tag:gh-pr-todo":label "pr:")))
```

This adds a "pr:&lt;count&gt;" to the mode line.
The count is the number of messages, not threads, but frankly, I want it to be 0.
The counter will be refreshed every 60 seconds.
And lastly, if the count is 0, the label will not be added to the mode line.


## The downsides {#the-downsides}

Notmuch comes with one significant downside: lack of multi-device support.
It's 2023, and most of us have more than one computer and those pesky mobile phones.

As for the mobile - I have no solution.
The read statuses will sync via mbsync(1), but not much else.
I try to purge myself from phone addiction, so maybe that's a plus?

As for the other computers, we have muchsync(1)[^fn:15].
It's an external application designed to sync entire mailboxes and tags between devices over ssh.
I have not tried it yet[^fn:16], but it looks promising.


## Summary {#summary}

With local email and tools like notmuch(1), we are not at the mercy of external tools for even sophisticated workflows. If you get transactional emails, you can extract actionable data. It can be JIRA tickets, Pager Duty alerts, heck - even Amazon deliveries. Here I have demonstrated how easy it is to leverage notmuch(1), simple shell script, and emacs(1) to have a  fully automated notification setup. It does not try to hijack your attention (like mobile notifications do) and is not hidden on some webpage (like GitHub notification), but it still gives actionable results. And all that without leaving the comfort of Emacs.

One cool thing I plan to apply soon is integrating notmuch(1) with Org-mode with the ol-notmich[oln] package. But for now, I am in the process of moving as many external services to a similar workflow as possible.
[^oln]: [ol-notmuch on sourcehut](<https://git.sr.ht/~tarsius/ol-notmuch>)

[^fn:1]: [mbsync on sourceforge](https://isync.sourceforge.io/mbsync.html)
[^fn:2]: I have my gripes with Gmail, but it is still the standard.
    While this article assumes Gmail, you can easily use mbsync(1) with any IMAP service.
[^fn:3]: Originally I took the config from [Frosty](https://frostyx.cz/posts/synchronize-your-2fa-gmail-with-mbsync)
[^fn:4]: [Thunderbird website](https://www.thunderbird.net/)
[^fn:5]: [Mutt website](https://www.mutt.org/)
[^fn:6]: [Neomutt website](https://neomutt.org/)
[^fn:7]: Luke Smith has a few excellent vlogs about Mutt/NeoMutt like [Email on the terminal with mutt](https://www.youtube.com/watch?v=2jMInHnpNfQ&t=683s)%22)
[^fn:8]: [Gnus on gnu.org](https://www.gnu.org/software/emacs/manual/html_node/gnus/)
[^fn:9]: [mu4e website](https://www.djcbsoftware.nl/code/mu/)
[^fn:10]: [notmuch website](https://notmuchmail.org/)
[^fn:11]: Thanks, [Alex](https://se30.xyz/)
[^fn:12]: There is the GitHub CLI which is amazing by itself - one of the best things that GitHub has done in the last few years.
[^fn:13]: [Afew on Github](https://github.com/afewmail/afew)
[^fn:14]: [Prot's website](<https://protesilaos.com/>)
[^fn:15]: [Muchsync homepage](https://www.muchsync.org/)
[^fn:16]: my work computer gets all work messages, and my private one gets all private ones—complete separation. When I get a second personal machine, I will set it up, but for now, there is no use case for me.