path: root/2022/captions/emacsconf-2022-mail--revisiting-the-anatomy-of-emacs-mail-user-agents--mohsen-banan--main.vtt

WEBVTT captioned by mohsen

NOTE Introduction

00:00:00.000 --> 00:00:09.719
Greetings. Salaam. This is Mohsen Banan. محسن بنان.

00:00:09.720 --> 00:00:12.559
I am a software and internet engineer.

00:00:12.560 --> 00:00:14.519
I have been interested in email and

00:00:14.520 --> 00:00:17.199
Emacs for a very long  time.

00:00:17.200 --> 00:00:21.159
My interest in email started with X.400

00:00:21.160 --> 00:00:27.599
and the Red and Blue CCITT books -- circa 1988.

00:00:27.600 --> 00:00:31.999
Early  on, in the very early 1990s, I jumped ship

00:00:32.000 --> 00:00:35.279
and joined the Internet email movement.

00:00:35.280 --> 00:00:38.399
I am the primary author of two mobile email

00:00:38.400 --> 00:00:45.839
related Internet RFCs, RFC-2188 and RFC-2524.

00:00:45.840 --> 00:00:49.919
My interest in Emacs started in 1986 --

00:00:49.920 --> 00:00:54.959
It was Emacs version 17 then. By around 1988

00:00:54.960 --> 00:00:58.479
when Emacs version 18 was well in place,

00:00:58.480 --> 00:01:01.799
I started living inside of Emacs.

00:01:01.800 --> 00:01:06.479
My primary digital environment has been Emacs ever since.

00:01:06.480 --> 00:01:11.319
It has been a good life.

00:01:11.320 --> 00:01:16.999
It turns out that Emacs and email mix up really well.

00:01:17.000 --> 00:01:21.519
Here, in this presentation and in the context of

00:01:21.520 --> 00:01:26.799
Revisiting The Anatomy of Emacs Mail User Agents,

00:01:26.800 --> 00:01:30.319
With MARMEE (Multi-Account Resident

00:01:30.320 --> 00:01:33.399
Message Exchange Environment)

00:01:33.400 --> 00:01:35.559
I am offering my thoughts on this topic

00:01:35.560 --> 00:01:41.079
in this Emacs Conference 2022.

NOTE Mail and the digital ecosystem

00:01:41.080 --> 00:01:43.359
Long ago, I asked myself:

00:01:43.360 --> 00:01:49.119
"What should my ultimate mail environment be?"

00:01:49.120 --> 00:01:52.140
Over the past 20+ years, I have been exploring

00:01:52.141 --> 00:01:58.519
the concept of the "Ultimate Mail User Agent (MUA)".

00:01:58.520 --> 00:02:01.439
We do care about privacy, autonomy,

00:02:01.440 --> 00:02:05.039
morality, ethics, society and philosophy,

00:02:05.040 --> 00:02:10.679
so from the get go, proprietary (Haraam) environments

00:02:10.680 --> 00:02:14.199
such as Microsoft Office's Outlook

00:02:14.200 --> 00:02:20.159
and Google Office's Gmail were non-starters for me.

00:02:20.160 --> 00:02:23.159
But these are significant realities

00:02:23.160 --> 00:02:27.879
and we need to deal with these realities.

00:02:27.880 --> 00:02:30.199
Notice how Microsoft and Google

00:02:30.200 --> 00:02:36.079
have both framed their MUAs in the context of "office".

00:02:36.080 --> 00:02:38.719
That type of framing is correct.

00:02:38.720 --> 00:02:41.719
an MUA must be fully integrated

00:02:41.720 --> 00:02:46.559
in the totality of one's digital ecosystem.

00:02:46.560 --> 00:02:49.759
So, the Ultimate Mail User Agent

00:02:49.760 --> 00:02:53.399
must be part of the Ultimate Usage Environment

00:02:53.400 --> 00:02:57.599
of the Ultimate Digital Ecosystem.

00:02:57.600 --> 00:03:02.799
In the non-proprietary (Halaal) universe, clearly

00:03:02.800 --> 00:03:07.199
the ultimate usage environment is Emacs.

00:03:07.200 --> 00:03:10.879
Emacs is today's most potent and convivial

00:03:10.880 --> 00:03:15.159
non-proprietary usage environment.

00:03:15.160 --> 00:03:19.399
So, clearly, the ultimate Mail User Agent

00:03:19.400 --> 00:03:22.879
must be an integral part of Emacs.

NOTE Emacs and mail

00:03:22.880 --> 00:03:24.157
Having reached that conclusion,

00:03:24.158 --> 00:03:28.124
we then need to determine the specifics

00:03:28.125 --> 00:03:33.599
of the shape and the anatomy of Emacs' MUAs.

00:03:33.600 --> 00:03:36.039
We could have arrived at this conclusion

00:03:36.040 --> 00:03:38.239
from the reverse direction as well.

00:03:38.240 --> 00:03:41.599
Zawinski's Law states:

00:03:41.600 --> 00:03:46.479
Every program attempts to expand until it can read mail.

00:03:46.480 --> 00:03:49.119
Those programs which cannot so expand

00:03:49.120 --> 00:03:52.679
are replaced by ones which can.

00:03:52.680 --> 00:03:56.719
Jamie's point is very simple and obvious.

00:03:56.720 --> 00:03:59.759
The "App" that you "live in" all day

00:03:59.760 --> 00:04:05.439
should be your MUA and mail environment.

00:04:05.440 --> 00:04:09.559
I ask those who jumped ship, who abandoned Emacs

00:04:09.560 --> 00:04:15.879
in favor of VS Code: What about mail?

00:04:15.880 --> 00:04:21.279
Long ago, Emacs expanded to including MUAs.

00:04:21.280 --> 00:04:27.039
In fact there are many Emacs MUAs that you can choose from.

00:04:27.040 --> 00:04:30.599
If you are already hip with Emacs And Linux,

00:04:30.600 --> 00:04:34.639
you should definitely consider doing email in Emacs.

00:04:34.640 --> 00:04:39.079
But if you are not already hip with Emacs,

00:04:39.080 --> 00:04:43.359
I mean for new Emacs users, unfortunately,

00:04:43.360 --> 00:04:48.599
setting up and using email is not straight forward.

00:04:48.600 --> 00:04:54.519
We (I mean, Emacs developers) should work on that!

00:04:54.520 --> 00:04:57.359
Emacs offers a good number of MUAs with

00:04:57.360 --> 00:05:01.959
different characteristics to suit differing tastes.

00:05:01.960 --> 00:05:06.119
As of 2022, you can choose from the following MUAs:

00:05:06.120 --> 00:05:15.079
Gnus, VM, WanderLust, Mew, mu4e, notmuch.el, mh-e and Rmail.

00:05:15.080 --> 00:05:17.719
Over the years I have tried several of these

00:05:17.720 --> 00:05:20.119
and eventually landed on Gnus.

00:05:20.120 --> 00:05:25.519
The relevance column in this table simply and only

00:05:25.520 --> 00:05:28.079
reflects my taste.

00:05:28.080 --> 00:05:32.399
Throughout the rest of this presentation, I focus on Gnus.

NOTE The audience for this presentation

00:05:32.400 --> 00:05:36.199
I have 3 types of audiences in mind for this presentation.

00:05:36.200 --> 00:05:39.959
First, if you are already using Emacs

00:05:39.960 --> 00:05:42.439
as more than an editor,

00:05:42.440 --> 00:05:47.399
it makes good sense for you to also use Emacs as your MUA.

00:05:47.400 --> 00:05:50.759
There may well be some relevant information here for you

00:05:50.760 --> 00:05:52.319
in that situation.

00:05:52.320 --> 00:05:57.599
Second, for those interested in philosophy of Emacs,

00:05:57.600 --> 00:06:00.759
I go on some bigger picture tangents

00:06:00.760 --> 00:06:02.839
that may be of value to you.

00:06:02.840 --> 00:06:08.839
Third, I address some Emacs developers with some feedback,

00:06:08.840 --> 00:06:11.319
some suggestions, and some requests.

00:06:11.320 --> 00:06:14.599
The general model here is that

00:06:14.600 --> 00:06:17.079
we would collectively work towards

00:06:17.080 --> 00:06:19.799
improving what is on the table.

NOTE Monolithic mail user agents

00:06:19.800 --> 00:06:22.719
When a Mail User Agent is self-contained

00:06:22.720 --> 00:06:26.519
and includes implementation of mail protocols,

00:06:26.520 --> 00:06:29.079
we call it a Monolithic-MUA.

00:06:29.080 --> 00:06:33.879
Just as it is with the physical mail postal service,

00:06:33.880 --> 00:06:36.919
sending mail and receiving mail

00:06:36.920 --> 00:06:40.639
are fundamentally separate activities.

00:06:40.640 --> 00:06:42.839
And then there is mail processing.

00:06:42.840 --> 00:06:45.959
Based on these categorizations,

00:06:45.960 --> 00:06:49.679
Emacs has a set of mature libraries

00:06:49.680 --> 00:06:53.359
for composing mail, sending mail, and receiving mail.

00:06:53.360 --> 00:06:58.119
These are all independently well-documented

00:06:58.120 --> 00:07:02.479
and are part of the basic emacs Distribution.

00:07:02.480 --> 00:07:06.239
Emacs MUAs then use these common libraries

00:07:06.240 --> 00:07:09.519
to process mail (each somewhat differently).

00:07:09.520 --> 00:07:15.399
The primary benefit of the Monolithic-MUA approach

00:07:15.400 --> 00:07:19.599
is that Emacs MUAs then become self-contained

00:07:19.600 --> 00:07:22.959
and therefore multi-platform.

NOTE Split mail user agents

00:07:22.960 --> 00:07:25.559
But, when it comes to the question of merits of

00:07:25.560 --> 00:07:30.559
implementation of mail protocols in Elisp inside of Emacs,

00:07:30.560 --> 00:07:33.959
there is also another approach:

00:07:33.960 --> 00:07:36.159
that of a Split-MUA.

00:07:36.160 --> 00:07:40.559
Concept of a split-MUA is that of

00:07:40.560 --> 00:07:44.959
splitting the MUA into two different parts:

00:07:44.960 --> 00:07:47.159
One being the usage environment,

00:07:47.160 --> 00:07:50.319
and the other being mail protocols processing.

00:07:50.320 --> 00:07:54.719
The interface between these can be either

00:07:54.720 --> 00:07:57.839
direct (the upper box)

00:07:57.840 --> 00:08:00.279
or through protocols (the lower box).

00:08:00.280 --> 00:08:05.159
With Gnus, we primarily use the direct interface.

00:08:05.160 --> 00:08:09.639
The split-MUA model has many advantages

00:08:09.640 --> 00:08:12.599
over the monolithic-MUA model.

00:08:12.600 --> 00:08:17.279
With Split-MUAs, your messages are local,

00:08:17.280 --> 00:08:19.839
you can search them privately

00:08:19.840 --> 00:08:22.319
and access to your email is faster.

NOTE Gnus can be both

00:08:22.320 --> 00:08:28.119
Gnus can be used as both a Monolithic-MUA

00:08:28.120 --> 00:08:30.599
and also as a Split-MUA.

00:08:30.600 --> 00:08:35.399
Gnus and other Emacs MUAs are flexible enough

00:08:35.400 --> 00:08:39.119
to allow you to create your own split-MUA.

00:08:39.120 --> 00:08:42.519
For outgoing mail, Gnus can

00:08:42.520 --> 00:08:45.119
invoke a sendmail-like interface program.

00:08:45.120 --> 00:08:50.007
For incoming mail, Gnus can access Maildirs directly

00:08:50.008 --> 00:08:53.724
and let other programs imap-retrieve

00:08:53.725 --> 00:08:56.439
and update into maildirs.

00:08:56.440 --> 00:08:59.719
You can then search through your maildirs

00:08:59.720 --> 00:09:01.374
locally and privately

00:09:01.375 --> 00:09:04.559
with various mail-oriented search engines,

00:09:04.560 --> 00:09:07.319
and many have done so.

00:09:07.320 --> 00:09:11.399
For example, what we are seeing on this slide

00:09:11.400 --> 00:09:15.879
is from a 2014 Do It Yourself (DIY) recipe

00:09:15.880 --> 00:09:20.319
that one of our fellow Emacs conference participants,

00:09:20.320 --> 00:09:24.879
Adolfo, had published at the mentioned URL.

00:09:24.880 --> 00:09:29.719
The recipe in that slide is based on the following tools:

00:09:29.720 --> 00:09:35.079
mbsync, mu, mu4e, and msmtp.

00:09:35.080 --> 00:09:38.279
All our choices are different.

00:09:38.280 --> 00:09:42.399
There are many such recipes out there on the web.

NOTE Proprietary universes

00:09:42.400 --> 00:09:46.199
So, here, I don't want to provide

00:09:46.200 --> 00:09:50.439
yet another Emacs Split-MUA recipe.

00:09:50.440 --> 00:09:51.559
I want to do more.

00:09:51.560 --> 00:09:56.679
Instead, I want to target the contours of the ultimate MUA

00:09:56.680 --> 00:10:01.319
in the non-proprietary universe of digital ecosystems.

00:10:01.320 --> 00:10:05.439
But, first, let's take a look at what is

00:10:05.440 --> 00:10:07.479
happening in the proprietary universe.

00:10:07.480 --> 00:10:11.439
The 5 big American proprietary tech companies

00:10:11.440 --> 00:10:14.399
(Google, Microsoft, Apple, Facebook and Amazon)

00:10:14.400 --> 00:10:20.439
have created 5 competing enclaves as mostly separate

00:10:20.440 --> 00:10:22.999
and isolated digital ecosystem.

00:10:23.000 --> 00:10:27.839
In this slide, I am focusing on the first 3

00:10:27.840 --> 00:10:31.439
and each of their office and email environments.

00:10:31.440 --> 00:10:36.319
Let's clearly recognize that the economic model

00:10:36.320 --> 00:10:40.159
of these proprietary digital ecosystems is:

00:10:40.160 --> 00:10:42.159
"Surveillance Capitalism".

00:10:42.160 --> 00:10:45.799
So, when any of us goes there to get

00:10:45.800 --> 00:10:47.959
a free-of-charge email account,

00:10:47.960 --> 00:10:53.039
he has chosen to voluntarily forgo much of his privacy.

00:10:53.040 --> 00:10:55.799
And many have done so.

00:10:55.800 --> 00:10:59.919
Sadly, the rest of the world is becoming

00:10:59.920 --> 00:11:02.719
Americanized through the American Internet.

00:11:02.720 --> 00:11:08.439
As of 2022, almost %90 of Facebook's

00:11:08.440 --> 00:11:11.959
daily active users come from outside of the US.

00:11:11.960 --> 00:11:17.159
Also, with respect to email, each of the enclaves

00:11:17.160 --> 00:11:20.039
have MUAs that are fully integrated

00:11:20.040 --> 00:11:22.439
in their digital ecosystems

00:11:22.440 --> 00:11:24.599
in the form of an office environment

00:11:24.600 --> 00:11:29.839
comprising of address book, calendar, time management

00:11:29.840 --> 00:11:33.199
and planning tools and multi-lingual authoring

00:11:33.200 --> 00:11:36.239
and various other integrated tools.

NOTE Non-proprietary universes

00:11:36.240 --> 00:11:40.839
Now, let's focus on the right side of this picture.

00:11:40.840 --> 00:11:43.519
On the non-proprietary side,

00:11:43.520 --> 00:11:46.239
based on the Western FLOSS model,

00:11:46.240 --> 00:11:49.479
we have ended up with lots of components.

00:11:49.480 --> 00:11:52.239
We have Debian as a platform,

00:11:52.240 --> 00:11:58.919
we have Emacs as an editor-centered office environment

00:11:58.920 --> 00:12:03.439
and we have Gnus as an incredibly powerful MUA.

00:12:03.440 --> 00:12:07.839
But on the non-proprietary side we don't have anything

00:12:07.840 --> 00:12:12.079
that can reasonably be considered a digital ecosystem.

00:12:12.080 --> 00:12:16.439
I mean, the services aspect is missing.

NOTE By*

00:12:16.440 --> 00:12:20.799
Over the past two decades I have created

00:12:20.800 --> 00:12:24.399
quite an elaborate digital ecosystem for myself.

00:12:24.400 --> 00:12:26.759
It is called: By*.

00:12:26.760 --> 00:12:30.839
The Libre-Halaal ByStar Digital Ecosystem

00:12:30.840 --> 00:12:35.439
is being built to provide autonomy-oriented services

00:12:35.440 --> 00:12:37.359
on internet scale.

00:12:37.360 --> 00:12:42.919
The * in ByStar stands for Unix's globbing symbol,

00:12:42.920 --> 00:12:46.919
signifying that our scope is everything.

00:12:46.920 --> 00:12:52.039
Notice in this bigger picture that in the red box,

00:12:52.040 --> 00:12:58.439
our focus remains to be Emacs, Gnus and the ultimate MUA.

00:12:58.440 --> 00:13:03.439
I am not here to sell you ByStar, but perhaps

00:13:03.440 --> 00:13:06.239
you should be in the market for something like that.

00:13:06.240 --> 00:13:10.879
We need non-proprietary digital ecosystems.

00:13:10.880 --> 00:13:13.359
Very briefly, I'll give you

00:13:13.360 --> 00:13:16.319
some pointers to the full ByStar story.

00:13:16.320 --> 00:13:23.239
The full ByStar story is a 250 plus pages book titled:

00:13:23.240 --> 00:13:26.079
Nature Of Polyexistentials,

00:13:26.080 --> 00:13:28.399
Basis For Abolishment Of

00:13:28.400 --> 00:13:31.319
The Western Intellectual Property Rights Regime,

00:13:31.320 --> 00:13:33.479
And Introduction Of

00:13:33.480 --> 00:13:36.359
The Libre-Halaal ByStar Digital Ecosystem.

00:13:36.360 --> 00:13:42.039
I have it self-published on my own ByName public web page.

00:13:42.040 --> 00:13:46.639
The ByStar story starts with understanding of the

00:13:46.640 --> 00:13:48.959
Nature Of Polyexistentials.

00:13:48.960 --> 00:13:53.839
Polyexistentials inherently exist in multiples.

00:13:53.840 --> 00:13:56.959
Software is a polyexistential.

00:13:56.960 --> 00:14:01.919
Polyexistentials are naturally non-scarce,

00:14:01.920 --> 00:14:06.279
and making polyexistential artificially scarce,

00:14:06.280 --> 00:14:08.599
which is what the Western

00:14:08.600 --> 00:14:11.119
intellectual property rights regime does,

00:14:11.120 --> 00:14:13.599
is counter to nature.

00:14:13.600 --> 00:14:17.639
Polyexistentials are unownable

00:14:17.640 --> 00:14:20.039
and should not be considered property.

00:14:20.040 --> 00:14:25.079
The Western IPR regime is in conflict with nature.

00:14:25.080 --> 00:14:29.039
But, the book is more than just philosophy.

00:14:29.040 --> 00:14:32.359
In that book I also cover

00:14:32.360 --> 00:14:35.799
the bigger picture of healthy digital ecosystems

00:14:35.800 --> 00:14:39.359
which also includes the topic of this presentation.

00:14:39.360 --> 00:14:43.399
I'd be interested in your thoughts and your feedback,

00:14:43.400 --> 00:14:45.679
if you choose to dig deeper.

00:14:45.680 --> 00:14:50.999
And if you want to dig deeper, here are some links.

00:14:51.000 --> 00:14:55.079
By* is about re-decentralization

00:14:55.080 --> 00:14:57.159
of Internet application services.

00:14:57.160 --> 00:15:00.759
Among other things, ByStar provides

00:15:00.760 --> 00:15:05.159
complete own-your-email services. I mean,

00:15:05.160 --> 00:15:10.319
private Hillary-Clinton-Style mail servers for everyone.

00:15:10.320 --> 00:15:16.519
There is an overview of ByStar at by-star.net.

NOTE Libre-Halaal

00:15:16.520 --> 00:15:21.199
You may have noticed that I consistently use

00:15:21.200 --> 00:15:25.479
the "Libre-Halaal" label with ByStar.

00:15:25.480 --> 00:15:28.919
Halaal is a very sensitive word.

00:15:28.920 --> 00:15:30.719
I am a Moslem.

00:15:30.720 --> 00:15:35.919
But my use of Halaal is not in a religious context.

00:15:35.920 --> 00:15:39.079
It is in a philosphical context.

00:15:39.080 --> 00:15:42.759
And the scope of the "Libre-Halaal" label

00:15:42.760 --> 00:15:46.439
is manner-of-existence of Software and Services.

00:15:46.440 --> 00:15:50.599
It is not about Halaal-ness with respect to

00:15:50.600 --> 00:15:54.959
function and use of Software and Services.

00:15:54.960 --> 00:15:58.239
Unfortunately, the word Halaal

00:15:58.240 --> 00:16:02.319
and the concept of Halaal does not exist in English.

00:16:02.320 --> 00:16:06.759
So, first I introduce it into Globish.

00:16:06.760 --> 00:16:12.799
I have done so in PLPC-120039.

00:16:12.800 --> 00:16:18.199
Further, I explain as to why labels

00:16:18.200 --> 00:16:22.639
of Open Source and Free Software are both ill-directed.

00:16:22.640 --> 00:16:25.839
We then carefully define

00:16:25.840 --> 00:16:29.999
"Libre-Halaal Software" and "Libre-Halaal Services".

00:16:30.000 --> 00:16:33.919
Notice that last link.

00:16:33.920 --> 00:16:36.199
I bet, this is the first time

00:16:36.200 --> 00:16:40.679
that anyone includes a link to his "Open Business Plan"

00:16:40.680 --> 00:16:43.159
in an Emacs Conference.

00:16:43.160 --> 00:16:46.559
I hope others would do this as well.

00:16:46.560 --> 00:16:49.679
There is appetite out there

00:16:49.680 --> 00:16:53.639
for privacy- and autonomy-oriented digital ecosystems,

00:16:53.640 --> 00:16:57.759
and there is no conflict between honest business,

00:16:57.760 --> 00:17:01.039
honest profit, and Libre-Halaal Software

00:17:01.040 --> 00:17:02.919
and Libre-Halaal Services.

00:17:02.920 --> 00:17:08.039
The sub-title of our open business plan is:

00:17:08.040 --> 00:17:12.879
"An Inversion to the Proprietary Internet Services Model".

00:17:12.880 --> 00:17:20.879
And here are the same links as a native Reveal slide.

00:17:20.880 --> 00:17:24.679
If instead of a video, you are viewing

00:17:24.680 --> 00:17:27.399
this presentation as a Reveal web page,

00:17:27.400 --> 00:17:31.319
you can just click on the pointers and URLs.

NOTE The BISOS integration framework

00:17:31.320 --> 00:17:36.879
So, what was the point of bringing ByStar

00:17:36.880 --> 00:17:38.119
into this presentation?

00:17:38.120 --> 00:17:42.319
In tangible terms, what have we gotten out of

00:17:42.320 --> 00:17:45.639
the tangent we took on the ByStar bigger picture?

00:17:45.640 --> 00:17:50.839
Of course we have the ByStar Digital Ecosystem itself.

00:17:50.840 --> 00:17:54.879
But that is not immediately relevant to this presentation.

00:17:54.880 --> 00:17:59.959
Here, through BISOS we now have

00:17:59.960 --> 00:18:04.359
an integration framework, which we definitely needed.

00:18:04.360 --> 00:18:07.879
We now have BISOS-MARMEE,

00:18:07.880 --> 00:18:11.519
Multi-Account Resident Mail Exchange Environment,

00:18:11.520 --> 00:18:13.679
which is a consistent set

00:18:13.680 --> 00:18:17.639
of MUA-related software components --- which we need.

00:18:17.640 --> 00:18:23.159
We also needed to augment Emacs in our own terms,

00:18:23.160 --> 00:18:25.919
so we have Blee for that,

00:18:25.920 --> 00:18:29.959
ByStar Libre-Halaal Emacs Environment,

00:18:29.960 --> 00:18:32.919
is ByStar ecosystemized Emacs.

00:18:32.920 --> 00:18:37.679
And finally Blee-Gnus, which is

00:18:37.680 --> 00:18:40.639
Gnus and MARMEE integrated with Blee.

00:18:40.640 --> 00:18:46.479
With these in place, we can now dive deeper into MARMEE.

00:18:46.480 --> 00:18:53.119
The idea of MARMEE, is that of packaging together

00:18:53.120 --> 00:18:56.079
the mail protocols parts of the Split-MUA.

00:18:56.080 --> 00:19:00.799
MARMEE (which is of course in the context of BISOS)

00:19:00.800 --> 00:19:03.679
is the green box in this slide.

00:19:03.680 --> 00:19:08.519
For outgoing mail, we use an altered qmail.

00:19:08.520 --> 00:19:12.679
We will be looking deeper into qmail a bit later.

00:19:12.680 --> 00:19:17.559
For incoming mail, we are using offlineimap

00:19:17.560 --> 00:19:20.119
which is oauth2 aware.

NOTE BISOS-MARMEE

00:19:20.120 --> 00:19:23.439
Before going into more details,

00:19:23.440 --> 00:19:26.519
let's take a look at the parts lists for

00:19:26.520 --> 00:19:29.359
BISOS-MARMEE and Blee-Gnus.

00:19:29.360 --> 00:19:33.719
MARMEE is a collection of Python-based libraries

00:19:33.720 --> 00:19:37.479
and Debian packages that provide for rich sending

00:19:37.480 --> 00:19:40.279
and receiving of email outside of Emacs.

00:19:40.280 --> 00:19:44.239
Here is our BISOS-MARMEE parts list.

00:19:44.240 --> 00:19:48.599
MARMEE features include tracked mail Sending

00:19:48.600 --> 00:19:51.919
for confirmed mail communications

00:19:51.920 --> 00:19:54.599
and email distribution facilities

00:19:54.600 --> 00:19:57.559
(say, similar to Constant Contact).

00:19:57.560 --> 00:20:02.079
For Delivery Status Notification (DSN),

00:20:02.080 --> 00:20:06.039
we have adopted flufl.bounce.

00:20:06.040 --> 00:20:10.879
I'll be touching on everything that is qmail-related,

00:20:10.880 --> 00:20:17.319
namely qmail-remote.cs and mailfront, in a separate slide.

00:20:17.320 --> 00:20:21.759
notmuch is our choice of mail search engine.

NOTE Blee-Gnus

00:20:21.760 --> 00:20:27.319
Similarly, here is our Blee-Gnus Parts List.

00:20:27.320 --> 00:20:33.439
Blee-Gnus is Gnus and MARMEE integrated with BISOS and Blee.

00:20:33.440 --> 00:20:38.599
Notice mentions of org-msg and polymode here.

00:20:38.600 --> 00:20:42.879
Later, I'll expand on these in the context of

00:20:42.880 --> 00:20:47.679
transitioning from Message-Mode to Message-Polymode.

NOTE In combination

00:20:47.680 --> 00:20:52.199
With these parts in place,

00:20:52.200 --> 00:20:55.279
now let's see how they will all come together.

00:20:55.280 --> 00:20:59.999
Gnus is very flexible, and in combination with MARMEE,

00:21:00.000 --> 00:21:03.639
it can create an incredibly powerful MUA.

00:21:03.640 --> 00:21:07.279
On this slide, note the boxes

00:21:07.280 --> 00:21:10.319
that include the FPs label.

00:21:10.320 --> 00:21:14.839
FP stand for File Parameters.

00:21:14.840 --> 00:21:18.519
It is the basis of BISOS's configuration

00:21:18.520 --> 00:21:20.039
and secrets management.

00:21:20.040 --> 00:21:24.319
Notice that it has consistent agents

00:21:24.320 --> 00:21:27.839
inside of Emacs and on the OS.

00:21:27.840 --> 00:21:29.919
This is a big deal

00:21:29.920 --> 00:21:34.719
in that it can reduce user visible configuration complexity.

NOTE X822-Bus

00:21:34.720 --> 00:21:39.759
Also, notice the X822-Bus here.

00:21:39.760 --> 00:21:43.999
The idea of X822-Bus is that of

00:21:44.000 --> 00:21:49.839
allowing for communication among user's preferences, Gnus

00:21:49.840 --> 00:21:53.599
and MARMEE-qmail through addition of X- fields

00:21:53.600 --> 00:21:57.959
in RFC-822 message headers.

00:21:57.960 --> 00:22:03.599
X822-Bus is used for selection of mail sending agents

00:22:03.600 --> 00:22:08.839
and specification of delivery status parameters.

NOTE bystar-qmail

00:22:08.840 --> 00:22:12.279
Of key significance in this picture

00:22:12.280 --> 00:22:15.479
is our choice of qmail for outgoing mail.

00:22:15.480 --> 00:22:22.519
Compared to sendmail, postfix, exim,

00:22:22.520 --> 00:22:25.399
and other conventional MTAs;

00:22:25.400 --> 00:22:31.719
the qmail ecosystem is far more flexible and potent.

00:22:31.720 --> 00:22:34.599
We are not using qmail as is.

00:22:34.600 --> 00:22:37.999
Ours is called bystar-qmail.

00:22:38.000 --> 00:22:41.799
When we use it as a traditional MTA,

00:22:41.800 --> 00:22:45.439
we refer to it as PALS-qmail.

00:22:45.440 --> 00:22:52.599
And when we use it on the MUA side, we call it MARMEE-qmail.

00:22:52.600 --> 00:22:56.599
Just like Emacs, qmail has

00:22:56.600 --> 00:22:59.959
a solid core and a flexible periphery.

00:22:59.960 --> 00:23:04.479
All our alterations have been on the periphery.

00:23:04.480 --> 00:23:07.890
We have replaced qmail-remote

00:23:07.891 --> 00:23:14.479
with our own Python implementation called qmail-remote.cs.

00:23:14.480 --> 00:23:20.919
By being in Python, it can do a lot more a lot more easily.

00:23:20.920 --> 00:23:26.540
For example, qmail-remote.cs interacts with

00:23:26.541 --> 00:23:33.079
Google Oauth2 APIs and allows you to send through Gmail.

00:23:33.080 --> 00:23:36.399
This is shown with the red circle.

00:23:36.400 --> 00:23:43.639
We have also replaced qmail-smtpd with mailfront,

00:23:43.640 --> 00:23:46.159
shown with a blue circle.

00:23:46.160 --> 00:23:51.359
This allows us to use MARMEE Split-MUA

00:23:51.360 --> 00:23:53.999
through protocol interfaces.

00:23:54.000 --> 00:23:56.559
Let's take a look at that.

NOTE Using MARMEE with other MUAs outside Emacs

00:23:56.560 --> 00:24:02.719
Previously we looked at the "Direct Interface" of MARMEE,

00:24:02.720 --> 00:24:08.479
specifically, qmail-inject and Maildir for Gnus.

00:24:08.480 --> 00:24:11.479
But what if we wanted to use

00:24:11.480 --> 00:24:15.159
MARMEE with other MUAs outside of Emacs?

00:24:15.160 --> 00:24:18.799
That can be done through the "Protocol Interface".

00:24:18.800 --> 00:24:22.039
MARMEE also includes "mailfront"

00:24:22.040 --> 00:24:27.439
which can function as an SMTP submit server for localhost.

00:24:27.440 --> 00:24:30.599
This way, we can configure

00:24:30.600 --> 00:24:36.239
the outgoing mail part of any MUA to point to the localhost

00:24:36.240 --> 00:24:41.399
and have MARMEE-qmail function as an outgoing proxy.

NOTE Incoming mail

00:24:41.400 --> 00:24:47.919
For incoming mail, MARMEE-Split-MUA-Protocol-Interface

00:24:47.920 --> 00:24:51.359
includes "Courier", which can function

00:24:51.360 --> 00:24:54.079
as an IMAP server for localhost.

00:24:54.080 --> 00:24:58.519
This way, we can configure the incoming mail part

00:24:58.520 --> 00:25:02.319
of any MUA to point to the localhost

00:25:02.320 --> 00:25:06.519
and have MARMEE function as an incoming proxy

00:25:06.520 --> 00:25:10.759
by serving the local Maildir to the MUA.

NOTE Licensing and project status

00:25:10.760 --> 00:25:18.079
All sources for all of ByStar, BISOS,

00:25:18.080 --> 00:25:23.439
Blee and MARMEE are subject to Affero GPL.

00:25:23.440 --> 00:25:28.319
The sources and documentation are all republished

00:25:28.320 --> 00:25:30.559
under various "Organizations"

00:25:30.560 --> 00:25:35.439
under github.com/mohsenBanan

00:25:35.440 --> 00:25:40.679
All of ByStar, BISOS, Blee and MARMEE

00:25:40.680 --> 00:25:42.839
reflect work in progress,

00:25:42.840 --> 00:25:46.319
and we are NOT recruiting users at this time.

00:25:46.320 --> 00:25:49.279
For more than two decades,

00:25:49.280 --> 00:25:53.159
I have been using these all in that bigger context.

00:25:53.160 --> 00:25:56.239
They are mostly real,

00:25:56.240 --> 00:26:01.399
but so far, just for myself and a few other engineers.

00:26:01.400 --> 00:26:06.519
Our model is similar to God's early days.

00:26:06.520 --> 00:26:08.319
You may ask:

00:26:08.320 --> 00:26:12.119
"How did God create all of this in just 7 days?"

00:26:12.120 --> 00:26:17.199
Well, easy, He did not have an installed base to deal with.

NOTE Installing MARMEE

00:26:17.200 --> 00:26:24.519
You can obtain and install MARMEE in two ways. As is:

00:26:24.520 --> 00:26:29.439
as standalone-MARMEE, you can just

00:26:29.440 --> 00:26:31.919
pip install bisos.marmee.

00:26:31.920 --> 00:26:35.879
For the Gnus part you are completely on your own.

00:26:35.880 --> 00:26:40.279
Or on a Debian-11, you can just run

00:26:40.280 --> 00:26:43.399
the bisos bootstrap script.

00:26:43.400 --> 00:26:48.359
That way you will have all of BISOS, which includes MARMEE

00:26:48.360 --> 00:26:52.159
and you will have Blee, which includes Blee-Gnus.

00:26:52.160 --> 00:26:54.719
If you plan to do so,

00:26:54.720 --> 00:26:58.839
I suggest that you first try it in a disposable VM.

00:26:58.840 --> 00:27:02.159
BISOS and Blee are large.

00:27:02.160 --> 00:27:06.359
Many apt and pip packages will be installed!

00:27:06.360 --> 00:27:11.839
And here are the same links as a native Reveal slide.

00:27:11.840 --> 00:27:17.519
If you are viewing this presentation as Reveal.js web page,

00:27:17.520 --> 00:27:20.479
you can just click on the pointers and URLs.

NOTE MARMEE as an Emacs "Common Agent"

00:27:20.480 --> 00:27:25.359
Let's consider MARMEE as an Emacs "Common Agent".

00:27:25.360 --> 00:27:28.919
By "Common-Agent" I mean a capability

00:27:28.920 --> 00:27:30.959
which Emacs builds on

00:27:30.960 --> 00:27:33.759
and which other Apps can also use.

00:27:33.760 --> 00:27:38.479
Emacs has a very rich applications development framework

00:27:38.480 --> 00:27:41.119
for absorbing common-agents.

00:27:41.120 --> 00:27:45.474
Consider how magit has absorbed git,

00:27:45.475 --> 00:27:49.774
or how flycheck has absorbed mypy

00:27:49.775 --> 00:27:54.199
or how EAF does its work outside of Emacs ---

00:27:54.200 --> 00:27:57.919
that too can be considered a common-agent.

00:27:57.920 --> 00:28:02.039
The common-agent model permits us

00:28:02.040 --> 00:28:04.599
to do more outside of Emacs.

00:28:04.600 --> 00:28:08.639
Common-agents maximize social benefits

00:28:08.640 --> 00:28:10.559
and are more convivial.

00:28:10.560 --> 00:28:15.479
For example, any MUA can profit from MARMEE.

00:28:15.480 --> 00:28:18.439
But we don't have good ways of

00:28:18.440 --> 00:28:21.399
packaging Emacs and its packages

00:28:21.400 --> 00:28:23.639
with their common-agents.

00:28:23.640 --> 00:28:28.359
Instead, we usually end up with DIY recipes.

00:28:28.360 --> 00:28:32.479
This is why I am contextualizing

00:28:32.480 --> 00:28:35.439
Emacs inside of Blee and BISOS.

00:28:35.440 --> 00:28:37.999
That is what they are for.

00:28:38.000 --> 00:28:40.424
And that is why I consider them

00:28:40.425 --> 00:28:43.199
immediately relevant to this presentation.

00:28:43.200 --> 00:28:47.719
With an incredibly powerful Display Engine,

00:28:47.720 --> 00:28:51.279
and an incredibly powerful Elisp Engine,

00:28:51.280 --> 00:28:55.039
and an incredibly powerful Input Methods Engine,

00:28:55.040 --> 00:28:59.559
and an incredibly powerful Common-Agents paradigm,

00:28:59.560 --> 00:29:02.707
Emacs has the potential of being

00:29:02.708 --> 00:29:06.407
any non-proprietary digital ecosystem's

00:29:06.408 --> 00:29:08.879
preferred usage environment.

00:29:08.880 --> 00:29:14.090
I am in favor of putting more around Emacs

00:29:14.091 --> 00:29:17.440
and strengthening integration of Emacs

00:29:17.441 --> 00:29:19.719
with Debian, explicitly,

00:29:19.720 --> 00:29:22.839
perhaps even at the cost of

00:29:22.840 --> 00:29:26.599
de-emphasizing its multi-platform attribute.

00:29:26.600 --> 00:29:30.319
A smaller Emacs is a better Emacs.

00:29:30.320 --> 00:29:33.719
Notice that in this slide,

00:29:33.720 --> 00:29:37.279
I have used many arrows in many colors.

00:29:37.280 --> 00:29:45.599
Much of Emacs's power comes from its ability

00:29:45.600 --> 00:29:47.759
to absorb and to integrate.

00:29:47.760 --> 00:29:51.919
Tomohiro is right on the mark when he says,

00:29:51.920 --> 00:29:55.359
"The reason why Emacs platform is good

00:29:55.360 --> 00:29:58.119
is that it cooperates with OS,

00:29:58.120 --> 00:30:00.919
not because it is good by itself."

00:30:00.920 --> 00:30:03.999
I am suggesting that we should

00:30:04.000 --> 00:30:06.519
raise the bar from the OS

00:30:06.520 --> 00:30:09.839
to the entirety of our digital ecosystem.

00:30:09.840 --> 00:30:13.039
There are many models

00:30:13.040 --> 00:30:15.839
for Emacs to cooperate with the OS

00:30:15.840 --> 00:30:19.319
and with applications and with services.

00:30:19.320 --> 00:30:25.439
The colors of arrows in the previous slide correspond to

00:30:25.440 --> 00:30:28.879
the model of interface of the common-agent:

00:30:28.880 --> 00:30:33.999
for example, sub-process invocation, pipe-based

00:30:34.000 --> 00:30:39.359
asynchronous interface, or file-based interactions.

NOTE Consistent configuration

00:30:39.360 --> 00:30:44.759
One important aspect of common-agent paradigm is that

00:30:44.760 --> 00:30:50.399
both the common-agent and its Emacs App

00:30:50.400 --> 00:30:53.039
need to be configured consistently.

00:30:53.040 --> 00:30:57.959
In MARMEE and Blee-Gnus,

00:30:57.960 --> 00:31:01.919
we use File-Params to accomplish this.

00:31:01.920 --> 00:31:06.959
In BISOS, there is a Python interface to File-Params,

00:31:06.960 --> 00:31:10.239
there is a Bash interface to File-Params,

00:31:10.240 --> 00:31:15.719
and in Blee, there is an Elisp interface to File-Params.

00:31:15.720 --> 00:31:18.919
So, configurations are extended.

00:31:18.920 --> 00:31:23.279
Furthermore, File-Params can be encrypted,

00:31:23.280 --> 00:31:26.959
and credentials can be protected and shared.

00:31:26.960 --> 00:31:33.919
This is a significant improvement over .authinfo

00:31:33.920 --> 00:31:36.959
and its more recent incarnations.

NOTE Feedback and requests

00:31:36.960 --> 00:31:41.199
EmacsConf could be a great place

00:31:41.200 --> 00:31:44.374
for users to provide feedback to developers

00:31:44.375 --> 00:31:47.879
and for developers to suggest to developers.

00:31:47.880 --> 00:31:52.759
In that spirit, my primary audience in this part

00:31:52.760 --> 00:31:54.839
are fellow Emacs developers.

00:31:54.840 --> 00:32:00.039
BISOS-MARMEE and Blee-Gnus are starting points.

00:32:00.040 --> 00:32:02.439
We can collectively work

00:32:02.440 --> 00:32:04.679
towards improving what is in place.

00:32:04.680 --> 00:32:07.919
Some such improvements involve

00:32:07.920 --> 00:32:11.919
collaboration among various Emacs developers.

00:32:11.920 --> 00:32:16.599
Here, I am making some explicit requests

00:32:16.600 --> 00:32:19.559
from some of the relevant emacs developers.

00:32:19.560 --> 00:32:24.159
At most, these are requests and invitations.

00:32:24.160 --> 00:32:28.519
For each of these requests, I am providing links

00:32:28.520 --> 00:32:30.119
for additional details.

00:32:30.120 --> 00:32:33.039
In due course, I'll follow up

00:32:33.040 --> 00:32:35.279
in the Emacs developers mailing list.

NOTE X-Message-SMTP-Method: qmail

00:32:35.280 --> 00:32:41.879
Gnus uses X-Message-SMTP-Method

00:32:41.880 --> 00:32:45.119
for selection of Mail-Sending-Agent.

00:32:45.120 --> 00:32:50.519
Even though all the qmail injection code is still in Gnus,

00:32:50.520 --> 00:32:58.279
support for "X-Message-SMTP-Method: qmail" is missing.

00:32:58.280 --> 00:33:02.759
It takes 2 lines of code to revive it.

00:33:02.760 --> 00:33:07.399
With regards to (1), qmail was previously supported in Gnus.

00:33:07.400 --> 00:33:11.319
Lars, can you please reactivate it? Thanks.

NOTE X-Message-Send-Method

00:33:11.320 --> 00:33:16.439
(2) is a terminology suggestion.

00:33:16.440 --> 00:33:21.559
The term X-Message-SMTP-Method violates conceptual layering.

00:33:21.560 --> 00:33:27.079
Please consider changing it to X-Message-Send-Method.

00:33:27.080 --> 00:33:33.719
In a Split-MUA setup, Gnus need not know about SMTP at all.

00:33:33.720 --> 00:33:36.599
We just need to pass information

00:33:36.600 --> 00:33:39.319
to a Mail-Sending-Agent selector.

NOTE Sharing config info and secrets with common agents

00:33:39.320 --> 00:33:44.439
(3) is simply a design suggestion for

00:33:44.440 --> 00:33:46.759
which I prepared the context.

00:33:46.760 --> 00:33:51.839
.authinfo and Emacs auth-source library

00:33:51.840 --> 00:33:54.199
are too Emacs-centric.

00:33:54.200 --> 00:33:57.839
We need to share config info and secrets

00:33:57.840 --> 00:34:00.799
between common-agents and Emacs.

00:34:00.800 --> 00:34:03.639
The File Parameters approach

00:34:03.640 --> 00:34:05.799
can be a general-purpose solution.

00:34:05.800 --> 00:34:10.839
Is it reasonable to extend auth-source library to

00:34:10.840 --> 00:34:12.719
support File Params?

00:34:12.720 --> 00:34:16.519
I'll cover (4) in the next slide.

00:34:16.520 --> 00:34:21.159
(5) is a philosophical common suggestion

00:34:21.160 --> 00:34:26.439
to all Emacs developers. We need to better cultivate

00:34:26.440 --> 00:34:30.879
the model of Common-Agents integration with Emacs.

00:34:30.880 --> 00:34:39.599
And here are the same links as a native Reveal slide.

NOTE message-polymode

00:34:39.600 --> 00:34:42.799
A mail message comprises of

00:34:42.800 --> 00:34:45.599
Envelope, Header and BodyParts.

00:34:45.600 --> 00:34:49.439
Each of these have their own syntax (their own mode).

00:34:49.440 --> 00:34:53.639
Conceivably Each BodyPart has its own mode.

00:34:53.640 --> 00:34:59.439
So, we need to evolve Message-Mode into Message-Polymode.

00:34:59.440 --> 00:35:03.719
More or less by default, org-mode has become

00:35:03.720 --> 00:35:08.999
the beginnings of "Emacs Native Markup Language -- ENML".

00:35:09.000 --> 00:35:14.399
With org-msg you can write your emails in org-mode ---

00:35:14.400 --> 00:35:16.559
destined as html.

00:35:16.560 --> 00:35:19.559
org-msg needs to become

00:35:19.560 --> 00:35:22.239
an integral part of Message-Polymode.

00:35:22.240 --> 00:35:25.119
It would be heavenly

00:35:25.120 --> 00:35:29.959
if Lars, Jérémy and Vitalie could collaborate

00:35:29.960 --> 00:35:34.079
and give us the needed Message-Polymode. Thank you.

NOTE Vertical slice use cases

00:35:34.080 --> 00:35:38.119
One way to verify that we have not gone astray

00:35:38.120 --> 00:35:42.759
in our horizontal bigger pictures is to verify them

00:35:42.760 --> 00:35:46.919
through the concept of "Vertical Slice Use Cases".

00:35:46.920 --> 00:35:50.799
Let one use case be reading and writing

00:35:50.800 --> 00:35:54.639
of mail on multiple gmail accounts with Gnus.

00:35:54.640 --> 00:35:59.319
Google now requires use of oauth2 tokens

00:35:59.320 --> 00:36:02.439
which MARMEE can do outside of emacs.

00:36:02.440 --> 00:36:05.679
There is a recent email thread

00:36:05.680 --> 00:36:09.119
on that in the emacs-devel mailing list.

00:36:09.120 --> 00:36:14.279
Let another use case be that of tracking delivery

00:36:14.280 --> 00:36:18.679
and non-delivery reports for custom envelope addresses

00:36:18.680 --> 00:36:26.039
of byname.net (part of ByStar) autonomous mail services.

00:36:26.040 --> 00:36:30.319
I would have loved to walk you through these

00:36:30.320 --> 00:36:32.959
vertical slice use cases

00:36:32.960 --> 00:36:36.439
as screen captures of my Blee environment.

00:36:36.440 --> 00:36:40.639
For that, I need at least another 20 minutes.

00:36:40.640 --> 00:36:43.079
But my time is up.

00:36:43.080 --> 00:36:46.719
So, let's consider this as the first

00:36:46.720 --> 00:36:48.919
in a series of presentations

00:36:48.920 --> 00:36:51.799
where next in this series could be

00:36:51.800 --> 00:36:55.479
the mentioned two vertical slice use cases.

00:36:55.480 --> 00:36:59.279
Perhaps there could be another presentation

00:36:59.280 --> 00:37:02.879
on this topic in EmacsConf 2023.

00:37:02.880 --> 00:37:06.759
This document was produced entirely with

00:37:06.760 --> 00:37:10.799
Libre-Halaal Software, and is published using

00:37:10.800 --> 00:37:13.079
Libre-Halaal Internet Services.

00:37:13.080 --> 00:37:17.959
I want to thank all the EmacsConf Organizers

00:37:17.960 --> 00:37:19.519
for their great work,

00:37:19.520 --> 00:37:30.960
and Sacha, Leo, and Amin in particular.