Greetings. Salaam. This is Mohsen Banan. محسن بنان.
I am a software and internet engineer.
I have been interested in email and
Emacs for a very long time.
My interest in email started with X.400
and the Red and Blue CCITT books -- circa 1988.
Early on, in the very early 1990s, I jumped ship
and joined the Internet email movement.
I am the primary author of two mobile email
related Internet RFCs, RFC-2188 and RFC-2524.
My interest in Emacs started in 1986 --
It was Emacs version 17 then. By around 1988
when Emacs version 18 was well in place,
I started living inside of Emacs.
My primary digital environment has been Emacs ever since.
It has been a good life.
It turns out that Emacs and email mix up really well.
Here, in this presentation and in the context of
Revisiting The Anatomy of Emacs Mail User Agents,
With MARMEE (Multi-Account Resident
Message Exchange Environment)
I am offering my thoughts on this topic
in this Emacs Conference 2022.
Long ago, I asked myself:
"What should my ultimate mail environment be?"
Over the past 20+ years, I have been exploring
the concept of the "Ultimate Mail User Agent (MUA)".
We do care about privacy, autonomy,
morality, ethics, society and philosophy,
so from the get go, proprietary (Haraam) environments
such as Microsoft Office's Outlook
and Google Office's Gmail were non-starters for me.
But these are significant realities
and we need to deal with these realities.
Notice how Microsoft and Google
have both framed their MUAs in the context of "office".
That type of framing is correct.
an MUA must be fully integrated
in the totality of one's digital ecosystem.
So, the Ultimate Mail User Agent
must be part of the Ultimate Usage Environment
of the Ultimate Digital Ecosystem.
In the non-proprietary (Halaal) universe, clearly
the ultimate usage environment is Emacs.
Emacs is today's most potent and convivial
non-proprietary usage environment.
So, clearly, the ultimate Mail User Agent
must be an integral part of Emacs.
Having reached that conclusion,
we then need to determine the specifics
of the shape and the anatomy of Emacs' MUAs.
We could have arrived at this conclusion
from the reverse direction as well.
Zawinski's Law states:
Every program attempts to expand until it can read mail.
Those programs which cannot so expand
are replaced by ones which can.
Jamie's point is very simple and obvious.
The "App" that you "live in" all day
should be your MUA and mail environment.
I ask those who jumped ship, who abandoned Emacs
in favor of VS Code: What about mail?
Long ago, Emacs expanded to including MUAs.
In fact there are many Emacs MUAs that you can choose from.
If you are already hip with Emacs And Linux,
you should definitely consider doing email in Emacs.
But if you are not already hip with Emacs,
I mean for new Emacs users, unfortunately,
setting up and using email is not straight forward.
We (I mean, Emacs developers) should work on that!
Emacs offers a good number of MUAs with
different characteristics to suit differing tastes.
As of 2022, you can choose from the following MUAs:
Gnus, VM, WanderLust, Mew, mu4e, notmuch.el, mh-e and Rmail.
Over the years I have tried several of these
and eventually landed on Gnus.
The relevance column in this table simply and only
reflects my taste.
Throughout the rest of this presentation, I focus on Gnus.
I have 3 types of audiences in mind for this presentation.
First, if you are already using Emacs
as more than an editor,
it makes good sense for you to also use Emacs as your MUA.
There may well be some relevant information here for you
in that situation.
Second, for those interested in philosophy of Emacs,
I go on some bigger picture tangents
that may be of value to you.
Third, I address some Emacs developers with some feedback,
some suggestions, and some requests.
The general model here is that
we would collectively work towards
improving what is on the table.
When a Mail User Agent is self-contained
and includes implementation of mail protocols,
we call it a Monolithic-MUA.
Just as it is with the physical mail postal service,
sending mail and receiving mail
are fundamentally separate activities.
And then there is mail processing.
Based on these categorizations,
Emacs has a set of mature libraries
for composing mail, sending mail, and receiving mail.
These are all independently well-documented
and are part of the basic emacs Distribution.
Emacs MUAs then use these common libraries
to process mail (each somewhat differently).
The primary benefit of the Monolithic-MUA approach
is that Emacs MUAs then become self-contained
and therefore multi-platform.
But, when it comes to the question of merits of
implementation of mail protocols in Elisp inside of Emacs,
there is also another approach:
that of a Split-MUA.
Concept of a split-MUA is that of
splitting the MUA into two different parts:
One being the usage environment,
and the other being mail protocols processing.
The interface between these can be either
direct (the upper box)
or through protocols (the lower box).
With Gnus, we primarily use the direct interface.
The split-MUA model has many advantages
over the monolithic-MUA model.
With Split-MUAs, your messages are local,
you can search them privately
and access to your email is faster.
Gnus can be used as both a Monolithic-MUA
and also as a Split-MUA.
Gnus and other Emacs MUAs are flexible enough
to allow you to create your own split-MUA.
For outgoing mail, Gnus can
invoke a sendmail-like interface program.
For incoming mail, Gnus can access Maildirs directly
and let other programs imap-retrieve
and update into maildirs.
You can then search through your maildirs
locally and privately
with various mail-oriented search engines,
and many have done so.
For example, what we are seeing on this slide
is from a 2014 Do It Yourself (DIY) recipe
that one of our fellow Emacs conference participants,
Adolfo, had published at the mentioned URL.
The recipe in that slide is based on the following tools:
mbsync, mu, mu4e, and msmtp.
All our choices are different.
There are many such recipes out there on the web.
So, here, I don't want to provide
yet another Emacs Split-MUA recipe.
I want to do more.
Instead, I want to target the contours of the ultimate MUA
in the non-proprietary universe of digital ecosystems.
But, first, let's take a look at what is
happening in the proprietary universe.
The 5 big American proprietary tech companies
(Google, Microsoft, Apple, Facebook and Amazon)
have created 5 competing enclaves as mostly separate
and isolated digital ecosystem.
In this slide, I am focusing on the first 3
and each of their office and email environments.
Let's clearly recognize that the economic model
of these proprietary digital ecosystems is:
"Surveillance Capitalism".
So, when any of us goes there to get
a free-of-charge email account,
he has chosen to voluntarily forgo much of his privacy.
And many have done so.
Sadly, the rest of the world is becoming
Americanized through the American Internet.
As of 2022, almost %90 of Facebook's
daily active users come from outside of the US.
Also, with respect to email, each of the enclaves
have MUAs that are fully integrated
in their digital ecosystems
in the form of an office environment
comprising of address book, calendar, time management
and planning tools and multi-lingual authoring
and various other integrated tools.
Now, let's focus on the right side of this picture.
On the non-proprietary side,
based on the Western FLOSS model,
we have ended up with lots of components.
We have Debian as a platform,
we have Emacs as an editor-centered office environment
and we have Gnus as an incredibly powerful MUA.
But on the non-proprietary side we don't have anything
that can reasonably be considered a digital ecosystem.
I mean, the services aspect is missing.
Over the past two decades I have created
quite an elaborate digital ecosystem for myself.
It is called: By*.
The Libre-Halaal ByStar Digital Ecosystem
is being built to provide autonomy-oriented services
on internet scale.
The * in ByStar stands for Unix's globbing symbol,
signifying that our scope is everything.
Notice in this bigger picture that in the red box,
our focus remains to be Emacs, Gnus and the ultimate MUA.
I am not here to sell you ByStar, but perhaps
you should be in the market for something like that.
We need non-proprietary digital ecosystems.
Very briefly, I'll give you
some pointers to the full ByStar story.
The full ByStar story is a 250 plus pages book titled:
Nature Of Polyexistentials,
Basis For Abolishment Of
The Western Intellectual Property Rights Regime,
And Introduction Of
The Libre-Halaal ByStar Digital Ecosystem.
I have it self-published on my own ByName public web page.
The ByStar story starts with understanding of the
Nature Of Polyexistentials.
Polyexistentials inherently exist in multiples.
Software is a polyexistential.
Polyexistentials are naturally non-scarce,
and making polyexistential artificially scarce,
which is what the Western
intellectual property rights regime does,
is counter to nature.
Polyexistentials are unownable
and should not be considered property.
The Western IPR regime is in conflict with nature.
But, the book is more than just philosophy.
In that book I also cover
the bigger picture of healthy digital ecosystems
which also includes the topic of this presentation.
I'd be interested in your thoughts and your feedback,
if you choose to dig deeper.
And if you want to dig deeper, here are some links.
By* is about re-decentralization
of Internet application services.
Among other things, ByStar provides
complete own-your-email services. I mean,
private Hillary-Clinton-Style mail servers for everyone.
There is an overview of ByStar at by-star.net.
You may have noticed that I consistently use
the "Libre-Halaal" label with ByStar.
Halaal is a very sensitive word.
I am a Moslem.
But my use of Halaal is not in a religious context.
It is in a philosphical context.
And the scope of the "Libre-Halaal" label
is manner-of-existence of Software and Services.
It is not about Halaal-ness with respect to
function and use of Software and Services.
Unfortunately, the word Halaal
and the concept of Halaal does not exist in English.
So, first I introduce it into Globish.
I have done so in PLPC-120039.
Further, I explain as to why labels
of Open Source and Free Software are both ill-directed.
We then carefully define
"Libre-Halaal Software" and "Libre-Halaal Services".
Notice that last link.
I bet, this is the first time
that anyone includes a link to his "Open Business Plan"
in an Emacs Conference.
I hope others would do this as well.
There is appetite out there
for privacy- and autonomy-oriented digital ecosystems,
and there is no conflict between honest business,
honest profit, and Libre-Halaal Software
and Libre-Halaal Services.
The sub-title of our open business plan is:
"An Inversion to the Proprietary Internet Services Model".
And here are the same links as a native Reveal slide.
If instead of a video, you are viewing
this presentation as a Reveal web page,
you can just click on the pointers and URLs.
So, what was the point of bringing ByStar
into this presentation?
In tangible terms, what have we gotten out of
the tangent we took on the ByStar bigger picture?
Of course we have the ByStar Digital Ecosystem itself.
But that is not immediately relevant to this presentation.
Here, through BISOS we now have
an integration framework, which we definitely needed.
We now have BISOS-MARMEE,
Multi-Account Resident Mail Exchange Environment,
which is a consistent set
of MUA-related software components --- which we need.
We also needed to augment Emacs in our own terms,
so we have Blee for that,
ByStar Libre-Halaal Emacs Environment,
is ByStar ecosystemized Emacs.
And finally Blee-Gnus, which is
Gnus and MARMEE integrated with Blee.
With these in place, we can now dive deeper into MARMEE.
The idea of MARMEE, is that of packaging together
the mail protocols parts of the Split-MUA.
MARMEE (which is of course in the context of BISOS)
is the green box in this slide.
For outgoing mail, we use an altered qmail.
We will be looking deeper into qmail a bit later.
For incoming mail, we are using offlineimap
which is oauth2 aware.
Before going into more details,
let's take a look at the parts lists for
BISOS-MARMEE and Blee-Gnus.
MARMEE is a collection of Python-based libraries
and Debian packages that provide for rich sending
and receiving of email outside of Emacs.
Here is our BISOS-MARMEE parts list.
MARMEE features include tracked mail Sending
for confirmed mail communications
and email distribution facilities
(say, similar to Constant Contact).
For Delivery Status Notification (DSN),
we have adopted flufl.bounce.
I'll be touching on everything that is qmail-related,
namely qmail-remote.cs and mailfront, in a separate slide.
notmuch is our choice of mail search engine.
Similarly, here is our Blee-Gnus Parts List.
Blee-Gnus is Gnus and MARMEE integrated with BISOS and Blee.
Notice mentions of org-msg and polymode here.
Later, I'll expand on these in the context of
transitioning from Message-Mode to Message-Polymode.
With these parts in place,
now let's see how they will all come together.
Gnus is very flexible, and in combination with MARMEE,
it can create an incredibly powerful MUA.
On this slide, note the boxes
that include the FPs label.
FP stand for File Parameters.
It is the basis of BISOS's configuration
and secrets management.
Notice that it has consistent agents
inside of Emacs and on the OS.
This is a big deal
in that it can reduce user visible configuration complexity.
Also, notice the X822-Bus here.
The idea of X822-Bus is that of
allowing for communication among user's preferences, Gnus
and MARMEE-qmail through addition of X- fields
in RFC-822 message headers.
X822-Bus is used for selection of mail sending agents
and specification of delivery status parameters.
Of key significance in this picture
is our choice of qmail for outgoing mail.
Compared to sendmail, postfix, exim,
and other conventional MTAs;
the qmail ecosystem is far more flexible and potent.
We are not using qmail as is.
Ours is called bystar-qmail.
When we use it as a traditional MTA,
we refer to it as PALS-qmail.
And when we use it on the MUA side, we call it MARMEE-qmail.
Just like Emacs, qmail has
a solid core and a flexible periphery.
All our alterations have been on the periphery.
We have replaced qmail-remote
with our own Python implementation called qmail-remote.cs.
By being in Python, it can do a lot more a lot more easily.
For example, qmail-remote.cs interacts with
Google Oauth2 APIs and allows you to send through Gmail.
This is shown with the red circle.
We have also replaced qmail-smtpd with mailfront,
shown with a blue circle.
This allows us to use MARMEE Split-MUA
through protocol interfaces.
Let's take a look at that.
Previously we looked at the "Direct Interface" of MARMEE,
specifically, qmail-inject and Maildir for Gnus.
But what if we wanted to use
MARMEE with other MUAs outside of Emacs?
That can be done through the "Protocol Interface".
MARMEE also includes "mailfront"
which can function as an SMTP submit server for localhost.
This way, we can configure
the outgoing mail part of any MUA to point to the localhost
and have MARMEE-qmail function as an outgoing proxy.
For incoming mail, MARMEE-Split-MUA-Protocol-Interface
includes "Courier", which can function
as an IMAP server for localhost.
This way, we can configure the incoming mail part
of any MUA to point to the localhost
and have MARMEE function as an incoming proxy
by serving the local Maildir to the MUA.
All sources for all of ByStar, BISOS,
Blee and MARMEE are subject to Affero GPL.
The sources and documentation are all republished
under various "Organizations"
under github.com/mohsenBanan
All of ByStar, BISOS, Blee and MARMEE
reflect work in progress,
and we are NOT recruiting users at this time.
For more than two decades,
I have been using these all in that bigger context.
They are mostly real,
but so far, just for myself and a few other engineers.
Our model is similar to God's early days.
You may ask:
"How did God create all of this in just 7 days?"
Well, easy, He did not have an installed base to deal with.
You can obtain and install MARMEE in two ways. As is:
as standalone-MARMEE, you can just
pip install bisos.marmee.
For the Gnus part you are completely on your own.
Or on a Debian-11, you can just run
the bisos bootstrap script.
That way you will have all of BISOS, which includes MARMEE
and you will have Blee, which includes Blee-Gnus.
If you plan to do so,
I suggest that you first try it in a disposable VM.
BISOS and Blee are large.
Many apt and pip packages will be installed!
And here are the same links as a native Reveal slide.
If you are viewing this presentation as Reveal.js web page,
you can just click on the pointers and URLs.
Let's consider MARMEE as an Emacs "Common Agent".
By "Common-Agent" I mean a capability
which Emacs builds on
and which other Apps can also use.
Emacs has a very rich applications development framework
for absorbing common-agents.
Consider how magit has absorbed git,
or how flycheck has absorbed mypy
or how EAF does its work outside of Emacs ---
that too can be considered a common-agent.
The common-agent model permits us
to do more outside of Emacs.
Common-agents maximize social benefits
and are more convivial.
For example, any MUA can profit from MARMEE.
But we don't have good ways of
packaging Emacs and its packages
with their common-agents.
Instead, we usually end up with DIY recipes.
This is why I am contextualizing
Emacs inside of Blee and BISOS.
That is what they are for.
And that is why I consider them
immediately relevant to this presentation.
With an incredibly powerful Display Engine,
and an incredibly powerful Elisp Engine,
and an incredibly powerful Input Methods Engine,
and an incredibly powerful Common-Agents paradigm,
Emacs has the potential of being
any non-proprietary digital ecosystem's
preferred usage environment.
I am in favor of putting more around Emacs
and strengthening integration of Emacs
with Debian, explicitly,
perhaps even at the cost of
de-emphasizing its multi-platform attribute.
A smaller Emacs is a better Emacs.
Notice that in this slide,
I have used many arrows in many colors.
Much of Emacs's power comes from its ability
to absorb and to integrate.
Tomohiro is right on the mark when he says,
"The reason why Emacs platform is good
is that it cooperates with OS,
not because it is good by itself."
I am suggesting that we should
raise the bar from the OS
to the entirety of our digital ecosystem.
There are many models
for Emacs to cooperate with the OS
and with applications and with services.
The colors of arrows in the previous slide correspond to
the model of interface of the common-agent:
for example, sub-process invocation, pipe-based
asynchronous interface, or file-based interactions.
One important aspect of common-agent paradigm is that
both the common-agent and its Emacs App
need to be configured consistently.
In MARMEE and Blee-Gnus,
we use File-Params to accomplish this.
In BISOS, there is a Python interface to File-Params,
there is a Bash interface to File-Params,
and in Blee, there is an Elisp interface to File-Params.
So, configurations are extended.
Furthermore, File-Params can be encrypted,
and credentials can be protected and shared.
This is a significant improvement over .authinfo
and its more recent incarnations.
EmacsConf could be a great place
for users to provide feedback to developers
and for developers to suggest to developers.
In that spirit, my primary audience in this part
are fellow Emacs developers.
BISOS-MARMEE and Blee-Gnus are starting points.
We can collectively work
towards improving what is in place.
Some such improvements involve
collaboration among various Emacs developers.
Here, I am making some explicit requests
from some of the relevant emacs developers.
At most, these are requests and invitations.
For each of these requests, I am providing links
for additional details.
In due course, I'll follow up
in the Emacs developers mailing list.
Gnus uses X-Message-SMTP-Method
for selection of Mail-Sending-Agent.
Even though all the qmail injection code is still in Gnus,
support for "X-Message-SMTP-Method: qmail" is missing.
It takes 2 lines of code to revive it.
With regards to (1), qmail was previously supported in Gnus.
Lars, can you please reactivate it? Thanks.
(2) is a terminology suggestion.
The term X-Message-SMTP-Method violates conceptual layering.
Please consider changing it to X-Message-Send-Method.
In a Split-MUA setup, Gnus need not know about SMTP at all.
We just need to pass information
to a Mail-Sending-Agent selector.
(3) is simply a design suggestion for
which I prepared the context.
.authinfo and Emacs auth-source library
are too Emacs-centric.
We need to share config info and secrets
between common-agents and Emacs.
The File Parameters approach
can be a general-purpose solution.
Is it reasonable to extend auth-source library to
support File Params?
I'll cover (4) in the next slide.
(5) is a philosophical common suggestion
to all Emacs developers. We need to better cultivate
the model of Common-Agents integration with Emacs.
And here are the same links as a native Reveal slide.
A mail message comprises of
Envelope, Header and BodyParts.
Each of these have their own syntax (their own mode).
Conceivably Each BodyPart has its own mode.
So, we need to evolve Message-Mode into Message-Polymode.
More or less by default, org-mode has become
the beginnings of "Emacs Native Markup Language -- ENML".
With org-msg you can write your emails in org-mode ---
destined as html.
org-msg needs to become
an integral part of Message-Polymode.
It would be heavenly
if Lars, Jérémy and Vitalie could collaborate
and give us the needed Message-Polymode. Thank you.
One way to verify that we have not gone astray
in our horizontal bigger pictures is to verify them
through the concept of "Vertical Slice Use Cases".
Let one use case be reading and writing
of mail on multiple gmail accounts with Gnus.
Google now requires use of oauth2 tokens
which MARMEE can do outside of emacs.
There is a recent email thread
on that in the emacs-devel mailing list.
Let another use case be that of tracking delivery
and non-delivery reports for custom envelope addresses
of byname.net (part of ByStar) autonomous mail services.
I would have loved to walk you through these
vertical slice use cases
as screen captures of my Blee environment.
For that, I need at least another 20 minutes.
But my time is up.
So, let's consider this as the first
in a series of presentations
where next in this series could be
the mentioned two vertical slice use cases.
Perhaps there could be another presentation
on this topic in EmacsConf 2023.
This document was produced entirely with
Libre-Halaal Software, and is published using
Libre-Halaal Internet Services.
I want to thank all the EmacsConf Organizers
for their great work,
and Sacha, Leo, and Amin in particular.