Hello, EmacsConf!
Thanks very much, first of all,
to the organizers of the conference
and to the audience,
who I hope is out there somewhere,
for giving me this chance
to talk about Emacs
and some of my poking around with Emacs Lisp.
My name is Eric Abrahamsen.
I'm not a professional programmer,
but I use Emacs all day, every day,
for writing, for translating,
for project management,
and most importantly, for email,
which will be the subject of my talk today.
So I'm talking about
object-oriented code
in Emacs' most famous, possibly oldest,
definitely most notorious
news reader / email client,
so, in particular,
object-oriented code in Gnus.
Why object-oriented code?
The way Gnus works is
it started off as a news reader,
so for accessing NNTP servers
and later on grew a whole bunch
of new functionality as a mail client,
so it can talk to IMAP servers,
Maildir directories,
folders on your file system,
all kinds of stuff,
but it presents a unified interface
to all those things,
so it's basically polymorphism,
one of the the basic fundamental principles
of object oriented code.
So it's a good fit.
Second reason is it already is object-oriented,
and I'll get into what that means in a second.
So the background that you should know
is that most of this code
was written in the 90s.
Emacs Lisp has only grown sort of
official object orientation support libraries
over the past 10 years or so,
from about 2010 to the present.
So what does Gnus do?
So the basics of object orientation
in most languages are:
you define a class of some sort,
and then you instantiate that class.
These class instances have two things:
they have data attributes (or slots,
or members, or whatever
you're going to call them),
and they have methods
which operate on individual instances.
So you could say that
you create or instantiate
an instance of a class,
and that instance owns two things.
That owns its set of attributes,
and it owns some methods,
which also work on the instance.
Both in Gnus' existing code
and in the more standard object-oriented
Emacs Lisp libraries, this relationship
is turned on its head a little bit,
in that data slots and instance methods
are defined outside of the class
or the instances themselves.
They are top-level definitions.
We'll get to what that means
in the newer libraries in a bit,
but first I want to talk about
how Gnus does this.
In order to do that,
we are going to go deep into
the darkest corner
of the Gnus source code tree
to a library called nnoo.el,
very cryptically-titled library,
and when we open it up, we find
a library with no code comments
and almost no doc strings.
Almost as if Lars was a little ashamed--
not ashamed, but knew he was doing
something a little bit crazy
and didn't want anyone to see.
So this file contains
the object-oriented mechanism
whereby you can define
different kinds of backends for Gnus,
and then those backends
can be instantiated as individual servers.
As you define these backends,
you're supposed to use two macros,
which you can see here.
One is called defvoo,
and one is called deffoo.
If you look at the definitions,
the definitions look pretty simple.
Here, defvoo basically turns into a defvar
and foo turns into a defun.
Along with those basic definitions,
the library also does some registration,
memoization, caching of those variables.
It saves them in the structure for later use,
so that we know that those are meant to be
attributes and methods
that are used with instances,
with server instances.
But you can see that
there's no server instance definition here.
There's no, like, no nothing.
These are top-level definitions,
so really, data attributes for new servers
and methods or functions
that operate on those instances
are completely separate mechanisms.
They don't really have anything to do
with each other.
They don't belong to the same data structures.
So how do they work?
Follow me. deffoo and defvoo,
aka methods and attributes,
these are all the things I just said.
So when you define a a backend type in Gnus,
what you get is this: a definition, a list.
It'll say, there is such a backend as nnml,
and these are its data attributes
that any given instance can have,
and then these are the functions or methods
that are defined to operate on
an instance of this backend,
so a server that belongs to
the nnml backend.
So at least we have this data here.
That's handy. We don't really touch that.
That's, like, very, very, very deep Gnus code
that doesn't really come up
even as a bug squasher or whatever.
We don't touch that very often,
but there they are,
and that's how they work.
Now the next thing that obviously
you want to know is, okay,
where are... if I've started up Gnus,
where are my servers?
Where are these server objects,
since this is object-oriented programming?
And the weird thing
that you will eventually figure out
(in some cases, after years of poking around)
in the Gnus source code
is that servers do not exist
in an ontological, philosophical sense,
as objects. The primary data structures of Gnus
are groups, and in sort of
an object-oriented hierarchical mindset,
you'd think, well, groups belong to servers,
so servers must exist, but they don't.
Each group... And here you can see
some examples of groups...
These are basically the data structures
that represent a group.
Each group also has a little entry here
that tells you what server it belongs to,
and each group replicates that data,
saying which server it belongs to.
So when Gnus is going through
doing its business,
trying to figure out updating mail
from the groups or whatever,
almost every time, it will cycle through
all the list of groups.
It'll look at all the server definitions,
and it will categorize the groups by server,
which is just weird,
because you're sort of looking for...
okay, where does the server exist?
It doesn't exist.
It's put together every time
out of code elsewhere in the Gnus code base,
specifically from these group definitions.
So this is very odd,
because in some sense...
Like here, this one, its server is nnml
and an empty string,
so there's a certain sense here
in which this server is not really
an object at all. What it is
is a set of instructions
for how to find messages,
and this set of instructions is:
go to the default place
where the user might have their mail
and expect to find messages there
in an nnml format, which is basically
just one message per file.
Any number of groups could have
those same instructions, but they're not...
It's not really a thing.
It's really just a...
It's more of a procedural instruction.
On the other end of the spectrum,
you might have an nnimap server,
which very much is a thing.
It has its own server, its own port,
its own authentication system.
So some of the servers are more like things,
some of the servers are more like instructions.
As Gnus works right now,
most of the servers are treated like
just instruction sets,
and there's no place where you can
go and find them.
There's no one central variable
that defines them all. So how do the...
We'll talk about the methods in a second.
How do the data attributes work?
Put very crudely,
your servers, when they're put together,
they are kept in a variable,
and it's called nnoo
nno
state a list and there's a concept to
this of the current server
so when we go here let's go back to
our nnno definition a list so when we
have an nnml
server say we have one here and it's
just this blank string
these are all when you define that in
your own uh server definition code
you can put in different values for all
of these
various attributes and when noose comes
when it comes time for news to operate
on this server in particular ask it to
you know open a group or get new mail
what it will do is it will take
that particular server's data from these
symbols
and it will copy all that information
into the global devfars
so for the time that you are operating
on this particular server
its individual data becomes the values
of these global
variables which when you realize what's
happening is sort of terrifying you
think oh my god
but at the same time it's actually kind
of impressive and it's amazing that it
works as well as it does
I'm actually a little bit in awe of the
of the code in this in this library I
think it's pretty impressive
so as you nno change server
uh this function here these values get
copied into the global value into the
global variables and then as you go on
the next server
that gets you know cleaned out and
recopied there are a few
um a few other slot types
or attribute types which do because all
of these attributes see they all start
with the nml
or in this case and in folder prefix
but there are a few slot types that all
servers need for
for instance their most recent status
message a status symbol like open denied
whatever
and that data is sort of scattered
around the rest of the news
code base in various variables or
various places
so that's that sort of just contributes
to the confusion when you're trying to
figure out why
things are going wrong so that is our
um nnoo which is and
sort of how the attributes and these
global variables work
if we want to talk about defu and the
methods we
go to
and so this is the place where all the
server level methods are defined
and what we have here are things like
here's an example there's closed server
this closed server is given a
a server as a an argument
it looks at the server and basically it
finds the proper function to call on
this particular server
using the function new skip function by
taking
the sort of latter half of this function
symbol
and pasting it together with the symbol
that represents the back end so
if you were calling this on an nni map
server your skip function would look at
your imap server look at closed server I
knew what
it would come up with
server sure enough there's an imac close
server and it'll call this code and then
it'll
it'll go and do its other bookkeeping
another sort of
surrounding code and so that's not that
actually works pretty well
uh as as things go uh defu
makes a record that this this function
exists and nus gets function get
function looks on that cache finds the
function and calls it
now what's particularly confusing is
that you don't actually even have to use
defu
so whoever wrote and then mail gear
which is a weird library
said to heck with you I'm not using any
of these
any of this machinery I'm going to do it
myself so we have
def structs to hold uh the instance
data and then we have just plain old
defense
for things like animal your close server
request close all of these
these server level uh variables and it
just turns out that
news in its belt and suspenders
approach to uh to coding it'll actually
just go out if it can't find
the memoized function it'll just go out
and say has anybody defined a function
that looks like this pattern and then
and then melder says yes I did and then
we call it and then we go so it's just
it's fine it works it just adds to the
confusion why
why does it work we don't know sometimes
the only thing worse than not knowing
why something doesn't work is
not knowing why something does work um
and then
a last little bit I want to touch on
here is inheritance which is another
sort of cornerstone of object-oriented
coding as far as I can tell only uh
the only inheritance that goes on is in
something called nn male
which provides sort of common functions
for back ends that keep their mail on
your
local machine and you can spool it you
can delete it you can you know you own
the messages it's not like an nntp
server
and so a lot of those male deer nnml
whatever
a lot of those have sort of similar code
which they
which they share via this nn mail
you call it an abstract parent class I
guess so if you have something like nnml
it has a request scan uh when it goes
into request scan
it ends up calling nnmail.newmail and it
says I am calling this as an nml server
and here are some of my callback
functions and my variables that I would
like you to use
when you are getting your email so in
this way the code is sort of you know
inter-interleaved between the the child
class and the parent class even though
we're not talking in terms of classes
here at all really
so that's how noose works right now I
hope that's clear
it certainly wasn't to me and I still
have to go refresh my
memory I'd like to talk a little bit
about sort of the newer
libraries that are available now for
doing object-oriented code
uh as I mentioned I think earlier nno
the copyright headers for 1996 so that's
pretty venerable
coincidentally around the same time eric
ludlum started developing e-I-e-I-o
which is a which is sort of inspired by
a common lisp's
common lisp object system um I got a
very good introduction to that from this
book
practical common lisp which I would
encourage you to look at if you haven't
which you probably have anyway
e-I-e-I-o was incorporated into Emacs in
2010
so that yeah e-I-e-I-o provides um
the deaf class statements it provides
deaf generics deaf methods all that sort
of stuff
sort of a common lisp object-oriented
code
at some point stephan monier's money
money another name I haven't pronounced
it all out
started either cleaning up that code or
for one reason or another writing a
re-implementation of generic functions
which was added
uh in 2015 and then throughout this time
another
sort of object-oriented style
declaration is
defstruct which started off in the cl
libraries
implemented with vectors later was
implemented with records so they're
easier to target
anyway that's another option so how
would we this is I'm probably out of
time already but
we're only getting to the part
the whole point of this is how would we
rewrite someone news's code to use these
newer libraries
if we didn't have to support third party
libraries this wouldn't be that hard
but out there noose is really up on uh
you know backwards compatibility and not
breaking people's stuff and you know
multi-decade support for things so there
are people out there who have written
third-party libraries
um defining new backends for you can use
like hacker news or whatever as
a as a server so we want to be able to
support those if you didn't have to
support those it'd be fine you'd
re-implement you'd use generic functions
you'd use either structure classes
whatever but we got a
it's a little bit tricky to support
these other people's
libraries so one of the things we can do
is rewrite the defu
so if you remember defu is the thing
that uh
or sorry uh defu
is the thing that defines methods that
operate on object answers
instances and we can uh rewrite that to
use
cldef generic and that's this is fairly
fairly simple
it looks like a lot of code it's not a
lot of good for instance we have the
new closed server code that we looked at
earlier
and we have this phone call and the new
skip function so this would look
like using generic functions and methods
it would look like this we'd have
a generic def generic which is just a
sort of a declaration
and a doc string and then we have those
implementations
so we can see what the original code
does here is it first says okay what
type is our our is our argument here and
if it's a string
then go and get the proper s the proper
method definition
from that string so the way we do that
with methods is we
we say if the server is a string so if
it matches this type
then what we're going to do is just
recall we're going to call this function
all over again
using uh basically the same code here
the same code that takes a string and
gets the object so
this does this can add extra function
calls
depending on how you've written the rest
of your code um but this is sort of the
canonical way of doing this
uh using methods then our next part here
is
nishkit function we're going to get a
function called closed server
the difference here is that all these
functions are all going to be called
close
news close server they're not going to
be called news like nni my
closed server and ntp close server
they're all going to have the same name
and what we do is uh we have an around
method
for any server that is a const which is
which is as close as we care to get uh
for you know zeroing in on the type that
we're looking for
we put in a round method on that so that
we can call the next method which we'll
call the more specific
method and then we have our other
bookkeeping code to clean up you know
set up tear down code we'll go around
that
and then in one of the back-end
definitions for instance in an imap
we have another news closed server thing
this looks at the server
and it says is this server a list that
starts with a symbol and an imap and if
it is then we're almost guaranteed that
this is what we wanted
and then this is where we would insert
all the rest of the code from anonymous
closed server
where we'd re-redefine that to look like
this so it's not that hard
theoretically so what we would do
is take the defu macro macro and then
rewrite that
so that it actually defines a cl def
method like
one of these now there's a couple of
these things
unfortunately it's not that easy get rid
of you
a couple of these things
[Music]
that don't use their server as the first
argument
or any of the arguments or it's an
optional argument
and we need the server to be in there to
dispatch on its type
if the server doesn't show up as a as a
required
argument we're not going to be able to
locate the the proper
function call so in the case of
noose request group here we start with
the group it's the group that matters
and we get
the newscommand method as an optional
argument
so that's not cool we don't want that so
what we need instead is something that
looks like this
what we're going to do with uh this is
gonna be just terrible terrible code but
hopefully it won't get used very often
it's gonna be really embarrassing
um defu what's what definitely was gonna
have to do is
say okay is this a function that doesn't
have the server as the first argument
and if it does
it's gonna say oh it's news request
group what happens has to happen with
news request group
is we take the news command method and
we're going to move it up to the front
to the first argument here and it's
either going to be
it's either going to be given or it's
going to be nil because it's it is
optional
okay I briefly edited the space time
continuum there to conceal the fact that
I had
actually not finished writing the code
that I was supposed to write anyway
um so now we have once we've reordered
the
the arguments to the function then we
have to check our various possible
values one is
uh that the server was not passed in in
which case we recall
request group with the server um the
other is that it's just a string in
which case we do that and then this is
sort of the the normal
the normal case that we would expect to
cons so that's not that bad it's not you
know
it's not beautiful um I would be sort of
ashamed to
let anybody see that particular macro
but I think that it would work okay
now the more difficult thing is going to
be
the data variables so
the equivalent of def vu because our two
options for defining classes here are
def struct and def class both of which
required you
to define the slots inside this macro
itself
so defu is top level um how do we get
the top level this top level macro
uh to insert slot names into these
definitions it's
it's possible that it'll be um that I
could monkey patch
uh an existing struct or an existing
class to add a new slot into it that
sounds
ugly the other option would be to give
it a server variable slot which is just
a generalized bucket
that holds anything that gets defined
via def loop
I don't like either of those solutions
but I'm I don't see any other
any other way of doing that so we re
rewrite the nno declare macro to either
be a destructor or a def class
and we rewrite the def boom macro to
somehow
associate that variable name the symbol
with the with the resulting class
definition
then the last question is do we use
structure classes
they both got their their strengths and
their weaknesses
the nice thing is that I mean I've got
how many servers you're going to have
really I've got I think less than 10
uh truly deranged mine might have as as
many as
50 let's double that to 100 100 of
anything is not going to matter it
doesn't matter what we use
death structures are simpler they're
lighter weight they're defined on top of
the direct the c records so you know
that's nice
the slots don't carry very much
information with them there's no type
information there's no doc string for
the slots themselves
they can also only do single inheritance
which some might say
was an advantage def class each slot
gets a lot more information associated
with it with it which I think can be
nice
it can do multiple inheritance if you're
going to go there
they are heavier weight in particular
their printed representation is gross
it's enormous
so if you see one show up in a back
trace or in your messages buffer can
really
it can really blow that up and make it
hard to read this of course won't be an
issue because our code won't have any
errors in it um my argument for multiple
inheritance here is that I can imagine
new servers falling into sort of like a
little two by two matrix of
of parent classes one being news versus
mail
so news the messages belong to somebody
else
you can't touch them you can't delete
them mail meaning
the messages are under your command
either a local mail dealer a remote imap
you're allowed to spool them copy them
delete them at will
and then the other sort of line of the
matrix would be a local file system
versus some kind of a you know server
port
remote access and that second the server
port remote access thing might require
authentication it might require a keep
alive for a connection
um it's there's going to be a process
there rather than just file system
commands
so I could see if I was going to do
multiple inheritance that's what I would
do those two
those two possible parent classes anyway
that's as far as I've gotten
I thought that I would be able to write
more of this code before I did this talk
but instead I spent the whole time
messing with video codecs but that's
where we're at and I'm going to cut
myself off now
I hope there are questions I hope I'm
there to to answer your questions
and thanks very much again to everyone
involved
bye