Hey everyone, I'm Michael
and I'm going to be talking to you today
about asynchronous programming in Emacs Lisp.
I'm located in the San Francisco Bay Area
where I'm a developer as well as
a long time Emacs user.
You may have heard of async or asynchronous programming.
The idea has been around for decades
but it first gained widespread attention in JavaScript
back in the aughts.
Then in the teens it gained tremendous popularity
in the DevOps world with Golang.
And just in the last few years
support for async programming has landed in Rust.
Well it can be done in Emacs as well
and this talk will demonstrate that
by walking you through a little problem
that I actually solved for myself.
Like a lot of these stories
it begins with scratching a personal itch.
In my case automating my music server.
I use something called the music player daemon locally
and as the name suggests
it just kind of hangs out in the background.
Reads music files and talks to assorted sound drivers.
In fact it is so focused on
that mission that it doesn't even offer a user interface.
Instead it serves an API
and invites application developers
to build clients on top of that API.
Okay so let's hop into a vterm
and I'd like to show you the MPD client I use
for my daily driver.
Something called ncmpcpp.
Doesn't exactly roll off the tongue
but I've got a playlist.
I can browse the file system.
Looks like I can search my music library.
Yada yada yada.
It's got all the basic features.
The point that I want to make is that
ncmpcpp is a completely independent project of MPD.
Separate and distinct.
It does all of its work
by simply communicating with
the music player daemon over the API.
Well I wanted to program to that API
only from within Emacs.
Now there are already Emacs MPD clients out there
but I didn't really want a full blown client.
I just wanted a few small tweaks
over my current configuration.
A command to skip to the next song.
Maybe shove the current track into the mode line.
Things like this.
I needed an elisp API that would let me do this.
Okay well let's get out of ncmpcpp
and let's get into a netcat session
with my local MPD server.
As you can see we get a welcome string.
So it is a server goes first protocol.
But after that it's a very familiar
text based request response oriented protocol.
I can ask for the volume.
I can ask for the status.
But in particular I wanted an asynchronous API.
If I issue a command like
find every track in my library
that's going to produce a lot of data
that's a human perceptible pause
as Emacs processes all the input.
What I wanted was a style of programming
where I could fire off my command
have the Emacs command loop keep working
and only invoke some callback
when there was data available.
Well Emacs is famously single threaded
so it shouldn't come as a surprise
that it offers a rich set of primitives
that enable the sort of network programming
that I wanted to do.
In particular it offers
a function called make network process.
Now this method offers a bewildering variety of options.
But at the heart of the matter
it opens a network connection
to some endpoint out there
and we can configure it to be non blocking.
It returns a handle that you can use to refer to
this network connection with other methods.
Other methods such as process and string
which as the name suggests
allows you to send textual data
to the remote endpoint of your network connection.
You can also use it with set process filter
which allows you to associate a callback
with your network connection.
That callback will be invoked
when there is data available
in the processes read buffer.
In other words in a request response oriented protocol
like that of MPD you open your socket
with make network process
send your request via process send string
and life will just continue in emacs
until some data shows up
in the processes read buffer
at which point your callback will be invoked.
It turns out this was enough
for a purpose built async runtime.
Let's work through the sequence of events
when opening a connection
and firing off a few commands in this style.
So let's imagine a library
that offers a connection object of some sort
a caller and an MPD server out on the network.
The caller will presumably get themselves
a connection object by invoking some sort of
connect method on our library.
We can handle this through make network process
but we're going to invoke make network process
with no weight equal to true
in other words asynchronously.
That means the method is going to return immediately.
We won't even know if the connection is up
let alone what the response would be.
This has some implications.
At this point we've returned control to the caller
the emacs event loop is proceeding quite happily
and so the caller is free
to start using our connection object.
They might say issue a status command.
Okay well in our library
we don't have a connection yet.
How on earth are we going to service this?
Well we can simply give ourselves a queue
and note down the fact that we owe a status command.
That's pretty quick.
We've now returned control back to our caller
and they are again free to issue more commands.
Maybe they issue a play command.
Okay well we're going to go deeper into debt
and note that we also owe a play command.
At some point in the indeterminate future MPDU
is the connection will get up
MPDU will allocate resources to track a new client.
They will write the welcome string into the socket
and those bytes are going to show up
in the emacs process read buffer
at which point our callback will be invoked.
We can parse the welcome string maybe
note the version that connection object
that might come in handy
but the key point is
our callback needs to take a look at the queue
and notice
oh we owe a status command
and so we'll invoke process and string
and send the status command down the pipe.
Again at some indeterminate time in the future
some bytes are going to show up
in our processes read buffer
and our callback will again be invoked.
We've got volume is 75 plus a lot of other stuff
and here we come to the next problem.
If our caller invoked status
they probably wanted to know about the status
so how shall we get them to them?
Well there's really not a lot of options at this point
except the callback.
Okay so change of plan our queue
is no longer a queue of commands
it's going to be a queue of commands
with associated callbacks.
We read the response off the socket
invoke our caller supplied callback
and then pop the queue.
At this point our callback
the library callback needs to know
that we still have a pending command
we fire that off down the pipe
at some indeterminate time in the future
we get a call we get a response
our callback is invoked
we invoke the caller supplied callback
and we pop the queue.
The structure of such a program
is best viewed as a finite state machine
and this is typically where you end up
in asynchronous programming at least
when you don't have a runtime grafted onto your program
the way you do with Golang
or when you don't have sort of extensive library support
the way you do with Rust.
Your data structure exists in one of these states
at any given time
and when input shows up on your file descriptor
you transition along one of these edges to a new state.
Cool so let's take a look at some of the code
that flows from this.
Okay let's hop over to an Emacs
and take a look at how we might code this up.
If you recall the sequence diagrams I shared
we're going to be scribbling down the command
and the callback that will be invoking
upon its completion.
So the first thing I did was give myself
a little command struct
with that I was able to define the connection object.
We're going to be storing the handle to the connection.
We're going to write down the protocol version
that we harvest from the welcome message
and of course we'll be recording
the command queue as well.
And so I gave myself a little connection object
with a connection struct
with those three attributes.
With the data model squared away
it was really pretty easy to code up
the connect implementation.
I'm deleting some details for exposition purposes
but in the event it's really not that more complex
than what you see here.
We're going to unpack the arguments,
figure out where the MPD server is
to which you would like us to connect.
We'll connect via make network process.
We'll associate a library defined callback
with that connection via set process filter.
Then we'll instantiate the connection object
and return it to the caller.
Once the caller has a connection object
they're free to send commands down that connection.
So what we're doing here
is simply instantiating a command object
on the basis of the caller supplied arguments
and appending it to the queue.
And then the last thing we do
and I've just indicated this with a comment
is we kick the queue.
This kind of goes back to
the state transition diagram I laid out earlier.
What this means is the logic for saying well
if we're waiting the completion
of a previously sent command
there's really not much more to be done.
We're just going to push this command onto the queue
and return.
On the other hand
if the queue was empty on entry to LMPD send
there's no reason not to just
immediately send the command.
And this is an example of
the sort of client side code
that results from this API.
So you can see here we are giving ourselves
a connection to the MPD server on the local host
and we're going to send the get volume command
down that connection.
And if that command completes and all is well
we'll just send a message to Emacs.
Unfortunately you can't see my mini buffer
so I'll hop over to the messages buffer
and there's our result.
The volume is 43.
Great I thought.
Simple clean responsive easy to code to.
That is unfortunately not the end of the story.
Let's continue this example a little bit.
Let's imagine that
if the volume comes back from the server
and it is less than 50
we would like to set it to 50.
So this is interesting
because we have two commands
and whether or not we send the second command
is going to depend on
the response we get from the first.
Okay I thought well that's fine
I can simply put that logic in the callback
that I specified for the get volume command.
So here we are we check the return code
we parse the volume we compare it to 50
and if it's less we just invoke LMPD send again
from the first command's callback.
Okay I could live with that
it's not the worst thing I've ever seen.
Let's extend this example a little further
and this is contrived but bear with me.
Let us suppose that if we do set the volume to 50
we'd like to get the volume one more time
just to make sure that our change took on the server.
Okay we can play the same game.
We will put that logic in the callback
that we specified for the set volume command.
And here we are we check the return code
we send a message to Emacs
we send the get volume command again
along with its own callback
and at this point I think you know I hope it's clear
the problem that is emerging
and if it's not yet let's let me note that so far
we're only handling the happy path
in each of these callbacks.
We really ought to do something about the error path
for purposes of illustration let's just say
we send a message to Emacs
that means it would look like this
and it's at this point
that I really think it's impossible to deny
that this API is actually not that easy to program to
and if there are any JavaScript devs watching
you're probably chuckling right now
because I have discovered for myself
what they call callback hell.
If you are returning
the results of asynchronous function invocations
to their caller via callbacks
you pretty much inevitably end up in this sort of
deeply nested sequence of callbacks
that is difficult to write difficult to read
and difficult to reason about.
And yet when I was stuck in this situation
it just seemed like it really shouldn't be this bad.
If I give myself this sort of tabular data structure
I felt that this expressed precisely the same logic
just in a much easier to read manner.
I could in my mind's eye
see the code for transforming this data structure
which is really just a list
into the code that you just saw in the previous slide
and really if Lisp is good at anything
it is list processing right
and it was really at this point
that a little bit of enlightenment dawned.
I learned that Lisp is homo iconic
which is just means that the language itself
is a data structure in that language.
Lisp code is after all just a list
and the power of Lisp macros
is taking that data structure
some data structure that you've defined
and doing exactly what I wanted to do
transforming it from one list into another
the destination list being Lisp code.
So I got busy and I coded up my first Lisp macro
which I called LMPD chain
and that lengthy list of you know
three or four nested callbacks
gets turned into this
which I hope you'll agree is much simpler
much easier to read much easier to reason about.
And if you're morbidly curious
you can you can expand your macros
and this invocation of LMPD chain expands to this.
So that's my story.
In all fairness I should note that
the MPD protocol has some subtleties and complexities
that I didn't really get into
both due to time constraints
and because they're not terribly relevant
to the points I wanted to touch on
I should also note that there's
a fair amount of work in the library itself
around accumulating partial responses
as they show up in the buffer
and dispatching them piecemeal to the caller
that was really too complex to get into here.
If you would like to see the code
it's available on GitHub as well as Melpa.
I'll be putting a version of this talk
on my personal site
and you can always reach out to me personally
I hang out on IRC as SPIF
or you can just email me as SPIF at P.O.Box dot com.
Thank you very much.