path: root/2022/captions/emacsconf-2022-async--emacs-was-async-before-async-was-cool--michael-herstine--main.vtt

WEBVTT

1
00:00:00.000 --> 00:00:02.720
Hey everyone, I'm Michael

2
00:00:02.720 --> 00:00:04.480
and I'm going to be talking to you today

3
00:00:04.480 --> 00:00:07.640
about asynchronous programming in Emacs Lisp.

4
00:00:07.640 --> 00:00:10.360
I'm located in the San Francisco Bay Area

5
00:00:10.360 --> 00:00:12.040
where I'm a developer as well as

6
00:00:12.040 --> 00:00:14.160
a long time Emacs user.

7
00:00:14.160 --> 00:00:18.760
You may have heard of async or asynchronous programming.

8
00:00:18.760 --> 00:00:21.360
The idea has been around for decades

9
00:00:21.360 --> 00:00:24.400
but it first gained widespread attention in JavaScript

10
00:00:24.400 --> 00:00:26.720
back in the aughts.

11
00:00:26.720 --> 00:00:29.680
Then in the teens it gained tremendous popularity

12
00:00:29.680 --> 00:00:31.720
in the DevOps world with Golang.

13
00:00:31.720 --> 00:00:33.800
And just in the last few years

14
00:00:33.800 --> 00:00:37.880
support for async programming has landed in Rust.

15
00:00:37.880 --> 00:00:40.080
Well it can be done in Emacs as well

16
00:00:40.080 --> 00:00:42.040
and this talk will demonstrate that

17
00:00:42.040 --> 00:00:44.600
by walking you through a little problem

18
00:00:44.600 --> 00:00:47.200
that I actually solved for myself.

19
00:00:47.200 --> 00:00:49.040
Like a lot of these stories

20
00:00:49.040 --> 00:00:51.920
it begins with scratching a personal itch.

21
00:00:51.920 --> 00:00:55.320
In my case automating my music server.

22
00:00:55.320 --> 00:00:58.240
I use something called the music player daemon locally

23
00:00:58.240 --> 00:01:00.240
and as the name suggests

24
00:01:00.240 --> 00:01:03.560
it just kind of hangs out in the background.

25
00:01:03.560 --> 00:01:08.040
Reads music files and talks to assorted sound drivers.

26
00:01:08.040 --> 00:01:09.640
In fact it is so focused on

27
00:01:09.640 --> 00:01:12.440
that mission that it doesn't even offer a user interface.

28
00:01:12.440 --> 00:01:14.400
Instead it serves an API

29
00:01:14.400 --> 00:01:16.120
and invites application developers

30
00:01:16.120 --> 00:01:19.360
to build clients on top of that API.

31
00:01:19.360 --> 00:01:22.200
Okay so let's hop into a vterm

32
00:01:22.200 --> 00:01:25.080
and I'd like to show you the MPD client I use

33
00:01:25.080 --> 00:01:26.600
for my daily driver.

34
00:01:26.600 --> 00:01:29.520
Something called ncmpcpp.

35
00:01:29.520 --> 00:01:31.800
Doesn't exactly roll off the tongue

36
00:01:31.800 --> 00:01:33.720
but I've got a playlist.

37
00:01:33.720 --> 00:01:36.560
I can browse the file system.

38
00:01:36.560 --> 00:01:39.240
Looks like I can search my music library.

39
00:01:39.240 --> 00:01:40.000
Yada yada yada.

40
00:01:40.000 --> 00:01:42.600
It's got all the basic features.

41
00:01:42.600 --> 00:01:44.640
The point that I want to make is that

42
00:01:44.640 --> 00:01:51.920
ncmpcpp is a completely independent project of MPD.

43
00:01:51.920 --> 00:01:53.720
Separate and distinct.

44
00:01:53.720 --> 00:01:55.680
It does all of its work

45
00:01:55.680 --> 00:01:57.200
by simply communicating with

46
00:01:57.200 --> 00:02:01.400
the music player daemon over the API.

47
00:02:01.400 --> 00:02:03.440
Well I wanted to program to that API

48
00:02:03.440 --> 00:02:05.840
only from within Emacs.

49
00:02:05.840 --> 00:02:09.520
Now there are already Emacs MPD clients out there

50
00:02:09.520 --> 00:02:11.560
but I didn't really want a full blown client.

51
00:02:11.560 --> 00:02:14.320
I just wanted a few small tweaks

52
00:02:14.320 --> 00:02:16.320
over my current configuration.

53
00:02:16.320 --> 00:02:19.280
A command to skip to the next song.

54
00:02:19.280 --> 00:02:22.360
Maybe shove the current track into the mode line.

55
00:02:22.360 --> 00:02:24.160
Things like this.

56
00:02:24.160 --> 00:02:28.560
I needed an elisp API that would let me do this.

57
00:02:28.560 --> 00:02:32.000
Okay well let's get out of ncmpcpp

58
00:02:32.000 --> 00:02:37.560
and let's get into a netcat session

59
00:02:37.560 --> 00:02:39.400
with my local MPD server.

60
00:02:39.400 --> 00:02:43.840
As you can see we get a welcome string.

61
00:02:43.840 --> 00:02:46.800
So it is a server goes first protocol.

62
00:02:46.800 --> 00:02:49.640
But after that it's a very familiar

63
00:02:49.640 --> 00:02:53.960
text based request response oriented protocol.

64
00:02:53.960 --> 00:02:56.240
I can ask for the volume.

65
00:02:56.240 --> 00:02:58.160
I can ask for the status.

66
00:02:58.160 --> 00:03:06.000
But in particular I wanted an asynchronous API.

67
00:03:06.000 --> 00:03:07.800
If I issue a command like

68
00:03:07.800 --> 00:03:11.840
find every track in my library

69
00:03:11.840 --> 00:03:15.360
that's going to produce a lot of data

70
00:03:15.360 --> 00:03:18.920
that's a human perceptible pause

71
00:03:18.920 --> 00:03:22.080
as Emacs processes all the input.

72
00:03:22.080 --> 00:03:25.560
What I wanted was a style of programming

73
00:03:25.560 --> 00:03:28.080
where I could fire off my command

74
00:03:28.080 --> 00:03:31.560
have the Emacs command loop keep working

75
00:03:31.560 --> 00:03:33.440
and only invoke some callback

76
00:03:33.440 --> 00:03:35.280
when there was data available.

77
00:03:35.280 --> 00:03:39.560
Well Emacs is famously single threaded

78
00:03:39.560 --> 00:03:41.840
so it shouldn't come as a surprise

79
00:03:41.840 --> 00:03:44.080
that it offers a rich set of primitives

80
00:03:44.080 --> 00:03:46.720
that enable the sort of network programming

81
00:03:46.720 --> 00:03:49.320
that I wanted to do.

82
00:03:49.320 --> 00:03:50.760
In particular it offers

83
00:03:50.760 --> 00:03:53.280
a function called make network process.

84
00:03:53.280 --> 00:03:57.800
Now this method offers a bewildering variety of options.

85
00:03:57.800 --> 00:03:59.320
But at the heart of the matter

86
00:03:59.320 --> 00:04:01.040
it opens a network connection

87
00:04:01.040 --> 00:04:03.120
to some endpoint out there

88
00:04:03.120 --> 00:04:06.640
and we can configure it to be non blocking.

89
00:04:06.640 --> 00:04:09.840
It returns a handle that you can use to refer to

90
00:04:09.840 --> 00:04:14.880
this network connection with other methods.

91
00:04:14.880 --> 00:04:17.760
Other methods such as process and string

92
00:04:17.760 --> 00:04:19.600
which as the name suggests

93
00:04:19.600 --> 00:04:21.960
allows you to send textual data

94
00:04:21.960 --> 00:04:26.320
to the remote endpoint of your network connection.

95
00:04:26.320 --> 00:04:29.400
You can also use it with set process filter

96
00:04:29.400 --> 00:04:32.160
which allows you to associate a callback

97
00:04:32.160 --> 00:04:33.240
with your network connection.

98
00:04:33.240 --> 00:04:35.920
That callback will be invoked

99
00:04:35.920 --> 00:04:40.480
when there is data available

100
00:04:40.480 --> 00:04:41.960
in the processes read buffer.

101
00:04:41.960 --> 00:04:44.960
In other words in a request response oriented protocol

102
00:04:44.960 --> 00:04:47.800
like that of MPD you open your socket

103
00:04:47.800 --> 00:04:50.960
with make network process

104
00:04:50.960 --> 00:04:53.760
send your request via process send string

105
00:04:53.760 --> 00:04:56.360
and life will just continue in emacs

106
00:04:56.360 --> 00:04:57.560
until some data shows up

107
00:04:57.560 --> 00:05:00.720
in the processes read buffer

108
00:05:00.720 --> 00:05:05.200
at which point your callback will be invoked.

109
00:05:05.200 --> 00:05:07.560
It turns out this was enough

110
00:05:07.560 --> 00:05:12.280
for a purpose built async runtime.

111
00:05:12.280 --> 00:05:14.800
Let's work through the sequence of events

112
00:05:14.800 --> 00:05:16.480
when opening a connection

113
00:05:16.480 --> 00:05:18.720
and firing off a few commands in this style.

114
00:05:18.720 --> 00:05:22.120
So let's imagine a library

115
00:05:22.120 --> 00:05:25.520
that offers a connection object of some sort

116
00:05:25.520 --> 00:05:28.720
a caller and an MPD server out on the network.

117
00:05:28.720 --> 00:05:31.880
The caller will presumably get themselves

118
00:05:31.880 --> 00:05:34.760
a connection object by invoking some sort of

119
00:05:34.760 --> 00:05:38.080
connect method on our library.

120
00:05:38.080 --> 00:05:41.160
We can handle this through make network process

121
00:05:41.160 --> 00:05:45.360
but we're going to invoke make network process

122
00:05:45.360 --> 00:05:47.200
with no weight equal to true

123
00:05:47.200 --> 00:05:48.520
in other words asynchronously.

124
00:05:48.520 --> 00:05:52.240
That means the method is going to return immediately.

125
00:05:52.240 --> 00:05:56.320
We won't even know if the connection is up

126
00:05:56.320 --> 00:05:57.920
let alone what the response would be.

127
00:05:57.920 --> 00:06:01.560
This has some implications.

128
00:06:01.560 --> 00:06:05.280
At this point we've returned control to the caller

129
00:06:05.280 --> 00:06:09.400
the emacs event loop is proceeding quite happily

130
00:06:09.400 --> 00:06:11.320
and so the caller is free

131
00:06:11.320 --> 00:06:14.920
to start using our connection object.

132
00:06:14.920 --> 00:06:17.640
They might say issue a status command.

133
00:06:17.640 --> 00:06:20.600
Okay well in our library

134
00:06:20.600 --> 00:06:22.680
we don't have a connection yet.

135
00:06:22.680 --> 00:06:25.920
How on earth are we going to service this?

136
00:06:25.920 --> 00:06:29.440
Well we can simply give ourselves a queue

137
00:06:29.440 --> 00:06:33.360
and note down the fact that we owe a status command.

138
00:06:33.360 --> 00:06:35.560
That's pretty quick.

139
00:06:35.560 --> 00:06:38.120
We've now returned control back to our caller

140
00:06:38.120 --> 00:06:40.640
and they are again free to issue more commands.

141
00:06:40.640 --> 00:06:41.840
Maybe they issue a play command.

142
00:06:41.840 --> 00:06:45.160
Okay well we're going to go deeper into debt

143
00:06:45.160 --> 00:06:48.160
and note that we also owe a play command.

144
00:06:48.160 --> 00:06:56.160
At some point in the indeterminate future MPDU

145
00:06:56.160 --> 00:06:57.320
is the connection will get up

146
00:06:57.320 --> 00:07:03.000
MPDU will allocate resources to track a new client.

147
00:07:03.000 --> 00:07:06.160
They will write the welcome string into the socket

148
00:07:06.160 --> 00:07:07.920
and those bytes are going to show up

149
00:07:07.920 --> 00:07:10.360
in the emacs process read buffer

150
00:07:10.360 --> 00:07:13.160
at which point our callback will be invoked.

151
00:07:13.160 --> 00:07:17.440
We can parse the welcome string maybe

152
00:07:17.440 --> 00:07:19.240
note the version that connection object

153
00:07:19.240 --> 00:07:20.400
that might come in handy

154
00:07:20.400 --> 00:07:21.720
but the key point is

155
00:07:21.720 --> 00:07:24.080
our callback needs to take a look at the queue

156
00:07:24.080 --> 00:07:25.240
and notice

157
00:07:25.240 --> 00:07:27.200
oh we owe a status command

158
00:07:27.200 --> 00:07:29.880
and so we'll invoke process and string

159
00:07:29.880 --> 00:07:32.280
and send the status command down the pipe.

160
00:07:32.280 --> 00:07:36.760
Again at some indeterminate time in the future

161
00:07:36.760 --> 00:07:38.600
some bytes are going to show up

162
00:07:38.600 --> 00:07:41.200
in our processes read buffer

163
00:07:41.200 --> 00:07:43.160
and our callback will again be invoked.

164
00:07:43.160 --> 00:07:48.560
We've got volume is 75 plus a lot of other stuff

165
00:07:48.560 --> 00:07:50.480
and here we come to the next problem.

166
00:07:50.480 --> 00:07:54.440
If our caller invoked status

167
00:07:54.440 --> 00:07:56.960
they probably wanted to know about the status

168
00:07:56.960 --> 00:07:59.880
so how shall we get them to them?

169
00:07:59.880 --> 00:08:03.040
Well there's really not a lot of options at this point

170
00:08:03.040 --> 00:08:04.280
except the callback.

171
00:08:04.280 --> 00:08:09.000
Okay so change of plan our queue

172
00:08:09.000 --> 00:08:11.720
is no longer a queue of commands

173
00:08:11.720 --> 00:08:13.840
it's going to be a queue of commands

174
00:08:13.840 --> 00:08:15.880
with associated callbacks.

175
00:08:15.880 --> 00:08:20.280
We read the response off the socket

176
00:08:20.280 --> 00:08:23.440
invoke our caller supplied callback

177
00:08:23.440 --> 00:08:26.080
and then pop the queue.

178
00:08:26.080 --> 00:08:28.920
At this point our callback

179
00:08:28.920 --> 00:08:32.160
the library callback needs to know

180
00:08:32.160 --> 00:08:34.040
that we still have a pending command

181
00:08:34.040 --> 00:08:35.720
we fire that off down the pipe

182
00:08:35.720 --> 00:08:38.520
at some indeterminate time in the future

183
00:08:38.520 --> 00:08:40.360
we get a call we get a response

184
00:08:40.360 --> 00:08:42.640
our callback is invoked

185
00:08:42.640 --> 00:08:45.720
we invoke the caller supplied callback

186
00:08:45.720 --> 00:08:47.240
and we pop the queue.

187
00:08:47.240 --> 00:08:53.760
The structure of such a program

188
00:08:53.760 --> 00:08:55.800
is best viewed as a finite state machine

189
00:08:55.800 --> 00:08:57.640
and this is typically where you end up

190
00:08:57.640 --> 00:08:59.200
in asynchronous programming at least

191
00:08:59.200 --> 00:09:03.360
when you don't have a runtime grafted onto your program

192
00:09:03.360 --> 00:09:04.960
the way you do with Golang

193
00:09:04.960 --> 00:09:08.240
or when you don't have sort of extensive library support

194
00:09:08.240 --> 00:09:09.680
the way you do with Rust.

195
00:09:09.680 --> 00:09:14.480
Your data structure exists in one of these states

196
00:09:14.480 --> 00:09:15.440
at any given time

197
00:09:15.440 --> 00:09:18.960
and when input shows up on your file descriptor

198
00:09:18.960 --> 00:09:24.240
you transition along one of these edges to a new state.

199
00:09:24.240 --> 00:09:28.160
Cool so let's take a look at some of the code

200
00:09:28.160 --> 00:09:29.480
that flows from this.

201
00:09:29.480 --> 00:09:32.240
Okay let's hop over to an Emacs

202
00:09:32.240 --> 00:09:33.920
and take a look at how we might code this up.

203
00:09:33.920 --> 00:09:38.360
If you recall the sequence diagrams I shared

204
00:09:38.360 --> 00:09:40.120
we're going to be scribbling down the command

205
00:09:40.120 --> 00:09:42.160
and the callback that will be invoking

206
00:09:42.160 --> 00:09:43.240
upon its completion.

207
00:09:43.240 --> 00:09:45.440
So the first thing I did was give myself

208
00:09:45.440 --> 00:09:47.400
a little command struct

209
00:09:47.400 --> 00:09:52.280
with that I was able to define the connection object.

210
00:09:52.280 --> 00:09:56.280
We're going to be storing the handle to the connection.

211
00:09:56.280 --> 00:09:59.400
We're going to write down the protocol version

212
00:09:59.400 --> 00:10:02.000
that we harvest from the welcome message

213
00:10:02.000 --> 00:10:03.560
and of course we'll be recording

214
00:10:03.560 --> 00:10:05.760
the command queue as well.

215
00:10:05.760 --> 00:10:08.640
And so I gave myself a little connection object

216
00:10:08.640 --> 00:10:10.960
with a connection struct

217
00:10:10.960 --> 00:10:12.240
with those three attributes.

218
00:10:12.240 --> 00:10:15.000
With the data model squared away

219
00:10:15.000 --> 00:10:17.840
it was really pretty easy to code up

220
00:10:17.840 --> 00:10:21.160
the connect implementation.

221
00:10:21.160 --> 00:10:24.880
I'm deleting some details for exposition purposes

222
00:10:24.880 --> 00:10:29.520
but in the event it's really not that more complex

223
00:10:29.520 --> 00:10:30.520
than what you see here.

224
00:10:30.520 --> 00:10:32.840
We're going to unpack the arguments,

225
00:10:32.840 --> 00:10:35.040
figure out where the MPD server is

226
00:10:35.040 --> 00:10:37.280
to which you would like us to connect.

227
00:10:37.280 --> 00:10:39.920
We'll connect via make network process.

228
00:10:39.920 --> 00:10:42.640
We'll associate a library defined callback

229
00:10:42.640 --> 00:10:45.920
with that connection via set process filter.

230
00:10:45.920 --> 00:10:48.440
Then we'll instantiate the connection object

231
00:10:48.440 --> 00:10:50.120
and return it to the caller.

232
00:10:50.120 --> 00:10:53.800
Once the caller has a connection object

233
00:10:53.800 --> 00:10:56.880
they're free to send commands down that connection.

234
00:10:56.880 --> 00:10:59.120
So what we're doing here

235
00:10:59.120 --> 00:11:02.320
is simply instantiating a command object

236
00:11:02.320 --> 00:11:05.200
on the basis of the caller supplied arguments

237
00:11:05.200 --> 00:11:06.640
and appending it to the queue.

238
00:11:06.640 --> 00:11:07.920
And then the last thing we do

239
00:11:07.920 --> 00:11:11.040
and I've just indicated this with a comment

240
00:11:11.040 --> 00:11:12.040
is we kick the queue.

241
00:11:12.040 --> 00:11:14.560
This kind of goes back to

242
00:11:14.560 --> 00:11:18.200
the state transition diagram I laid out earlier.

243
00:11:18.200 --> 00:11:22.680
What this means is the logic for saying well

244
00:11:22.680 --> 00:11:24.280
if we're waiting the completion

245
00:11:24.280 --> 00:11:25.480
of a previously sent command

246
00:11:25.480 --> 00:11:27.280
there's really not much more to be done.

247
00:11:27.280 --> 00:11:31.000
We're just going to push this command onto the queue

248
00:11:31.000 --> 00:11:31.600
and return.

249
00:11:31.600 --> 00:11:33.120
On the other hand

250
00:11:33.120 --> 00:11:37.120
if the queue was empty on entry to LMPD send

251
00:11:37.120 --> 00:11:39.160
there's no reason not to just

252
00:11:39.160 --> 00:11:43.400
immediately send the command.

253
00:11:43.400 --> 00:11:44.680
And this is an example of

254
00:11:44.680 --> 00:11:46.520
the sort of client side code

255
00:11:46.520 --> 00:11:48.080
that results from this API.

256
00:11:48.080 --> 00:11:51.360
So you can see here we are giving ourselves

257
00:11:51.360 --> 00:11:54.240
a connection to the MPD server on the local host

258
00:11:54.240 --> 00:11:56.600
and we're going to send the get volume command

259
00:11:56.600 --> 00:11:58.160
down that connection.

260
00:11:58.160 --> 00:12:02.840
And if that command completes and all is well

261
00:12:02.840 --> 00:12:05.360
we'll just send a message to Emacs.

262
00:12:05.360 --> 00:12:07.800
Unfortunately you can't see my mini buffer

263
00:12:07.800 --> 00:12:10.960
so I'll hop over to the messages buffer

264
00:12:10.960 --> 00:12:12.720
and there's our result.

265
00:12:12.720 --> 00:12:15.160
The volume is 43.

266
00:12:15.160 --> 00:12:17.960
Great I thought.

267
00:12:17.960 --> 00:12:22.520
Simple clean responsive easy to code to.

268
00:12:22.520 --> 00:12:27.760
That is unfortunately not the end of the story.

269
00:12:27.760 --> 00:12:32.320
Let's continue this example a little bit.

270
00:12:32.320 --> 00:12:33.560
Let's imagine that

271
00:12:33.560 --> 00:12:35.920
if the volume comes back from the server

272
00:12:35.920 --> 00:12:37.360
and it is less than 50

273
00:12:37.360 --> 00:12:38.600
we would like to set it to 50.

274
00:12:38.600 --> 00:12:41.560
So this is interesting

275
00:12:41.560 --> 00:12:43.200
because we have two commands

276
00:12:43.200 --> 00:12:45.840
and whether or not we send the second command

277
00:12:45.840 --> 00:12:46.840
is going to depend on

278
00:12:46.840 --> 00:12:48.560
the response we get from the first.

279
00:12:48.560 --> 00:12:51.640
Okay I thought well that's fine

280
00:12:51.640 --> 00:12:55.080
I can simply put that logic in the callback

281
00:12:55.080 --> 00:12:57.920
that I specified for the get volume command.

282
00:12:57.920 --> 00:13:01.560
So here we are we check the return code

283
00:13:01.560 --> 00:13:04.400
we parse the volume we compare it to 50

284
00:13:04.400 --> 00:13:08.360
and if it's less we just invoke LMPD send again

285
00:13:08.360 --> 00:13:10.800
from the first command's callback.

286
00:13:10.800 --> 00:13:13.440
Okay I could live with that

287
00:13:13.440 --> 00:13:15.520
it's not the worst thing I've ever seen.

288
00:13:15.520 --> 00:13:19.400
Let's extend this example a little further

289
00:13:19.400 --> 00:13:21.480
and this is contrived but bear with me.

290
00:13:21.480 --> 00:13:25.480
Let us suppose that if we do set the volume to 50

291
00:13:25.480 --> 00:13:27.800
we'd like to get the volume one more time

292
00:13:27.800 --> 00:13:30.640
just to make sure that our change took on the server.

293
00:13:30.640 --> 00:13:33.560
Okay we can play the same game.

294
00:13:33.560 --> 00:13:37.280
We will put that logic in the callback

295
00:13:37.280 --> 00:13:39.520
that we specified for the set volume command.

296
00:13:39.520 --> 00:13:43.480
And here we are we check the return code

297
00:13:43.480 --> 00:13:45.480
we send a message to Emacs

298
00:13:45.480 --> 00:13:49.200
we send the get volume command again

299
00:13:49.200 --> 00:13:51.080
along with its own callback

300
00:13:51.080 --> 00:13:55.280
and at this point I think you know I hope it's clear

301
00:13:55.280 --> 00:13:57.520
the problem that is emerging

302
00:13:57.520 --> 00:14:01.360
and if it's not yet let's let me note that so far

303
00:14:01.360 --> 00:14:03.000
we're only handling the happy path

304
00:14:03.000 --> 00:14:04.520
in each of these callbacks.

305
00:14:04.520 --> 00:14:06.840
We really ought to do something about the error path

306
00:14:06.840 --> 00:14:10.120
for purposes of illustration let's just say

307
00:14:10.120 --> 00:14:12.120
we send a message to Emacs

308
00:14:12.120 --> 00:14:14.320
that means it would look like this

309
00:14:14.320 --> 00:14:16.560
and it's at this point

310
00:14:16.560 --> 00:14:19.400
that I really think it's impossible to deny

311
00:14:19.400 --> 00:14:23.280
that this API is actually not that easy to program to

312
00:14:23.280 --> 00:14:27.160
and if there are any JavaScript devs watching

313
00:14:27.160 --> 00:14:28.840
you're probably chuckling right now

314
00:14:28.840 --> 00:14:30.720
because I have discovered for myself

315
00:14:30.720 --> 00:14:33.880
what they call callback hell.

316
00:14:33.880 --> 00:14:36.040
If you are returning

317
00:14:36.040 --> 00:14:40.160
the results of asynchronous function invocations

318
00:14:40.160 --> 00:14:42.200
to their caller via callbacks

319
00:14:42.200 --> 00:14:45.640
you pretty much inevitably end up in this sort of

320
00:14:45.640 --> 00:14:48.040
deeply nested sequence of callbacks

321
00:14:48.040 --> 00:14:49.880
that is difficult to write difficult to read

322
00:14:49.880 --> 00:14:53.520
and difficult to reason about.

323
00:14:53.520 --> 00:14:57.480
And yet when I was stuck in this situation

324
00:14:57.480 --> 00:15:00.080
it just seemed like it really shouldn't be this bad.

325
00:15:00.080 --> 00:15:05.320
If I give myself this sort of tabular data structure

326
00:15:05.320 --> 00:15:10.160
I felt that this expressed precisely the same logic

327
00:15:10.160 --> 00:15:11.960
just in a much easier to read manner.

328
00:15:11.960 --> 00:15:15.840
I could in my mind's eye

329
00:15:15.840 --> 00:15:19.720
see the code for transforming this data structure

330
00:15:19.720 --> 00:15:21.040
which is really just a list

331
00:15:21.040 --> 00:15:25.600
into the code that you just saw in the previous slide

332
00:15:25.600 --> 00:15:29.440
and really if Lisp is good at anything

333
00:15:29.440 --> 00:15:31.080
it is list processing right

334
00:15:31.080 --> 00:15:33.080
and it was really at this point

335
00:15:33.080 --> 00:15:35.240
that a little bit of enlightenment dawned.

336
00:15:35.240 --> 00:15:40.800
I learned that Lisp is homo iconic

337
00:15:40.800 --> 00:15:46.040
which is just means that the language itself

338
00:15:46.040 --> 00:15:49.360
is a data structure in that language.

339
00:15:49.360 --> 00:15:53.160
Lisp code is after all just a list

340
00:15:53.160 --> 00:15:57.160
and the power of Lisp macros

341
00:15:57.160 --> 00:15:59.760
is taking that data structure

342
00:15:59.760 --> 00:16:02.400
some data structure that you've defined

343
00:16:02.400 --> 00:16:04.640
and doing exactly what I wanted to do

344
00:16:04.640 --> 00:16:07.520
transforming it from one list into another

345
00:16:07.520 --> 00:16:11.080
the destination list being Lisp code.

346
00:16:11.080 --> 00:16:16.000
So I got busy and I coded up my first Lisp macro

347
00:16:16.000 --> 00:16:19.160
which I called LMPD chain

348
00:16:19.160 --> 00:16:21.600
and that lengthy list of you know

349
00:16:21.600 --> 00:16:24.200
three or four nested callbacks

350
00:16:24.200 --> 00:16:25.920
gets turned into this

351
00:16:25.920 --> 00:16:29.520
which I hope you'll agree is much simpler

352
00:16:29.520 --> 00:16:32.240
much easier to read much easier to reason about.

353
00:16:32.240 --> 00:16:36.000
And if you're morbidly curious

354
00:16:36.000 --> 00:16:40.160
you can you can expand your macros

355
00:16:40.160 --> 00:16:44.200
and this invocation of LMPD chain expands to this.

356
00:16:44.200 --> 00:16:46.400
So that's my story.

357
00:16:46.400 --> 00:16:50.840
In all fairness I should note that

358
00:16:50.840 --> 00:16:55.160
the MPD protocol has some subtleties and complexities

359
00:16:55.160 --> 00:16:56.880
that I didn't really get into

360
00:16:56.880 --> 00:16:58.360
both due to time constraints

361
00:16:58.360 --> 00:17:00.520
and because they're not terribly relevant

362
00:17:00.520 --> 00:17:02.000
to the points I wanted to touch on

363
00:17:02.000 --> 00:17:05.360
I should also note that there's

364
00:17:05.360 --> 00:17:07.720
a fair amount of work in the library itself

365
00:17:07.720 --> 00:17:11.240
around accumulating partial responses

366
00:17:11.240 --> 00:17:12.560
as they show up in the buffer

367
00:17:12.560 --> 00:17:16.120
and dispatching them piecemeal to the caller

368
00:17:16.120 --> 00:17:19.720
that was really too complex to get into here.

369
00:17:19.720 --> 00:17:22.360
If you would like to see the code

370
00:17:22.360 --> 00:17:25.080
it's available on GitHub as well as Melpa.

371
00:17:25.080 --> 00:17:29.200
I'll be putting a version of this talk

372
00:17:29.200 --> 00:17:30.480
on my personal site

373
00:17:30.480 --> 00:17:33.720
and you can always reach out to me personally

374
00:17:33.720 --> 00:17:36.960
I hang out on IRC as SPIF

375
00:17:36.960 --> 00:17:41.920
or you can just email me as SPIF at P.O.Box dot com.

376
00:17:41.920 --> 00:17:47.880
Thank you very much.