path: root/2024/captions/emacsconf-2024-guile--beguiling-emacs-guileemacs-relaunched--robin-templeton--main.vtt

WEBVTT captioned by sachac and robin

00:00:00.000 --> 00:00:03.839
Hello everyone. I'm Robin Templeton, and I'm going to talk

00:00:03.840 --> 00:00:07.919
about Emacs Beguiled and recent progress on the Guile-Emacs

00:00:07.920 --> 00:00:13.919
project.

00:00:13.920 --> 00:00:16.839
First of all, if you're not familiar with Guile, it's an

00:00:16.840 --> 00:00:20.239
implementation of the Scheme programming language, which

00:00:20.240 --> 00:00:24.799
is a dialect of Lisp, and in the same family as Emacs Lisp, and

00:00:24.800 --> 00:00:28.759
Guile is GNU's official extension language. The goal of

00:00:28.760 --> 00:00:32.679
the Guile-Emacs project is to use Guile as the basis for

00:00:32.680 --> 00:00:37.599
Emacs's Lisp support. It has two main components: a new

00:00:37.600 --> 00:00:41.919
Emacs Lisp compiler built on top of Guile, and a variant of

00:00:41.920 --> 00:00:45.199
Emacs in which the built-in Lisp implementation is

00:00:45.200 --> 00:00:50.239
entirely replaced with Guile Elisp. We expect the

00:00:50.240 --> 00:00:53.439
combination of these two projects to have several

00:00:53.440 --> 00:00:57.999
benefits. One is improved performance. Another is

00:00:58.000 --> 00:01:03.839
increased expressiveness for Elisp and making it easier to

00:01:03.840 --> 00:01:07.839
extend and experiment with the language. Finally, it

00:01:07.840 --> 00:01:13.079
will reduce Emacs's reliance on C for two reasons. Guile will

00:01:13.080 --> 00:01:16.959
be responsible for the language implementation, so Emacs

00:01:16.960 --> 00:01:21.559
will no longer have to include a Lisp interpreter. It

00:01:21.560 --> 00:01:25.759
will also become possible to implement much more of Emacs in

00:01:25.760 --> 00:01:30.279
Lisp than is currently feasible. Of course, this raises

00:01:30.280 --> 00:01:34.119
the question of why Guile is suitable for this project. And

00:01:34.120 --> 00:01:38.079
we chose Guile for a few reasons. Guile is primarily a Scheme

00:01:38.080 --> 00:01:41.119
implementation, but it also has built-in support for

00:01:41.120 --> 00:01:44.399
multiple languages using its compiler tower. To add

00:01:44.400 --> 00:01:50.079
support for a new language to Guile, you only have to write a

00:01:50.080 --> 00:01:53.399
compiler from the source language to Tree-IL, which is

00:01:53.400 --> 00:01:57.439
essentially a low-level, minimal representation of

00:01:57.440 --> 00:02:02.479
Scheme. All of Guile's compiler optimizations occur at the

00:02:02.480 --> 00:02:07.599
Tree-IL layer or lower, so you don't need to worry about the

00:02:07.600 --> 00:02:10.159
lower-level details of the compiler when initially

00:02:10.160 --> 00:02:14.639
implementing your language. Guile also has some Lisp

00:02:14.640 --> 00:02:18.879
features that are very rare in Scheme implementations. For

00:02:18.880 --> 00:02:22.599
example, it has a nil value that counts as both false and an

00:02:22.600 --> 00:02:27.759
empty list, just like in Elisp, and it also has a version of

00:02:27.760 --> 00:02:32.319
the Common Lisp Object System and its metaobject protocol,

00:02:32.320 --> 00:02:37.239
which is called GOOPS.

00:02:37.240 --> 00:02:42.199
The idea of Guile-Emacs has a pretty long history, going back

00:02:42.200 --> 00:02:45.319
at least three decades. There have been about half a dozen

00:02:45.320 --> 00:02:48.519
previous implementation attempts. But the current

00:02:48.520 --> 00:02:51.519
iteration began with a series of six Summer of Code

00:02:51.520 --> 00:02:56.279
internships: Daniel Kraft's in 2009, and then my

00:02:56.280 --> 00:03:02.519
internships from 2010 to 2014. My basic implementation

00:03:02.520 --> 00:03:06.319
strategy was pretty straightforward. I implemented a core

00:03:06.320 --> 00:03:09.679
subset of Elisp, which was enough to run some batch mode

00:03:09.680 --> 00:03:15.399
programs outside of Emacs. In Emacs, I modified the garbage

00:03:15.400 --> 00:03:19.679
collector and the data structures for Lisp objects to use

00:03:19.680 --> 00:03:24.679
their libguile equivalents. I replaced Emacs' Lisp

00:03:24.680 --> 00:03:32.199
evaluator with the one provided by Guile Elisp.

00:03:32.200 --> 00:03:35.919
After a little over a year of work, at the end of the 2014

00:03:35.920 --> 00:03:41.079
internship, I ended up with a fully functional prototype of

00:03:41.080 --> 00:03:46.039
Guile-Emacs. It used Guile Elisp alone as its Lisp

00:03:46.040 --> 00:03:53.319
implementation and was completely compatible with Emacs

00:03:53.320 --> 00:03:57.559
functionality and with external extensions. One caveat

00:03:57.560 --> 00:04:01.399
was that performance was pretty bad, because I was focused

00:04:01.400 --> 00:04:05.639
on correctness, as well as ease of integration with the

00:04:05.640 --> 00:04:10.559
Emacs C code. But it was nonetheless a major milestone for

00:04:10.560 --> 00:04:15.759
the project. Let's take just a moment to look at

00:04:15.760 --> 00:04:19.599
Guile-Elisp.

00:04:19.600 --> 00:04:23.879
For starters, we have access to Guile modules. If we call

00:04:23.880 --> 00:04:26.959
Guile's <i>version</i> function, we can see that we're running

00:04:26.960 --> 00:04:33.879
under Guile 3.0. We have access to some of the numeric tower via

00:04:33.880 --> 00:04:41.279
the arithmetic functions. We also have multiple values. We

00:04:41.280 --> 00:04:45.599
have to be careful to use Guile's <i>values</i> procedure here, not

00:04:45.600 --> 00:04:48.839
the CL library's, but you can see that this works properly

00:04:48.840 --> 00:04:52.879
rather than being an emulation. Finally, we have tail

00:04:52.880 --> 00:04:57.999
call elimination. Naturally, we're going to use factorial

00:04:58.000 --> 00:05:07.159
to demonstrate it. If <i>n</i> is zero, return the answer, else

00:05:07.160 --> 00:05:14.199
recurse with <i>n</i> less one and <i>n</i> times <i>a</i>.

00:05:14.200 --> 00:05:17.119
Of course, this definition works correctly, but it gets

00:05:17.120 --> 00:05:21.759
more interesting if we communicate the answer with an

00:05:21.760 --> 00:05:27.759
error,

00:05:27.760 --> 00:05:32.359
in order to look at a backtrace. You can see here that there are no

00:05:32.360 --> 00:05:37.839
calls to <i>fact</i> visible in between the request to evaluate and

00:05:37.840 --> 00:05:42.199
the error communicating the answer. That's because

00:05:42.200 --> 00:05:53.319
this tail call has been optimized into effectively a goto.

00:05:53.320 --> 00:05:55.759
This is essential for any kind of serious functional

00:05:55.760 --> 00:06:00.279
programming.

00:06:00.280 --> 00:06:05.359
That's a peek at Guile-Elisp. In 2015, I left university

00:06:05.360 --> 00:06:09.479
to go work on web technologies, and the project was dormant

00:06:09.480 --> 00:06:14.679
for a very long time. But that's been changing recently.

00:06:14.680 --> 00:06:17.039
During the last few months, I've been working with Larry

00:06:17.040 --> 00:06:23.399
Valkama to rebase Guile-Emacs onto the development branch

00:06:23.400 --> 00:06:28.319
of upstream Emacs, including the past decade's worth of

00:06:28.320 --> 00:06:33.399
upstream development. What we've ended up with is a series

00:06:33.400 --> 00:06:38.839
of rebases onto different versions of Emacs. The older ones

00:06:38.840 --> 00:06:44.239
tend to work pretty well. The newer ones have increasingly

00:06:44.240 --> 00:06:49.799
bad problems where they haven't been properly adjusted for

00:06:49.800 --> 00:06:55.599
changes in the Emacs implementation. But we do have by now a

00:06:55.600 --> 00:06:58.919
version of Emacs 30 which boots correctly and can be used for

00:06:58.920 --> 00:07:04.959
interactive debugging, as well as the ability to bisect the

00:07:04.960 --> 00:07:08.919
revisions of Emacs and find out where regressions were

00:07:08.920 --> 00:07:13.199
introduced. Our immediate goal is of course to complete

00:07:13.200 --> 00:07:19.719
the rebase. At the same time, we want to improve Guile Elisp's

00:07:19.720 --> 00:07:22.799
performance to at least be competitive with ordinary Emacs

00:07:22.800 --> 00:07:29.279
Lisp. Just to characterize the performance situation,

00:07:29.280 --> 00:07:34.479
Guile Elisp is usually about half as fast as ordinary Elisp,

00:07:34.480 --> 00:07:37.839
while Guile Scheme is quite often an order of magnitude

00:07:37.840 --> 00:07:43.319
faster than ordinary Elisp, and that's based on micro

00:07:43.320 --> 00:07:47.799
benchmarks like the Gabriel benchmarks. But there's

00:07:47.800 --> 00:07:52.319
clearly a lot of room to improve our compiler's output.

00:07:52.320 --> 00:07:57.759
If you want to mark your calendars, we're expecting to have a

00:07:57.760 --> 00:08:04.199
usable version of Guile-Emacs 30 out sometime next spring. We're

00:08:04.200 --> 00:08:06.799
also going to put some effort into either extracting old

00:08:06.800 --> 00:08:13.599
work or doing new work that could be contributed upstream.

00:08:13.600 --> 00:08:17.559
On the Guile side, we'll probably start out with optimizing

00:08:17.560 --> 00:08:22.839
the dynamic binding facilities, which are used very seldom

00:08:22.840 --> 00:08:27.199
in Scheme, but are used all the time in traditional Lisp

00:08:27.200 --> 00:08:31.399
dialects. On the Emacs side, we'll be working initially on

00:08:31.400 --> 00:08:35.919
abstracting away the details of the Lisp implementation

00:08:35.920 --> 00:08:39.999
where they're not relevant. And that will clean up the Emacs

00:08:40.000 --> 00:08:44.279
code base a bit. It'll make it easier to integrate Emacs and

00:08:44.280 --> 00:08:49.919
Guile Elisp. It will probably be helpful for anyone who

00:08:49.920 --> 00:08:51.559
is working on ordinary Elisp on their own.

00:08:51.560 --> 00:08:57.199
We're also going to be adding new features to Emacs Lisp.

00:08:57.200 --> 00:09:01.639
We've seen a few of them already. The numeric tower, tail

00:09:01.640 --> 00:09:05.919
call optimization, Common Lisp compatibility. We're also

00:09:05.920 --> 00:09:10.359
going to provide access to Fibers, which is a Guile library

00:09:10.360 --> 00:09:14.639
based on ideas from Concurrent ML that provides much more

00:09:14.640 --> 00:09:17.679
powerful facilities for concurrent and parallel

00:09:17.680 --> 00:09:20.679
programming than what Emacs currently offers.

00:09:20.680 --> 00:09:33.759
This plan meets Guile-Emacs' basic goals, and it's work

00:09:33.760 --> 00:09:36.879
that we could maybe get integrated upstream in a reasonable

00:09:36.880 --> 00:09:41.799
amount of time. But it's also worth considering what more we

00:09:41.800 --> 00:09:47.239
can do, and what effect Guile-Emacs might have on Emacs if it

00:09:47.240 --> 00:09:49.079
becomes simply Emacs.

00:09:49.080 --> 00:09:54.599
For context, the amount of C code in Emacs has increased by

00:09:54.600 --> 00:09:58.559
around 50% in the last decade, and now it constitutes around

00:09:58.560 --> 00:10:06.399
a quarter of the code base. C can be a bit of a barrier to

00:10:06.400 --> 00:10:13.279
customizing and extending Emacs. For example, there are

00:10:13.280 --> 00:10:20.439
about 1500 C subroutines. Around 500 are used in C code, as

00:10:20.440 --> 00:10:26.519
well as available to Lisp code, and being written in C means

00:10:26.520 --> 00:10:31.519
that they can't be practically redefined. The use of C can

00:10:31.520 --> 00:10:35.839
become a barrier to extending Emacs or customizing its

00:10:35.840 --> 00:10:40.479
behavior. We might consider writing as much of Emacs as

00:10:40.480 --> 00:10:46.039
possible in Lisp. One way to speed up this process would

00:10:46.040 --> 00:10:52.199
be to provide a Common Lisp implementation for Guile. Note

00:10:52.200 --> 00:10:56.199
that between Guile Elisp and Guile Scheme, we have all of

00:10:56.200 --> 00:10:58.839
the essential ingredients for a Common Lisp environment.

00:10:58.840 --> 00:11:03.279
We can also share code with other Common Lisp

00:11:03.280 --> 00:11:12.479
implementations such as SBCL and SICL. Overall, the

00:11:12.480 --> 00:11:15.959
duration of the project will be better measured in months

00:11:15.960 --> 00:11:19.479
rather than years, despite Common Lisp's reputation for

00:11:19.480 --> 00:11:23.959
being a large language. This could have multiple uses, of

00:11:23.960 --> 00:11:29.199
course. It could be a model for future improvements to

00:11:29.200 --> 00:11:38.399
Elisp, because Elisp and CL can interact directly without

00:11:38.400 --> 00:11:41.319
problems. And it would be very easy for Elisp to borrow

00:11:41.320 --> 00:11:45.479
language features from Common Lisp. But for the purpose of a

00:11:45.480 --> 00:11:49.559
C to Lisp transition, it would also provide us with instant

00:11:49.560 --> 00:11:52.599
access to a huge number of high-quality libraries for

00:11:52.600 --> 00:11:58.159
things that Guile is not necessarily equipped to deal with,

00:11:58.160 --> 00:12:03.879
such as access to low-level Windows APIs, as well as lots of

00:12:03.880 --> 00:12:08.799
other libraries, such as interfaces to GUI toolkits for a

00:12:08.800 --> 00:12:12.079
variety of operating systems.

00:12:12.080 --> 00:12:21.799
At a certain point, this has technical advantages. If

00:12:21.800 --> 00:12:26.119
most of Emacs is written in Lisp, then we could consider

00:12:26.120 --> 00:12:30.759
using Guile Hoot to compile Emacs to WebAssembly, making it

00:12:30.760 --> 00:12:35.159
available perhaps in web browsers or on systems with the

00:12:35.160 --> 00:12:40.679
WebAssembly System Interface. But it would also be a great

00:12:40.680 --> 00:12:44.759
victory for practical software freedom. That's the idea

00:12:44.760 --> 00:12:48.359
that Freedom One, the freedom to study and modify programs,

00:12:48.360 --> 00:12:52.039
should not just be legally and technically possible, but

00:12:52.040 --> 00:12:54.719
should be actively encouraged by our computing

00:12:54.720 --> 00:12:58.959
environments. Emacs is really one of the archetypal

00:12:58.960 --> 00:13:02.519
examples of this, but we can and should go further.

00:13:02.520 --> 00:13:10.919
When Emacs is implemented primarily in Lisp, the entirety

00:13:10.920 --> 00:13:14.599
of the system will be transparent to examination and open to

00:13:14.600 --> 00:13:20.359
modification. Every part of Emacs will be instantaneously

00:13:20.360 --> 00:13:23.319
inspectable, redefinable, and debuggable.

00:13:23.320 --> 00:13:30.559
This will be a fundamental change in what is possible to

00:13:30.560 --> 00:13:36.159
do with Emacs extensions. For example, one experiment I'd

00:13:36.160 --> 00:13:40.319
be interested in is using the Common Lisp Interface Manager

00:13:40.320 --> 00:13:46.479
as the basis for Emacs's user interface. Screwlisp is

00:13:46.480 --> 00:13:52.879
giving a talk about McCLIM later today, but for present

00:13:52.880 --> 00:13:55.919
purposes, just think of it as a super-powered version of

00:13:55.920 --> 00:14:01.279
Emacs's concept of interactive functions. It would be a

00:14:01.280 --> 00:14:04.799
pretty long-term project in Emacs as it currently exists,

00:14:04.800 --> 00:14:10.519
but it would be almost trivial if Emacs were customizable at

00:14:10.520 --> 00:14:11.599
the lowest layers via Lisp.

00:14:11.600 --> 00:14:19.599
We'll certainly be looking at the practicality of these

00:14:19.600 --> 00:14:23.839
kinds of changes as we continue developing Guile-Emacs.

00:14:23.840 --> 00:14:31.719
Finally, how can you get involved with and support Guile

00:14:31.720 --> 00:14:35.999
Emacs? One way to help is just by trying it out and letting us

00:14:36.000 --> 00:14:40.519
know what your experiences are like. There will be a

00:14:40.520 --> 00:14:44.079
snapshot available on the Codeberg project site of the

00:14:44.080 --> 00:14:48.759
version that I'm using to give this presentation. It will be

00:14:48.760 --> 00:14:52.719
available both as a Guix package and as a portable tarball.

00:14:52.720 --> 00:14:58.799
This will be more interesting as we get closer to a complete

00:14:58.800 --> 00:15:05.479
rebase. We're also always happy to talk to potential

00:15:05.480 --> 00:15:10.879
contributors or potential collaborators from other

00:15:10.880 --> 00:15:11.599
projects.

00:15:11.600 --> 00:15:18.159
We can always use bug reports, and we're interested in what

00:15:18.160 --> 00:15:21.719
kind of features people actually want to see in Guile-Emacs.

00:15:21.720 --> 00:15:27.359
Guile-Emacs is also being developed by a small worker

00:15:27.360 --> 00:15:32.159
cooperative, so donations are a pretty direct way to

00:15:32.160 --> 00:15:36.039
support the project. If you do nothing else, I recommend

00:15:36.040 --> 00:15:40.719
going to the website and subscribing to our mailing lists so

00:15:40.720 --> 00:15:45.879
that you can keep up with news on the project. If you're

00:15:45.880 --> 00:15:49.239
watching this at EmacsConf, there will be a Q&A session

00:15:49.240 --> 00:15:57.080
immediately following this, and thanks for watching!