Hello and welcome everyone on EmacsConf 2023.
I'm Andrew Tropin.
I work on operating systems and programming languages.
Today, we discuss Lisps, Schemes, REPLs,
interactive development,
and how to make your own cozy development environment.
Let's start from interactive development.
Lisps are famous for a nice
Interactive Development Experience.
They have REPLs.
Emacs Lisp has its own Lisp machine,
and a lot of cool IDE with different functionality
is already here and providing
a nice and pleasant experience.
The question is, is it enough?
In most cases, yes, but for some languages,
we have some white spaces, some missing pieces.
And for example, in Scheme world,
we already have a few tools.
We have REPL, we have integration for REPL in Emacs,
but is it enough?
Let's see.
We know that Emacs is very good for Lisps and REPL.
Lisp and Emacs should be a perfect setup.
But let's see how REPL basically works.
It's an event loop which does three things.
It reads an expression, it evaluates the expression,
and it prints the result.
We can take a simple expression, input it into REPL,
and evaluate it and see the result.
Very nice, very convenient.
You can experiment and see immediately what is happening.
You can even run a long-running process
which does something.
You can interrupt it and everything will be okay.
But the problem appears
when you start to develop a bigger project.
And in most cases, you don't do
your whole development in REPL.
You do only a small part of it.
In most cases, you just write
the source code in text files,
and after that, you run those snippets of code
from those text files, or run the whole project.
It's not very convenient to copy and paste
every time the snippets of code to the REPL,
see the result, modify the snippet of code,
copy it again, and so on.
So people invented some integration
between REPL and your text editor.
So you can evaluate expressions inside your text editor
and see the result here.
Works good so far, but what happens
if we run a long-lasting loop,
which does a lot of operations.
As you can see here with a simple example,
the output of the function,
stdout of the function is presented here,
and the resulting value is here.
If you run a long-running process,
you don't see anything happening.
And you see there's a watch instead of my cursor.
Maybe you don't see it, but nothing actually happens,
at least from the point of view of the user.
But if we interrupt the evaluation,
we will see that some process in the background
was launched, but we didn't see anything.
Because the REPL is a single-threaded blocking process,
which reads stdin and prints stdout,
make the integration
between the REPL and your text editor
is not an easy task.
And even if you do it,
you have a lot of downsides, usually.
First of all, the process is not interruptible.
If you have a remote process which listens on the socket
to which you connect from your development environment,
and you run some infinite loop, for example,
you can't interrupt it.
Because interruption is done via signals,
and signals to remote processes are not usually
the thing in such integrations. NOT Output is not interactive
Output is also not interactive.
Usually, for example, here you can see
when I evaluate the expression,
the output is captured on the evaluation side,
and after that, after the whole evaluation
of the whole expression finished,
I get the result, all the stdout at once.
And if I run the process which evaluates for 5 seconds,
I will see the first signs of the life
only after 5 seconds of evaluation.
Okay, what else?
When you do such integrations, you have no protocol,
you have just stdin and stdout.
You print to stdin from your text editor.
You read from stdout of the process.
It's hard to tell if evaluation is finished,
if it requires stdin, and how to extend the REPL
to make it more featureful, and so on.
And also, such integrations are usually not very scalable.
For example, if you want to have a completion,
you type something, you have the completion. Cool.
But if you run the process and at the same time
try to have a completion, you don't have it,
because the evaluation is in progress,
and you can't calculate the completion candidates
at the same time. To make it more obvious,
I will start a completion here.
You see the completion pop-ups.
I start the evaluation process,
and when I try to complete something,
the evaluation freezes and there is no completion.
Not very convenient.
Usually, you have some long-running processes
and you want them to continue while you have
your go to definition, completion, and other things.
Overall, those issues make it quite inconvenient
to integrate REPL in text editors or development environments,
so you need something else
to make the work comfortable.
There is already a solution called nREPL.
It's a synchronous protocol which allows
to send operations to the server
and receive responses in a synchronous manner.
And here is a simple example of a few operations.
First one is cloning the existing session,
and as a response you will get a new session.
Also you send the evaluation request with code
that you want to evaluate, and you get two responses.
First one says that output is captured
and it's equal to "hi\n",
and after that, you receive an "Evaluation completed",
the value of this expression.
This protocol was developed
for CIDER development environment.
It's a Clojure development environment for Emacs.
It's very cool, featureful, reliable,
and I would say production-ready.
A lot of professional Clojure developers use it.
The nREPL protocol is very simple.
It has a few operations out of the box,
and you can extend it with any arbitrary operation you want.
I work a lot on Guix codebase and other Scheme projects,
so the experience I had previously with nREPL
was not satisfying. I decided
to just implement nREPL protocol.
First of all, I implemented nREPL server in Guile.
I called it `guile-ares-rs`, and used it
with a generic nREPL client for Emacs.
It worked.
It had some rough edges, but overall it was okay.
And after that, to add more features
to make the implementation more complete,
I wrote my own nREPL client for Emacs and called it `arei`.
And I got almost complete Guile IDE in two months.
So `ares-rs` is nREPL server implementation.
`arei` is Emacs client, which uses the same nREPL protocol.
It utilizes `sesman` package for managing sessions,
the association of buffers with nREPL connection.
It has some roots.
The implementation has some roots
in Geiser, CIDER, Monroe, and Rail.
I took small snippets for some parts of functionality.
I used the CAPF and xref infrastructure
for completion at point and cross-reference capabilities.
And by the time of conference, I hope
that README will be complete enough
so you will be able to try it yourself.
Let's see what is possible with it already.
Let's connect to nREPL server.
After that, you can evaluate the expression.
And you see the stdout and the result.
Very nice, very convenient.
You have different expression, you evaluate it,
you get the value of the evaluation.
You can run an infinite loop
which prints to stderr and stdout
and you see all necessary stuff.
Very cool.
But also, you can interrupt the evaluation,
which is very convenient if you accidentally
run an infinite loop.
Also, do you remember here we have a few more examples
that we didn't try yet?
For example, on usual REPL implementation,
if I evaluate this expression, I get return value.
I make a continuation and save it to this variable
and I try to call this evaluation
and I get an exception,
because the environment in which this continuation
was created was different and it has redefined
stdout and stderr to capture it.
But when I run it one more time,
when I resume the continuation,
the environment changed and it doesn't work.
What happens in `arei`?
I define continuation, I save the continuation
for the simple expression
and I resume the continuation with a new argument,
and you can see at the top of the screen
that it works perfectly fine.
Also, with a usual REPL implementation,
let's see what happens when we have a process
which reads from stdin.
I evaluate the expression and nothing visible happens.
I can try to type `C-g`, `C-c`,
and after some time it will say user interrupt.
What actually I expect in such a case
to have a minibuffer which prompts me for the input.
When I evaluate the same expression in the `arei`,
you see the prompt at the minibuffer
and here I can tell, "Hello I'm a message from minibuffer".
Cool. You will see that this message is printed to stdout,
and unspecified was returned
as a result of this expression.
Let's make some fancy example with continuations.
Continuations is a very cool mechanism
which is not the topic of today's talk,
but you can find a lot of interesting information
in Scheme documentation or in related books,
and I advise you to do it because it's really nice thing
that is actually applicable
in many different programming languages.
Here you can see the infinite loop
which just prints values increasing one by one.
And here we save a continuation on each iteration.
I can call the continuation
and it will resume from the previous saved step.
And you can see, it resumed from the same step
we interrupted earlier, but we provided a new value for it. another value for it.
We can provide another value
and it resumed from the same spot it was saved earlier.
But I also can provide a `read-i` value
and if I provide `read-i` value,
the infinite loop will read the input from stdin
and will continue the evaluation
with a different `i` provided in this input.
So let's try to type some arbitrary value
and you see that the loop continued with this value.
Very nice.
And every time we could easily interrupt it.
Okay, what most annoying thing that I had previously
with the usual REPL implementation
that I have a quite nice Guix API
where I can build packages, systems and other stuff.
But if I evaluate this expression, I will get an error.
Okay. I will get an error
because I don't have an appropriate environment.
But what I can do, I can connect to the remote REPL
by creating a server with `guix repl --listen` command
and connecting to it with `geiser-connect` command.
And now I can evaluate this expression.
Right?
Wow.
Okay.
It actually doesn't matter for my example.
I will explain how it doesn't work easily.
This is a long-running process which prints something
and it can take up to a few minutes.
And for the whole few minutes I don't see any results,
the same as with this infinite loop which prints to stdout
but I don't see anything interactively.
With `arei`, I can run
the evaluation of the same expression,
and you will see instantly
that stdout is presented here in slightly yellowish color.
I can interrupt the evaluation
if I don't want to wait until it's finished,
and just after that, I can evaluate another value.
So that's cool.
And let's see one more thing.
We have an infinite loop and we have some completion here.
And completion still works,
very nice,
while the infinite loop is running.
Okay.
Actually it took me around two months
of full-time work funded by my own savings,
and you can support and help to the project
using OpenCollective or by contributing on SourceHut.
The future steps for the project
include an experimental workflow where you have
multiple simultaneous evaluation in different contexts.
For example, you have Fibers, you have Goblins,
you have some HTTP server or some other thing,
and you want to run all of them independently
in slightly isolated sessions,
and you want to have the ability
to still interact with them.
For example, if they require standard input
or something else, you want to be able to provide it.
You want to see the stderr and stdout
of those long-running processes and so on.
The second thing is tree-sitter integration
for better syntax highlighting, code navigation,
and other features.
And after that, probably we will do a full-fledged debugger
so you can jump expressions one by one
and see the results and see some intermediate values
during the evaluation.
And it's very possible
because nREPL is a very extensible protocol
and you can implement
whatever you want on top of it.
I will answer two probably very frequent questions.
Does it support other Scheme implementations?
At the moment, it doesn't,
but the Scheme implementation is not restricted.
You have a server which is implemented in your language
and you have a client--in our case, `arei`--
which communicates with this protocol.
So if you implement nREPL server in a different language,
it should work with already implemented `arei` client.
And is it possible to use the same functionality
in other text editors, for example in VS Code,
Vim, whatever?
Yes, it's possible and the case is similar here.
You have already implemented nREPL server
and you can write your own nREPL client
in a different text editor and it will work.
I would like to thank the authors and maintainers
and contributors of Guile, Geiser, CIDER, Clojure,
and Emacs, and all other people
who are somehow related to the work on those projects
involved in this talk.
And I hope the Scheme programming will be enjoyable.
If you want to contact me,
join #tropin IRC channel at libera.chat,
or drop me a message via email or feediverse
using `andrew@trop.in` handle.
I will see you in a bit in Q&A session.