Hi, I'm Austin Theriault,
and this is writing a language server in OCaml
for Emacs, fun, and profit.
Real quick, who am I?
Well, I'm a software engineer at Semgrep.
I work on our editor integrations,
and I love working on programming languages, editors,
and cryptography.
What is Semgrep?
We're a small cybersecurity startup
whose core product is a SaaS tool,
which is static application security testing.
You can think of it as like a security linter.
Normal linters will say, hey,
you wrote ugly code, fix it.
We'll say, hey, you wrote a SQL injection, fix that.
We support 30+ languages,
and we have lots of customers all using different IDEs.
Why does that matter?
Well, our goal is to show security bugs
as early as possible in the development cycle.
In the industry, we call this shifting left.
And so how far left can we shift? The editor.
So that's why it matters
that our customers have different editors.
Our goal is to have Semgrep and the editor
show up like other language tooling.
And what I mean by that is I wrote some bad OCaml up here,
and the editor gave me that red squiggly and said,
fix your OCaml, and we want Semgrep to do something similar.
And so our goal then is to provide a similar experience
to normal language checking.
And then since we're a small startup,
and there's a ton of different IDEs that our customers use,
ideally, we don't want to have to rewrite a plugin
for every single type of editor out there.
Our other goal is abstract away
editing and language features for editors to one code base.
Ideally, we write it once
and then plug it into all of them.
So how can we do that, though?
Well, in the process of working on this stuff,
I found out about
the Language Server Protocol.
And what's great about the Language Server Protocol is
it's a specification that defines all the ways
that these language tools might interact
with a development tool. And by development tool,
I mean like VS Code, Sublime, Emacs, any of those.
And by language tool, I mean something like PyRight, MyPy.
So what's cool about LSP is that
you can separate out those tools into language servers
and the development tools into language clients.
And because they share this common specification,
they can now interact without knowing each other.
So it's this great abstraction that means
all you have to do is go write one language server
and you can hook it up to a bunch of language clients
and it'll just work.
So let's do a quick case study on language servers in LSP,
just so you get an idea of why this is super cool.
So there's this language server called Rust Analyzer.
It's a language server for the Rust language.
If you've ever developed in Rust,
you'll know that takes a really long time to compile,
but the compiler gives you fantastic feedback.
Rust has a lot of advanced language features,
so that feedback is super important for developing.
And so Rust Analyzer will give you that feedback instantly.
Here's a ton of things that it gives you.
Code completion, fixes, compiler errors, warnings,
type signatures. Rust has a pretty strong type system.
It also has this thing called lifetimes.
A bunch of advanced language features in Rust Analyzer
helps you manage all that
and gives you all that info
without having to wait for it to compile.
Developing with the Rust Analyzer
is just orders of magnitude easier
than just trying to write Rust straight.
Rust Analyzer, fantastic. They went and they developed it,
and now you can go use that in Emacs, NeoVim,
VS Code, wherever.
So you can develop Rust in a way that's relatively efficient
without having to give up your favorite editor.
So here's a quick little demo
of all the cool things it can do.
So you can see I typed an error.
It tells me that I wrote an error.
I used the incorrect lifetime,
which is some advanced language feature,
and it'll let me know that.
I expanded a Rust macro just there,
which is similar to Lisp macros,
and then I ran a single unit test,
and that's really cool because I ran a single unit test
from my editor.
I didn't have to go and type any commands or anything.
It just worked.
So why is this just useful in general for a user?
Well, you get the same experience across editors.
Like I was saying, you don't have to give up
one editor for another
so you get some sort of cool language feature.
You can easily set up and use language servers
made for other editors
if developers don't support your editor of choice.
Performance is not dependent on the editor.
That's fantastic because to do all that Rust stuff,
it takes a lot of CPU power,
and so that's going to be slow
if your editor language is not great, not fast.
And then bug fixes, updates, all that,
it all comes out at the same time.
And then from the developer perspective, well,
adding new editors is quick and easy.
For reference, when I wrote the Semgrep language server,
it took me maybe two or three weeks,
but then actually going and setting it up for VS Code,
that took an hour. For Emacs, 30 minutes.
IntelliJ, maybe another hour.
So it took me a day to add support
for three different editors,
which was I think something like 75% of the market share
or something crazy like that.
So very quick. You only need one mental model.
You don't have to figure out
all these different extension mental models,
how those editors work, anything like that.
And another thing that's cool is
you only have to write tests for the language server,
not necessarily for the editor.
It's great to have just one set of tests
that you have to pass.
So why does a language server protocol matter with Emacs?
Well, like I was saying before,
Emacs gets the benefit from work put into other editors.
So we get all this language support,
and no one actually has to go and write the list for it
or write those tools specific to Emacs.
You get the language tooling,
the CPU-intensive part of the editors.
It can be written in something else.
Lisp is fast. It's not that fast.
Having that speed is fantastic. It's all asynchronous.
It won't slow down Emacs.
And then there's this package called `lsp-mode`,
which is an LSP client commonly included
in popular Emacs distributions.
So a lot of people already have that.
If you're using Emacs 29 or greater, you have `eglot-mode`,
which is a lighter weight version of `lsp-mode`.
It's just another LSP client.
When I wrote the Semgrep language server,
Emacs 29 hadn't come out yet.
I'm not going to talk too much about `eglot-mode`
because I did everything in `lsp-mode`,
but I would imagine a lot of this stuff is very similar.
Here's a list of some supported languages.
Now let's get into the technical part.
How does LSP actually work?
So let's go over how it communicates first.
It uses JSONRPC,
which is just kind of like HTTP,
but instead of sending plain text, you're sending JSON.
So it's just sending JSON back and forth.
It's great because it's a way
for two programs to communicate
without sharing a common programming language.
Transport platform agnostic,
so it could be stdin, stdout,
sockets, whatever. It's just JSON.
You can send it over whatever.
There's two different types of messages,
a request, which requires a response from the other party,
and a notification, which does not expect a response.
So just a quick little example,
a user might open a document,
and then it'll send like a text document did open
and what document it was to the language server,
and then they'll change it.
Maybe they edit some code and introduce a syntax error.
The changes will be sent to the language server,
and then the language server will publish diagnostics,
which is those red squigglies
I was talking about earlier,
and say, hey, syntax error or whatever here,
or maybe the user says,
I want to go to the definition of this function,
and then the language server will spit back,
hey, this is where that function lives.
All very useful,
and the communication is relatively simple,
which is great.
This is what it looks like, what a request looks like.
Notifications look somewhat similar.
So now we know how LSP communication works,
but how does the actual protocol work?
Well, almost all of the protocol is opt-in,
meaning you don't have to support the entire specification,
you can just pick and choose.
Servers and clients will then communicate
what part of the protocol they both support,
so they'll both say, hey,
we support being notified when a user opens a document,
or if they're looking for documentation.
And so then once they agree upon what they'll both support,
then they'll send that stuff,
those notifications and requests back and forth.
Things like opening and closing files, diagnostics,
code completion, hovering over stuff, type signatures,
all of that. And what's cool is
even though the specification is huge
and probably has everything you need,
you can go ahead and add custom capabilities
if you really want to.
So you can just define a custom method,
and then now that works for you,
and now you can have that in all your editors.
For example, Rust Analyzer
has structural search and replace,
which is like find and replace,
but with respect to the structure of the code.
And if you choose to go down this route
with the custom capabilities,
you do have to remember you're going to have to
implement it in every client.
And that's a little bit more work,
but it's better than where we were without LSP.
So some quick tips on writing a language server.
I'm not going to get too into this
because it's very application-specific.
I wrote Semgrep's in OCaml
since our code base was almost all OCaml already,
and I wanted to leverage that.
Would not recommend
unless you also have a code base all in OCaml.
Structure is similar to a Rust server,
so a bunch of independent endpoints.
I would do everything functionally if I were you.
This is EmacsConf.
We're all hopefully used to writing functional Lisp.
I would recommend TypeScript or Rust, though,
depending on your level of performance
that you really need or whatever language
you're trying to support ideally.
Most languages have
some sort of language server protocol already.
But if they don't, then it might be easier
to do it in that language.
TypeScript has a lot of support, a lot of documentation,
a lot of examples out there
because it was what Microsoft originally intended
the language server protocol to be for, for VS Code,
which is written in TypeScript.
Rust is fast, it's going to take more effort,
but it's very fast, and Rust Analyzer has a great library
that they use and that they support.
So support there, examples there are great.
The hard part is not really the language server protocol,
but the actual logic. So, like, if you're doing, like,
language tooling, you're going to have to do
analysis on the code, so you need to do parsing,
possibly compiling, all these different advanced features,
all these advanced different things.
For example, Rust Analyzer will do incremental compilation,
which is really, really cool,
but that's, like, a whole separate talk.
If you're adapting an existing language tool,
this stuff is really easy.
You're basically just wiring stuff up.
But, yeah. So, now we know all about
LSP and language servers.
Say you want to actually
add support for a language server in Emacs.
How do you do that? Well, let's look at LSP mode,
because, like I said, this is what I'm most familiar with.
I'm sure `eglot-mode` is pretty similar.
So, `lsp-mode`'s repository is on GitHub,
like everything, and it has a ton of different clients
for a ton of different languages and frameworks and tools,
like Semgrep, and these are available
to anyone who installs LSP mode.
Alternatively, you can make a separate package
and just use LSP mode as a library,
but I'm not going to focus on this,
because there's already a ton of resources out there
on packaging and Emacs.
So, our steps, very quickly, are going to look like
adding an Emacs Lisp file that contains some logic,
add an entry somewhere, so we added a new client
to the list of clients, and then do some documentation,
because documentation's great.
First, creating a client.
In the `clients/` folder in `lsp-mode/`,
literally just add, like, `lsp-` whatever it is,
`require` the library, and register a client.
Registering a client just means, like,
saying what kind of connection it is.
It's most likely going to be standard I/O,
because that's pretty easy to implement,
and then you just pass it the executable
that you actually want to run.
Say what the activation function is,
so this is when the client should start,
so you can specify the language
or the major mode or whatever,
and now your client will start whenever that's triggered,
and then finally provide just a server ID,
so that way it's easy to keep track of,
and then run this LSP consistency check function.
This just makes sure everything up there is good.
You can do more advanced stuff with making an LSP client
that I'm not going to get into,
but just know that these aren't your only options,
and then finally provide your client.
Next, you just have to add your client
to the list of clients that `lsp-mode` supports,
and now you've added support for a whole new language,
whole new framework, whole new tool to Emacs,
and it's taking you, what, like, what is that,
20 lines of Lisp? No, not even, like, 15.
15 lines of Lisp, whole new language for Emacs.
It's really exciting. Now that you have your client,
let's do some documentation. Go fill out this, like, name,
where the repository, the source code is,
because free software is great,
and you should open source your stuff.
Specify the installation command.
What's cool about this is
this can be run automatically from Emacs,
so if it's, like, `pip install pyright`, right,
you can put that there, and Emacs will ask you,
do you want to install the language server,
and you can hit yes
and users will just have it installed for them,
and then you can say whether or not it's a debugger.
This is completely separate,
so there's this thing called DAP,
which is the debugger adapter protocol,
and it's similar to LSP but for debuggers,
which is very cool,
and then finally link to your documentation.
Please, please document your stuff.
If you want to add, like, a custom Emacs function
or custom capabilities, it's super easy.
It's literally just, like, calling a normal Emacs function.
For example, Semgrep normally only scans files
when you open them, but we added a Emacs function
that will scan your entire project, right,
and so that was just a client notification.
It was just `lsp-notify` and then a custom method,
and it's great because now you can just scan your project
from a simple Emacs function.
Requests, very similar to notifications.
You send it and then pass it a lambda
and do something with the result,
and so that's adding custom capabilities.
That's pretty much it. Thank you for listening.
Some resources here.
These links are clickable if you get the PDF,
if you get the slides. Semgrep: we're hiring!
If you want to work on, like,
programming language theory stuff,
compilers, parsers, editors,
email me or go look at our jobs.
The LSP specification, this is, like, the holy Bible.
It has all the specs, all the types, everything.
`lsp-mode` and the docs.
`lsp-mode`, right, that's where you want to add your client.
The docs are great, super useful.
Rust Analyzer is just a great reference
for language servers in general
if you want to write one or if you just want to, like,
see how they work. It's all just really well done.
It's great code, very readable.
And then down here is just a long video tutorial,
a longer video tutorial, not by me,
by someone else, on how to add a language client to Emacs,
but hopefully this is sufficient for y'all,
and now it's time for some Q&A.