Hello, I'm Andrew Hyatt and I'm going to talk to you
about large language models and how
they relate to Emacs.
And I'm going to talk to you about the technology
and how we're going to use it in Emacs.
There'll be demos and there'll be talks about,
I'll finish up by kind of talking about where
I think this should go in the future.
So to start off with, let's just talk like,
I just want to make sure everyone's on the same page.
What are large language models?
Not everyone may be caught up on this.
Large language models are a way... Basically,
the current versions of large language models
are all based on the similar architecture
called the transformer.
It's just an efficient way to train and produce output.
So these things are basically models
that predict the next word or something like that.
And they're trained on an enormous corpus of information
and they get extremely good
at predicting the next word.
And from that basic ability, you can train
through further tuning from human input,
human ratings and things like that.
You can train different models based on that
that will do question answering.
And this is how basically ChatGPT works.
There's a base LLM, like GPT.
And then you have a chat version of that,
which is just trained to just... You give
it a prompt, like what do you want it to do?
And it gives you an output that does what you told it to do,
or at least attempts to do it.
Those are the power of large language models is
they're extremely, extremely impressive.
Certainly this is, in AI,
this has been the biggest thing to happen
probably in my lifetime,
or at least my lifetime as my working lifetime.
So let me give you a demonstration of
what kinds of stuff it could do in Emacs.
So here I have a Emacs file.
So this is my Emacs init file.
I have a change.
Let's commit that change.
And, you know, I don't like writing commit messages,
so I can generate it.
And it did an actually just looking.
So all it does is it's looking, it's just reading the diff.
I'm just feeding it the diff with some instructions.
And it is this a incredible commit message?
It's not bad, actually.
You can see that it actually has really extracted
the meaning of what I'm doing and has written
a reasonably good commit message.
Now I have to edit it because this is not quite correct.
But it's kind of impressive how good it is.
And my editing, it's kind of easier for me to edit this
than just to write a new one.
And quite often it's good enough to just submit as is.
So this is kind of, you know, you could say
this is just commit messages.
You could respond to emails.
You could, you know, using your own custom instructions
about what you want your email to say.
It'll write the email for you.
It could do like this
Emacs is a way to interact with buffers.
This could basically just output text.
So it's super useful for
understanding something and outputting text based on that,
which is just useful for Emacs.
So the drawback is, yeah, it's good,
but it's not that reliable.
And you'd think it's very easy to get caught up in like,
oh my gosh, like this is so powerful.
I bet it could work this, whatever idea could work.
And these ideas, like they almost can.
For example, I was thinking, you know what I could do?
I don't like writing regexes.
Why can't I have a regex replace that's powered by LLMs?
And that way I could give just an instruction
to regex replace.
And so for example, I could do Emacs LLM regex replace.
This is not checked in anywhere.
These are just my own kind of private functions.
My description lowercase all the org headings.
Let's see if it works.
It might work.
No, it doesn't work.
So if I, I'm not going to bother to show you
what it actually came up with, but it's something,
if you looked at it, it'd be like, wow,
this is very close to being...
It looks like it should work, but it doesn't.
Okay.
It's not quite good enough to get it right.
And it's possible that perhaps by giving it
a few examples of, or explaining more
what makes Emacs regexes different.
It could do a better job
and maybe could solve these problems,
but it's always a little bit random.
You're never quite sure what you're going to get.
So this is the drawback.
Like there's a lot of things that look like you could do it,
but when it actually comes down to trying it,
it's surprisingly hard.
And, you know, and whatever you're doing,
it's surprisingly hard to get something
that is repeatably, that's, that is always good.
So yeah, that's currently the problem.
So I want to talk about embeddings.
They're another thing that LLMs offer
and that are extremely useful.
They are, what they do is they encode from
a input text that could be a word, a sentence,
a small document.
It encodes a vector about what the meaning,
the semantic meaning of that is.
That means you could, something that is,
uses completely different words,
but is basically talking about the same thing,
perhaps in a different language, should be pretty close
as a vector to the other vector.
You know, as long as they're similarly semantic things,
like the words
highway and Camino are two different words.
They mean the same thing.
They should have very similar embeddings.
So it is a way to kind of encode this
and then you could use this for search.
For example, I haven't tried to do this yet,
but you could probably just make an embedding
for every paragraph in the Emacs manual
and the Elisp manual.
And then, and then there's a very standard technique.
You just... You find that you have a query,
oh, how do I do whatever, whatever in Emacs again?
And you could, you just find that 20 things
that are closest to whatever you're
trying to... the embedding of your query.
You send those things to the LLM, as you know,
with the original query,
and you're basically telling the--asking the LLM,
look, the user is trying to do this.
Here's what I found in the Emacs manual.
That's on the Elisp manual.
That's close to what they're trying to do.
So can you kind of just tell the user what to do?
And from this, and you could say,
just use things from this, you know, that I give you.
Don't just make up your own idea.
You know, don't use your own ideas,
because sometimes it likes to do that
and those things are wrong.
So you could try to, you know, do this and you get,
you could get quite good results using this.
So no one has done this yet,
but that should not be hard to do.
Image generation is something that's, you know,
it's not quite an LLM in the sense of...
These are... It's a different technology,
but these things are kind of packaged together
in a sense.
And you'll see that when I talk about Emacs packages,
a lot of them bundle image generation
and large language models.
You know, the APIs are often bundled together by providers.
And the general idea is it's kind of similar
because it's very similar to large, you know,
doing a chat thing where you, you know,
the chat is like, you give it a text request,
like write me a sonnet about, you know,
the battle between Emacs and vi.
And it could, it could do it.
It could do a very good job of that.
But you could also say, you know,
draw me a picture of Emacs and vi as boxers,
as a character-character boxing in a ring,
like a, you know, political cartoon style.
And it can do that as well.
And so you could basically think of this
as just sort of... it's kind of the
same thing with what you're doing
with large language models,
but instead of outputting a text,
you're outputting a picture.
There's also, I want to mention the concept of fine-tuning.
Fine-tuning is a way to take your--
take a corpus of inputs and outputs and just from
a large language model, you're like, okay,
given this base large language model,
I want to make sure that when I give you input,
you give me something like output.
And this is what I'm just going to
train you further on these,
these mappings between input and output.
And for example, you could do this. Like,
let's say you wanted to fix that regex demo
I had to make it good.
I don't think it, I think it'd be
relatively effective to train,
to have regex descriptions
and regex examples, Emacs regex examples
as inputs and outputs.
You could get, you know, maybe a hundred,
a few hundreds of these things.
You could train it.
I think that is a reasonable way to,
let's just say, I don't know how well it would work,
but these things definitely work some of the time
and produce pretty good results.
And you could do this on your own machine.
Corporations like OpenAI offer APIs with, you know,
to build your fine tunes on top of OpenAI.
And I think, I'm not a hundred percent sure,
but I think then you can share your model
with other people.
But if not, then you just, you know,
you could use your model for your own specialized purposes.
But in the world of models that you could run,
for example, based on Llama, which is like...
Llama is this model you can run on your own machine from Meta.
There's many fine-tuned models that you could download
and you could run on your own.
They can do very different things too.
Some output Python programs, for example,
that you could just run.
So you just say...
Tell me how old... Let's just say
you have a random task, like
tell me how old these five cities are in minutes,
based on historical evidence.
It's kind of a weird query, but it probably can figure,
it could probably run that for you.
It'll encode its knowledge into whatever
the Python program, then use the Python program
to do the correct calculations.
So pretty, pretty useful stuff.
So I also want to mention open source
and basically free software here.
These LLMs are mostly not free software.
They're sometimes open source,
but they're generally not free
without restrictions to use.
Most of these things, even Llama,
which you can use on your own machine,
have restrictions that you cannot use it
to train your own model.
This is something that, you know,
it costs millions and millions of dollars
to train and produce these models.
And that's just computation costs.
They do not want you
stealing all that work by training your own models
based on their output.
But there are research LLMs that do, I believe,
conform to free software principles.
They're just not as good yet.
And I think that might change in the future.
So speaking of the future,
one of the things I'd like to point out
is that like the demos I showed you are based on,
I'm using OpenAI 3.5 model.
That's more than, well, no,
it's like a year old basically at this point.
And things are moving fast.
They came out with 4.0.
4.0 is significantly better.
I don't have access to it.
Even though I'm using the API and I'm paying money for it,
you only can get access to 4.0
if you can spend a dollar.
And I've never been able to spend,
use so much API use that I've spent a dollar.
So I have, I don't have 4.0, but I've tried it
because I do pay for this
so I could get access to 4.0
and it is substantially better.
By all reports, it's,
the difference is extremely significant.
I would not be surprised
if some of the limitations and drawbacks I described
mostly went away with 4.0.
We're probably at a stage
where regexes will work maybe 5% of the time
if you try them.
But with 4.0, it could work like 80% of the time.
Now, is that good enough?
Probably not, but it's a,
I wouldn't be surprised if you got results like that.
And in a year's time, in two years time,
no one knows how much this is going to play out
before progress stalls,
but there are a lot of interesting research.
I don't think, research wise,
I don't think things have slowed down.
You're still seeing a lot of advances.
You're still seeing a lot of models coming out
and that will come out.
That will be each one, one upping the other one
in terms of quality.
It'll be really interesting to see how this all plays out.
I think that message here is that
we're at the beginning here.
This is why I think this talk is important.
I think this is why we should be
paying attention to this stuff.
Let's talk about the existing packages.
Because there's a lot out there, people have,
I think people have been integrating with
these LLMs that often have a relatively easy to use API.
So it's kind of natural that people
have already put out a lot of packages.
Coming off this problem from a lot of different angles,
I don't have time to go through
all of these packages.
These are great packages though.
If you're not familiar with them,
please check them out.
And they all are doing slightly different things.
Some of these are relatively straightforward.
Interactions, just a way to
almost in a comment sort of way to kind of
have just an interaction,
long running interaction with an LLM
where you kind of build off previous responses,
kind of like the OpenAI's UI.
Two very more Emacsy things where you can sort of
embed these LLM responses within a org-mode block
using the org-mode's context.
Or GitHub Copilot integration where you can use it
for auto completion in a very powerful,
you know, this stuff is very useful if it could figure out
what you're trying to do based on the context.
It's quite effective.
But I want to kind of call out one thing
that I'd like to see change.
Which is that users right now,
not all of these have a choice of,
first of all, there's a lot of them.
Each one of them is doing their own calls.
And each one of them is, so each one of them
has their own interfaces.
They're rewriting the interface to OpenAI or wherever.
And they're not, they don't, most of these
do not make it that configurable or at all configurable
what LLM use.
This is not good.
It is important that we use,
we give the user a way to change the LLM they use.
And that is because you might not be comfortable
sending your requests over to a private corporation
where you don't get to see how they use their data.
Your data, really.
That's especially true with things like embeddings
where you might be sending over your documents.
You're just giving them your documents, basically.
And, you know, that does happen.
I don't think really that there's a reason
to be uncomfortable with this,
but that, you know, people are uncomfortable and that's okay.
People might want to use a local machine,
a local LLM for maximum privacy.
That's something we should allow.
People might want to especially use free software.
That's something we should definitely allow.
This is Emacs.
We need to encourage that.
But right now, as most of these things are written,
you can't do it.
And they're spending precious time
just doing things themselves.
This is why I wrote LLM, which is...
it will just make that connection to the LLM for you
and it will connect to, you know, it has plugins.
So if you can, the user can configure what plugin
it actually goes to.
Does it go to OpenAI?
Does it go to Google Cloud Vertex?
Does it go to Llama on your machine?
We're using Ollama,
which is just a way to run Llama locally.
And more things in the future, I hope.
So this is, I'm hoping that we use this.
It's designed to be sort of maximally usable.
You don't need to install anything.
It's on GNU ELPA.
So even if you write something
that you want to contribute to GNU ELPA,
you can use it because it's on GNU ELPA.
It's part of the Emacs package, Emacs core packages.
So, but it has no functionality.
It's really just there as a library
to use by other things offering functionality. Okay.
And it's a little bit difficult to abstract.
I want to point this out
because I think it's an important point
is that the, it's, some of these LLMs, for example,
have image generation.
Some do not.
Some of them have very large context windows, even for chat.
You say, okay, all these things can do chat.
Okay.
Yeah, kind of.
Some of these things you could pass a book to,
like Anthropic's API.
Most, you cannot.
So there really are big differences
in how these things work.
I hope those differences diminish in the future.
But it's just one of the challenges
that I hope we can work through in the LLM library.
So it's compatible, but there's definitely
limits to that compatibility.
I want to point out just to finish off,
Emacs is the, Emacs has real power here
that nothing else I think in the industry is offering.
First of all, people that use Emacs
tend to do a lot of things in Emacs.
We have our to-dos in Emacs with the org mode.
We have mail.
We, you know, we might read email and we might,
and respond to email in Emacs.
We might have notes in Emacs.
This is very powerful.
Using... there's not other stuff like that.
And you could feed this stuff to an LLM.
You could do interesting things
using a combination of all this data.
No one else could do this.
We need to start thinking about it.
Secondly, Emacs can execute commands.
This might be a bad idea.
This might be how the robots take over,
but you could have the LLMs respond with Emacs
commands and run those Emacs commands
and tell the LLM the response and have it do things
as your agent in the editor.
I think we need to explore ideas like this.
And I think we need to share these ideas
and we need to make sure that we're pushing the
envelope for Emacs and actually, you know, doing things,
sharing ideas, sharing progress,
and kind of seeing how far we can push this stuff.
Let's really help Emacs out, be sort of,
take advantage of this super powerful technique.
Thank you for listening.