[Speaker 0]: Okay. Hello, everyone.
I think this is the start of the Q&A session.
So people can just ask me questions here.
Or I think maybe these questions are going to
be read by someone. Yes,
thank you. Should I start doing that?
I also know that there's questions in the
either pad room, so I could start out
answering those as well.
[Speaker 1]: Right, sure. Whichever way you prefer.
If you prefer to read the questions yourself,
by all means, or if you would prefer me to
read them to you, that also works.
[Speaker 2]: Oh, I see.
[Speaker 0]: Why don't you read them to me?
I think it'll just be more interesting then.
[Speaker 1]: Sure. OK, let's see. The first question is,
what is your use case for embedding,
mainly for searching?
[Speaker 0]: Yeah, I mean, I think the use case really is
searching. And I think it is very useful when
you're searching for something in a vague
way. Just to give you an example,
I have a note system called EKG.
I type all my notes on it.
You can find it on GitHub and Melba.
But I wrote something at some point a year
ago or something. I wrote something that I
just vaguely remembered.
Oh, this was about a certain kind of
communication. I wanted communicating to
large audiences. There's some interesting tip
that I wrote down that was really cool.
And I was like, well, I need to find it.
So I did an embedding search for something
like, you know, tips for communicating.
Like those words may not have been in what I
was trying to find at all,
But it was able to find it.
And that is something that's very hard to do
in other ways. Like, you know,
if you had to do this with normal search,
you have to do synonyms.
And like maybe those synonyms wouldn't cover
it. Like with embedding,
you can basically get at like the vague
sentiment. You're like,
you know, you're, you know,
you can really query on like what things are
about as opposed to what words they have.
Also, it's super good for similarity search.
So you could say, look,
I have a bunch of things that are encoded
with embeddings that I want to show.
For example, you can make an embedding for
every buffer. You'd be like,
well, show me buffers that are similar to
this buffer. That doesn't sound super useful,
but this is the kind of thing you could do.
And so if you have a bunch of notes or
something else that you want to search on,
you'd be like, what's similar to this buffer?
Or what notes are similar to each other?
What buffers are similar to each other?
It's super good for this sort of thing.
And it's also good for this kind of retrieval
augmented generation, where you sort of,
you retrieve things and the purpose is not
for you to see them, but then you pass that
to the LLM. And then it's able to be a little
bit more accurate because it has the actual
text that you're trying to,
that is relevant, and it can cite from and
things like that. And then it could give you
a much better answer that's kind of,
you know, not just from its own little neural
nets and memory.
[Speaker 1]: Cool, thanks. Let's see,
next question. What do you think about embed
Emacs manual versus GPT's Emacs manual?
[Speaker 0]: I'm not exactly sure what this question is
trying to say. So I mean,
if someone wrote that and wants to expand on
it a little bit, but I think that maybe
you're saying like you could embed,
have embeddings for like various,
like every paragraph or something of the
Emacs manual. But it's also the case that
like GPT is already for sure already read it,
right? And so you could ask questions that
are about Emacs and our ELISP or whatever
part of the manual you want to find.
And it will do a reasonably good job,
especially the better models will do a
reasonably good job of saying you something
that is vaguely accurate.
But if you do this retrieval augmented
generation with embeddings,
you can get something that is very accurate.
At least I think. I haven't tried it,
but this is a technique that works in other
similar cases. So you can also imagine like,
oh, this whole thing I said,
like, oh, you can query for vague things and
get parts of the manual,
perhaps. I'm not exactly sure if that would
be useful, but maybe. Usually when I'm
looking things up in the Emacs manual or
Elist manual, I have something extremely
specific and I kind of know where to look.
But having other ways to get at this
information is always good.
[Speaker 1]: Right. Looks like they added a clarification
if you would like to read that yourself,
or would you like me to read it for you?
[Speaker 0]: Yeah. Yes, OK. It says,
I've never tried. Yeah,
the question is like OK,
there is a difference between the kind of
thing as I just described.
I have not tried the difference with the EMAX
manual itself. It'd be interesting to see
what this is, but I would expect like these
techniques, the retrieval augmented
generation is generally pretty good.
And I suspect it would,
I would bet money on the fact that it's gonna
give you, you know, better results than just,
you know, doing a free form query without any
retrieval augmented generation.
[Speaker 1]: Cool. Let's see. Next question.
When deferring commit messages to an LLM,
what, if anything, do you find you might have
[Speaker 0]: lost? Yeah, it's a good question.
When deferring anything to a computer,
like, you know, I used to have to remember
how to get places, and now,
you know, on the few occasions which I drive,
like, It could just tell me how to get
places. So similar things could occur here
where like, okay, I'm just leaving the LLM.
And so I'm kind of missing out on some
opportunity to think coherently about a
particular commit. Particular commits are
kind of low level. I don't think it's usually
relatively obvious and what they're doing.
And in this case, I think there's not much
loss. But for sure, in other cases,
if you're starting to get into situations
where it's writing your emails and all this
stuff. First of all, it's in 1 sense,
I'm not sure you might be losing something by
delegating things. On the other hand,
you know, when you're interacting with these
LLMs, you have to be extremely specific about
what you want, or else it's just not going to
do a good job. And that might actually be a
good thing. So the question might be that
maybe you might gain things by using an LLM
to do your work. It might not actually even
save you that much time,
at least initially, because you have to kind
of practice again super specific about what
you want to get out of the output it's going
to give you so like oh I'm you know maybe you
know you're on the emacs devel mailing list
and you're like okay write this email about
this about this And here's what I want to
say. And here's the kind of tone I want to
use. And here's the like,
oh, you might want to specify like everything
that you kind of want to get into this.
Usually it's easier just to write the email.
But I think that practice of kind of
understanding what you want is not something
you normally do. And I think it's going to be
an interesting exercise that will help people
understand. That said,
I haven't done that much of that,
so I can't say, oh, yeah,
I've done this and it works for me.
Maybe. I think it's an interesting thing to
explore.
[Speaker 1]: Sure. Thanks. Let's see.
Let's see. Can you share your font settings
in your Emacs config? Those are some nice
fonts for reading.
[Speaker 0]: Yeah, I think I was using Menlo at the time.
Unfortunately, I don't save those kinds of
things, like a history of this.
I've kind of switched now to,
what was that? I think I wrote it down in
the, I switched to MunaSpace,
which just came out like a week or 2 ago,
and is also pretty cool.
So I think it's Menlo.
The internal question,
what font are you using?
[Speaker 1]: Indeed, yeah. It looks like someone guessed
as well that it might be Menlo.
OK, Cool. Yeah, next question.
In terms of standardization,
do you see a need for the medium to large
scale effort needed? And then they also
elaborate about it.
[Speaker 0]: Yeah, I mean, I do think,
I don't know if it's large scale,
but at least it's probably medium scale.
There's a lot of things that are missing that
we don't have right now in emacs when you're
dealing with LLMs. 1 is,
a prompting system. And by that,
I mean, you know, prompts are just like big
blocks of text, but there's also senses that
like prompts need to be composable and you
need to be able to iterate on parts of the
prompt. And so it's also customizable.
Users might want to customize it.
On the other hand, it's not super easy to
write the prompt. So you want really good
defaults. So the whole prompt system is kind
of complicated. That needs to be kind of
standardized, because I don't think there's
any tools for doing something like that right
now. I personally use my system,
my note system for EKG.
I don't think that's appropriate for
everyone, but it does,
I did write it to have some of these
capabilities of composability that I think
are useful for a prompt generation.
It'd be nice to have a system like that,
but for general use. I don't,
this is something I've been meaning to think
about, like how to do it,
but like this, you know,
if someone's interested in getting this area,
like, I would love to chat about that or,
you know, I think there's a lot of
interesting ideas that we could have to have
a system that allows us to make progress
here. And also, I think there's more to
standardization to be done.
1 thing I'd also like to see that we haven't
done yet is a system for standardizing on
getting structured output.
This is gonna be super useful.
I have this for open AIs API,
cause they support it.
And it's really nice, cause then you can
write elist functions that like,
okay, I'm going to call the LLM.
I'm gonna get structured output.
I know what that structure is going to be.
It's not going to be just a big block of
text. I could turn it into a,
you know, a P list or something.
And then I could get the values out of that P
list. And I know that way I could do,
I could write actual apps that are,
you know, very, very sort of,
you know, useful for very specific purposes
and not just for text generation.
And I think that's 1 of the most important
things we want to do. And I have some ideas
about how to do it. I just haven't pursued
those yet. But if other people have ideas,
I think this would be really interesting to
add to the LLM package.
So contact me there.
[Speaker 1]: Awesome. Quick note before we continue.
So I'm not sure how long we're going to be on
stream for, because this is the last talk
before the break. If we are on the stream
long-term, then great.
But if not, folks are welcome to continue
writing questions on the pad.
And hopefully, Andrew will get to them at
some point. Or if Andrew maybe has some extra
time available and wants to stay on
BigBlueButton here, then folks are also
welcome to join here and chat with Andrew
directly as well. Okay,
awesome. So yeah, the next question is,
what are your thoughts on the carbon
footprint of LLM usage?
[Speaker 0]: Yeah, it's a really interesting question.
I don't have any particular knowledge or
opinions about that. It's something I think
we should all be educating ourselves more
about. It is really, I mean,
there's 2 parts of this,
right? They take a, there's a huge amount of
carbon footprint involved in training these
things. Then running them is relatively
lightweight. So the question is not
necessarily like once it's trained,
like I don't feel like it's a big deal to
keep using it, but like training these things
is kind of like the big carbon cost of it.
But like right now, the way everything's
going, like every, you know,
all, you know, the top 5 or 6 tech companies
are all training their LLMs,
and this is all costing a giant amount of
carbon probably. On the other hand these same
companies are pretty good about using the
least amount of carbon necessary you know
they have their own their tricks for doing
things very efficiently.
[Speaker 1]: Cool next question, LLMs are slow and
responding. Do you think Emacs should provide
more async primitives to keep it responsive?
Like the URL retrieve is quite bad at
building API clients with it.
Building API clients with it?
[Speaker 0]: Yeah. Well, OK, so first of all,
people should be using the LLM client.
And So right now, 1 thing I should have
mentioned at the top is that there are new
packages that I recorded this talk that you
just saw several months ago.
And so like Elama, there's this package Elama
that came out that is using the LM package.
And so for example, it doesn't need to worry
about this sort of thing because it just uses
LLM and package and the LLM package worries
about this. And while I'm on the subject of
things I forgot to mention,
I also should just mention very quickly that
there is now an open source model,
Mistral. And so that's kind of this new thing
on the scene that happened after I recorded
my talk. And I think it's super important to
the community and important that we have the
opportunity to use that if we want to.
Okay, but to answer the actual question,
there has been some talk about the problems
with URL retrieve in the URL package in
general in EmacsDevEl.
It's not great. I would like to have better
primitives. And I've asked the author of
Please PLZ to kind of provide some necessary
callbacks. I think that's a great library.
And I'd like to see that kind of like,
It's nice that we have options,
and that is an option that uses curl on the
back end, and that has some benefits.
So there's this big debate about whether we
should have primitives or just use curl.
I'm not exactly sure what the right call is,
but there has been discussions about this.
[Speaker 1]: Excellent. And someone commented that GPTEL
is async and apparently very good at tracking
the point.
[Speaker 0]: Yes, yes, GPTEL has similar functionalities
to LLM, although I believe it's going to move
to LLM itself sometime soon.
[Speaker 1]: Next question, speaking of which,
anyone trained or fine-tuned or prompted a
model with their org data yet and applied it
to interesting use cases like planning,
scheduling, et cetera,
and maybe care to comment?
[Speaker 0]: I don't know anyone who is doing that.
I think it is interesting.
Like this is what I kind of mentioned at the
very end of the talk. There is a lot of stuff
there like you could you know if you
especially mean an LLM can kind of work as
sort of like a secretary kind of person that
could help you prioritize Still it's a
slightly unclear how what the best way to use
it is So I think there's more of a question
for the community about like what people have
been trying. I see someone has mentioned that
they are using it for weekly review.
And it's kind of nice to like,
maybe you could read your agenda or maybe
this for like weekly review.
It could like read all the stuff you've done
and ask you questions about it.
And like, what should happen next?
Or like, is this going to cause a problem?
Like, I can, I can understand if that could
happen? That's like, that's kind of nice.
And this kind of people have had good success
out of using these LLMs to bounce ideas off
of are, you know, for,
you know, I've seen people say that like they
want, they use it for reading and they kind
of dialogue with the LM to kind of like do
sort of active reading.
So you can imagine doing something similar
with your tasks where it's sort of you're
engaged in dialogue about like planning your
tax with some with a alum that could kind of
understand what those are and ask you some
questions I think it. You know,
if it'd be nice. So, the problem is like
there's no great way to share all this stuff.
I guess if you have something like this,
put it on Reddit. If you don't have Reddit,
I don't know what to do.
I would say put it somewhere.
At the very least, I could maybe open up like
an LLM discussion session on the LLM package
GitHub, But not everyone likes to use GitHub.
I don't know. It'd be nice if there's a
mailing list or IRC chat for this sort of
thing. But there isn't at the moment.
[Speaker 1]: All right. Let's see. I think that's the end
of the questions on the pad so far.
There was also some discussion or some
chatter, I believe, on IRC.
I'm not sure. Andrew, are you on IRC right
[Speaker 0]: I am, but I don't think I'm on any place that
has the chatter. So if there's chatter,
then I'm not seeing it.
[Speaker 1]: now? Okay. Yeah, it was in the emacsconf-dev
channel.
[Speaker 0]: Okay, let me see if I can.
Oh, yes. I mean, I could see the channel,
but I missed whatever came before.
So if there's anything you want to kind of
call out, I can try to answer it here.
[Speaker 1]: OK, cool. I believe at least 2 other folks
who are participating in the discussion there
who have also joined here on BigBlueButton,
Codin Quark and AeonTurn92.
So you folks, if Andrew is still available
and has time, you're welcome to chat here and
ask questions or discuss here as well.
[Speaker 0]: 1 Thank you. Thank you for your help,
and thank you for reading all the questions.
[Speaker 1]: AUDIENCE 2 Cheers, and thanks to you for a
great talk and the discussion.
[Speaker 0]: AUDIENCE AUDIENCE 1 Thank you.
[Speaker 1]: AUDIENCE 2 Cheers.
[Speaker 0]: So I'll just, I will wait here and see if
there's any questions.
If not, I will log off after a few minutes.
[Speaker 2]: Well, I guess since we were mentioned that
there was a small chat about local alarms.
Because chat dpt is nice,
no, but privacy concerns,
and it's not free and stuff.
Which, so The question is,
what is the promise for local models?
[Speaker 0]: Yeah, so local is definitely...
[Speaker 2]: Or at least open source.
[Speaker 0]: Yeah, so there is a local open source model,
Misral, which you could run.
The LLM package allows you to use,
I think there's 3 kind of local things you
could use. Like many of these things,
there's like many kind of ways to do the same
sort of thing. So LLM is supporting OLAMMA
and LLAMMA-CPP. And let's see,
1 other. Which 1 is it?
And maybe that's it. Maybe the,
oh, GPT for all. So each 1 of these kind of
has slightly different functionality.
For example, I think GPT for all doesn't
support embeddings. And I hear that Olama's
embeddings are kind of currently broken.
But basically they should support everything.
And the open source models are,
so the local models are reasonably good.
Like I don't think you'd use them and be
like, what is this horrible nonsense?
Like it's, it gives you relatively good
results. Like it's not gonna be at the level
of like GPT 3.5 or 4, but it's not far away
from GPT 3.5, I think.
[Speaker 2]: I'm just saying that Olam has like a presets
for connecting the actual working servers for
Olama?
[Speaker 0]: So, I'll try. Yeah, so you could,
what you could do is you could like for
example you could download Olama which is
just a way of setting up local models and
running local models on your machine.
So typically what it does,
you like download a program,
let's say Olama. Then Olama will have the
ability to download models.
And so you could choose from just a host of
different models. Each 1 of these things has
a bunch of different models.
So it downloads all these things to your
machine. But I would say that the key problem
here is that it requires a fairly beefy
machine.
[Speaker 2]: So. Yeah, yeah, of course.
Why I was asking, because you briefly
mentioned that there are some Israeli
servers. I understand that they run it like a
government or stuff like that?
No, no, sorry. People want everyone?
[Speaker 0]: I don't, I mean, maybe you've said something
that sounded like Israeli servers.
[Speaker 2]: Okay, okay.
[Speaker 0]: I think- There's no government LLMs as far as
I know. Although, I'm sure the governments
are working on their own LLMs,
et cetera. But yeah, basically your choices
are spend a, I mean, if you use open AI or
something or anything else,
you're really not spending any money.
Like I've never been able to spend any money
on OpenAI. Like unless you're doing something
very intensive and really are using it to,
you know, if you're using it for your
personal use, it's just hard to spend any
money. But on the other hand,
it's not free. So you can,
you know,
[Speaker 2]: Actually, it's rather cheap.
There's no question about that.
The problem is that it has a bad track record
on privacy.
[Speaker 0]: Yes, that's, I think that is a key problem.
This is probably the number 1 reason why you
might want to use a local AI,
a local LLM. Another 1 is like,
you may not agree with the decisions.
You know, there's a lot of trust and safety
stuff that these companies have to do.
Like they don't want like the LMs to kind of
like give you, like tell you how you can make
meth or how you can make a bomb,
which they would do. They would totally do
it. So, But each time you kind of restrict
what is happening with what you can get out
of the LM, it gets a little worse.
So some people
[Speaker 2]: want to have local. That's expected.
I guess even open source language modules
will soon have HR spaces because it's simply
a legal issue.
[Speaker 0]: I think that's true. I also think that there
probably will be, although I don't know of
any offhand, that will are completely
uncensored. I know people are interested and
are running uncensored models.
I don't know how to do it.
I think it's a little bit dubious,
but some people do want to do it.
There's another reason for using local
servers. Do you have any recommendation for
models to run locally and also comments on
whether a GPU is required?
Usually a GPU, well, you can run it without a
GPU, but it does run much better.
Like for example, I think when I used,
Lama is sort of like a standard.
This was the model for that Facebook came out
with for local use. And It was,
yeah, it's good. It's,
but it's now it's I think,
Mistral is kind of like has a better
performance, But there's also different model
sizes. There's 7B, like the Lama 7B is OK.
The Mistral 7B, 7 billion,
are like, basically it'll take like,
you can run it with like 16 gigs of RAM,
is pretty good. It's probably about as equal
to the LLAMA13B. Those are the number of
parameters, if I remember correctly.
And then there's a 7B,
which I've never been able to run.
And even if the 7B, if you run it without a
GPU, it takes quite a while to answer.
I think I've had experiences where it took
literally like several,
like 5 minutes before it even started
responding, but you do eventually get
something. And it could be that like things
have gotten better since the last time I
tried this, because things are moving fast.
But it is super recommended to have a GPU.
This is the problem. It's kind of like,
yes, free software is great.
But if free software is requiring that you
have these kind of beefy servers and have all
this hardware, that's not great.
I think there's a case to be made.
[Speaker 1]: a hardware
[Speaker 0]: with slots instead of a laptop.
Yeah, yeah, that's right.
[Speaker 2]: Ideally, you can have Ideally,
it would be nice if FSL for all things could
run something for open source model.
And not free, but the key point is that it's
Libre?
[Speaker 0]: Yes, so actually I think Google does do that.
I'll have to look it up,
but I haven't explored this yet.
But Google's server, which LLM does support,
supports arbitrary models.
So you can run LLMA or things like that.
The problem is that even if you're running
Mistral, which has no restrictions.
So this is the kind of thing that like the
Free Software Foundation cares a lot about.
Like you want it to be like no restrictions,
legal restrictions on you as you run the
model. So even if it's running Mistral,
just by using the server,
the company server, it will impose some
restrictions on you probably,
right? There's gonna be some license that you
have to, or something you have to abide by.
So I think, yes, it depends on how much you
care about it, I guess.
I should find out more about that and make
sure that it's a good point that I should,
you know, people should be able to run free
models over the server.
So I should make sure we support that in the
LLM package. So, is there any other questions
Or is otherwise we can end the session.
Yeah, all right. Thank you.
Thank you. Thank you everyone who listened.
I'm super happy like I,
the interest is great.
I think there's great stuff to be done here
and I'm kind of super excited what we're
going to do in the next year,
so hopefully, like next year,
and the conference we have something even
more exciting to say about LLM and how they
can be used with Emacs.
So thank
you