path: root/2023/captions/emacsconf-2023-llm--llm-clients-in-emacs-functionality-and-standardization--andrew-hyatt--answers.vtt

WEBVTT


00:00:13.099 --> 00:00:13.599
[Speaker 0]: Okay. Hello, everyone.

00:00:16.560 --> 00:00:17.060
I think this is the start of the Q&A session.

00:00:25.119 --> 00:00:25.599
So people can just ask me questions here.

00:00:28.259 --> 00:00:28.380
Or I think maybe these questions are going to

00:00:30.560 --> 00:00:30.980
be read by someone. Yes,

00:00:34.680 --> 00:00:35.060
thank you. Should I start doing that?

00:00:39.280 --> 00:00:39.400
I also know that there's questions in the

00:00:41.320 --> 00:00:41.480
either pad room, so I could start out

00:00:42.280 --> 00:00:42.780
answering those as well.

00:00:45.020 --> 00:00:45.300
[Speaker 1]: Right, sure. Whichever way you prefer.

00:00:46.860 --> 00:00:47.220
If you prefer to read the questions yourself,

00:00:48.940 --> 00:00:49.080
by all means, or if you would prefer me to

00:00:50.080 --> 00:00:50.280
read them to you, that also works.

00:00:50.580 --> 00:00:50.920
[Speaker 2]: Oh, I see.

00:00:51.600 --> 00:00:51.760
[Speaker 0]: Why don't you read them to me?

00:00:53.260 --> 00:00:53.760
I think it'll just be more interesting then.

00:00:56.820 --> 00:00:57.160
[Speaker 1]: Sure. OK, let's see. The first question is,

00:00:58.360 --> 00:00:58.860
what is your use case for embedding,

00:01:00.060 --> 00:01:00.560
mainly for searching?

00:01:06.180 --> 00:01:06.340
[Speaker 0]: Yeah, I mean, I think the use case really is

00:01:12.100 --> 00:01:12.320
searching. And I think it is very useful when

00:01:15.060 --> 00:01:15.320
you're searching for something in a vague

00:01:18.280 --> 00:01:18.780
way. Just to give you an example,

00:01:23.860 --> 00:01:24.360
I have a note system called EKG.

00:01:25.760 --> 00:01:26.260
I type all my notes on it.

00:01:28.620 --> 00:01:29.120
You can find it on GitHub and Melba.

00:01:34.140 --> 00:01:34.400
But I wrote something at some point a year

00:01:35.840 --> 00:01:36.020
ago or something. I wrote something that I

00:01:36.600 --> 00:01:36.980
just vaguely remembered.

00:01:38.800 --> 00:01:38.940
Oh, this was about a certain kind of

00:01:41.580 --> 00:01:41.760
communication. I wanted communicating to

00:01:43.280 --> 00:01:43.479
large audiences. There's some interesting tip

00:01:44.700 --> 00:01:45.060
that I wrote down that was really cool.

00:01:49.920 --> 00:01:50.080
And I was like, well, I need to find it.

00:01:52.260 --> 00:01:52.640
So I did an embedding search for something

00:01:55.479 --> 00:01:55.979
like, you know, tips for communicating.

00:01:58.979 --> 00:01:59.100
Like those words may not have been in what I

00:02:00.020 --> 00:02:00.520
was trying to find at all,

00:02:02.680 --> 00:02:03.180
But it was able to find it.

00:02:05.840 --> 00:02:06.260
And that is something that's very hard to do

00:02:07.200 --> 00:02:07.360
in other ways. Like, you know,

00:02:08.520 --> 00:02:08.720
if you had to do this with normal search,

00:02:09.199 --> 00:02:09.660
you have to do synonyms.

00:02:10.940 --> 00:02:11.200
And like maybe those synonyms wouldn't cover

00:02:11.960 --> 00:02:12.340
it. Like with embedding,

00:02:13.940 --> 00:02:14.160
you can basically get at like the vague

00:02:14.960 --> 00:02:15.460
sentiment. You're like,

00:02:17.320 --> 00:02:17.560
you know, you're, you know,

00:02:19.520 --> 00:02:19.700
you can really query on like what things are

00:02:21.760 --> 00:02:22.260
about as opposed to what words they have.

00:02:25.600 --> 00:02:26.100
Also, it's super good for similarity search.

00:02:27.720 --> 00:02:27.900
So you could say, look,

00:02:30.040 --> 00:02:30.240
I have a bunch of things that are encoded

00:02:31.400 --> 00:02:31.900
with embeddings that I want to show.

00:02:34.120 --> 00:02:34.280
For example, you can make an embedding for

00:02:35.220 --> 00:02:35.380
every buffer. You'd be like,

00:02:37.060 --> 00:02:37.200
well, show me buffers that are similar to

00:02:38.740 --> 00:02:38.980
this buffer. That doesn't sound super useful,

00:02:40.440 --> 00:02:40.940
but this is the kind of thing you could do.

00:02:45.300 --> 00:02:45.480
And so if you have a bunch of notes or

00:02:46.720 --> 00:02:46.920
something else that you want to search on,

00:02:48.240 --> 00:02:48.740
you'd be like, what's similar to this buffer?

00:02:51.500 --> 00:02:51.760
Or what notes are similar to each other?

00:02:53.040 --> 00:02:53.540
What buffers are similar to each other?

00:02:55.380 --> 00:02:55.880
It's super good for this sort of thing.

00:03:00.780 --> 00:03:01.280
And it's also good for this kind of retrieval

00:03:03.080 --> 00:03:03.520
augmented generation, where you sort of,

00:03:05.080 --> 00:03:05.280
you retrieve things and the purpose is not

00:03:06.600 --> 00:03:06.880
for you to see them, but then you pass that

00:03:12.040 --> 00:03:12.180
to the LLM. And then it's able to be a little

00:03:14.340 --> 00:03:14.800
bit more accurate because it has the actual

00:03:15.760 --> 00:03:16.260
text that you're trying to,

00:03:18.960 --> 00:03:19.180
that is relevant, and it can cite from and

00:03:20.720 --> 00:03:20.820
things like that. And then it could give you

00:03:22.260 --> 00:03:22.660
a much better answer that's kind of,

00:03:25.520 --> 00:03:25.680
you know, not just from its own little neural

00:03:26.320 --> 00:03:26.820
nets and memory.

00:03:31.920 --> 00:03:32.120
[Speaker 1]: Cool, thanks. Let's see,

00:03:35.740 --> 00:03:36.100
next question. What do you think about embed

00:03:40.160 --> 00:03:40.660
Emacs manual versus GPT's Emacs manual?

00:03:45.480 --> 00:03:45.660
[Speaker 0]: I'm not exactly sure what this question is

00:03:46.980 --> 00:03:47.480
trying to say. So I mean,

00:03:51.000 --> 00:03:51.140
if someone wrote that and wants to expand on

00:03:55.080 --> 00:03:55.320
it a little bit, but I think that maybe

00:03:58.420 --> 00:03:58.920
you're saying like you could embed,

00:04:00.280 --> 00:04:00.780
have embeddings for like various,

00:04:02.520 --> 00:04:02.720
like every paragraph or something of the

00:04:04.540 --> 00:04:04.680
Emacs manual. But it's also the case that

00:04:06.500 --> 00:04:07.000
like GPT is already for sure already read it,

00:04:09.760 --> 00:04:09.960
right? And so you could ask questions that

00:04:13.460 --> 00:04:13.780
are about Emacs and our ELISP or whatever

00:04:15.200 --> 00:04:15.700
part of the manual you want to find.

00:04:19.760 --> 00:04:20.240
And it will do a reasonably good job,

00:04:22.280 --> 00:04:22.780
especially the better models will do a

00:04:24.620 --> 00:04:24.940
reasonably good job of saying you something

00:04:26.040 --> 00:04:26.540
that is vaguely accurate.

00:04:29.440 --> 00:04:29.860
But if you do this retrieval augmented

00:04:30.580 --> 00:04:31.080
generation with embeddings,

00:04:32.640 --> 00:04:33.140
you can get something that is very accurate.

00:04:36.700 --> 00:04:36.960
At least I think. I haven't tried it,

00:04:38.760 --> 00:04:39.020
but this is a technique that works in other

00:04:43.040 --> 00:04:43.260
similar cases. So you can also imagine like,

00:04:44.320 --> 00:04:44.500
oh, this whole thing I said,

00:04:47.860 --> 00:04:48.120
like, oh, you can query for vague things and

00:04:49.140 --> 00:04:49.600
get parts of the manual,

00:04:52.680 --> 00:04:53.000
perhaps. I'm not exactly sure if that would

00:04:55.120 --> 00:04:55.520
be useful, but maybe. Usually when I'm

00:04:57.040 --> 00:04:57.180
looking things up in the Emacs manual or

00:04:58.320 --> 00:04:58.780
Elist manual, I have something extremely

00:05:00.020 --> 00:05:00.300
specific and I kind of know where to look.

00:05:02.960 --> 00:05:03.080
But having other ways to get at this

00:05:04.000 --> 00:05:04.500
information is always good.

00:05:10.240 --> 00:05:10.740
[Speaker 1]: Right. Looks like they added a clarification

00:05:12.280 --> 00:05:12.720
if you would like to read that yourself,

00:05:14.180 --> 00:05:14.680
or would you like me to read it for you?

00:05:17.640 --> 00:05:18.140
[Speaker 0]: Yeah. Yes, OK. It says,

00:05:20.460 --> 00:05:20.740
I've never tried. Yeah,

00:05:21.500 --> 00:05:21.820
the question is like OK,

00:05:23.100 --> 00:05:23.240
there is a difference between the kind of

00:05:23.860 --> 00:05:24.360
thing as I just described.

00:05:26.200 --> 00:05:26.600
I have not tried the difference with the EMAX

00:05:31.560 --> 00:05:31.980
manual itself. It'd be interesting to see

00:05:33.700 --> 00:05:33.960
what this is, but I would expect like these

00:05:35.140 --> 00:05:35.600
techniques, the retrieval augmented

00:05:38.840 --> 00:05:39.340
generation is generally pretty good.

00:05:41.240 --> 00:05:41.740
And I suspect it would,

00:05:43.580 --> 00:05:43.780
I would bet money on the fact that it's gonna

00:05:45.820 --> 00:05:46.240
give you, you know, better results than just,

00:05:48.160 --> 00:05:48.360
you know, doing a free form query without any

00:05:49.440 --> 00:05:49.940
retrieval augmented generation.

00:05:54.240 --> 00:05:54.640
[Speaker 1]: Cool. Let's see. Next question.

00:05:56.380 --> 00:05:56.880
When deferring commit messages to an LLM,

00:05:59.700 --> 00:05:59.920
what, if anything, do you find you might have

00:06:02.940 --> 00:06:03.440
[Speaker 0]: lost? Yeah, it's a good question.

00:06:06.060 --> 00:06:06.560
When deferring anything to a computer,

00:06:08.860 --> 00:06:09.360
like, you know, I used to have to remember

00:06:11.200 --> 00:06:11.700
how to get places, and now,

00:06:14.540 --> 00:06:15.040
you know, on the few occasions which I drive,

00:06:16.560 --> 00:06:16.720
like, It could just tell me how to get

00:06:21.960 --> 00:06:22.280
places. So similar things could occur here

00:06:24.960 --> 00:06:25.460
where like, okay, I'm just leaving the LLM.

00:06:27.380 --> 00:06:27.680
And so I'm kind of missing out on some

00:06:30.040 --> 00:06:30.220
opportunity to think coherently about a

00:06:32.440 --> 00:06:32.680
particular commit. Particular commits are

00:06:36.140 --> 00:06:36.540
kind of low level. I don't think it's usually

00:06:39.340 --> 00:06:39.840
relatively obvious and what they're doing.

00:06:42.600 --> 00:06:42.800
And in this case, I think there's not much

00:06:44.220 --> 00:06:44.540
loss. But for sure, in other cases,

00:06:46.400 --> 00:06:46.900
if you're starting to get into situations

00:06:48.640 --> 00:06:48.800
where it's writing your emails and all this

00:06:52.920 --> 00:06:53.300
stuff. First of all, it's in 1 sense,

00:06:55.580 --> 00:06:56.040
I'm not sure you might be losing something by

00:06:57.520 --> 00:06:57.780
delegating things. On the other hand,

00:06:59.120 --> 00:06:59.280
you know, when you're interacting with these

00:07:01.280 --> 00:07:01.560
LLMs, you have to be extremely specific about

00:07:03.120 --> 00:07:03.240
what you want, or else it's just not going to

00:07:07.540 --> 00:07:07.680
do a good job. And that might actually be a

00:07:09.440 --> 00:07:09.860
good thing. So the question might be that

00:07:11.820 --> 00:07:12.240
maybe you might gain things by using an LLM

00:07:13.860 --> 00:07:14.060
to do your work. It might not actually even

00:07:15.060 --> 00:07:15.400
save you that much time,

00:07:18.480 --> 00:07:18.640
at least initially, because you have to kind

00:07:20.460 --> 00:07:20.660
of practice again super specific about what

00:07:22.740 --> 00:07:22.900
you want to get out of the output it's going

00:07:26.940 --> 00:07:26.980
to give you so like oh I'm you know maybe you

00:07:29.600 --> 00:07:29.820
know you're on the emacs devel mailing list

00:07:31.780 --> 00:07:31.980
and you're like okay write this email about

00:07:34.000 --> 00:07:34.140
this about this And here's what I want to

00:07:35.370 --> 00:07:35.460
say. And here's the kind of tone I want to

00:07:36.020 --> 00:07:36.420
use. And here's the like,

00:07:37.660 --> 00:07:38.160
oh, you might want to specify like everything

00:07:39.620 --> 00:07:40.120
that you kind of want to get into this.

00:07:42.180 --> 00:07:42.680
Usually it's easier just to write the email.

00:07:45.600 --> 00:07:46.100
But I think that practice of kind of

00:07:48.080 --> 00:07:48.420
understanding what you want is not something

00:07:52.680 --> 00:07:53.180
you normally do. And I think it's going to be

00:07:56.040 --> 00:07:56.480
an interesting exercise that will help people

00:07:57.280 --> 00:07:57.540
understand. That said,

00:07:58.860 --> 00:07:59.020
I haven't done that much of that,

00:07:59.900 --> 00:08:00.040
so I can't say, oh, yeah,

00:08:01.080 --> 00:08:01.440
I've done this and it works for me.

00:08:03.000 --> 00:08:03.120
Maybe. I think it's an interesting thing to

00:08:03.120 --> 00:08:03.620
explore.

00:08:07.720 --> 00:08:08.220
[Speaker 1]: Sure. Thanks. Let's see.

00:08:10.840 --> 00:08:11.140
Let's see. Can you share your font settings

00:08:13.440 --> 00:08:13.740
in your Emacs config? Those are some nice

00:08:14.200 --> 00:08:14.700
fonts for reading.

00:08:18.900 --> 00:08:19.200
[Speaker 0]: Yeah, I think I was using Menlo at the time.

00:08:20.840 --> 00:08:20.980
Unfortunately, I don't save those kinds of

00:08:21.940 --> 00:08:22.200
things, like a history of this.

00:08:24.000 --> 00:08:24.500
I've kind of switched now to,

00:08:27.340 --> 00:08:27.540
what was that? I think I wrote it down in

00:08:29.440 --> 00:08:29.940
the, I switched to MunaSpace,

00:08:31.920 --> 00:08:32.420
which just came out like a week or 2 ago,

00:08:33.340 --> 00:08:33.840
and is also pretty cool.

00:08:35.440 --> 00:08:35.940
So I think it's Menlo.

00:08:37.380 --> 00:08:37.760
The internal question,

00:08:38.400 --> 00:08:38.900
what font are you using?

00:08:42.020 --> 00:08:42.340
[Speaker 1]: Indeed, yeah. It looks like someone guessed

00:08:43.780 --> 00:08:44.280
as well that it might be Menlo.

00:08:47.680 --> 00:08:48.040
OK, Cool. Yeah, next question.

00:08:48.900 --> 00:08:49.400
In terms of standardization,

00:08:53.260 --> 00:08:53.520
do you see a need for the medium to large

00:08:55.840 --> 00:08:56.160
scale effort needed? And then they also

00:08:56.960 --> 00:08:57.460
elaborate about it.

00:09:03.600 --> 00:09:04.100
[Speaker 0]: Yeah, I mean, I do think,

00:09:06.040 --> 00:09:06.300
I don't know if it's large scale,

00:09:08.000 --> 00:09:08.500
but at least it's probably medium scale.

00:09:10.320 --> 00:09:10.520
There's a lot of things that are missing that

00:09:12.260 --> 00:09:12.400
we don't have right now in emacs when you're

00:09:13.660 --> 00:09:14.160
dealing with LLMs. 1 is,

00:09:18.240 --> 00:09:18.420
a prompting system. And by that,

00:09:21.820 --> 00:09:22.020
I mean, you know, prompts are just like big

00:09:24.520 --> 00:09:24.720
blocks of text, but there's also senses that

00:09:28.260 --> 00:09:28.420
like prompts need to be composable and you

00:09:30.480 --> 00:09:30.660
need to be able to iterate on parts of the

00:09:36.600 --> 00:09:37.100
prompt. And so it's also customizable.

00:09:38.940 --> 00:09:39.060
Users might want to customize it.

00:09:41.260 --> 00:09:41.360
On the other hand, it's not super easy to

00:09:43.820 --> 00:09:44.320
write the prompt. So you want really good

00:09:47.900 --> 00:09:48.040
defaults. So the whole prompt system is kind

00:09:51.360 --> 00:09:51.460
of complicated. That needs to be kind of

00:09:52.580 --> 00:09:52.760
standardized, because I don't think there's

00:09:54.380 --> 00:09:54.720
any tools for doing something like that right

00:09:58.380 --> 00:09:58.880
now. I personally use my system,

00:10:00.220 --> 00:10:00.600
my note system for EKG.

00:10:01.720 --> 00:10:01.920
I don't think that's appropriate for

00:10:02.800 --> 00:10:03.120
everyone, but it does,

00:10:04.480 --> 00:10:04.640
I did write it to have some of these

00:10:06.540 --> 00:10:06.760
capabilities of composability that I think

00:10:08.360 --> 00:10:08.860
are useful for a prompt generation.

00:10:11.940 --> 00:10:12.280
It'd be nice to have a system like that,

00:10:15.660 --> 00:10:16.160
but for general use. I don't,

00:10:17.840 --> 00:10:18.060
this is something I've been meaning to think

00:10:18.840 --> 00:10:19.000
about, like how to do it,

00:10:19.760 --> 00:10:19.920
but like this, you know,

00:10:21.260 --> 00:10:21.660
if someone's interested in getting this area,

00:10:26.120 --> 00:10:26.420
like, I would love to chat about that or,

00:10:27.600 --> 00:10:27.980
you know, I think there's a lot of

00:10:31.020 --> 00:10:31.260
interesting ideas that we could have to have

00:10:34.080 --> 00:10:34.540
a system that allows us to make progress

00:10:38.860 --> 00:10:39.360
here. And also, I think there's more to

00:10:40.520 --> 00:10:40.900
standardization to be done.

00:10:42.820 --> 00:10:43.140
1 thing I'd also like to see that we haven't

00:10:47.020 --> 00:10:47.220
done yet is a system for standardizing on

00:10:48.060 --> 00:10:48.560
getting structured output.

00:10:49.640 --> 00:10:50.140
This is gonna be super useful.

00:10:52.280 --> 00:10:52.780
I have this for open AIs API,

00:10:53.560 --> 00:10:54.060
cause they support it.

00:10:55.940 --> 00:10:56.040
And it's really nice, cause then you can

00:10:57.440 --> 00:10:57.660
write elist functions that like,

00:10:59.380 --> 00:10:59.880
okay, I'm going to call the LLM.

00:11:00.760 --> 00:11:01.000
I'm gonna get structured output.

00:11:02.040 --> 00:11:02.160
I know what that structure is going to be.

00:11:03.480 --> 00:11:03.680
It's not going to be just a big block of

00:11:05.660 --> 00:11:06.040
text. I could turn it into a,

00:11:07.000 --> 00:11:07.480
you know, a P list or something.

00:11:09.280 --> 00:11:09.480
And then I could get the values out of that P

00:11:11.880 --> 00:11:12.260
list. And I know that way I could do,

00:11:14.220 --> 00:11:14.720
I could write actual apps that are,

00:11:18.300 --> 00:11:18.720
you know, very, very sort of,

00:11:20.200 --> 00:11:20.680
you know, useful for very specific purposes

00:11:21.900 --> 00:11:22.400
and not just for text generation.

00:11:24.000 --> 00:11:24.320
And I think that's 1 of the most important

00:11:27.100 --> 00:11:27.540
things we want to do. And I have some ideas

00:11:28.840 --> 00:11:29.160
about how to do it. I just haven't pursued

00:11:31.640 --> 00:11:32.040
those yet. But if other people have ideas,

00:11:34.340 --> 00:11:34.540
I think this would be really interesting to

00:11:35.520 --> 00:11:36.020
add to the LLM package.

00:11:37.260 --> 00:11:37.760
So contact me there.

00:11:42.100 --> 00:11:42.600
[Speaker 1]: Awesome. Quick note before we continue.

00:11:44.440 --> 00:11:44.540
So I'm not sure how long we're going to be on

00:11:46.040 --> 00:11:46.280
stream for, because this is the last talk

00:11:48.640 --> 00:11:49.120
before the break. If we are on the stream

00:11:49.840 --> 00:11:50.200
long-term, then great.

00:11:51.820 --> 00:11:52.300
But if not, folks are welcome to continue

00:11:53.320 --> 00:11:53.680
writing questions on the pad.

00:11:55.140 --> 00:11:55.280
And hopefully, Andrew will get to them at

00:11:58.020 --> 00:11:58.280
some point. Or if Andrew maybe has some extra

00:11:59.960 --> 00:12:00.140
time available and wants to stay on

00:12:01.640 --> 00:12:01.920
BigBlueButton here, then folks are also

00:12:03.940 --> 00:12:04.340
welcome to join here and chat with Andrew

00:12:08.940 --> 00:12:09.240
directly as well. Okay,

00:12:10.740 --> 00:12:10.900
awesome. So yeah, the next question is,

00:12:12.040 --> 00:12:12.400
what are your thoughts on the carbon

00:12:14.060 --> 00:12:14.560
footprint of LLM usage?

00:12:17.200 --> 00:12:17.700
[Speaker 0]: Yeah, it's a really interesting question.

00:12:23.180 --> 00:12:23.360
I don't have any particular knowledge or

00:12:25.440 --> 00:12:25.580
opinions about that. It's something I think

00:12:26.980 --> 00:12:27.180
we should all be educating ourselves more

00:12:32.240 --> 00:12:32.380
about. It is really, I mean,

00:12:33.040 --> 00:12:33.220
there's 2 parts of this,

00:12:35.380 --> 00:12:35.500
right? They take a, there's a huge amount of

00:12:37.160 --> 00:12:37.360
carbon footprint involved in training these

00:12:38.720 --> 00:12:39.220
things. Then running them is relatively

00:12:42.540 --> 00:12:42.880
lightweight. So the question is not

00:12:44.440 --> 00:12:44.920
necessarily like once it's trained,

00:12:46.480 --> 00:12:46.640
like I don't feel like it's a big deal to

00:12:48.280 --> 00:12:48.560
keep using it, but like training these things

00:12:50.680 --> 00:12:51.180
is kind of like the big carbon cost of it.

00:12:53.680 --> 00:12:54.160
But like right now, the way everything's

00:12:56.040 --> 00:12:56.260
going, like every, you know,

00:12:59.060 --> 00:12:59.560
all, you know, the top 5 or 6 tech companies

00:13:00.900 --> 00:13:01.400
are all training their LLMs,

00:13:03.580 --> 00:13:03.740
and this is all costing a giant amount of

00:13:06.820 --> 00:13:07.060
carbon probably. On the other hand these same

00:13:08.560 --> 00:13:08.680
companies are pretty good about using the

00:13:10.260 --> 00:13:10.440
least amount of carbon necessary you know

00:13:12.340 --> 00:13:12.740
they have their own their tricks for doing

00:13:13.260 --> 00:13:13.760
things very efficiently.

00:13:22.100 --> 00:13:22.360
[Speaker 1]: Cool next question, LLMs are slow and

00:13:24.000 --> 00:13:24.340
responding. Do you think Emacs should provide

00:13:26.680 --> 00:13:27.180
more async primitives to keep it responsive?

00:13:29.380 --> 00:13:29.880
Like the URL retrieve is quite bad at

00:13:31.720 --> 00:13:31.760
building API clients with it.

00:13:31.920 --> 00:13:32.420
Building API clients with it?

00:13:36.400 --> 00:13:36.900
[Speaker 0]: Yeah. Well, OK, so first of all,

00:13:40.240 --> 00:13:40.740
people should be using the LLM client.

00:13:48.740 --> 00:13:48.900
And So right now, 1 thing I should have

00:13:50.220 --> 00:13:50.440
mentioned at the top is that there are new

00:13:52.500 --> 00:13:52.640
packages that I recorded this talk that you

00:13:54.480 --> 00:13:54.920
just saw several months ago.

00:13:57.780 --> 00:13:58.180
And so like Elama, there's this package Elama

00:13:59.700 --> 00:14:00.080
that came out that is using the LM package.

00:14:02.440 --> 00:14:02.680
And so for example, it doesn't need to worry

00:14:05.140 --> 00:14:05.580
about this sort of thing because it just uses

00:14:07.560 --> 00:14:07.920
LLM and package and the LLM package worries

00:14:11.680 --> 00:14:11.820
about this. And while I'm on the subject of

00:14:12.540 --> 00:14:12.720
things I forgot to mention,

00:14:15.140 --> 00:14:15.340
I also should just mention very quickly that

00:14:17.020 --> 00:14:17.520
there is now an open source model,

00:14:21.680 --> 00:14:21.960
Mistral. And so that's kind of this new thing

00:14:23.860 --> 00:14:24.240
on the scene that happened after I recorded

00:14:26.240 --> 00:14:26.420
my talk. And I think it's super important to

00:14:28.660 --> 00:14:28.820
the community and important that we have the

00:14:30.620 --> 00:14:31.120
opportunity to use that if we want to.

00:14:33.160 --> 00:14:33.660
Okay, but to answer the actual question,

00:14:37.660 --> 00:14:38.100
there has been some talk about the problems

00:14:40.680 --> 00:14:40.840
with URL retrieve in the URL package in

00:14:42.200 --> 00:14:42.700
general in EmacsDevEl.

00:14:46.760 --> 00:14:47.080
It's not great. I would like to have better

00:14:50.900 --> 00:14:51.040
primitives. And I've asked the author of

00:14:54.060 --> 00:14:54.560
Please PLZ to kind of provide some necessary

00:14:56.120 --> 00:14:56.620
callbacks. I think that's a great library.

00:15:00.280 --> 00:15:00.360
And I'd like to see that kind of like,

00:15:01.320 --> 00:15:01.680
It's nice that we have options,

00:15:03.340 --> 00:15:03.520
and that is an option that uses curl on the

00:15:05.140 --> 00:15:05.640
back end, and that has some benefits.

00:15:09.060 --> 00:15:09.280
So there's this big debate about whether we

00:15:10.600 --> 00:15:11.100
should have primitives or just use curl.

00:15:13.340 --> 00:15:13.420
I'm not exactly sure what the right call is,

00:15:15.320 --> 00:15:15.820
but there has been discussions about this.

00:15:19.540 --> 00:15:20.040
[Speaker 1]: Excellent. And someone commented that GPTEL

00:15:21.820 --> 00:15:22.200
is async and apparently very good at tracking

00:15:22.300 --> 00:15:22.800
the point.

00:15:26.680 --> 00:15:27.180
[Speaker 0]: Yes, yes, GPTEL has similar functionalities

00:15:29.800 --> 00:15:30.040
to LLM, although I believe it's going to move

00:15:33.040 --> 00:15:33.540
to LLM itself sometime soon.

00:15:39.480 --> 00:15:39.860
[Speaker 1]: Next question, speaking of which,

00:15:42.440 --> 00:15:42.560
anyone trained or fine-tuned or prompted a

00:15:44.680 --> 00:15:44.760
model with their org data yet and applied it

00:15:46.560 --> 00:15:47.040
to interesting use cases like planning,

00:15:47.920 --> 00:15:48.340
scheduling, et cetera,

00:15:49.320 --> 00:15:49.820
and maybe care to comment?

00:15:54.620 --> 00:15:55.120
[Speaker 0]: I don't know anyone who is doing that.

00:15:55.860 --> 00:15:56.360
I think it is interesting.

00:15:57.800 --> 00:15:58.300
Like this is what I kind of mentioned at the

00:16:01.060 --> 00:16:01.300
very end of the talk. There is a lot of stuff

00:16:02.440 --> 00:16:02.540
there like you could you know if you

00:16:04.760 --> 00:16:04.920
especially mean an LLM can kind of work as

00:16:07.940 --> 00:16:08.160
sort of like a secretary kind of person that

00:16:12.180 --> 00:16:12.440
could help you prioritize Still it's a

00:16:14.760 --> 00:16:14.920
slightly unclear how what the best way to use

00:16:16.480 --> 00:16:16.720
it is So I think there's more of a question

00:16:18.340 --> 00:16:18.480
for the community about like what people have

00:16:21.140 --> 00:16:21.320
been trying. I see someone has mentioned that

00:16:23.400 --> 00:16:23.900
they are using it for weekly review.

00:16:26.940 --> 00:16:27.180
And it's kind of nice to like,

00:16:29.060 --> 00:16:29.380
maybe you could read your agenda or maybe

00:16:30.480 --> 00:16:30.780
this for like weekly review.

00:16:32.040 --> 00:16:32.240
It could like read all the stuff you've done

00:16:33.340 --> 00:16:33.480
and ask you questions about it.

00:16:35.020 --> 00:16:35.280
And like, what should happen next?

00:16:36.520 --> 00:16:36.780
Or like, is this going to cause a problem?

00:16:39.060 --> 00:16:39.280
Like, I can, I can understand if that could

00:16:40.860 --> 00:16:41.180
happen? That's like, that's kind of nice.

00:16:43.660 --> 00:16:44.160
And this kind of people have had good success

00:16:48.540 --> 00:16:48.760
out of using these LLMs to bounce ideas off

00:16:49.920 --> 00:16:50.420
of are, you know, for,

00:16:52.680 --> 00:16:52.800
you know, I've seen people say that like they

00:16:55.360 --> 00:16:55.600
want, they use it for reading and they kind

00:16:58.520 --> 00:16:58.740
of dialogue with the LM to kind of like do

00:16:59.500 --> 00:17:00.000
sort of active reading.

00:17:02.500 --> 00:17:02.860
So you can imagine doing something similar

00:17:04.400 --> 00:17:04.740
with your tasks where it's sort of you're

00:17:06.560 --> 00:17:06.760
engaged in dialogue about like planning your

00:17:08.880 --> 00:17:09.000
tax with some with a alum that could kind of

00:17:10.800 --> 00:17:11.180
understand what those are and ask you some

00:17:13.780 --> 00:17:13.940
questions I think it. You know,

00:17:16.839 --> 00:17:17.040
if it'd be nice. So, the problem is like

00:17:18.480 --> 00:17:18.980
there's no great way to share all this stuff.

00:17:20.720 --> 00:17:21.220
I guess if you have something like this,

00:17:23.300 --> 00:17:23.720
put it on Reddit. If you don't have Reddit,

00:17:24.599 --> 00:17:24.880
I don't know what to do.

00:17:26.000 --> 00:17:26.500
I would say put it somewhere.

00:17:28.840 --> 00:17:29.020
At the very least, I could maybe open up like

00:17:31.320 --> 00:17:31.820
an LLM discussion session on the LLM package

00:17:34.000 --> 00:17:34.500
GitHub, But not everyone likes to use GitHub.

00:17:36.100 --> 00:17:36.180
I don't know. It'd be nice if there's a

00:17:38.940 --> 00:17:39.060
mailing list or IRC chat for this sort of

00:17:40.840 --> 00:17:41.340
thing. But there isn't at the moment.

00:17:46.560 --> 00:17:46.720
[Speaker 1]: All right. Let's see. I think that's the end

00:17:48.080 --> 00:17:48.580
of the questions on the pad so far.

00:17:51.020 --> 00:17:51.180
There was also some discussion or some

00:17:52.260 --> 00:17:52.760
chatter, I believe, on IRC.

00:17:54.560 --> 00:17:54.820
I'm not sure. Andrew, are you on IRC right

00:18:00.060 --> 00:18:00.260
[Speaker 0]: I am, but I don't think I'm on any place that

00:18:01.400 --> 00:18:01.640
has the chatter. So if there's chatter,

00:18:02.440 --> 00:18:02.940
then I'm not seeing it.

00:18:04.600 --> 00:18:05.100
[Speaker 1]: now? Okay. Yeah, it was in the emacsconf-dev

00:18:06.760 --> 00:18:07.260
channel.

00:18:09.600 --> 00:18:10.100
[Speaker 0]: Okay, let me see if I can.

00:18:25.600 --> 00:18:25.840
Oh, yes. I mean, I could see the channel,

00:18:27.520 --> 00:18:27.840
but I missed whatever came before.

00:18:29.340 --> 00:18:29.480
So if there's anything you want to kind of

00:18:30.840 --> 00:18:31.340
call out, I can try to answer it here.

00:18:35.320 --> 00:18:35.640
[Speaker 1]: OK, cool. I believe at least 2 other folks

00:18:37.500 --> 00:18:38.000
who are participating in the discussion there

00:18:40.120 --> 00:18:40.620
who have also joined here on BigBlueButton,

00:18:42.440 --> 00:18:42.940
Codin Quark and AeonTurn92.

00:18:47.000 --> 00:18:47.480
So you folks, if Andrew is still available

00:18:50.460 --> 00:18:50.640
and has time, you're welcome to chat here and

00:18:53.000 --> 00:18:53.320
ask questions or discuss here as well.

00:18:55.580 --> 00:18:55.840
[Speaker 0]: 1 Thank you. Thank you for your help,

00:18:57.740 --> 00:18:58.080
and thank you for reading all the questions.

00:18:59.700 --> 00:18:59.820
[Speaker 1]: AUDIENCE 2 Cheers, and thanks to you for a

00:19:00.540 --> 00:19:01.040
great talk and the discussion.

00:19:01.880 --> 00:19:02.380
[Speaker 0]: AUDIENCE AUDIENCE 1 Thank you.

00:19:03.140 --> 00:19:03.640
[Speaker 1]: AUDIENCE 2 Cheers.

00:19:07.900 --> 00:19:08.040
[Speaker 0]: So I'll just, I will wait here and see if

00:19:08.320 --> 00:19:08.760
there's any questions.

00:19:10.760 --> 00:19:11.260
If not, I will log off after a few minutes.

00:19:15.900 --> 00:19:16.080
[Speaker 2]: Well, I guess since we were mentioned that

00:19:18.480 --> 00:19:18.980
there was a small chat about local alarms.

00:19:22.640 --> 00:19:23.000
Because chat dpt is nice,

00:19:25.600 --> 00:19:26.100
no, but privacy concerns,

00:19:27.380 --> 00:19:27.880
and it's not free and stuff.

00:19:31.000 --> 00:19:31.500
Which, so The question is,

00:19:36.960 --> 00:19:37.460
what is the promise for local models?

00:19:39.660 --> 00:19:40.160
[Speaker 0]: Yeah, so local is definitely...

00:19:41.380 --> 00:19:41.880
[Speaker 2]: Or at least open source.

00:19:45.680 --> 00:19:46.120
[Speaker 0]: Yeah, so there is a local open source model,

00:19:47.960 --> 00:19:48.460
Misral, which you could run.

00:19:51.340 --> 00:19:51.840
The LLM package allows you to use,

00:19:56.120 --> 00:19:56.260
I think there's 3 kind of local things you

00:19:58.100 --> 00:19:58.440
could use. Like many of these things,

00:20:00.220 --> 00:20:00.480
there's like many kind of ways to do the same

00:20:03.960 --> 00:20:04.460
sort of thing. So LLM is supporting OLAMMA

00:20:10.240 --> 00:20:10.520
and LLAMMA-CPP. And let's see,

00:20:12.240 --> 00:20:12.740
1 other. Which 1 is it?

00:20:18.420 --> 00:20:18.700
And maybe that's it. Maybe the,

00:20:21.820 --> 00:20:21.940
oh, GPT for all. So each 1 of these kind of

00:20:23.100 --> 00:20:23.600
has slightly different functionality.

00:20:26.820 --> 00:20:27.180
For example, I think GPT for all doesn't

00:20:31.780 --> 00:20:32.280
support embeddings. And I hear that Olama's

00:20:33.740 --> 00:20:34.240
embeddings are kind of currently broken.

00:20:35.920 --> 00:20:36.420
But basically they should support everything.

00:20:39.100 --> 00:20:39.600
And the open source models are,

00:20:43.180 --> 00:20:43.380
so the local models are reasonably good.

00:20:44.760 --> 00:20:44.900
Like I don't think you'd use them and be

00:20:46.200 --> 00:20:46.639
like, what is this horrible nonsense?

00:20:50.200 --> 00:20:50.380
Like it's, it gives you relatively good

00:20:51.820 --> 00:20:52.120
results. Like it's not gonna be at the level

00:20:56.060 --> 00:20:56.320
of like GPT 3.5 or 4, but it's not far away

00:20:57.720 --> 00:20:58.220
from GPT 3.5, I think.

00:21:02.380 --> 00:21:02.880
[Speaker 2]: I'm just saying that Olam has like a presets

00:21:05.940 --> 00:21:06.300
for connecting the actual working servers for

00:21:06.300 --> 00:21:06.800
Olama?

00:21:08.560 --> 00:21:08.760
[Speaker 0]: So, I'll try. Yeah, so you could,

00:21:09.860 --> 00:21:10.040
what you could do is you could like for

00:21:11.940 --> 00:21:12.100
example you could download Olama which is

00:21:15.780 --> 00:21:15.940
just a way of setting up local models and

00:21:17.320 --> 00:21:17.780
running local models on your machine.

00:21:18.580 --> 00:21:18.820
So typically what it does,

00:21:19.720 --> 00:21:20.020
you like download a program,

00:21:23.720 --> 00:21:23.800
let's say Olama. Then Olama will have the

00:21:24.940 --> 00:21:25.440
ability to download models.

00:21:27.240 --> 00:21:27.360
And so you could choose from just a host of

00:21:29.280 --> 00:21:29.440
different models. Each 1 of these things has

00:21:30.200 --> 00:21:30.440
a bunch of different models.

00:21:31.920 --> 00:21:32.080
So it downloads all these things to your

00:21:36.600 --> 00:21:37.020
machine. But I would say that the key problem

00:21:40.200 --> 00:21:40.580
here is that it requires a fairly beefy

00:21:40.580 --> 00:21:41.080
machine.

00:21:42.600 --> 00:21:43.100
[Speaker 2]: So. Yeah, yeah, of course.

00:21:45.060 --> 00:21:45.300
Why I was asking, because you briefly

00:21:46.440 --> 00:21:46.880
mentioned that there are some Israeli

00:21:52.300 --> 00:21:52.440
servers. I understand that they run it like a

00:21:53.680 --> 00:21:54.000
government or stuff like that?

00:21:55.440 --> 00:21:55.940
No, no, sorry. People want everyone?

00:21:59.340 --> 00:21:59.620
[Speaker 0]: I don't, I mean, maybe you've said something

00:22:00.620 --> 00:22:01.020
that sounded like Israeli servers.

00:22:01.620 --> 00:22:02.120
[Speaker 2]: Okay, okay.

00:22:04.920 --> 00:22:05.080
[Speaker 0]: I think- There's no government LLMs as far as

00:22:06.820 --> 00:22:07.280
I know. Although, I'm sure the governments

00:22:08.200 --> 00:22:08.700
are working on their own LLMs,

00:22:10.980 --> 00:22:11.480
et cetera. But yeah, basically your choices

00:22:15.060 --> 00:22:15.220
are spend a, I mean, if you use open AI or

00:22:16.080 --> 00:22:16.580
something or anything else,

00:22:17.960 --> 00:22:18.460
you're really not spending any money.

00:22:20.560 --> 00:22:20.840
Like I've never been able to spend any money

00:22:23.680 --> 00:22:24.020
on OpenAI. Like unless you're doing something

00:22:25.840 --> 00:22:26.280
very intensive and really are using it to,

00:22:28.000 --> 00:22:28.180
you know, if you're using it for your

00:22:29.620 --> 00:22:29.780
personal use, it's just hard to spend any

00:22:31.720 --> 00:22:31.960
money. But on the other hand,

00:22:32.780 --> 00:22:32.860
it's not free. So you can,

00:22:33.040 --> 00:22:33.540
you know,

00:22:36.300 --> 00:22:36.680
[Speaker 2]: Actually, it's rather cheap.

00:22:37.680 --> 00:22:38.180
There's no question about that.

00:22:40.580 --> 00:22:40.920
The problem is that it has a bad track record

00:22:41.580 --> 00:22:42.080
on privacy.

00:22:45.540 --> 00:22:46.040
[Speaker 0]: Yes, that's, I think that is a key problem.

00:22:48.120 --> 00:22:48.280
This is probably the number 1 reason why you

00:22:51.840 --> 00:22:52.340
might want to use a local AI,

00:22:54.720 --> 00:22:55.220
a local LLM. Another 1 is like,

00:22:57.400 --> 00:22:57.900
you may not agree with the decisions.

00:23:00.360 --> 00:23:00.820
You know, there's a lot of trust and safety

00:23:05.140 --> 00:23:05.440
stuff that these companies have to do.

00:23:09.020 --> 00:23:09.240
Like they don't want like the LMs to kind of

00:23:11.400 --> 00:23:11.640
like give you, like tell you how you can make

00:23:13.180 --> 00:23:13.580
meth or how you can make a bomb,

00:23:14.960 --> 00:23:15.460
which they would do. They would totally do

00:23:19.580 --> 00:23:20.080
it. So, But each time you kind of restrict

00:23:22.540 --> 00:23:22.680
what is happening with what you can get out

00:23:23.860 --> 00:23:24.360
of the LM, it gets a little worse.

00:23:24.960 --> 00:23:25.440
So some people

00:23:27.040 --> 00:23:27.540
[Speaker 2]: want to have local. That's expected.

00:23:31.080 --> 00:23:31.400
I guess even open source language modules

00:23:33.540 --> 00:23:34.040
will soon have HR spaces because it's simply

00:23:34.720 --> 00:23:35.220
a legal issue.

00:23:40.760 --> 00:23:41.140
[Speaker 0]: I think that's true. I also think that there

00:23:42.880 --> 00:23:43.040
probably will be, although I don't know of

00:23:45.060 --> 00:23:45.300
any offhand, that will are completely

00:23:46.960 --> 00:23:47.200
uncensored. I know people are interested and

00:23:48.240 --> 00:23:48.480
are running uncensored models.

00:23:49.440 --> 00:23:49.940
I don't know how to do it.

00:23:52.280 --> 00:23:52.780
I think it's a little bit dubious,

00:23:54.960 --> 00:23:55.040
but some people do want to do it.

00:23:56.280 --> 00:23:56.780
There's another reason for using local

00:24:02.280 --> 00:24:02.780
servers. Do you have any recommendation for

00:24:05.500 --> 00:24:05.720
models to run locally and also comments on

00:24:06.780 --> 00:24:07.280
whether a GPU is required?

00:24:14.040 --> 00:24:14.160
Usually a GPU, well, you can run it without a

00:24:16.960 --> 00:24:17.460
GPU, but it does run much better.

00:24:19.480 --> 00:24:19.980
Like for example, I think when I used,

00:24:22.560 --> 00:24:23.060
Lama is sort of like a standard.

00:24:27.160 --> 00:24:27.320
This was the model for that Facebook came out

00:24:31.380 --> 00:24:31.880
with for local use. And It was,

00:24:37.260 --> 00:24:37.760
yeah, it's good. It's,

00:24:40.400 --> 00:24:40.900
but it's now it's I think,

00:24:44.620 --> 00:24:44.920
Mistral is kind of like has a better

00:24:46.480 --> 00:24:46.800
performance, But there's also different model

00:24:51.000 --> 00:24:51.500
sizes. There's 7B, like the Lama 7B is OK.

00:24:52.940 --> 00:24:53.440
The Mistral 7B, 7 billion,

00:24:54.800 --> 00:24:55.300
are like, basically it'll take like,

00:24:58.380 --> 00:24:58.880
you can run it with like 16 gigs of RAM,

00:25:02.720 --> 00:25:03.040
is pretty good. It's probably about as equal

00:25:06.900 --> 00:25:07.000
to the LLAMA13B. Those are the number of

00:25:08.360 --> 00:25:08.860
parameters, if I remember correctly.

00:25:10.680 --> 00:25:11.180
And then there's a 7B,

00:25:12.340 --> 00:25:12.840
which I've never been able to run.

00:25:16.120 --> 00:25:16.620
And even if the 7B, if you run it without a

00:25:19.640 --> 00:25:20.140
GPU, it takes quite a while to answer.

00:25:22.080 --> 00:25:22.580
I think I've had experiences where it took

00:25:23.940 --> 00:25:24.440
literally like several,

00:25:26.480 --> 00:25:26.780
like 5 minutes before it even started

00:25:28.880 --> 00:25:29.100
responding, but you do eventually get

00:25:32.220 --> 00:25:32.580
something. And it could be that like things

00:25:33.840 --> 00:25:33.960
have gotten better since the last time I

00:25:35.440 --> 00:25:35.940
tried this, because things are moving fast.

00:25:38.360 --> 00:25:38.860
But it is super recommended to have a GPU.

00:25:42.440 --> 00:25:42.620
This is the problem. It's kind of like,

00:25:43.840 --> 00:25:44.180
yes, free software is great.

00:25:46.880 --> 00:25:47.120
But if free software is requiring that you

00:25:50.460 --> 00:25:50.760
have these kind of beefy servers and have all

00:25:52.000 --> 00:25:52.500
this hardware, that's not great.

00:25:53.600 --> 00:25:54.100
I think there's a case to be made.

00:25:55.680 --> 00:25:56.180
[Speaker 1]: a hardware

00:25:59.040 --> 00:25:59.540
[Speaker 0]: with slots instead of a laptop.

00:26:01.560 --> 00:26:02.060
Yeah, yeah, that's right.

00:26:03.660 --> 00:26:03.960
[Speaker 2]: Ideally, you can have Ideally,

00:26:07.400 --> 00:26:07.660
it would be nice if FSL for all things could

00:26:12.040 --> 00:26:12.540
run something for open source model.

00:26:16.320 --> 00:26:16.640
And not free, but the key point is that it's

00:26:16.640 --> 00:26:17.140
Libre?

00:26:22.580 --> 00:26:23.080
[Speaker 0]: Yes, so actually I think Google does do that.

00:26:24.720 --> 00:26:24.800
I'll have to look it up,

00:26:27.560 --> 00:26:27.820
but I haven't explored this yet.

00:26:31.220 --> 00:26:31.720
But Google's server, which LLM does support,

00:26:33.800 --> 00:26:34.300
supports arbitrary models.

00:26:36.420 --> 00:26:36.600
So you can run LLMA or things like that.

00:26:38.940 --> 00:26:39.200
The problem is that even if you're running

00:26:40.320 --> 00:26:40.820
Mistral, which has no restrictions.

00:26:42.940 --> 00:26:43.140
So this is the kind of thing that like the

00:26:44.900 --> 00:26:45.060
Free Software Foundation cares a lot about.

00:26:47.240 --> 00:26:47.740
Like you want it to be like no restrictions,

00:26:49.740 --> 00:26:49.840
legal restrictions on you as you run the

00:26:52.080 --> 00:26:52.580
model. So even if it's running Mistral,

00:26:54.800 --> 00:26:55.300
just by using the server,

00:26:58.460 --> 00:26:58.660
the company server, it will impose some

00:26:59.440 --> 00:26:59.900
restrictions on you probably,

00:27:02.320 --> 00:27:02.480
right? There's gonna be some license that you

00:27:04.760 --> 00:27:05.260
have to, or something you have to abide by.

00:27:08.480 --> 00:27:08.600
So I think, yes, it depends on how much you

00:27:09.280 --> 00:27:09.780
care about it, I guess.

00:27:19.500 --> 00:27:19.640
I should find out more about that and make

00:27:21.580 --> 00:27:22.080
sure that it's a good point that I should,

00:27:23.980 --> 00:27:24.180
you know, people should be able to run free

00:27:25.920 --> 00:27:26.280
models over the server.

00:27:28.320 --> 00:27:28.440
So I should make sure we support that in the

00:27:40.360 --> 00:27:40.860
LLM package. So, is there any other questions

00:27:48.240 --> 00:27:48.740
Or is otherwise we can end the session.

00:28:00.800 --> 00:28:01.040
Yeah, all right. Thank you.

00:28:02.440 --> 00:28:02.940
Thank you. Thank you everyone who listened.

00:28:04.540 --> 00:28:05.040
I'm super happy like I,

00:28:06.560 --> 00:28:07.060
the interest is great.

00:28:08.900 --> 00:28:09.220
I think there's great stuff to be done here

00:28:10.960 --> 00:28:11.140
and I'm kind of super excited what we're

00:28:11.940 --> 00:28:12.160
going to do in the next year,

00:28:13.140 --> 00:28:13.440
so hopefully, like next year,

00:28:14.600 --> 00:28:14.760
and the conference we have something even

00:28:16.440 --> 00:28:16.560
more exciting to say about LLM and how they

00:28:17.320 --> 00:28:17.820
can be used with Emacs.

00:28:19.620 --> 00:28:20.120
So thank

00:28:30.060 --> 00:28:30.560
you