WEBVTT
00:00:13.099 --> 00:00:13.599
[Speaker 0]: Okay. Hello, everyone.
00:00:16.560 --> 00:00:17.060
I think this is the start of the Q&A session.
00:00:25.119 --> 00:00:25.599
So people can just ask me questions here.
00:00:28.259 --> 00:00:28.380
Or I think maybe these questions are going to
00:00:30.560 --> 00:00:30.980
be read by someone. Yes,
00:00:34.680 --> 00:00:35.060
thank you. Should I start doing that?
00:00:39.280 --> 00:00:39.400
I also know that there's questions in the
00:00:41.320 --> 00:00:41.480
either pad room, so I could start out
00:00:42.280 --> 00:00:42.780
answering those as well.
00:00:45.020 --> 00:00:45.300
[Speaker 1]: Right, sure. Whichever way you prefer.
00:00:46.860 --> 00:00:47.220
If you prefer to read the questions yourself,
00:00:48.940 --> 00:00:49.080
by all means, or if you would prefer me to
00:00:50.080 --> 00:00:50.280
read them to you, that also works.
00:00:50.580 --> 00:00:50.920
[Speaker 2]: Oh, I see.
00:00:51.600 --> 00:00:51.760
[Speaker 0]: Why don't you read them to me?
00:00:53.260 --> 00:00:53.760
I think it'll just be more interesting then.
00:00:56.820 --> 00:00:57.160
[Speaker 1]: Sure. OK, let's see. The first question is,
00:00:58.360 --> 00:00:58.860
what is your use case for embedding,
00:01:00.060 --> 00:01:00.560
mainly for searching?
00:01:06.180 --> 00:01:06.340
[Speaker 0]: Yeah, I mean, I think the use case really is
00:01:12.100 --> 00:01:12.320
searching. And I think it is very useful when
00:01:15.060 --> 00:01:15.320
you're searching for something in a vague
00:01:18.280 --> 00:01:18.780
way. Just to give you an example,
00:01:23.860 --> 00:01:24.360
I have a note system called EKG.
00:01:25.760 --> 00:01:26.260
I type all my notes on it.
00:01:28.620 --> 00:01:29.120
You can find it on GitHub and Melba.
00:01:34.140 --> 00:01:34.400
But I wrote something at some point a year
00:01:35.840 --> 00:01:36.020
ago or something. I wrote something that I
00:01:36.600 --> 00:01:36.980
just vaguely remembered.
00:01:38.800 --> 00:01:38.940
Oh, this was about a certain kind of
00:01:41.580 --> 00:01:41.760
communication. I wanted communicating to
00:01:43.280 --> 00:01:43.479
large audiences. There's some interesting tip
00:01:44.700 --> 00:01:45.060
that I wrote down that was really cool.
00:01:49.920 --> 00:01:50.080
And I was like, well, I need to find it.
00:01:52.260 --> 00:01:52.640
So I did an embedding search for something
00:01:55.479 --> 00:01:55.979
like, you know, tips for communicating.
00:01:58.979 --> 00:01:59.100
Like those words may not have been in what I
00:02:00.020 --> 00:02:00.520
was trying to find at all,
00:02:02.680 --> 00:02:03.180
But it was able to find it.
00:02:05.840 --> 00:02:06.260
And that is something that's very hard to do
00:02:07.200 --> 00:02:07.360
in other ways. Like, you know,
00:02:08.520 --> 00:02:08.720
if you had to do this with normal search,
00:02:09.199 --> 00:02:09.660
you have to do synonyms.
00:02:10.940 --> 00:02:11.200
And like maybe those synonyms wouldn't cover
00:02:11.960 --> 00:02:12.340
it. Like with embedding,
00:02:13.940 --> 00:02:14.160
you can basically get at like the vague
00:02:14.960 --> 00:02:15.460
sentiment. You're like,
00:02:17.320 --> 00:02:17.560
you know, you're, you know,
00:02:19.520 --> 00:02:19.700
you can really query on like what things are
00:02:21.760 --> 00:02:22.260
about as opposed to what words they have.
00:02:25.600 --> 00:02:26.100
Also, it's super good for similarity search.
00:02:27.720 --> 00:02:27.900
So you could say, look,
00:02:30.040 --> 00:02:30.240
I have a bunch of things that are encoded
00:02:31.400 --> 00:02:31.900
with embeddings that I want to show.
00:02:34.120 --> 00:02:34.280
For example, you can make an embedding for
00:02:35.220 --> 00:02:35.380
every buffer. You'd be like,
00:02:37.060 --> 00:02:37.200
well, show me buffers that are similar to
00:02:38.740 --> 00:02:38.980
this buffer. That doesn't sound super useful,
00:02:40.440 --> 00:02:40.940
but this is the kind of thing you could do.
00:02:45.300 --> 00:02:45.480
And so if you have a bunch of notes or
00:02:46.720 --> 00:02:46.920
something else that you want to search on,
00:02:48.240 --> 00:02:48.740
you'd be like, what's similar to this buffer?
00:02:51.500 --> 00:02:51.760
Or what notes are similar to each other?
00:02:53.040 --> 00:02:53.540
What buffers are similar to each other?
00:02:55.380 --> 00:02:55.880
It's super good for this sort of thing.
00:03:00.780 --> 00:03:01.280
And it's also good for this kind of retrieval
00:03:03.080 --> 00:03:03.520
augmented generation, where you sort of,
00:03:05.080 --> 00:03:05.280
you retrieve things and the purpose is not
00:03:06.600 --> 00:03:06.880
for you to see them, but then you pass that
00:03:12.040 --> 00:03:12.180
to the LLM. And then it's able to be a little
00:03:14.340 --> 00:03:14.800
bit more accurate because it has the actual
00:03:15.760 --> 00:03:16.260
text that you're trying to,
00:03:18.960 --> 00:03:19.180
that is relevant, and it can cite from and
00:03:20.720 --> 00:03:20.820
things like that. And then it could give you
00:03:22.260 --> 00:03:22.660
a much better answer that's kind of,
00:03:25.520 --> 00:03:25.680
you know, not just from its own little neural
00:03:26.320 --> 00:03:26.820
nets and memory.
00:03:31.920 --> 00:03:32.120
[Speaker 1]: Cool, thanks. Let's see,
00:03:35.740 --> 00:03:36.100
next question. What do you think about embed
00:03:40.160 --> 00:03:40.660
Emacs manual versus GPT's Emacs manual?
00:03:45.480 --> 00:03:45.660
[Speaker 0]: I'm not exactly sure what this question is
00:03:46.980 --> 00:03:47.480
trying to say. So I mean,
00:03:51.000 --> 00:03:51.140
if someone wrote that and wants to expand on
00:03:55.080 --> 00:03:55.320
it a little bit, but I think that maybe
00:03:58.420 --> 00:03:58.920
you're saying like you could embed,
00:04:00.280 --> 00:04:00.780
have embeddings for like various,
00:04:02.520 --> 00:04:02.720
like every paragraph or something of the
00:04:04.540 --> 00:04:04.680
Emacs manual. But it's also the case that
00:04:06.500 --> 00:04:07.000
like GPT is already for sure already read it,
00:04:09.760 --> 00:04:09.960
right? And so you could ask questions that
00:04:13.460 --> 00:04:13.780
are about Emacs and our ELISP or whatever
00:04:15.200 --> 00:04:15.700
part of the manual you want to find.
00:04:19.760 --> 00:04:20.240
And it will do a reasonably good job,
00:04:22.280 --> 00:04:22.780
especially the better models will do a
00:04:24.620 --> 00:04:24.940
reasonably good job of saying you something
00:04:26.040 --> 00:04:26.540
that is vaguely accurate.
00:04:29.440 --> 00:04:29.860
But if you do this retrieval augmented
00:04:30.580 --> 00:04:31.080
generation with embeddings,
00:04:32.640 --> 00:04:33.140
you can get something that is very accurate.
00:04:36.700 --> 00:04:36.960
At least I think. I haven't tried it,
00:04:38.760 --> 00:04:39.020
but this is a technique that works in other
00:04:43.040 --> 00:04:43.260
similar cases. So you can also imagine like,
00:04:44.320 --> 00:04:44.500
oh, this whole thing I said,
00:04:47.860 --> 00:04:48.120
like, oh, you can query for vague things and
00:04:49.140 --> 00:04:49.600
get parts of the manual,
00:04:52.680 --> 00:04:53.000
perhaps. I'm not exactly sure if that would
00:04:55.120 --> 00:04:55.520
be useful, but maybe. Usually when I'm
00:04:57.040 --> 00:04:57.180
looking things up in the Emacs manual or
00:04:58.320 --> 00:04:58.780
Elist manual, I have something extremely
00:05:00.020 --> 00:05:00.300
specific and I kind of know where to look.
00:05:02.960 --> 00:05:03.080
But having other ways to get at this
00:05:04.000 --> 00:05:04.500
information is always good.
00:05:10.240 --> 00:05:10.740
[Speaker 1]: Right. Looks like they added a clarification
00:05:12.280 --> 00:05:12.720
if you would like to read that yourself,
00:05:14.180 --> 00:05:14.680
or would you like me to read it for you?
00:05:17.640 --> 00:05:18.140
[Speaker 0]: Yeah. Yes, OK. It says,
00:05:20.460 --> 00:05:20.740
I've never tried. Yeah,
00:05:21.500 --> 00:05:21.820
the question is like OK,
00:05:23.100 --> 00:05:23.240
there is a difference between the kind of
00:05:23.860 --> 00:05:24.360
thing as I just described.
00:05:26.200 --> 00:05:26.600
I have not tried the difference with the EMAX
00:05:31.560 --> 00:05:31.980
manual itself. It'd be interesting to see
00:05:33.700 --> 00:05:33.960
what this is, but I would expect like these
00:05:35.140 --> 00:05:35.600
techniques, the retrieval augmented
00:05:38.840 --> 00:05:39.340
generation is generally pretty good.
00:05:41.240 --> 00:05:41.740
And I suspect it would,
00:05:43.580 --> 00:05:43.780
I would bet money on the fact that it's gonna
00:05:45.820 --> 00:05:46.240
give you, you know, better results than just,
00:05:48.160 --> 00:05:48.360
you know, doing a free form query without any
00:05:49.440 --> 00:05:49.940
retrieval augmented generation.
00:05:54.240 --> 00:05:54.640
[Speaker 1]: Cool. Let's see. Next question.
00:05:56.380 --> 00:05:56.880
When deferring commit messages to an LLM,
00:05:59.700 --> 00:05:59.920
what, if anything, do you find you might have
00:06:02.940 --> 00:06:03.440
[Speaker 0]: lost? Yeah, it's a good question.
00:06:06.060 --> 00:06:06.560
When deferring anything to a computer,
00:06:08.860 --> 00:06:09.360
like, you know, I used to have to remember
00:06:11.200 --> 00:06:11.700
how to get places, and now,
00:06:14.540 --> 00:06:15.040
you know, on the few occasions which I drive,
00:06:16.560 --> 00:06:16.720
like, It could just tell me how to get
00:06:21.960 --> 00:06:22.280
places. So similar things could occur here
00:06:24.960 --> 00:06:25.460
where like, okay, I'm just leaving the LLM.
00:06:27.380 --> 00:06:27.680
And so I'm kind of missing out on some
00:06:30.040 --> 00:06:30.220
opportunity to think coherently about a
00:06:32.440 --> 00:06:32.680
particular commit. Particular commits are
00:06:36.140 --> 00:06:36.540
kind of low level. I don't think it's usually
00:06:39.340 --> 00:06:39.840
relatively obvious and what they're doing.
00:06:42.600 --> 00:06:42.800
And in this case, I think there's not much
00:06:44.220 --> 00:06:44.540
loss. But for sure, in other cases,
00:06:46.400 --> 00:06:46.900
if you're starting to get into situations
00:06:48.640 --> 00:06:48.800
where it's writing your emails and all this
00:06:52.920 --> 00:06:53.300
stuff. First of all, it's in 1 sense,
00:06:55.580 --> 00:06:56.040
I'm not sure you might be losing something by
00:06:57.520 --> 00:06:57.780
delegating things. On the other hand,
00:06:59.120 --> 00:06:59.280
you know, when you're interacting with these
00:07:01.280 --> 00:07:01.560
LLMs, you have to be extremely specific about
00:07:03.120 --> 00:07:03.240
what you want, or else it's just not going to
00:07:07.540 --> 00:07:07.680
do a good job. And that might actually be a
00:07:09.440 --> 00:07:09.860
good thing. So the question might be that
00:07:11.820 --> 00:07:12.240
maybe you might gain things by using an LLM
00:07:13.860 --> 00:07:14.060
to do your work. It might not actually even
00:07:15.060 --> 00:07:15.400
save you that much time,
00:07:18.480 --> 00:07:18.640
at least initially, because you have to kind
00:07:20.460 --> 00:07:20.660
of practice again super specific about what
00:07:22.740 --> 00:07:22.900
you want to get out of the output it's going
00:07:26.940 --> 00:07:26.980
to give you so like oh I'm you know maybe you
00:07:29.600 --> 00:07:29.820
know you're on the emacs devel mailing list
00:07:31.780 --> 00:07:31.980
and you're like okay write this email about
00:07:34.000 --> 00:07:34.140
this about this And here's what I want to
00:07:35.370 --> 00:07:35.460
say. And here's the kind of tone I want to
00:07:36.020 --> 00:07:36.420
use. And here's the like,
00:07:37.660 --> 00:07:38.160
oh, you might want to specify like everything
00:07:39.620 --> 00:07:40.120
that you kind of want to get into this.
00:07:42.180 --> 00:07:42.680
Usually it's easier just to write the email.
00:07:45.600 --> 00:07:46.100
But I think that practice of kind of
00:07:48.080 --> 00:07:48.420
understanding what you want is not something
00:07:52.680 --> 00:07:53.180
you normally do. And I think it's going to be
00:07:56.040 --> 00:07:56.480
an interesting exercise that will help people
00:07:57.280 --> 00:07:57.540
understand. That said,
00:07:58.860 --> 00:07:59.020
I haven't done that much of that,
00:07:59.900 --> 00:08:00.040
so I can't say, oh, yeah,
00:08:01.080 --> 00:08:01.440
I've done this and it works for me.
00:08:03.000 --> 00:08:03.120
Maybe. I think it's an interesting thing to
00:08:03.120 --> 00:08:03.620
explore.
00:08:07.720 --> 00:08:08.220
[Speaker 1]: Sure. Thanks. Let's see.
00:08:10.840 --> 00:08:11.140
Let's see. Can you share your font settings
00:08:13.440 --> 00:08:13.740
in your Emacs config? Those are some nice
00:08:14.200 --> 00:08:14.700
fonts for reading.
00:08:18.900 --> 00:08:19.200
[Speaker 0]: Yeah, I think I was using Menlo at the time.
00:08:20.840 --> 00:08:20.980
Unfortunately, I don't save those kinds of
00:08:21.940 --> 00:08:22.200
things, like a history of this.
00:08:24.000 --> 00:08:24.500
I've kind of switched now to,
00:08:27.340 --> 00:08:27.540
what was that? I think I wrote it down in
00:08:29.440 --> 00:08:29.940
the, I switched to MunaSpace,
00:08:31.920 --> 00:08:32.420
which just came out like a week or 2 ago,
00:08:33.340 --> 00:08:33.840
and is also pretty cool.
00:08:35.440 --> 00:08:35.940
So I think it's Menlo.
00:08:37.380 --> 00:08:37.760
The internal question,
00:08:38.400 --> 00:08:38.900
what font are you using?
00:08:42.020 --> 00:08:42.340
[Speaker 1]: Indeed, yeah. It looks like someone guessed
00:08:43.780 --> 00:08:44.280
as well that it might be Menlo.
00:08:47.680 --> 00:08:48.040
OK, Cool. Yeah, next question.
00:08:48.900 --> 00:08:49.400
In terms of standardization,
00:08:53.260 --> 00:08:53.520
do you see a need for the medium to large
00:08:55.840 --> 00:08:56.160
scale effort needed? And then they also
00:08:56.960 --> 00:08:57.460
elaborate about it.
00:09:03.600 --> 00:09:04.100
[Speaker 0]: Yeah, I mean, I do think,
00:09:06.040 --> 00:09:06.300
I don't know if it's large scale,
00:09:08.000 --> 00:09:08.500
but at least it's probably medium scale.
00:09:10.320 --> 00:09:10.520
There's a lot of things that are missing that
00:09:12.260 --> 00:09:12.400
we don't have right now in emacs when you're
00:09:13.660 --> 00:09:14.160
dealing with LLMs. 1 is,
00:09:18.240 --> 00:09:18.420
a prompting system. And by that,
00:09:21.820 --> 00:09:22.020
I mean, you know, prompts are just like big
00:09:24.520 --> 00:09:24.720
blocks of text, but there's also senses that
00:09:28.260 --> 00:09:28.420
like prompts need to be composable and you
00:09:30.480 --> 00:09:30.660
need to be able to iterate on parts of the
00:09:36.600 --> 00:09:37.100
prompt. And so it's also customizable.
00:09:38.940 --> 00:09:39.060
Users might want to customize it.
00:09:41.260 --> 00:09:41.360
On the other hand, it's not super easy to
00:09:43.820 --> 00:09:44.320
write the prompt. So you want really good
00:09:47.900 --> 00:09:48.040
defaults. So the whole prompt system is kind
00:09:51.360 --> 00:09:51.460
of complicated. That needs to be kind of
00:09:52.580 --> 00:09:52.760
standardized, because I don't think there's
00:09:54.380 --> 00:09:54.720
any tools for doing something like that right
00:09:58.380 --> 00:09:58.880
now. I personally use my system,
00:10:00.220 --> 00:10:00.600
my note system for EKG.
00:10:01.720 --> 00:10:01.920
I don't think that's appropriate for
00:10:02.800 --> 00:10:03.120
everyone, but it does,
00:10:04.480 --> 00:10:04.640
I did write it to have some of these
00:10:06.540 --> 00:10:06.760
capabilities of composability that I think
00:10:08.360 --> 00:10:08.860
are useful for a prompt generation.
00:10:11.940 --> 00:10:12.280
It'd be nice to have a system like that,
00:10:15.660 --> 00:10:16.160
but for general use. I don't,
00:10:17.840 --> 00:10:18.060
this is something I've been meaning to think
00:10:18.840 --> 00:10:19.000
about, like how to do it,
00:10:19.760 --> 00:10:19.920
but like this, you know,
00:10:21.260 --> 00:10:21.660
if someone's interested in getting this area,
00:10:26.120 --> 00:10:26.420
like, I would love to chat about that or,
00:10:27.600 --> 00:10:27.980
you know, I think there's a lot of
00:10:31.020 --> 00:10:31.260
interesting ideas that we could have to have
00:10:34.080 --> 00:10:34.540
a system that allows us to make progress
00:10:38.860 --> 00:10:39.360
here. And also, I think there's more to
00:10:40.520 --> 00:10:40.900
standardization to be done.
00:10:42.820 --> 00:10:43.140
1 thing I'd also like to see that we haven't
00:10:47.020 --> 00:10:47.220
done yet is a system for standardizing on
00:10:48.060 --> 00:10:48.560
getting structured output.
00:10:49.640 --> 00:10:50.140
This is gonna be super useful.
00:10:52.280 --> 00:10:52.780
I have this for open AIs API,
00:10:53.560 --> 00:10:54.060
cause they support it.
00:10:55.940 --> 00:10:56.040
And it's really nice, cause then you can
00:10:57.440 --> 00:10:57.660
write elist functions that like,
00:10:59.380 --> 00:10:59.880
okay, I'm going to call the LLM.
00:11:00.760 --> 00:11:01.000
I'm gonna get structured output.
00:11:02.040 --> 00:11:02.160
I know what that structure is going to be.
00:11:03.480 --> 00:11:03.680
It's not going to be just a big block of
00:11:05.660 --> 00:11:06.040
text. I could turn it into a,
00:11:07.000 --> 00:11:07.480
you know, a P list or something.
00:11:09.280 --> 00:11:09.480
And then I could get the values out of that P
00:11:11.880 --> 00:11:12.260
list. And I know that way I could do,
00:11:14.220 --> 00:11:14.720
I could write actual apps that are,
00:11:18.300 --> 00:11:18.720
you know, very, very sort of,
00:11:20.200 --> 00:11:20.680
you know, useful for very specific purposes
00:11:21.900 --> 00:11:22.400
and not just for text generation.
00:11:24.000 --> 00:11:24.320
And I think that's 1 of the most important
00:11:27.100 --> 00:11:27.540
things we want to do. And I have some ideas
00:11:28.840 --> 00:11:29.160
about how to do it. I just haven't pursued
00:11:31.640 --> 00:11:32.040
those yet. But if other people have ideas,
00:11:34.340 --> 00:11:34.540
I think this would be really interesting to
00:11:35.520 --> 00:11:36.020
add to the LLM package.
00:11:37.260 --> 00:11:37.760
So contact me there.
00:11:42.100 --> 00:11:42.600
[Speaker 1]: Awesome. Quick note before we continue.
00:11:44.440 --> 00:11:44.540
So I'm not sure how long we're going to be on
00:11:46.040 --> 00:11:46.280
stream for, because this is the last talk
00:11:48.640 --> 00:11:49.120
before the break. If we are on the stream
00:11:49.840 --> 00:11:50.200
long-term, then great.
00:11:51.820 --> 00:11:52.300
But if not, folks are welcome to continue
00:11:53.320 --> 00:11:53.680
writing questions on the pad.
00:11:55.140 --> 00:11:55.280
And hopefully, Andrew will get to them at
00:11:58.020 --> 00:11:58.280
some point. Or if Andrew maybe has some extra
00:11:59.960 --> 00:12:00.140
time available and wants to stay on
00:12:01.640 --> 00:12:01.920
BigBlueButton here, then folks are also
00:12:03.940 --> 00:12:04.340
welcome to join here and chat with Andrew
00:12:08.940 --> 00:12:09.240
directly as well. Okay,
00:12:10.740 --> 00:12:10.900
awesome. So yeah, the next question is,
00:12:12.040 --> 00:12:12.400
what are your thoughts on the carbon
00:12:14.060 --> 00:12:14.560
footprint of LLM usage?
00:12:17.200 --> 00:12:17.700
[Speaker 0]: Yeah, it's a really interesting question.
00:12:23.180 --> 00:12:23.360
I don't have any particular knowledge or
00:12:25.440 --> 00:12:25.580
opinions about that. It's something I think
00:12:26.980 --> 00:12:27.180
we should all be educating ourselves more
00:12:32.240 --> 00:12:32.380
about. It is really, I mean,
00:12:33.040 --> 00:12:33.220
there's 2 parts of this,
00:12:35.380 --> 00:12:35.500
right? They take a, there's a huge amount of
00:12:37.160 --> 00:12:37.360
carbon footprint involved in training these
00:12:38.720 --> 00:12:39.220
things. Then running them is relatively
00:12:42.540 --> 00:12:42.880
lightweight. So the question is not
00:12:44.440 --> 00:12:44.920
necessarily like once it's trained,
00:12:46.480 --> 00:12:46.640
like I don't feel like it's a big deal to
00:12:48.280 --> 00:12:48.560
keep using it, but like training these things
00:12:50.680 --> 00:12:51.180
is kind of like the big carbon cost of it.
00:12:53.680 --> 00:12:54.160
But like right now, the way everything's
00:12:56.040 --> 00:12:56.260
going, like every, you know,
00:12:59.060 --> 00:12:59.560
all, you know, the top 5 or 6 tech companies
00:13:00.900 --> 00:13:01.400
are all training their LLMs,
00:13:03.580 --> 00:13:03.740
and this is all costing a giant amount of
00:13:06.820 --> 00:13:07.060
carbon probably. On the other hand these same
00:13:08.560 --> 00:13:08.680
companies are pretty good about using the
00:13:10.260 --> 00:13:10.440
least amount of carbon necessary you know
00:13:12.340 --> 00:13:12.740
they have their own their tricks for doing
00:13:13.260 --> 00:13:13.760
things very efficiently.
00:13:22.100 --> 00:13:22.360
[Speaker 1]: Cool next question, LLMs are slow and
00:13:24.000 --> 00:13:24.340
responding. Do you think Emacs should provide
00:13:26.680 --> 00:13:27.180
more async primitives to keep it responsive?
00:13:29.380 --> 00:13:29.880
Like the URL retrieve is quite bad at
00:13:31.720 --> 00:13:31.760
building API clients with it.
00:13:31.920 --> 00:13:32.420
Building API clients with it?
00:13:36.400 --> 00:13:36.900
[Speaker 0]: Yeah. Well, OK, so first of all,
00:13:40.240 --> 00:13:40.740
people should be using the LLM client.
00:13:48.740 --> 00:13:48.900
And So right now, 1 thing I should have
00:13:50.220 --> 00:13:50.440
mentioned at the top is that there are new
00:13:52.500 --> 00:13:52.640
packages that I recorded this talk that you
00:13:54.480 --> 00:13:54.920
just saw several months ago.
00:13:57.780 --> 00:13:58.180
And so like Elama, there's this package Elama
00:13:59.700 --> 00:14:00.080
that came out that is using the LM package.
00:14:02.440 --> 00:14:02.680
And so for example, it doesn't need to worry
00:14:05.140 --> 00:14:05.580
about this sort of thing because it just uses
00:14:07.560 --> 00:14:07.920
LLM and package and the LLM package worries
00:14:11.680 --> 00:14:11.820
about this. And while I'm on the subject of
00:14:12.540 --> 00:14:12.720
things I forgot to mention,
00:14:15.140 --> 00:14:15.340
I also should just mention very quickly that
00:14:17.020 --> 00:14:17.520
there is now an open source model,
00:14:21.680 --> 00:14:21.960
Mistral. And so that's kind of this new thing
00:14:23.860 --> 00:14:24.240
on the scene that happened after I recorded
00:14:26.240 --> 00:14:26.420
my talk. And I think it's super important to
00:14:28.660 --> 00:14:28.820
the community and important that we have the
00:14:30.620 --> 00:14:31.120
opportunity to use that if we want to.
00:14:33.160 --> 00:14:33.660
Okay, but to answer the actual question,
00:14:37.660 --> 00:14:38.100
there has been some talk about the problems
00:14:40.680 --> 00:14:40.840
with URL retrieve in the URL package in
00:14:42.200 --> 00:14:42.700
general in EmacsDevEl.
00:14:46.760 --> 00:14:47.080
It's not great. I would like to have better
00:14:50.900 --> 00:14:51.040
primitives. And I've asked the author of
00:14:54.060 --> 00:14:54.560
Please PLZ to kind of provide some necessary
00:14:56.120 --> 00:14:56.620
callbacks. I think that's a great library.
00:15:00.280 --> 00:15:00.360
And I'd like to see that kind of like,
00:15:01.320 --> 00:15:01.680
It's nice that we have options,
00:15:03.340 --> 00:15:03.520
and that is an option that uses curl on the
00:15:05.140 --> 00:15:05.640
back end, and that has some benefits.
00:15:09.060 --> 00:15:09.280
So there's this big debate about whether we
00:15:10.600 --> 00:15:11.100
should have primitives or just use curl.
00:15:13.340 --> 00:15:13.420
I'm not exactly sure what the right call is,
00:15:15.320 --> 00:15:15.820
but there has been discussions about this.
00:15:19.540 --> 00:15:20.040
[Speaker 1]: Excellent. And someone commented that GPTEL
00:15:21.820 --> 00:15:22.200
is async and apparently very good at tracking
00:15:22.300 --> 00:15:22.800
the point.
00:15:26.680 --> 00:15:27.180
[Speaker 0]: Yes, yes, GPTEL has similar functionalities
00:15:29.800 --> 00:15:30.040
to LLM, although I believe it's going to move
00:15:33.040 --> 00:15:33.540
to LLM itself sometime soon.
00:15:39.480 --> 00:15:39.860
[Speaker 1]: Next question, speaking of which,
00:15:42.440 --> 00:15:42.560
anyone trained or fine-tuned or prompted a
00:15:44.680 --> 00:15:44.760
model with their org data yet and applied it
00:15:46.560 --> 00:15:47.040
to interesting use cases like planning,
00:15:47.920 --> 00:15:48.340
scheduling, et cetera,
00:15:49.320 --> 00:15:49.820
and maybe care to comment?
00:15:54.620 --> 00:15:55.120
[Speaker 0]: I don't know anyone who is doing that.
00:15:55.860 --> 00:15:56.360
I think it is interesting.
00:15:57.800 --> 00:15:58.300
Like this is what I kind of mentioned at the
00:16:01.060 --> 00:16:01.300
very end of the talk. There is a lot of stuff
00:16:02.440 --> 00:16:02.540
there like you could you know if you
00:16:04.760 --> 00:16:04.920
especially mean an LLM can kind of work as
00:16:07.940 --> 00:16:08.160
sort of like a secretary kind of person that
00:16:12.180 --> 00:16:12.440
could help you prioritize Still it's a
00:16:14.760 --> 00:16:14.920
slightly unclear how what the best way to use
00:16:16.480 --> 00:16:16.720
it is So I think there's more of a question
00:16:18.340 --> 00:16:18.480
for the community about like what people have
00:16:21.140 --> 00:16:21.320
been trying. I see someone has mentioned that
00:16:23.400 --> 00:16:23.900
they are using it for weekly review.
00:16:26.940 --> 00:16:27.180
And it's kind of nice to like,
00:16:29.060 --> 00:16:29.380
maybe you could read your agenda or maybe
00:16:30.480 --> 00:16:30.780
this for like weekly review.
00:16:32.040 --> 00:16:32.240
It could like read all the stuff you've done
00:16:33.340 --> 00:16:33.480
and ask you questions about it.
00:16:35.020 --> 00:16:35.280
And like, what should happen next?
00:16:36.520 --> 00:16:36.780
Or like, is this going to cause a problem?
00:16:39.060 --> 00:16:39.280
Like, I can, I can understand if that could
00:16:40.860 --> 00:16:41.180
happen? That's like, that's kind of nice.
00:16:43.660 --> 00:16:44.160
And this kind of people have had good success
00:16:48.540 --> 00:16:48.760
out of using these LLMs to bounce ideas off
00:16:49.920 --> 00:16:50.420
of are, you know, for,
00:16:52.680 --> 00:16:52.800
you know, I've seen people say that like they
00:16:55.360 --> 00:16:55.600
want, they use it for reading and they kind
00:16:58.520 --> 00:16:58.740
of dialogue with the LM to kind of like do
00:16:59.500 --> 00:17:00.000
sort of active reading.
00:17:02.500 --> 00:17:02.860
So you can imagine doing something similar
00:17:04.400 --> 00:17:04.740
with your tasks where it's sort of you're
00:17:06.560 --> 00:17:06.760
engaged in dialogue about like planning your
00:17:08.880 --> 00:17:09.000
tax with some with a alum that could kind of
00:17:10.800 --> 00:17:11.180
understand what those are and ask you some
00:17:13.780 --> 00:17:13.940
questions I think it. You know,
00:17:16.839 --> 00:17:17.040
if it'd be nice. So, the problem is like
00:17:18.480 --> 00:17:18.980
there's no great way to share all this stuff.
00:17:20.720 --> 00:17:21.220
I guess if you have something like this,
00:17:23.300 --> 00:17:23.720
put it on Reddit. If you don't have Reddit,
00:17:24.599 --> 00:17:24.880
I don't know what to do.
00:17:26.000 --> 00:17:26.500
I would say put it somewhere.
00:17:28.840 --> 00:17:29.020
At the very least, I could maybe open up like
00:17:31.320 --> 00:17:31.820
an LLM discussion session on the LLM package
00:17:34.000 --> 00:17:34.500
GitHub, But not everyone likes to use GitHub.
00:17:36.100 --> 00:17:36.180
I don't know. It'd be nice if there's a
00:17:38.940 --> 00:17:39.060
mailing list or IRC chat for this sort of
00:17:40.840 --> 00:17:41.340
thing. But there isn't at the moment.
00:17:46.560 --> 00:17:46.720
[Speaker 1]: All right. Let's see. I think that's the end
00:17:48.080 --> 00:17:48.580
of the questions on the pad so far.
00:17:51.020 --> 00:17:51.180
There was also some discussion or some
00:17:52.260 --> 00:17:52.760
chatter, I believe, on IRC.
00:17:54.560 --> 00:17:54.820
I'm not sure. Andrew, are you on IRC right
00:18:00.060 --> 00:18:00.260
[Speaker 0]: I am, but I don't think I'm on any place that
00:18:01.400 --> 00:18:01.640
has the chatter. So if there's chatter,
00:18:02.440 --> 00:18:02.940
then I'm not seeing it.
00:18:04.600 --> 00:18:05.100
[Speaker 1]: now? Okay. Yeah, it was in the emacsconf-dev
00:18:06.760 --> 00:18:07.260
channel.
00:18:09.600 --> 00:18:10.100
[Speaker 0]: Okay, let me see if I can.
00:18:25.600 --> 00:18:25.840
Oh, yes. I mean, I could see the channel,
00:18:27.520 --> 00:18:27.840
but I missed whatever came before.
00:18:29.340 --> 00:18:29.480
So if there's anything you want to kind of
00:18:30.840 --> 00:18:31.340
call out, I can try to answer it here.
00:18:35.320 --> 00:18:35.640
[Speaker 1]: OK, cool. I believe at least 2 other folks
00:18:37.500 --> 00:18:38.000
who are participating in the discussion there
00:18:40.120 --> 00:18:40.620
who have also joined here on BigBlueButton,
00:18:42.440 --> 00:18:42.940
Codin Quark and AeonTurn92.
00:18:47.000 --> 00:18:47.480
So you folks, if Andrew is still available
00:18:50.460 --> 00:18:50.640
and has time, you're welcome to chat here and
00:18:53.000 --> 00:18:53.320
ask questions or discuss here as well.
00:18:55.580 --> 00:18:55.840
[Speaker 0]: 1 Thank you. Thank you for your help,
00:18:57.740 --> 00:18:58.080
and thank you for reading all the questions.
00:18:59.700 --> 00:18:59.820
[Speaker 1]: AUDIENCE 2 Cheers, and thanks to you for a
00:19:00.540 --> 00:19:01.040
great talk and the discussion.
00:19:01.880 --> 00:19:02.380
[Speaker 0]: AUDIENCE AUDIENCE 1 Thank you.
00:19:03.140 --> 00:19:03.640
[Speaker 1]: AUDIENCE 2 Cheers.
00:19:07.900 --> 00:19:08.040
[Speaker 0]: So I'll just, I will wait here and see if
00:19:08.320 --> 00:19:08.760
there's any questions.
00:19:10.760 --> 00:19:11.260
If not, I will log off after a few minutes.
00:19:15.900 --> 00:19:16.080
[Speaker 2]: Well, I guess since we were mentioned that
00:19:18.480 --> 00:19:18.980
there was a small chat about local alarms.
00:19:22.640 --> 00:19:23.000
Because chat dpt is nice,
00:19:25.600 --> 00:19:26.100
no, but privacy concerns,
00:19:27.380 --> 00:19:27.880
and it's not free and stuff.
00:19:31.000 --> 00:19:31.500
Which, so The question is,
00:19:36.960 --> 00:19:37.460
what is the promise for local models?
00:19:39.660 --> 00:19:40.160
[Speaker 0]: Yeah, so local is definitely...
00:19:41.380 --> 00:19:41.880
[Speaker 2]: Or at least open source.
00:19:45.680 --> 00:19:46.120
[Speaker 0]: Yeah, so there is a local open source model,
00:19:47.960 --> 00:19:48.460
Misral, which you could run.
00:19:51.340 --> 00:19:51.840
The LLM package allows you to use,
00:19:56.120 --> 00:19:56.260
I think there's 3 kind of local things you
00:19:58.100 --> 00:19:58.440
could use. Like many of these things,
00:20:00.220 --> 00:20:00.480
there's like many kind of ways to do the same
00:20:03.960 --> 00:20:04.460
sort of thing. So LLM is supporting OLAMMA
00:20:10.240 --> 00:20:10.520
and LLAMMA-CPP. And let's see,
00:20:12.240 --> 00:20:12.740
1 other. Which 1 is it?
00:20:18.420 --> 00:20:18.700
And maybe that's it. Maybe the,
00:20:21.820 --> 00:20:21.940
oh, GPT for all. So each 1 of these kind of
00:20:23.100 --> 00:20:23.600
has slightly different functionality.
00:20:26.820 --> 00:20:27.180
For example, I think GPT for all doesn't
00:20:31.780 --> 00:20:32.280
support embeddings. And I hear that Olama's
00:20:33.740 --> 00:20:34.240
embeddings are kind of currently broken.
00:20:35.920 --> 00:20:36.420
But basically they should support everything.
00:20:39.100 --> 00:20:39.600
And the open source models are,
00:20:43.180 --> 00:20:43.380
so the local models are reasonably good.
00:20:44.760 --> 00:20:44.900
Like I don't think you'd use them and be
00:20:46.200 --> 00:20:46.639
like, what is this horrible nonsense?
00:20:50.200 --> 00:20:50.380
Like it's, it gives you relatively good
00:20:51.820 --> 00:20:52.120
results. Like it's not gonna be at the level
00:20:56.060 --> 00:20:56.320
of like GPT 3.5 or 4, but it's not far away
00:20:57.720 --> 00:20:58.220
from GPT 3.5, I think.
00:21:02.380 --> 00:21:02.880
[Speaker 2]: I'm just saying that Olam has like a presets
00:21:05.940 --> 00:21:06.300
for connecting the actual working servers for
00:21:06.300 --> 00:21:06.800
Olama?
00:21:08.560 --> 00:21:08.760
[Speaker 0]: So, I'll try. Yeah, so you could,
00:21:09.860 --> 00:21:10.040
what you could do is you could like for
00:21:11.940 --> 00:21:12.100
example you could download Olama which is
00:21:15.780 --> 00:21:15.940
just a way of setting up local models and
00:21:17.320 --> 00:21:17.780
running local models on your machine.
00:21:18.580 --> 00:21:18.820
So typically what it does,
00:21:19.720 --> 00:21:20.020
you like download a program,
00:21:23.720 --> 00:21:23.800
let's say Olama. Then Olama will have the
00:21:24.940 --> 00:21:25.440
ability to download models.
00:21:27.240 --> 00:21:27.360
And so you could choose from just a host of
00:21:29.280 --> 00:21:29.440
different models. Each 1 of these things has
00:21:30.200 --> 00:21:30.440
a bunch of different models.
00:21:31.920 --> 00:21:32.080
So it downloads all these things to your
00:21:36.600 --> 00:21:37.020
machine. But I would say that the key problem
00:21:40.200 --> 00:21:40.580
here is that it requires a fairly beefy
00:21:40.580 --> 00:21:41.080
machine.
00:21:42.600 --> 00:21:43.100
[Speaker 2]: So. Yeah, yeah, of course.
00:21:45.060 --> 00:21:45.300
Why I was asking, because you briefly
00:21:46.440 --> 00:21:46.880
mentioned that there are some Israeli
00:21:52.300 --> 00:21:52.440
servers. I understand that they run it like a
00:21:53.680 --> 00:21:54.000
government or stuff like that?
00:21:55.440 --> 00:21:55.940
No, no, sorry. People want everyone?
00:21:59.340 --> 00:21:59.620
[Speaker 0]: I don't, I mean, maybe you've said something
00:22:00.620 --> 00:22:01.020
that sounded like Israeli servers.
00:22:01.620 --> 00:22:02.120
[Speaker 2]: Okay, okay.
00:22:04.920 --> 00:22:05.080
[Speaker 0]: I think- There's no government LLMs as far as
00:22:06.820 --> 00:22:07.280
I know. Although, I'm sure the governments
00:22:08.200 --> 00:22:08.700
are working on their own LLMs,
00:22:10.980 --> 00:22:11.480
et cetera. But yeah, basically your choices
00:22:15.060 --> 00:22:15.220
are spend a, I mean, if you use open AI or
00:22:16.080 --> 00:22:16.580
something or anything else,
00:22:17.960 --> 00:22:18.460
you're really not spending any money.
00:22:20.560 --> 00:22:20.840
Like I've never been able to spend any money
00:22:23.680 --> 00:22:24.020
on OpenAI. Like unless you're doing something
00:22:25.840 --> 00:22:26.280
very intensive and really are using it to,
00:22:28.000 --> 00:22:28.180
you know, if you're using it for your
00:22:29.620 --> 00:22:29.780
personal use, it's just hard to spend any
00:22:31.720 --> 00:22:31.960
money. But on the other hand,
00:22:32.780 --> 00:22:32.860
it's not free. So you can,
00:22:33.040 --> 00:22:33.540
you know,
00:22:36.300 --> 00:22:36.680
[Speaker 2]: Actually, it's rather cheap.
00:22:37.680 --> 00:22:38.180
There's no question about that.
00:22:40.580 --> 00:22:40.920
The problem is that it has a bad track record
00:22:41.580 --> 00:22:42.080
on privacy.
00:22:45.540 --> 00:22:46.040
[Speaker 0]: Yes, that's, I think that is a key problem.
00:22:48.120 --> 00:22:48.280
This is probably the number 1 reason why you
00:22:51.840 --> 00:22:52.340
might want to use a local AI,
00:22:54.720 --> 00:22:55.220
a local LLM. Another 1 is like,
00:22:57.400 --> 00:22:57.900
you may not agree with the decisions.
00:23:00.360 --> 00:23:00.820
You know, there's a lot of trust and safety
00:23:05.140 --> 00:23:05.440
stuff that these companies have to do.
00:23:09.020 --> 00:23:09.240
Like they don't want like the LMs to kind of
00:23:11.400 --> 00:23:11.640
like give you, like tell you how you can make
00:23:13.180 --> 00:23:13.580
meth or how you can make a bomb,
00:23:14.960 --> 00:23:15.460
which they would do. They would totally do
00:23:19.580 --> 00:23:20.080
it. So, But each time you kind of restrict
00:23:22.540 --> 00:23:22.680
what is happening with what you can get out
00:23:23.860 --> 00:23:24.360
of the LM, it gets a little worse.
00:23:24.960 --> 00:23:25.440
So some people
00:23:27.040 --> 00:23:27.540
[Speaker 2]: want to have local. That's expected.
00:23:31.080 --> 00:23:31.400
I guess even open source language modules
00:23:33.540 --> 00:23:34.040
will soon have HR spaces because it's simply
00:23:34.720 --> 00:23:35.220
a legal issue.
00:23:40.760 --> 00:23:41.140
[Speaker 0]: I think that's true. I also think that there
00:23:42.880 --> 00:23:43.040
probably will be, although I don't know of
00:23:45.060 --> 00:23:45.300
any offhand, that will are completely
00:23:46.960 --> 00:23:47.200
uncensored. I know people are interested and
00:23:48.240 --> 00:23:48.480
are running uncensored models.
00:23:49.440 --> 00:23:49.940
I don't know how to do it.
00:23:52.280 --> 00:23:52.780
I think it's a little bit dubious,
00:23:54.960 --> 00:23:55.040
but some people do want to do it.
00:23:56.280 --> 00:23:56.780
There's another reason for using local
00:24:02.280 --> 00:24:02.780
servers. Do you have any recommendation for
00:24:05.500 --> 00:24:05.720
models to run locally and also comments on
00:24:06.780 --> 00:24:07.280
whether a GPU is required?
00:24:14.040 --> 00:24:14.160
Usually a GPU, well, you can run it without a
00:24:16.960 --> 00:24:17.460
GPU, but it does run much better.
00:24:19.480 --> 00:24:19.980
Like for example, I think when I used,
00:24:22.560 --> 00:24:23.060
Lama is sort of like a standard.
00:24:27.160 --> 00:24:27.320
This was the model for that Facebook came out
00:24:31.380 --> 00:24:31.880
with for local use. And It was,
00:24:37.260 --> 00:24:37.760
yeah, it's good. It's,
00:24:40.400 --> 00:24:40.900
but it's now it's I think,
00:24:44.620 --> 00:24:44.920
Mistral is kind of like has a better
00:24:46.480 --> 00:24:46.800
performance, But there's also different model
00:24:51.000 --> 00:24:51.500
sizes. There's 7B, like the Lama 7B is OK.
00:24:52.940 --> 00:24:53.440
The Mistral 7B, 7 billion,
00:24:54.800 --> 00:24:55.300
are like, basically it'll take like,
00:24:58.380 --> 00:24:58.880
you can run it with like 16 gigs of RAM,
00:25:02.720 --> 00:25:03.040
is pretty good. It's probably about as equal
00:25:06.900 --> 00:25:07.000
to the LLAMA13B. Those are the number of
00:25:08.360 --> 00:25:08.860
parameters, if I remember correctly.
00:25:10.680 --> 00:25:11.180
And then there's a 7B,
00:25:12.340 --> 00:25:12.840
which I've never been able to run.
00:25:16.120 --> 00:25:16.620
And even if the 7B, if you run it without a
00:25:19.640 --> 00:25:20.140
GPU, it takes quite a while to answer.
00:25:22.080 --> 00:25:22.580
I think I've had experiences where it took
00:25:23.940 --> 00:25:24.440
literally like several,
00:25:26.480 --> 00:25:26.780
like 5 minutes before it even started
00:25:28.880 --> 00:25:29.100
responding, but you do eventually get
00:25:32.220 --> 00:25:32.580
something. And it could be that like things
00:25:33.840 --> 00:25:33.960
have gotten better since the last time I
00:25:35.440 --> 00:25:35.940
tried this, because things are moving fast.
00:25:38.360 --> 00:25:38.860
But it is super recommended to have a GPU.
00:25:42.440 --> 00:25:42.620
This is the problem. It's kind of like,
00:25:43.840 --> 00:25:44.180
yes, free software is great.
00:25:46.880 --> 00:25:47.120
But if free software is requiring that you
00:25:50.460 --> 00:25:50.760
have these kind of beefy servers and have all
00:25:52.000 --> 00:25:52.500
this hardware, that's not great.
00:25:53.600 --> 00:25:54.100
I think there's a case to be made.
00:25:55.680 --> 00:25:56.180
[Speaker 1]: a hardware
00:25:59.040 --> 00:25:59.540
[Speaker 0]: with slots instead of a laptop.
00:26:01.560 --> 00:26:02.060
Yeah, yeah, that's right.
00:26:03.660 --> 00:26:03.960
[Speaker 2]: Ideally, you can have Ideally,
00:26:07.400 --> 00:26:07.660
it would be nice if FSL for all things could
00:26:12.040 --> 00:26:12.540
run something for open source model.
00:26:16.320 --> 00:26:16.640
And not free, but the key point is that it's
00:26:16.640 --> 00:26:17.140
Libre?
00:26:22.580 --> 00:26:23.080
[Speaker 0]: Yes, so actually I think Google does do that.
00:26:24.720 --> 00:26:24.800
I'll have to look it up,
00:26:27.560 --> 00:26:27.820
but I haven't explored this yet.
00:26:31.220 --> 00:26:31.720
But Google's server, which LLM does support,
00:26:33.800 --> 00:26:34.300
supports arbitrary models.
00:26:36.420 --> 00:26:36.600
So you can run LLMA or things like that.
00:26:38.940 --> 00:26:39.200
The problem is that even if you're running
00:26:40.320 --> 00:26:40.820
Mistral, which has no restrictions.
00:26:42.940 --> 00:26:43.140
So this is the kind of thing that like the
00:26:44.900 --> 00:26:45.060
Free Software Foundation cares a lot about.
00:26:47.240 --> 00:26:47.740
Like you want it to be like no restrictions,
00:26:49.740 --> 00:26:49.840
legal restrictions on you as you run the
00:26:52.080 --> 00:26:52.580
model. So even if it's running Mistral,
00:26:54.800 --> 00:26:55.300
just by using the server,
00:26:58.460 --> 00:26:58.660
the company server, it will impose some
00:26:59.440 --> 00:26:59.900
restrictions on you probably,
00:27:02.320 --> 00:27:02.480
right? There's gonna be some license that you
00:27:04.760 --> 00:27:05.260
have to, or something you have to abide by.
00:27:08.480 --> 00:27:08.600
So I think, yes, it depends on how much you
00:27:09.280 --> 00:27:09.780
care about it, I guess.
00:27:19.500 --> 00:27:19.640
I should find out more about that and make
00:27:21.580 --> 00:27:22.080
sure that it's a good point that I should,
00:27:23.980 --> 00:27:24.180
you know, people should be able to run free
00:27:25.920 --> 00:27:26.280
models over the server.
00:27:28.320 --> 00:27:28.440
So I should make sure we support that in the
00:27:40.360 --> 00:27:40.860
LLM package. So, is there any other questions
00:27:48.240 --> 00:27:48.740
Or is otherwise we can end the session.
00:28:00.800 --> 00:28:01.040
Yeah, all right. Thank you.
00:28:02.440 --> 00:28:02.940
Thank you. Thank you everyone who listened.
00:28:04.540 --> 00:28:05.040
I'm super happy like I,
00:28:06.560 --> 00:28:07.060
the interest is great.
00:28:08.900 --> 00:28:09.220
I think there's great stuff to be done here
00:28:10.960 --> 00:28:11.140
and I'm kind of super excited what we're
00:28:11.940 --> 00:28:12.160
going to do in the next year,
00:28:13.140 --> 00:28:13.440
so hopefully, like next year,
00:28:14.600 --> 00:28:14.760
and the conference we have something even
00:28:16.440 --> 00:28:16.560
more exciting to say about LLM and how they
00:28:17.320 --> 00:28:17.820
can be used with Emacs.
00:28:19.620 --> 00:28:20.120
So thank
00:28:30.060 --> 00:28:30.560
you
