WEBVTT 00:00:13.099 --> 00:00:13.599 [Speaker 0]: Okay. Hello, everyone. 00:00:16.560 --> 00:00:17.060 I think this is the start of the Q&A session. 00:00:25.119 --> 00:00:25.599 So people can just ask me questions here. 00:00:28.259 --> 00:00:28.380 Or I think maybe these questions are going to 00:00:30.560 --> 00:00:30.980 be read by someone. Yes, 00:00:34.680 --> 00:00:35.060 thank you. Should I start doing that? 00:00:39.280 --> 00:00:39.400 I also know that there's questions in the 00:00:41.320 --> 00:00:41.480 either pad room, so I could start out 00:00:42.280 --> 00:00:42.780 answering those as well. 00:00:45.020 --> 00:00:45.300 [Speaker 1]: Right, sure. Whichever way you prefer. 00:00:46.860 --> 00:00:47.220 If you prefer to read the questions yourself, 00:00:48.940 --> 00:00:49.080 by all means, or if you would prefer me to 00:00:50.080 --> 00:00:50.280 read them to you, that also works. 00:00:50.580 --> 00:00:50.920 [Speaker 2]: Oh, I see. 00:00:51.600 --> 00:00:51.760 [Speaker 0]: Why don't you read them to me? 00:00:53.260 --> 00:00:53.760 I think it'll just be more interesting then. 00:00:56.820 --> 00:00:57.160 [Speaker 1]: Sure. OK, let's see. The first question is, 00:00:58.360 --> 00:00:58.860 what is your use case for embedding, 00:01:00.060 --> 00:01:00.560 mainly for searching? 00:01:06.180 --> 00:01:06.340 [Speaker 0]: Yeah, I mean, I think the use case really is 00:01:12.100 --> 00:01:12.320 searching. And I think it is very useful when 00:01:15.060 --> 00:01:15.320 you're searching for something in a vague 00:01:18.280 --> 00:01:18.780 way. Just to give you an example, 00:01:23.860 --> 00:01:24.360 I have a note system called EKG. 00:01:25.760 --> 00:01:26.260 I type all my notes on it. 00:01:28.620 --> 00:01:29.120 You can find it on GitHub and Melba. 00:01:34.140 --> 00:01:34.400 But I wrote something at some point a year 00:01:35.840 --> 00:01:36.020 ago or something. I wrote something that I 00:01:36.600 --> 00:01:36.980 just vaguely remembered. 00:01:38.800 --> 00:01:38.940 Oh, this was about a certain kind of 00:01:41.580 --> 00:01:41.760 communication. I wanted communicating to 00:01:43.280 --> 00:01:43.479 large audiences. There's some interesting tip 00:01:44.700 --> 00:01:45.060 that I wrote down that was really cool. 00:01:49.920 --> 00:01:50.080 And I was like, well, I need to find it. 00:01:52.260 --> 00:01:52.640 So I did an embedding search for something 00:01:55.479 --> 00:01:55.979 like, you know, tips for communicating. 00:01:58.979 --> 00:01:59.100 Like those words may not have been in what I 00:02:00.020 --> 00:02:00.520 was trying to find at all, 00:02:02.680 --> 00:02:03.180 But it was able to find it. 00:02:05.840 --> 00:02:06.260 And that is something that's very hard to do 00:02:07.200 --> 00:02:07.360 in other ways. Like, you know, 00:02:08.520 --> 00:02:08.720 if you had to do this with normal search, 00:02:09.199 --> 00:02:09.660 you have to do synonyms. 00:02:10.940 --> 00:02:11.200 And like maybe those synonyms wouldn't cover 00:02:11.960 --> 00:02:12.340 it. Like with embedding, 00:02:13.940 --> 00:02:14.160 you can basically get at like the vague 00:02:14.960 --> 00:02:15.460 sentiment. You're like, 00:02:17.320 --> 00:02:17.560 you know, you're, you know, 00:02:19.520 --> 00:02:19.700 you can really query on like what things are 00:02:21.760 --> 00:02:22.260 about as opposed to what words they have. 00:02:25.600 --> 00:02:26.100 Also, it's super good for similarity search. 00:02:27.720 --> 00:02:27.900 So you could say, look, 00:02:30.040 --> 00:02:30.240 I have a bunch of things that are encoded 00:02:31.400 --> 00:02:31.900 with embeddings that I want to show. 00:02:34.120 --> 00:02:34.280 For example, you can make an embedding for 00:02:35.220 --> 00:02:35.380 every buffer. You'd be like, 00:02:37.060 --> 00:02:37.200 well, show me buffers that are similar to 00:02:38.740 --> 00:02:38.980 this buffer. That doesn't sound super useful, 00:02:40.440 --> 00:02:40.940 but this is the kind of thing you could do. 00:02:45.300 --> 00:02:45.480 And so if you have a bunch of notes or 00:02:46.720 --> 00:02:46.920 something else that you want to search on, 00:02:48.240 --> 00:02:48.740 you'd be like, what's similar to this buffer? 00:02:51.500 --> 00:02:51.760 Or what notes are similar to each other? 00:02:53.040 --> 00:02:53.540 What buffers are similar to each other? 00:02:55.380 --> 00:02:55.880 It's super good for this sort of thing. 00:03:00.780 --> 00:03:01.280 And it's also good for this kind of retrieval 00:03:03.080 --> 00:03:03.520 augmented generation, where you sort of, 00:03:05.080 --> 00:03:05.280 you retrieve things and the purpose is not 00:03:06.600 --> 00:03:06.880 for you to see them, but then you pass that 00:03:12.040 --> 00:03:12.180 to the LLM. And then it's able to be a little 00:03:14.340 --> 00:03:14.800 bit more accurate because it has the actual 00:03:15.760 --> 00:03:16.260 text that you're trying to, 00:03:18.960 --> 00:03:19.180 that is relevant, and it can cite from and 00:03:20.720 --> 00:03:20.820 things like that. And then it could give you 00:03:22.260 --> 00:03:22.660 a much better answer that's kind of, 00:03:25.520 --> 00:03:25.680 you know, not just from its own little neural 00:03:26.320 --> 00:03:26.820 nets and memory. 00:03:31.920 --> 00:03:32.120 [Speaker 1]: Cool, thanks. Let's see, 00:03:35.740 --> 00:03:36.100 next question. What do you think about embed 00:03:40.160 --> 00:03:40.660 Emacs manual versus GPT's Emacs manual? 00:03:45.480 --> 00:03:45.660 [Speaker 0]: I'm not exactly sure what this question is 00:03:46.980 --> 00:03:47.480 trying to say. So I mean, 00:03:51.000 --> 00:03:51.140 if someone wrote that and wants to expand on 00:03:55.080 --> 00:03:55.320 it a little bit, but I think that maybe 00:03:58.420 --> 00:03:58.920 you're saying like you could embed, 00:04:00.280 --> 00:04:00.780 have embeddings for like various, 00:04:02.520 --> 00:04:02.720 like every paragraph or something of the 00:04:04.540 --> 00:04:04.680 Emacs manual. But it's also the case that 00:04:06.500 --> 00:04:07.000 like GPT is already for sure already read it, 00:04:09.760 --> 00:04:09.960 right? And so you could ask questions that 00:04:13.460 --> 00:04:13.780 are about Emacs and our ELISP or whatever 00:04:15.200 --> 00:04:15.700 part of the manual you want to find. 00:04:19.760 --> 00:04:20.240 And it will do a reasonably good job, 00:04:22.280 --> 00:04:22.780 especially the better models will do a 00:04:24.620 --> 00:04:24.940 reasonably good job of saying you something 00:04:26.040 --> 00:04:26.540 that is vaguely accurate. 00:04:29.440 --> 00:04:29.860 But if you do this retrieval augmented 00:04:30.580 --> 00:04:31.080 generation with embeddings, 00:04:32.640 --> 00:04:33.140 you can get something that is very accurate. 00:04:36.700 --> 00:04:36.960 At least I think. I haven't tried it, 00:04:38.760 --> 00:04:39.020 but this is a technique that works in other 00:04:43.040 --> 00:04:43.260 similar cases. So you can also imagine like, 00:04:44.320 --> 00:04:44.500 oh, this whole thing I said, 00:04:47.860 --> 00:04:48.120 like, oh, you can query for vague things and 00:04:49.140 --> 00:04:49.600 get parts of the manual, 00:04:52.680 --> 00:04:53.000 perhaps. I'm not exactly sure if that would 00:04:55.120 --> 00:04:55.520 be useful, but maybe. Usually when I'm 00:04:57.040 --> 00:04:57.180 looking things up in the Emacs manual or 00:04:58.320 --> 00:04:58.780 Elist manual, I have something extremely 00:05:00.020 --> 00:05:00.300 specific and I kind of know where to look. 00:05:02.960 --> 00:05:03.080 But having other ways to get at this 00:05:04.000 --> 00:05:04.500 information is always good. 00:05:10.240 --> 00:05:10.740 [Speaker 1]: Right. Looks like they added a clarification 00:05:12.280 --> 00:05:12.720 if you would like to read that yourself, 00:05:14.180 --> 00:05:14.680 or would you like me to read it for you? 00:05:17.640 --> 00:05:18.140 [Speaker 0]: Yeah. Yes, OK. It says, 00:05:20.460 --> 00:05:20.740 I've never tried. Yeah, 00:05:21.500 --> 00:05:21.820 the question is like OK, 00:05:23.100 --> 00:05:23.240 there is a difference between the kind of 00:05:23.860 --> 00:05:24.360 thing as I just described. 00:05:26.200 --> 00:05:26.600 I have not tried the difference with the EMAX 00:05:31.560 --> 00:05:31.980 manual itself. It'd be interesting to see 00:05:33.700 --> 00:05:33.960 what this is, but I would expect like these 00:05:35.140 --> 00:05:35.600 techniques, the retrieval augmented 00:05:38.840 --> 00:05:39.340 generation is generally pretty good. 00:05:41.240 --> 00:05:41.740 And I suspect it would, 00:05:43.580 --> 00:05:43.780 I would bet money on the fact that it's gonna 00:05:45.820 --> 00:05:46.240 give you, you know, better results than just, 00:05:48.160 --> 00:05:48.360 you know, doing a free form query without any 00:05:49.440 --> 00:05:49.940 retrieval augmented generation. 00:05:54.240 --> 00:05:54.640 [Speaker 1]: Cool. Let's see. Next question. 00:05:56.380 --> 00:05:56.880 When deferring commit messages to an LLM, 00:05:59.700 --> 00:05:59.920 what, if anything, do you find you might have 00:06:02.940 --> 00:06:03.440 [Speaker 0]: lost? Yeah, it's a good question. 00:06:06.060 --> 00:06:06.560 When deferring anything to a computer, 00:06:08.860 --> 00:06:09.360 like, you know, I used to have to remember 00:06:11.200 --> 00:06:11.700 how to get places, and now, 00:06:14.540 --> 00:06:15.040 you know, on the few occasions which I drive, 00:06:16.560 --> 00:06:16.720 like, It could just tell me how to get 00:06:21.960 --> 00:06:22.280 places. So similar things could occur here 00:06:24.960 --> 00:06:25.460 where like, okay, I'm just leaving the LLM. 00:06:27.380 --> 00:06:27.680 And so I'm kind of missing out on some 00:06:30.040 --> 00:06:30.220 opportunity to think coherently about a 00:06:32.440 --> 00:06:32.680 particular commit. Particular commits are 00:06:36.140 --> 00:06:36.540 kind of low level. I don't think it's usually 00:06:39.340 --> 00:06:39.840 relatively obvious and what they're doing. 00:06:42.600 --> 00:06:42.800 And in this case, I think there's not much 00:06:44.220 --> 00:06:44.540 loss. But for sure, in other cases, 00:06:46.400 --> 00:06:46.900 if you're starting to get into situations 00:06:48.640 --> 00:06:48.800 where it's writing your emails and all this 00:06:52.920 --> 00:06:53.300 stuff. First of all, it's in 1 sense, 00:06:55.580 --> 00:06:56.040 I'm not sure you might be losing something by 00:06:57.520 --> 00:06:57.780 delegating things. On the other hand, 00:06:59.120 --> 00:06:59.280 you know, when you're interacting with these 00:07:01.280 --> 00:07:01.560 LLMs, you have to be extremely specific about 00:07:03.120 --> 00:07:03.240 what you want, or else it's just not going to 00:07:07.540 --> 00:07:07.680 do a good job. And that might actually be a 00:07:09.440 --> 00:07:09.860 good thing. So the question might be that 00:07:11.820 --> 00:07:12.240 maybe you might gain things by using an LLM 00:07:13.860 --> 00:07:14.060 to do your work. It might not actually even 00:07:15.060 --> 00:07:15.400 save you that much time, 00:07:18.480 --> 00:07:18.640 at least initially, because you have to kind 00:07:20.460 --> 00:07:20.660 of practice again super specific about what 00:07:22.740 --> 00:07:22.900 you want to get out of the output it's going 00:07:26.940 --> 00:07:26.980 to give you so like oh I'm you know maybe you 00:07:29.600 --> 00:07:29.820 know you're on the emacs devel mailing list 00:07:31.780 --> 00:07:31.980 and you're like okay write this email about 00:07:34.000 --> 00:07:34.140 this about this And here's what I want to 00:07:35.370 --> 00:07:35.460 say. And here's the kind of tone I want to 00:07:36.020 --> 00:07:36.420 use. And here's the like, 00:07:37.660 --> 00:07:38.160 oh, you might want to specify like everything 00:07:39.620 --> 00:07:40.120 that you kind of want to get into this. 00:07:42.180 --> 00:07:42.680 Usually it's easier just to write the email. 00:07:45.600 --> 00:07:46.100 But I think that practice of kind of 00:07:48.080 --> 00:07:48.420 understanding what you want is not something 00:07:52.680 --> 00:07:53.180 you normally do. And I think it's going to be 00:07:56.040 --> 00:07:56.480 an interesting exercise that will help people 00:07:57.280 --> 00:07:57.540 understand. That said, 00:07:58.860 --> 00:07:59.020 I haven't done that much of that, 00:07:59.900 --> 00:08:00.040 so I can't say, oh, yeah, 00:08:01.080 --> 00:08:01.440 I've done this and it works for me. 00:08:03.000 --> 00:08:03.120 Maybe. I think it's an interesting thing to 00:08:03.120 --> 00:08:03.620 explore. 00:08:07.720 --> 00:08:08.220 [Speaker 1]: Sure. Thanks. Let's see. 00:08:10.840 --> 00:08:11.140 Let's see. Can you share your font settings 00:08:13.440 --> 00:08:13.740 in your Emacs config? Those are some nice 00:08:14.200 --> 00:08:14.700 fonts for reading. 00:08:18.900 --> 00:08:19.200 [Speaker 0]: Yeah, I think I was using Menlo at the time. 00:08:20.840 --> 00:08:20.980 Unfortunately, I don't save those kinds of 00:08:21.940 --> 00:08:22.200 things, like a history of this. 00:08:24.000 --> 00:08:24.500 I've kind of switched now to, 00:08:27.340 --> 00:08:27.540 what was that? I think I wrote it down in 00:08:29.440 --> 00:08:29.940 the, I switched to MunaSpace, 00:08:31.920 --> 00:08:32.420 which just came out like a week or 2 ago, 00:08:33.340 --> 00:08:33.840 and is also pretty cool. 00:08:35.440 --> 00:08:35.940 So I think it's Menlo. 00:08:37.380 --> 00:08:37.760 The internal question, 00:08:38.400 --> 00:08:38.900 what font are you using? 00:08:42.020 --> 00:08:42.340 [Speaker 1]: Indeed, yeah. It looks like someone guessed 00:08:43.780 --> 00:08:44.280 as well that it might be Menlo. 00:08:47.680 --> 00:08:48.040 OK, Cool. Yeah, next question. 00:08:48.900 --> 00:08:49.400 In terms of standardization, 00:08:53.260 --> 00:08:53.520 do you see a need for the medium to large 00:08:55.840 --> 00:08:56.160 scale effort needed? And then they also 00:08:56.960 --> 00:08:57.460 elaborate about it. 00:09:03.600 --> 00:09:04.100 [Speaker 0]: Yeah, I mean, I do think, 00:09:06.040 --> 00:09:06.300 I don't know if it's large scale, 00:09:08.000 --> 00:09:08.500 but at least it's probably medium scale. 00:09:10.320 --> 00:09:10.520 There's a lot of things that are missing that 00:09:12.260 --> 00:09:12.400 we don't have right now in emacs when you're 00:09:13.660 --> 00:09:14.160 dealing with LLMs. 1 is, 00:09:18.240 --> 00:09:18.420 a prompting system. And by that, 00:09:21.820 --> 00:09:22.020 I mean, you know, prompts are just like big 00:09:24.520 --> 00:09:24.720 blocks of text, but there's also senses that 00:09:28.260 --> 00:09:28.420 like prompts need to be composable and you 00:09:30.480 --> 00:09:30.660 need to be able to iterate on parts of the 00:09:36.600 --> 00:09:37.100 prompt. And so it's also customizable. 00:09:38.940 --> 00:09:39.060 Users might want to customize it. 00:09:41.260 --> 00:09:41.360 On the other hand, it's not super easy to 00:09:43.820 --> 00:09:44.320 write the prompt. So you want really good 00:09:47.900 --> 00:09:48.040 defaults. So the whole prompt system is kind 00:09:51.360 --> 00:09:51.460 of complicated. That needs to be kind of 00:09:52.580 --> 00:09:52.760 standardized, because I don't think there's 00:09:54.380 --> 00:09:54.720 any tools for doing something like that right 00:09:58.380 --> 00:09:58.880 now. I personally use my system, 00:10:00.220 --> 00:10:00.600 my note system for EKG. 00:10:01.720 --> 00:10:01.920 I don't think that's appropriate for 00:10:02.800 --> 00:10:03.120 everyone, but it does, 00:10:04.480 --> 00:10:04.640 I did write it to have some of these 00:10:06.540 --> 00:10:06.760 capabilities of composability that I think 00:10:08.360 --> 00:10:08.860 are useful for a prompt generation. 00:10:11.940 --> 00:10:12.280 It'd be nice to have a system like that, 00:10:15.660 --> 00:10:16.160 but for general use. I don't, 00:10:17.840 --> 00:10:18.060 this is something I've been meaning to think 00:10:18.840 --> 00:10:19.000 about, like how to do it, 00:10:19.760 --> 00:10:19.920 but like this, you know, 00:10:21.260 --> 00:10:21.660 if someone's interested in getting this area, 00:10:26.120 --> 00:10:26.420 like, I would love to chat about that or, 00:10:27.600 --> 00:10:27.980 you know, I think there's a lot of 00:10:31.020 --> 00:10:31.260 interesting ideas that we could have to have 00:10:34.080 --> 00:10:34.540 a system that allows us to make progress 00:10:38.860 --> 00:10:39.360 here. And also, I think there's more to 00:10:40.520 --> 00:10:40.900 standardization to be done. 00:10:42.820 --> 00:10:43.140 1 thing I'd also like to see that we haven't 00:10:47.020 --> 00:10:47.220 done yet is a system for standardizing on 00:10:48.060 --> 00:10:48.560 getting structured output. 00:10:49.640 --> 00:10:50.140 This is gonna be super useful. 00:10:52.280 --> 00:10:52.780 I have this for open AIs API, 00:10:53.560 --> 00:10:54.060 cause they support it. 00:10:55.940 --> 00:10:56.040 And it's really nice, cause then you can 00:10:57.440 --> 00:10:57.660 write elist functions that like, 00:10:59.380 --> 00:10:59.880 okay, I'm going to call the LLM. 00:11:00.760 --> 00:11:01.000 I'm gonna get structured output. 00:11:02.040 --> 00:11:02.160 I know what that structure is going to be. 00:11:03.480 --> 00:11:03.680 It's not going to be just a big block of 00:11:05.660 --> 00:11:06.040 text. I could turn it into a, 00:11:07.000 --> 00:11:07.480 you know, a P list or something. 00:11:09.280 --> 00:11:09.480 And then I could get the values out of that P 00:11:11.880 --> 00:11:12.260 list. And I know that way I could do, 00:11:14.220 --> 00:11:14.720 I could write actual apps that are, 00:11:18.300 --> 00:11:18.720 you know, very, very sort of, 00:11:20.200 --> 00:11:20.680 you know, useful for very specific purposes 00:11:21.900 --> 00:11:22.400 and not just for text generation. 00:11:24.000 --> 00:11:24.320 And I think that's 1 of the most important 00:11:27.100 --> 00:11:27.540 things we want to do. And I have some ideas 00:11:28.840 --> 00:11:29.160 about how to do it. I just haven't pursued 00:11:31.640 --> 00:11:32.040 those yet. But if other people have ideas, 00:11:34.340 --> 00:11:34.540 I think this would be really interesting to 00:11:35.520 --> 00:11:36.020 add to the LLM package. 00:11:37.260 --> 00:11:37.760 So contact me there. 00:11:42.100 --> 00:11:42.600 [Speaker 1]: Awesome. Quick note before we continue. 00:11:44.440 --> 00:11:44.540 So I'm not sure how long we're going to be on 00:11:46.040 --> 00:11:46.280 stream for, because this is the last talk 00:11:48.640 --> 00:11:49.120 before the break. If we are on the stream 00:11:49.840 --> 00:11:50.200 long-term, then great. 00:11:51.820 --> 00:11:52.300 But if not, folks are welcome to continue 00:11:53.320 --> 00:11:53.680 writing questions on the pad. 00:11:55.140 --> 00:11:55.280 And hopefully, Andrew will get to them at 00:11:58.020 --> 00:11:58.280 some point. Or if Andrew maybe has some extra 00:11:59.960 --> 00:12:00.140 time available and wants to stay on 00:12:01.640 --> 00:12:01.920 BigBlueButton here, then folks are also 00:12:03.940 --> 00:12:04.340 welcome to join here and chat with Andrew 00:12:08.940 --> 00:12:09.240 directly as well. Okay, 00:12:10.740 --> 00:12:10.900 awesome. So yeah, the next question is, 00:12:12.040 --> 00:12:12.400 what are your thoughts on the carbon 00:12:14.060 --> 00:12:14.560 footprint of LLM usage? 00:12:17.200 --> 00:12:17.700 [Speaker 0]: Yeah, it's a really interesting question. 00:12:23.180 --> 00:12:23.360 I don't have any particular knowledge or 00:12:25.440 --> 00:12:25.580 opinions about that. It's something I think 00:12:26.980 --> 00:12:27.180 we should all be educating ourselves more 00:12:32.240 --> 00:12:32.380 about. It is really, I mean, 00:12:33.040 --> 00:12:33.220 there's 2 parts of this, 00:12:35.380 --> 00:12:35.500 right? They take a, there's a huge amount of 00:12:37.160 --> 00:12:37.360 carbon footprint involved in training these 00:12:38.720 --> 00:12:39.220 things. Then running them is relatively 00:12:42.540 --> 00:12:42.880 lightweight. So the question is not 00:12:44.440 --> 00:12:44.920 necessarily like once it's trained, 00:12:46.480 --> 00:12:46.640 like I don't feel like it's a big deal to 00:12:48.280 --> 00:12:48.560 keep using it, but like training these things 00:12:50.680 --> 00:12:51.180 is kind of like the big carbon cost of it. 00:12:53.680 --> 00:12:54.160 But like right now, the way everything's 00:12:56.040 --> 00:12:56.260 going, like every, you know, 00:12:59.060 --> 00:12:59.560 all, you know, the top 5 or 6 tech companies 00:13:00.900 --> 00:13:01.400 are all training their LLMs, 00:13:03.580 --> 00:13:03.740 and this is all costing a giant amount of 00:13:06.820 --> 00:13:07.060 carbon probably. On the other hand these same 00:13:08.560 --> 00:13:08.680 companies are pretty good about using the 00:13:10.260 --> 00:13:10.440 least amount of carbon necessary you know 00:13:12.340 --> 00:13:12.740 they have their own their tricks for doing 00:13:13.260 --> 00:13:13.760 things very efficiently. 00:13:22.100 --> 00:13:22.360 [Speaker 1]: Cool next question, LLMs are slow and 00:13:24.000 --> 00:13:24.340 responding. Do you think Emacs should provide 00:13:26.680 --> 00:13:27.180 more async primitives to keep it responsive? 00:13:29.380 --> 00:13:29.880 Like the URL retrieve is quite bad at 00:13:31.720 --> 00:13:31.760 building API clients with it. 00:13:31.920 --> 00:13:32.420 Building API clients with it? 00:13:36.400 --> 00:13:36.900 [Speaker 0]: Yeah. Well, OK, so first of all, 00:13:40.240 --> 00:13:40.740 people should be using the LLM client. 00:13:48.740 --> 00:13:48.900 And So right now, 1 thing I should have 00:13:50.220 --> 00:13:50.440 mentioned at the top is that there are new 00:13:52.500 --> 00:13:52.640 packages that I recorded this talk that you 00:13:54.480 --> 00:13:54.920 just saw several months ago. 00:13:57.780 --> 00:13:58.180 And so like Elama, there's this package Elama 00:13:59.700 --> 00:14:00.080 that came out that is using the LM package. 00:14:02.440 --> 00:14:02.680 And so for example, it doesn't need to worry 00:14:05.140 --> 00:14:05.580 about this sort of thing because it just uses 00:14:07.560 --> 00:14:07.920 LLM and package and the LLM package worries 00:14:11.680 --> 00:14:11.820 about this. And while I'm on the subject of 00:14:12.540 --> 00:14:12.720 things I forgot to mention, 00:14:15.140 --> 00:14:15.340 I also should just mention very quickly that 00:14:17.020 --> 00:14:17.520 there is now an open source model, 00:14:21.680 --> 00:14:21.960 Mistral. And so that's kind of this new thing 00:14:23.860 --> 00:14:24.240 on the scene that happened after I recorded 00:14:26.240 --> 00:14:26.420 my talk. And I think it's super important to 00:14:28.660 --> 00:14:28.820 the community and important that we have the 00:14:30.620 --> 00:14:31.120 opportunity to use that if we want to. 00:14:33.160 --> 00:14:33.660 Okay, but to answer the actual question, 00:14:37.660 --> 00:14:38.100 there has been some talk about the problems 00:14:40.680 --> 00:14:40.840 with URL retrieve in the URL package in 00:14:42.200 --> 00:14:42.700 general in EmacsDevEl. 00:14:46.760 --> 00:14:47.080 It's not great. I would like to have better 00:14:50.900 --> 00:14:51.040 primitives. And I've asked the author of 00:14:54.060 --> 00:14:54.560 Please PLZ to kind of provide some necessary 00:14:56.120 --> 00:14:56.620 callbacks. I think that's a great library. 00:15:00.280 --> 00:15:00.360 And I'd like to see that kind of like, 00:15:01.320 --> 00:15:01.680 It's nice that we have options, 00:15:03.340 --> 00:15:03.520 and that is an option that uses curl on the 00:15:05.140 --> 00:15:05.640 back end, and that has some benefits. 00:15:09.060 --> 00:15:09.280 So there's this big debate about whether we 00:15:10.600 --> 00:15:11.100 should have primitives or just use curl. 00:15:13.340 --> 00:15:13.420 I'm not exactly sure what the right call is, 00:15:15.320 --> 00:15:15.820 but there has been discussions about this. 00:15:19.540 --> 00:15:20.040 [Speaker 1]: Excellent. And someone commented that GPTEL 00:15:21.820 --> 00:15:22.200 is async and apparently very good at tracking 00:15:22.300 --> 00:15:22.800 the point. 00:15:26.680 --> 00:15:27.180 [Speaker 0]: Yes, yes, GPTEL has similar functionalities 00:15:29.800 --> 00:15:30.040 to LLM, although I believe it's going to move 00:15:33.040 --> 00:15:33.540 to LLM itself sometime soon. 00:15:39.480 --> 00:15:39.860 [Speaker 1]: Next question, speaking of which, 00:15:42.440 --> 00:15:42.560 anyone trained or fine-tuned or prompted a 00:15:44.680 --> 00:15:44.760 model with their org data yet and applied it 00:15:46.560 --> 00:15:47.040 to interesting use cases like planning, 00:15:47.920 --> 00:15:48.340 scheduling, et cetera, 00:15:49.320 --> 00:15:49.820 and maybe care to comment? 00:15:54.620 --> 00:15:55.120 [Speaker 0]: I don't know anyone who is doing that. 00:15:55.860 --> 00:15:56.360 I think it is interesting. 00:15:57.800 --> 00:15:58.300 Like this is what I kind of mentioned at the 00:16:01.060 --> 00:16:01.300 very end of the talk. There is a lot of stuff 00:16:02.440 --> 00:16:02.540 there like you could you know if you 00:16:04.760 --> 00:16:04.920 especially mean an LLM can kind of work as 00:16:07.940 --> 00:16:08.160 sort of like a secretary kind of person that 00:16:12.180 --> 00:16:12.440 could help you prioritize Still it's a 00:16:14.760 --> 00:16:14.920 slightly unclear how what the best way to use 00:16:16.480 --> 00:16:16.720 it is So I think there's more of a question 00:16:18.340 --> 00:16:18.480 for the community about like what people have 00:16:21.140 --> 00:16:21.320 been trying. I see someone has mentioned that 00:16:23.400 --> 00:16:23.900 they are using it for weekly review. 00:16:26.940 --> 00:16:27.180 And it's kind of nice to like, 00:16:29.060 --> 00:16:29.380 maybe you could read your agenda or maybe 00:16:30.480 --> 00:16:30.780 this for like weekly review. 00:16:32.040 --> 00:16:32.240 It could like read all the stuff you've done 00:16:33.340 --> 00:16:33.480 and ask you questions about it. 00:16:35.020 --> 00:16:35.280 And like, what should happen next? 00:16:36.520 --> 00:16:36.780 Or like, is this going to cause a problem? 00:16:39.060 --> 00:16:39.280 Like, I can, I can understand if that could 00:16:40.860 --> 00:16:41.180 happen? That's like, that's kind of nice. 00:16:43.660 --> 00:16:44.160 And this kind of people have had good success 00:16:48.540 --> 00:16:48.760 out of using these LLMs to bounce ideas off 00:16:49.920 --> 00:16:50.420 of are, you know, for, 00:16:52.680 --> 00:16:52.800 you know, I've seen people say that like they 00:16:55.360 --> 00:16:55.600 want, they use it for reading and they kind 00:16:58.520 --> 00:16:58.740 of dialogue with the LM to kind of like do 00:16:59.500 --> 00:17:00.000 sort of active reading. 00:17:02.500 --> 00:17:02.860 So you can imagine doing something similar 00:17:04.400 --> 00:17:04.740 with your tasks where it's sort of you're 00:17:06.560 --> 00:17:06.760 engaged in dialogue about like planning your 00:17:08.880 --> 00:17:09.000 tax with some with a alum that could kind of 00:17:10.800 --> 00:17:11.180 understand what those are and ask you some 00:17:13.780 --> 00:17:13.940 questions I think it. You know, 00:17:16.839 --> 00:17:17.040 if it'd be nice. So, the problem is like 00:17:18.480 --> 00:17:18.980 there's no great way to share all this stuff. 00:17:20.720 --> 00:17:21.220 I guess if you have something like this, 00:17:23.300 --> 00:17:23.720 put it on Reddit. If you don't have Reddit, 00:17:24.599 --> 00:17:24.880 I don't know what to do. 00:17:26.000 --> 00:17:26.500 I would say put it somewhere. 00:17:28.840 --> 00:17:29.020 At the very least, I could maybe open up like 00:17:31.320 --> 00:17:31.820 an LLM discussion session on the LLM package 00:17:34.000 --> 00:17:34.500 GitHub, But not everyone likes to use GitHub. 00:17:36.100 --> 00:17:36.180 I don't know. It'd be nice if there's a 00:17:38.940 --> 00:17:39.060 mailing list or IRC chat for this sort of 00:17:40.840 --> 00:17:41.340 thing. But there isn't at the moment. 00:17:46.560 --> 00:17:46.720 [Speaker 1]: All right. Let's see. I think that's the end 00:17:48.080 --> 00:17:48.580 of the questions on the pad so far. 00:17:51.020 --> 00:17:51.180 There was also some discussion or some 00:17:52.260 --> 00:17:52.760 chatter, I believe, on IRC. 00:17:54.560 --> 00:17:54.820 I'm not sure. Andrew, are you on IRC right 00:18:00.060 --> 00:18:00.260 [Speaker 0]: I am, but I don't think I'm on any place that 00:18:01.400 --> 00:18:01.640 has the chatter. So if there's chatter, 00:18:02.440 --> 00:18:02.940 then I'm not seeing it. 00:18:04.600 --> 00:18:05.100 [Speaker 1]: now? Okay. Yeah, it was in the emacsconf-dev 00:18:06.760 --> 00:18:07.260 channel. 00:18:09.600 --> 00:18:10.100 [Speaker 0]: Okay, let me see if I can. 00:18:25.600 --> 00:18:25.840 Oh, yes. I mean, I could see the channel, 00:18:27.520 --> 00:18:27.840 but I missed whatever came before. 00:18:29.340 --> 00:18:29.480 So if there's anything you want to kind of 00:18:30.840 --> 00:18:31.340 call out, I can try to answer it here. 00:18:35.320 --> 00:18:35.640 [Speaker 1]: OK, cool. I believe at least 2 other folks 00:18:37.500 --> 00:18:38.000 who are participating in the discussion there 00:18:40.120 --> 00:18:40.620 who have also joined here on BigBlueButton, 00:18:42.440 --> 00:18:42.940 Codin Quark and AeonTurn92. 00:18:47.000 --> 00:18:47.480 So you folks, if Andrew is still available 00:18:50.460 --> 00:18:50.640 and has time, you're welcome to chat here and 00:18:53.000 --> 00:18:53.320 ask questions or discuss here as well. 00:18:55.580 --> 00:18:55.840 [Speaker 0]: 1 Thank you. Thank you for your help, 00:18:57.740 --> 00:18:58.080 and thank you for reading all the questions. 00:18:59.700 --> 00:18:59.820 [Speaker 1]: AUDIENCE 2 Cheers, and thanks to you for a 00:19:00.540 --> 00:19:01.040 great talk and the discussion. 00:19:01.880 --> 00:19:02.380 [Speaker 0]: AUDIENCE AUDIENCE 1 Thank you. 00:19:03.140 --> 00:19:03.640 [Speaker 1]: AUDIENCE 2 Cheers. 00:19:07.900 --> 00:19:08.040 [Speaker 0]: So I'll just, I will wait here and see if 00:19:08.320 --> 00:19:08.760 there's any questions. 00:19:10.760 --> 00:19:11.260 If not, I will log off after a few minutes. 00:19:15.900 --> 00:19:16.080 [Speaker 2]: Well, I guess since we were mentioned that 00:19:18.480 --> 00:19:18.980 there was a small chat about local alarms. 00:19:22.640 --> 00:19:23.000 Because chat dpt is nice, 00:19:25.600 --> 00:19:26.100 no, but privacy concerns, 00:19:27.380 --> 00:19:27.880 and it's not free and stuff. 00:19:31.000 --> 00:19:31.500 Which, so The question is, 00:19:36.960 --> 00:19:37.460 what is the promise for local models? 00:19:39.660 --> 00:19:40.160 [Speaker 0]: Yeah, so local is definitely... 00:19:41.380 --> 00:19:41.880 [Speaker 2]: Or at least open source. 00:19:45.680 --> 00:19:46.120 [Speaker 0]: Yeah, so there is a local open source model, 00:19:47.960 --> 00:19:48.460 Misral, which you could run. 00:19:51.340 --> 00:19:51.840 The LLM package allows you to use, 00:19:56.120 --> 00:19:56.260 I think there's 3 kind of local things you 00:19:58.100 --> 00:19:58.440 could use. Like many of these things, 00:20:00.220 --> 00:20:00.480 there's like many kind of ways to do the same 00:20:03.960 --> 00:20:04.460 sort of thing. So LLM is supporting OLAMMA 00:20:10.240 --> 00:20:10.520 and LLAMMA-CPP. And let's see, 00:20:12.240 --> 00:20:12.740 1 other. Which 1 is it? 00:20:18.420 --> 00:20:18.700 And maybe that's it. Maybe the, 00:20:21.820 --> 00:20:21.940 oh, GPT for all. So each 1 of these kind of 00:20:23.100 --> 00:20:23.600 has slightly different functionality. 00:20:26.820 --> 00:20:27.180 For example, I think GPT for all doesn't 00:20:31.780 --> 00:20:32.280 support embeddings. And I hear that Olama's 00:20:33.740 --> 00:20:34.240 embeddings are kind of currently broken. 00:20:35.920 --> 00:20:36.420 But basically they should support everything. 00:20:39.100 --> 00:20:39.600 And the open source models are, 00:20:43.180 --> 00:20:43.380 so the local models are reasonably good. 00:20:44.760 --> 00:20:44.900 Like I don't think you'd use them and be 00:20:46.200 --> 00:20:46.639 like, what is this horrible nonsense? 00:20:50.200 --> 00:20:50.380 Like it's, it gives you relatively good 00:20:51.820 --> 00:20:52.120 results. Like it's not gonna be at the level 00:20:56.060 --> 00:20:56.320 of like GPT 3.5 or 4, but it's not far away 00:20:57.720 --> 00:20:58.220 from GPT 3.5, I think. 00:21:02.380 --> 00:21:02.880 [Speaker 2]: I'm just saying that Olam has like a presets 00:21:05.940 --> 00:21:06.300 for connecting the actual working servers for 00:21:06.300 --> 00:21:06.800 Olama? 00:21:08.560 --> 00:21:08.760 [Speaker 0]: So, I'll try. Yeah, so you could, 00:21:09.860 --> 00:21:10.040 what you could do is you could like for 00:21:11.940 --> 00:21:12.100 example you could download Olama which is 00:21:15.780 --> 00:21:15.940 just a way of setting up local models and 00:21:17.320 --> 00:21:17.780 running local models on your machine. 00:21:18.580 --> 00:21:18.820 So typically what it does, 00:21:19.720 --> 00:21:20.020 you like download a program, 00:21:23.720 --> 00:21:23.800 let's say Olama. Then Olama will have the 00:21:24.940 --> 00:21:25.440 ability to download models. 00:21:27.240 --> 00:21:27.360 And so you could choose from just a host of 00:21:29.280 --> 00:21:29.440 different models. Each 1 of these things has 00:21:30.200 --> 00:21:30.440 a bunch of different models. 00:21:31.920 --> 00:21:32.080 So it downloads all these things to your 00:21:36.600 --> 00:21:37.020 machine. But I would say that the key problem 00:21:40.200 --> 00:21:40.580 here is that it requires a fairly beefy 00:21:40.580 --> 00:21:41.080 machine. 00:21:42.600 --> 00:21:43.100 [Speaker 2]: So. Yeah, yeah, of course. 00:21:45.060 --> 00:21:45.300 Why I was asking, because you briefly 00:21:46.440 --> 00:21:46.880 mentioned that there are some Israeli 00:21:52.300 --> 00:21:52.440 servers. I understand that they run it like a 00:21:53.680 --> 00:21:54.000 government or stuff like that? 00:21:55.440 --> 00:21:55.940 No, no, sorry. People want everyone? 00:21:59.340 --> 00:21:59.620 [Speaker 0]: I don't, I mean, maybe you've said something 00:22:00.620 --> 00:22:01.020 that sounded like Israeli servers. 00:22:01.620 --> 00:22:02.120 [Speaker 2]: Okay, okay. 00:22:04.920 --> 00:22:05.080 [Speaker 0]: I think- There's no government LLMs as far as 00:22:06.820 --> 00:22:07.280 I know. Although, I'm sure the governments 00:22:08.200 --> 00:22:08.700 are working on their own LLMs, 00:22:10.980 --> 00:22:11.480 et cetera. But yeah, basically your choices 00:22:15.060 --> 00:22:15.220 are spend a, I mean, if you use open AI or 00:22:16.080 --> 00:22:16.580 something or anything else, 00:22:17.960 --> 00:22:18.460 you're really not spending any money. 00:22:20.560 --> 00:22:20.840 Like I've never been able to spend any money 00:22:23.680 --> 00:22:24.020 on OpenAI. Like unless you're doing something 00:22:25.840 --> 00:22:26.280 very intensive and really are using it to, 00:22:28.000 --> 00:22:28.180 you know, if you're using it for your 00:22:29.620 --> 00:22:29.780 personal use, it's just hard to spend any 00:22:31.720 --> 00:22:31.960 money. But on the other hand, 00:22:32.780 --> 00:22:32.860 it's not free. So you can, 00:22:33.040 --> 00:22:33.540 you know, 00:22:36.300 --> 00:22:36.680 [Speaker 2]: Actually, it's rather cheap. 00:22:37.680 --> 00:22:38.180 There's no question about that. 00:22:40.580 --> 00:22:40.920 The problem is that it has a bad track record 00:22:41.580 --> 00:22:42.080 on privacy. 00:22:45.540 --> 00:22:46.040 [Speaker 0]: Yes, that's, I think that is a key problem. 00:22:48.120 --> 00:22:48.280 This is probably the number 1 reason why you 00:22:51.840 --> 00:22:52.340 might want to use a local AI, 00:22:54.720 --> 00:22:55.220 a local LLM. Another 1 is like, 00:22:57.400 --> 00:22:57.900 you may not agree with the decisions. 00:23:00.360 --> 00:23:00.820 You know, there's a lot of trust and safety 00:23:05.140 --> 00:23:05.440 stuff that these companies have to do. 00:23:09.020 --> 00:23:09.240 Like they don't want like the LMs to kind of 00:23:11.400 --> 00:23:11.640 like give you, like tell you how you can make 00:23:13.180 --> 00:23:13.580 meth or how you can make a bomb, 00:23:14.960 --> 00:23:15.460 which they would do. They would totally do 00:23:19.580 --> 00:23:20.080 it. So, But each time you kind of restrict 00:23:22.540 --> 00:23:22.680 what is happening with what you can get out 00:23:23.860 --> 00:23:24.360 of the LM, it gets a little worse. 00:23:24.960 --> 00:23:25.440 So some people 00:23:27.040 --> 00:23:27.540 [Speaker 2]: want to have local. That's expected. 00:23:31.080 --> 00:23:31.400 I guess even open source language modules 00:23:33.540 --> 00:23:34.040 will soon have HR spaces because it's simply 00:23:34.720 --> 00:23:35.220 a legal issue. 00:23:40.760 --> 00:23:41.140 [Speaker 0]: I think that's true. I also think that there 00:23:42.880 --> 00:23:43.040 probably will be, although I don't know of 00:23:45.060 --> 00:23:45.300 any offhand, that will are completely 00:23:46.960 --> 00:23:47.200 uncensored. I know people are interested and 00:23:48.240 --> 00:23:48.480 are running uncensored models. 00:23:49.440 --> 00:23:49.940 I don't know how to do it. 00:23:52.280 --> 00:23:52.780 I think it's a little bit dubious, 00:23:54.960 --> 00:23:55.040 but some people do want to do it. 00:23:56.280 --> 00:23:56.780 There's another reason for using local 00:24:02.280 --> 00:24:02.780 servers. Do you have any recommendation for 00:24:05.500 --> 00:24:05.720 models to run locally and also comments on 00:24:06.780 --> 00:24:07.280 whether a GPU is required? 00:24:14.040 --> 00:24:14.160 Usually a GPU, well, you can run it without a 00:24:16.960 --> 00:24:17.460 GPU, but it does run much better. 00:24:19.480 --> 00:24:19.980 Like for example, I think when I used, 00:24:22.560 --> 00:24:23.060 Lama is sort of like a standard. 00:24:27.160 --> 00:24:27.320 This was the model for that Facebook came out 00:24:31.380 --> 00:24:31.880 with for local use. And It was, 00:24:37.260 --> 00:24:37.760 yeah, it's good. It's, 00:24:40.400 --> 00:24:40.900 but it's now it's I think, 00:24:44.620 --> 00:24:44.920 Mistral is kind of like has a better 00:24:46.480 --> 00:24:46.800 performance, But there's also different model 00:24:51.000 --> 00:24:51.500 sizes. There's 7B, like the Lama 7B is OK. 00:24:52.940 --> 00:24:53.440 The Mistral 7B, 7 billion, 00:24:54.800 --> 00:24:55.300 are like, basically it'll take like, 00:24:58.380 --> 00:24:58.880 you can run it with like 16 gigs of RAM, 00:25:02.720 --> 00:25:03.040 is pretty good. It's probably about as equal 00:25:06.900 --> 00:25:07.000 to the LLAMA13B. Those are the number of 00:25:08.360 --> 00:25:08.860 parameters, if I remember correctly. 00:25:10.680 --> 00:25:11.180 And then there's a 7B, 00:25:12.340 --> 00:25:12.840 which I've never been able to run. 00:25:16.120 --> 00:25:16.620 And even if the 7B, if you run it without a 00:25:19.640 --> 00:25:20.140 GPU, it takes quite a while to answer. 00:25:22.080 --> 00:25:22.580 I think I've had experiences where it took 00:25:23.940 --> 00:25:24.440 literally like several, 00:25:26.480 --> 00:25:26.780 like 5 minutes before it even started 00:25:28.880 --> 00:25:29.100 responding, but you do eventually get 00:25:32.220 --> 00:25:32.580 something. And it could be that like things 00:25:33.840 --> 00:25:33.960 have gotten better since the last time I 00:25:35.440 --> 00:25:35.940 tried this, because things are moving fast. 00:25:38.360 --> 00:25:38.860 But it is super recommended to have a GPU. 00:25:42.440 --> 00:25:42.620 This is the problem. It's kind of like, 00:25:43.840 --> 00:25:44.180 yes, free software is great. 00:25:46.880 --> 00:25:47.120 But if free software is requiring that you 00:25:50.460 --> 00:25:50.760 have these kind of beefy servers and have all 00:25:52.000 --> 00:25:52.500 this hardware, that's not great. 00:25:53.600 --> 00:25:54.100 I think there's a case to be made. 00:25:55.680 --> 00:25:56.180 [Speaker 1]: a hardware 00:25:59.040 --> 00:25:59.540 [Speaker 0]: with slots instead of a laptop. 00:26:01.560 --> 00:26:02.060 Yeah, yeah, that's right. 00:26:03.660 --> 00:26:03.960 [Speaker 2]: Ideally, you can have Ideally, 00:26:07.400 --> 00:26:07.660 it would be nice if FSL for all things could 00:26:12.040 --> 00:26:12.540 run something for open source model. 00:26:16.320 --> 00:26:16.640 And not free, but the key point is that it's 00:26:16.640 --> 00:26:17.140 Libre? 00:26:22.580 --> 00:26:23.080 [Speaker 0]: Yes, so actually I think Google does do that. 00:26:24.720 --> 00:26:24.800 I'll have to look it up, 00:26:27.560 --> 00:26:27.820 but I haven't explored this yet. 00:26:31.220 --> 00:26:31.720 But Google's server, which LLM does support, 00:26:33.800 --> 00:26:34.300 supports arbitrary models. 00:26:36.420 --> 00:26:36.600 So you can run LLMA or things like that. 00:26:38.940 --> 00:26:39.200 The problem is that even if you're running 00:26:40.320 --> 00:26:40.820 Mistral, which has no restrictions. 00:26:42.940 --> 00:26:43.140 So this is the kind of thing that like the 00:26:44.900 --> 00:26:45.060 Free Software Foundation cares a lot about. 00:26:47.240 --> 00:26:47.740 Like you want it to be like no restrictions, 00:26:49.740 --> 00:26:49.840 legal restrictions on you as you run the 00:26:52.080 --> 00:26:52.580 model. So even if it's running Mistral, 00:26:54.800 --> 00:26:55.300 just by using the server, 00:26:58.460 --> 00:26:58.660 the company server, it will impose some 00:26:59.440 --> 00:26:59.900 restrictions on you probably, 00:27:02.320 --> 00:27:02.480 right? There's gonna be some license that you 00:27:04.760 --> 00:27:05.260 have to, or something you have to abide by. 00:27:08.480 --> 00:27:08.600 So I think, yes, it depends on how much you 00:27:09.280 --> 00:27:09.780 care about it, I guess. 00:27:19.500 --> 00:27:19.640 I should find out more about that and make 00:27:21.580 --> 00:27:22.080 sure that it's a good point that I should, 00:27:23.980 --> 00:27:24.180 you know, people should be able to run free 00:27:25.920 --> 00:27:26.280 models over the server. 00:27:28.320 --> 00:27:28.440 So I should make sure we support that in the 00:27:40.360 --> 00:27:40.860 LLM package. So, is there any other questions 00:27:48.240 --> 00:27:48.740 Or is otherwise we can end the session. 00:28:00.800 --> 00:28:01.040 Yeah, all right. Thank you. 00:28:02.440 --> 00:28:02.940 Thank you. Thank you everyone who listened. 00:28:04.540 --> 00:28:05.040 I'm super happy like I, 00:28:06.560 --> 00:28:07.060 the interest is great. 00:28:08.900 --> 00:28:09.220 I think there's great stuff to be done here 00:28:10.960 --> 00:28:11.140 and I'm kind of super excited what we're 00:28:11.940 --> 00:28:12.160 going to do in the next year, 00:28:13.140 --> 00:28:13.440 so hopefully, like next year, 00:28:14.600 --> 00:28:14.760 and the conference we have something even 00:28:16.440 --> 00:28:16.560 more exciting to say about LLM and how they 00:28:17.320 --> 00:28:17.820 can be used with Emacs. 00:28:19.620 --> 00:28:20.120 So thank 00:28:30.060 --> 00:28:30.560 you