diff options
Diffstat (limited to '2025/info/private-ai-after.md')
| -rw-r--r-- | 2025/info/private-ai-after.md | 633 |
1 files changed, 0 insertions, 633 deletions
diff --git a/2025/info/private-ai-after.md b/2025/info/private-ai-after.md index f1a8149a..f256cbee 100644 --- a/2025/info/private-ai-after.md +++ b/2025/info/private-ai-after.md @@ -906,639 +906,6 @@ [[!template text="""Okay, I'm gonna go ahead and end the room there, thank you.""" start="00:39:29.940" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Excellent, thanks, bye.""" start="00:39:32.580" video="mainVideo-private-ai" id="subtitle"]] -</div><div class="transcript transcript-qanda"><a name="private-ai-qanda-transcript"></a><h1>Q&A transcript (unedited)</h1> - -[[!template text="""Hey, everybody. Welcome from frigid Omaha, Nebraska. I'm""" start="00:00:26.592" video="qanda-private-ai" id="subtitle"]] -[[!template text="""just going to kick off my talk here, and we'll see how it all""" start="00:00:30.007" video="qanda-private-ai" id="subtitle"]] -[[!template text="""goes. Thanks for attending.""" start="00:00:30.007" video="qanda-private-ai" id="subtitle"]] -[[!template text="""So the slides will be available on my site, growthy.us, in""" start="00:00:49.947" video="qanda-private-ai" id="subtitle"]] -[[!template text="""the presentation section tonight or tomorrow. This is a""" start="00:00:49.947" video="qanda-private-ai" id="subtitle"]] -[[!template text="""quick intro to one way to do private AI in Emacs. There are a""" start="00:00:55.997" video="qanda-private-ai" id="subtitle"]] -[[!template text="""lot of other ways to do it. This one is really just more or less""" start="00:00:59.162" video="qanda-private-ai" id="subtitle"]] -[[!template text="""the easiest way to do it. It's a minimal viable product to get""" start="00:01:01.446" video="qanda-private-ai" id="subtitle"]] -[[!template text="""you an idea of how to get started with it and how to give it a""" start="00:01:05.192" video="qanda-private-ai" id="subtitle"]] -[[!template text="""spin. Really hope some of you give it a shot and learn""" start="00:01:05.192" video="qanda-private-ai" id="subtitle"]] -[[!template text="""something along the way. So the overview of the talk. broke""" start="00:01:09.940" video="qanda-private-ai" id="subtitle"]] -[[!template text="""down these basic bullet points of why private AI, what do I""" start="00:01:16.289" video="qanda-private-ai" id="subtitle"]] -[[!template text="""need to do private AI, Emacs and private AI, pieces for an AI""" start="00:01:16.289" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Emacs solution, a demo of a minimal viable product, and the""" start="00:01:16.289" video="qanda-private-ai" id="subtitle"]] -[[!template text="""summary. Why private AI? This is pretty simple. Just read""" start="00:01:16.289" video="qanda-private-ai" id="subtitle"]] -[[!template text="""the terms and conditions for any AI system you're currently""" start="00:01:36.866" video="qanda-private-ai" id="subtitle"]] -[[!template text="""using. If you're using the free tiers, your queries, code""" start="00:01:36.866" video="qanda-private-ai" id="subtitle"]] -[[!template text="""uploaded information is being used to train the models. In""" start="00:01:40.951" video="qanda-private-ai" id="subtitle"]] -[[!template text="""some cases, you are giving the company a perpetual license""" start="00:01:46.819" video="qanda-private-ai" id="subtitle"]] -[[!template text="""to your data. You have no control over this, except for not""" start="00:01:46.819" video="qanda-private-ai" id="subtitle"]] -[[!template text="""using the engine. And keep in mind, the terms are changing""" start="00:01:51.505" video="qanda-private-ai" id="subtitle"]] -[[!template text="""all the time on that, and they're not normally changing for""" start="00:01:55.430" video="qanda-private-ai" id="subtitle"]] -[[!template text="""our benefit. So that's not necessarily a good thing. If""" start="00:01:55.430" video="qanda-private-ai" id="subtitle"]] -[[!template text="""you're using the paid tiers, you may be able to opt out of the""" start="00:02:04.298" video="qanda-private-ai" id="subtitle"]] -[[!template text="""data collection. But keep in mind, this can change, or they""" start="00:02:04.298" video="qanda-private-ai" id="subtitle"]] -[[!template text="""may start charging for that option. Every AI company wants""" start="00:02:09.496" video="qanda-private-ai" id="subtitle"]] -[[!template text="""more and more data. They need more and more data to train""" start="00:02:14.821" video="qanda-private-ai" id="subtitle"]] -[[!template text="""their models. It is just the way it is. They need more and more""" start="00:02:17.344" video="qanda-private-ai" id="subtitle"]] -[[!template text="""information to get it more and more accurate to keep it up to""" start="00:02:22.689" video="qanda-private-ai" id="subtitle"]] -[[!template text="""date. There's been a story about Stack Overflow. It has like""" start="00:02:22.689" video="qanda-private-ai" id="subtitle"]] -[[!template text="""half the number of queries they had a year ago because people""" start="00:02:29.396" video="qanda-private-ai" id="subtitle"]] -[[!template text="""are using AI. The problem with that is now there's less data""" start="00:02:29.396" video="qanda-private-ai" id="subtitle"]] -[[!template text="""going to Stack Overflow for the AI to get. vicious cycle,""" start="00:02:33.500" video="qanda-private-ai" id="subtitle"]] -[[!template text="""especially when you start looking at newer language like""" start="00:02:38.926" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Ruby and stuff like that. So it comes down to being an""" start="00:02:38.926" video="qanda-private-ai" id="subtitle"]] -[[!template text="""interesting time. Another reason why to go private AI is""" start="00:02:42.732" video="qanda-private-ai" id="subtitle"]] -[[!template text="""your costs are going to vary. Right now, these services are""" start="00:02:46.718" video="qanda-private-ai" id="subtitle"]] -[[!template text="""being heavily subsidized. If you're paying Claude $20 a""" start="00:02:50.824" video="qanda-private-ai" id="subtitle"]] -[[!template text="""month, it is not costing Claude, those guys $20 a month to""" start="00:02:53.067" video="qanda-private-ai" id="subtitle"]] -[[!template text="""host all the infrastructure to build all these data""" start="00:02:53.067" video="qanda-private-ai" id="subtitle"]] -[[!template text="""centers. They are severely subsidizing that at a very much a""" start="00:02:53.067" video="qanda-private-ai" id="subtitle"]] -[[!template text="""loss right now. When they start charging the real costs plus""" start="00:03:02.241" video="qanda-private-ai" id="subtitle"]] -[[!template text="""a profit, it's going to change. Right now, I use a bunch of""" start="00:03:07.327" video="qanda-private-ai" id="subtitle"]] -[[!template text="""different services. I've played with Grok and a bunch of""" start="00:03:11.591" video="qanda-private-ai" id="subtitle"]] -[[!template text="""other ones. But Grok right now is like $30 a month for a""" start="00:03:14.114" video="qanda-private-ai" id="subtitle"]] -[[!template text="""regular Super Grok. When they start charging the real cost""" start="00:03:16.696" video="qanda-private-ai" id="subtitle"]] -[[!template text="""of that, it's going to go from $30 to something a great deal""" start="00:03:20.240" video="qanda-private-ai" id="subtitle"]] -[[!template text="""more, perhaps, I think, $100 or $200 or whatever really""" start="00:03:20.240" video="qanda-private-ai" id="subtitle"]] -[[!template text="""turns out to be the cost when you figure everything into it.""" start="00:03:20.240" video="qanda-private-ai" id="subtitle"]] -[[!template text="""When you start adding that cost into that, a lot of people are""" start="00:03:32.032" video="qanda-private-ai" id="subtitle"]] -[[!template text="""using public AI right now are going to have no option but to""" start="00:03:32.032" video="qanda-private-ai" id="subtitle"]] -[[!template text="""move to private AI or give up on AI overall.""" start="00:03:32.032" video="qanda-private-ai" id="subtitle"]] -[[!template text="""What do you need to be able to do private AI? If you're going to""" start="00:03:42.275" video="qanda-private-ai" id="subtitle"]] -[[!template text="""run your own AI, you're going to need a system with either""" start="00:03:45.768" video="qanda-private-ai" id="subtitle"]] -[[!template text="""some cores, a graphics processor unit, or a neural""" start="00:03:45.768" video="qanda-private-ai" id="subtitle"]] -[[!template text="""processing unit, a GPU or an NPU. I currently have four""" start="00:03:45.768" video="qanda-private-ai" id="subtitle"]] -[[!template text="""systems I'm experimenting with and playing around with on a""" start="00:03:54.519" video="qanda-private-ai" id="subtitle"]] -[[!template text="""daily basis. I have a System76 Pangolin AMD Ryzen 7 78040U""" start="00:03:54.519" video="qanda-private-ai" id="subtitle"]] -[[!template text="""with a Radeon 7080M integrated graphics card. It's got 32""" start="00:03:59.145" video="qanda-private-ai" id="subtitle"]] -[[!template text="""gigs of RAM. It's a beautiful piece of hardware. I really do""" start="00:04:07.176" video="qanda-private-ai" id="subtitle"]] -[[!template text="""like it. I have my main workstation, it's an HP Z620 with dual""" start="00:04:10.139" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Intel Xeons with four NVIDIA K2200 graphics cards in it. Why""" start="00:04:11.401" video="qanda-private-ai" id="subtitle"]] -[[!template text="""the four NVIDIA K2200 graphics card on it? Because I could""" start="00:04:19.757" video="qanda-private-ai" id="subtitle"]] -[[!template text="""buy four of them on eBay for $100 and it was still supported by""" start="00:04:22.742" video="qanda-private-ai" id="subtitle"]] -[[!template text="""the NVIDIA drivers for Debian. So that's why that is. A""" start="00:04:22.742" video="qanda-private-ai" id="subtitle"]] -[[!template text="""MacBook Air with an M1 processor, a very nice piece of kit I""" start="00:04:30.959" video="qanda-private-ai" id="subtitle"]] -[[!template text="""picked up a couple years ago, very cheap, but it runs AI""" start="00:04:30.959" video="qanda-private-ai" id="subtitle"]] -[[!template text="""surprisingly well, and an Acer Aspire 1 with an AMD Ryzen""" start="00:04:30.959" video="qanda-private-ai" id="subtitle"]] -[[!template text="""5700H in it. This was my old laptop. It was a sturdy beast. It""" start="00:04:30.959" video="qanda-private-ai" id="subtitle"]] -[[!template text="""was able to do enough AI to do demos and stuff, and I liked it""" start="00:04:48.104" video="qanda-private-ai" id="subtitle"]] -[[!template text="""quite a bit for that. I'm using the Pangolin for this""" start="00:04:48.104" video="qanda-private-ai" id="subtitle"]] -[[!template text="""demonstration because it's just better. Apple's M4 chip""" start="00:04:52.611" video="qanda-private-ai" id="subtitle"]] -[[!template text="""has 38 teraflops of MPU performance. The Microsoft""" start="00:04:58.887" video="qanda-private-ai" id="subtitle"]] -[[!template text="""co-pilots are now requiring 45 teraflops of MPU to be able to""" start="00:05:03.933" video="qanda-private-ai" id="subtitle"]] -[[!template text="""have the co-pilot badge on it. And Raspberry Pi's new AI top""" start="00:05:03.933" video="qanda-private-ai" id="subtitle"]] -[[!template text="""is about 18 teraflops and is $70 on top of the cost of""" start="00:05:11.161" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Raspberry Pi 5. Keep in mind Raspberry recently raised the""" start="00:05:11.161" video="qanda-private-ai" id="subtitle"]] -[[!template text="""cost of their Pi 5s because of RAM pricing, which is going to""" start="00:05:18.009" video="qanda-private-ai" id="subtitle"]] -[[!template text="""be affecting a lot of these types of solutions in the near""" start="00:05:18.009" video="qanda-private-ai" id="subtitle"]] -[[!template text="""future. But there's going to be a lot of local power""" start="00:05:18.009" video="qanda-private-ai" id="subtitle"]] -[[!template text="""available in the future. That's what it really comes down""" start="00:05:29.178" video="qanda-private-ai" id="subtitle"]] -[[!template text="""to. A lot of people are going to have PCs on their desks.""" start="00:05:32.969" video="qanda-private-ai" id="subtitle"]] -[[!template text="""They're going to run a decent private AI without much issue.""" start="00:05:37.362" video="qanda-private-ai" id="subtitle"]] -[[!template text="""So for Emacs and private AI, there's a couple popular""" start="00:05:42.408" video="qanda-private-ai" id="subtitle"]] -[[!template text="""solutions. GPTEL, which is the one we're going to talk""" start="00:05:42.408" video="qanda-private-ai" id="subtitle"]] -[[!template text="""about. It's a simple interface. It's a minimal interface.""" start="00:05:46.473" video="qanda-private-ai" id="subtitle"]] -[[!template text="""It integrates easily into your workflow. It's just, quite""" start="00:05:50.959" video="qanda-private-ai" id="subtitle"]] -[[!template text="""honestly, chef's kiss, just a beautifully well-done piece""" start="00:05:53.021" video="qanda-private-ai" id="subtitle"]] -[[!template text="""of software. OlamaBuddy has more features, a menu""" start="00:05:53.021" video="qanda-private-ai" id="subtitle"]] -[[!template text="""interface, has quick access for things like code""" start="00:05:58.048" video="qanda-private-ai" id="subtitle"]] -[[!template text="""refactoring, text-free formatting, et cetera. This is the""" start="00:05:58.048" video="qanda-private-ai" id="subtitle"]] -[[!template text="""one that you spend a little more time with, but you also get a""" start="00:06:05.337" video="qanda-private-ai" id="subtitle"]] -[[!template text="""little bit more back from it. Elama is another one, has some""" start="00:06:05.337" video="qanda-private-ai" id="subtitle"]] -[[!template text="""really good features to it, more different capabilities,""" start="00:06:10.403" video="qanda-private-ai" id="subtitle"]] -[[!template text="""but it's a different set of rules and capabilities to it.""" start="00:06:10.403" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Itermac, which is programming with your AI and Emacs. The""" start="00:06:21.595" video="qanda-private-ai" id="subtitle"]] -[[!template text="""closest thing I can come up to comparing this to is Cursor,""" start="00:06:26.020" video="qanda-private-ai" id="subtitle"]] -[[!template text="""except it's an Emacs. It's really quite well done. These are""" start="00:06:26.020" video="qanda-private-ai" id="subtitle"]] -[[!template text="""all really quite well done. There's a bunch of other""" start="00:06:32.007" video="qanda-private-ai" id="subtitle"]] -[[!template text="""projects out there. If you go out to GitHub, type Emacs AI,""" start="00:06:33.188" video="qanda-private-ai" id="subtitle"]] -[[!template text="""you'll find a lot of different options. So what is a minimal""" start="00:06:34.550" video="qanda-private-ai" id="subtitle"]] -[[!template text="""viable product that can be done? A minimal viable product to""" start="00:06:41.555" video="qanda-private-ai" id="subtitle"]] -[[!template text="""show what an AI EMX solution is can be done with only needing""" start="00:06:45.081" video="qanda-private-ai" id="subtitle"]] -[[!template text="""two pieces of software. LLAMA file, this is an amazing piece""" start="00:06:45.081" video="qanda-private-ai" id="subtitle"]] -[[!template text="""of software. This is a whole LLM contained in one file. And""" start="00:06:52.575" video="qanda-private-ai" id="subtitle"]] -[[!template text="""the same file runs on Mac OS X, Linux, Windows, and the BSDs.""" start="00:06:59.667" video="qanda-private-ai" id="subtitle"]] -[[!template text="""It's a wonderful piece of kit based on these people who""" start="00:07:06.158" video="qanda-private-ai" id="subtitle"]] -[[!template text="""created this thing called Cosmopolitan that lets you""" start="00:07:06.158" video="qanda-private-ai" id="subtitle"]] -[[!template text="""create and execute while it runs on a bunch of different""" start="00:07:06.158" video="qanda-private-ai" id="subtitle"]] -[[!template text="""systems. And GPTEL, which is an easy plug-in for Emacs,""" start="00:07:06.158" video="qanda-private-ai" id="subtitle"]] -[[!template text="""which we talked about in the last slide a bit. So setting up""" start="00:07:15.375" video="qanda-private-ai" id="subtitle"]] -[[!template text="""the LLM, you have to just go out and just hit the a page for it""" start="00:07:22.509" video="qanda-private-ai" id="subtitle"]] -[[!template text="""and go out and do a wget of it. That's all it takes there.""" start="00:07:28.585" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Chmodding it so you can actually execute the executable.""" start="00:07:33.552" video="qanda-private-ai" id="subtitle"]] -[[!template text="""And then just go ahead and actually running it. And let's go""" start="00:07:36.876" video="qanda-private-ai" id="subtitle"]] -[[!template text="""ahead and do that. I've already downloaded it because I""" start="00:07:41.743" video="qanda-private-ai" id="subtitle"]] -[[!template text="""don't want to wait. And let's just take a look at it. I've""" start="00:07:43.144" video="qanda-private-ai" id="subtitle"]] -[[!template text="""actually downloaded several of them, but let's go ahead and""" start="00:07:47.550" video="qanda-private-ai" id="subtitle"]] -[[!template text="""just run lava 3.2b with the 3 billion instructions. And""" start="00:07:47.550" video="qanda-private-ai" id="subtitle"]] -[[!template text="""that's it firing up. And it is nice enough to actually be""" start="00:07:55.771" video="qanda-private-ai" id="subtitle"]] -[[!template text="""listening in port 8080, which we'll need in a minute.""" start="00:07:57.473" video="qanda-private-ai" id="subtitle"]] -[[!template text="""So once you do that, you have to install gptel and emacs.""" start="00:08:05.764" video="qanda-private-ai" id="subtitle"]] -[[!template text="""That's as simple as firing up emacs, doing the meta x install""" start="00:08:09.849" video="qanda-private-ai" id="subtitle"]] -[[!template text="""package, and then just typing gptel if you have your""" start="00:08:09.849" video="qanda-private-ai" id="subtitle"]] -[[!template text="""repository set up right, which hopefully you do. And then""" start="00:08:09.849" video="qanda-private-ai" id="subtitle"]] -[[!template text="""you just go ahead and have it. You also have to set up a config""" start="00:08:19.141" video="qanda-private-ai" id="subtitle"]] -[[!template text="""file. Here's my example config file as it currently set up,""" start="00:08:22.450" video="qanda-private-ai" id="subtitle"]] -[[!template text="""requiring ensuring GPTEL is loaded, defining the LLAMA""" start="00:08:24.333" video="qanda-private-ai" id="subtitle"]] -[[!template text="""file backend. You can put multiple backends into it, but I""" start="00:08:24.333" video="qanda-private-ai" id="subtitle"]] -[[!template text="""just have the one defined on this example. But it's pretty""" start="00:08:32.284" video="qanda-private-ai" id="subtitle"]] -[[!template text="""straightforward. LLAMA local file, name for it, stream,""" start="00:08:36.610" video="qanda-private-ai" id="subtitle"]] -[[!template text="""protocol HTTP. If you have HTTPS set up, that's obviously""" start="00:08:38.032" video="qanda-private-ai" id="subtitle"]] -[[!template text="""preferable, but a lot of people don't for their home labs.""" start="00:08:43.882" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Host is just 127.0.0.1 port 8080. Keep in mind, some of the""" start="00:08:49.253" video="qanda-private-ai" id="subtitle"]] -[[!template text="""AIs run on a different port, so you may be 8081 if you're""" start="00:08:53.000" video="qanda-private-ai" id="subtitle"]] -[[!template text="""running OpenWebView at the same time. The key, we don't need""" start="00:08:53.000" video="qanda-private-ai" id="subtitle"]] -[[!template text="""an API key because it's a local server. And the models just,""" start="00:09:00.295" video="qanda-private-ai" id="subtitle"]] -[[!template text="""uh, we can put multiple models on there if we want to. So if we""" start="00:09:03.541" video="qanda-private-ai" id="subtitle"]] -[[!template text="""create one with additional stuff or like rag and stuff like""" start="00:09:07.525" video="qanda-private-ai" id="subtitle"]] -[[!template text="""that, we can actually name those models by their domain,""" start="00:09:07.525" video="qanda-private-ai" id="subtitle"]] -[[!template text="""which is really kind of cool. But, uh, that's all that takes.""" start="00:09:07.525" video="qanda-private-ai" id="subtitle"]] -[[!template text="""So let's go ahead and go to a quick test of it.""" start="00:09:19.198" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Oops. Alt-X, GPTEL. And we're going to just choose the""" start="00:09:30.531" video="qanda-private-ai" id="subtitle"]] -[[!template text="""default buffer to make things easier. Going to resize it up a""" start="00:09:36.171" video="qanda-private-ai" id="subtitle"]] -[[!template text="""bit. And usually the go-to question I go to is, who was David""" start="00:09:40.202" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Bowie? This one is actually a question that's turned out to""" start="00:09:42.108" video="qanda-private-ai" id="subtitle"]] -[[!template text="""be really good for figuring out whether or not AI is""" start="00:09:49.627" video="qanda-private-ai" id="subtitle"]] -[[!template text="""complete. This is one that some engines do well on, other""" start="00:09:49.627" video="qanda-private-ai" id="subtitle"]] -[[!template text="""ones don't. And we can just do, we can either do the alt X and""" start="00:09:54.453" video="qanda-private-ai" id="subtitle"]] -[[!template text="""send the GPTEL send, or we can just do control C and hit enter.""" start="00:09:57.416" video="qanda-private-ai" id="subtitle"]] -[[!template text="""We'll just do control C and enter. And now it's going ahead""" start="00:10:04.084" video="qanda-private-ai" id="subtitle"]] -[[!template text="""and hitting our local AI system running on port 8080. And""" start="00:10:06.326" video="qanda-private-ai" id="subtitle"]] -[[!template text="""that looks pretty good, but let's go ahead and say, hey, it's""" start="00:10:11.472" video="qanda-private-ai" id="subtitle"]] -[[!template text="""set to terse mode right now. Please expand upon this.""" start="00:10:11.472" video="qanda-private-ai" id="subtitle"]] -[[!template text="""And there we go. We're getting a full description of the""" start="00:10:29.182" video="qanda-private-ai" id="subtitle"]] -[[!template text="""majority of, uh, about David Bowie's life and other""" start="00:10:29.923" video="qanda-private-ai" id="subtitle"]] -[[!template text="""information about him. So very, very happy with that.""" start="00:10:29.923" video="qanda-private-ai" id="subtitle"]] -[[!template text="""One thing to keep in mind is you look at things when you're""" start="00:10:47.946" video="qanda-private-ai" id="subtitle"]] -[[!template text="""looking for hallucinations, how accurate AI is, how it's""" start="00:10:47.946" video="qanda-private-ai" id="subtitle"]] -[[!template text="""compressed is it will tend to screw up on things like how many""" start="00:10:47.946" video="qanda-private-ai" id="subtitle"]] -[[!template text="""children he had and stuff like that. Let me see if it gets to""" start="00:10:47.946" video="qanda-private-ai" id="subtitle"]] -[[!template text="""that real quick.""" start="00:10:57.257" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Is it not actually on this one? Alright, so that's the first""" start="00:11:04.349" video="qanda-private-ai" id="subtitle"]] -[[!template text="""question I always ask one. The next one is what are sea""" start="00:11:06.552" video="qanda-private-ai" id="subtitle"]] -[[!template text="""monkeys? It gives you an idea of the breadth of the system.""" start="00:11:08.355" video="qanda-private-ai" id="subtitle"]] -[[!template text="""It's querying right now. Pulls it back correctly.""" start="00:11:19.011" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Yes. And it's smart enough to actually detect David Bowie""" start="00:11:35.197" video="qanda-private-ai" id="subtitle"]] -[[!template text="""even referenced see monkeys in the song sea of love, which""" start="00:11:36.380" video="qanda-private-ai" id="subtitle"]] -[[!template text="""came at hit single. So it's actually keeping the context""" start="00:11:36.380" video="qanda-private-ai" id="subtitle"]] -[[!template text="""alive and that which is very cool feature. I did not see that""" start="00:11:42.654" video="qanda-private-ai" id="subtitle"]] -[[!template text="""coming. Here's one that some people say is a really good one""" start="00:11:46.482" video="qanda-private-ai" id="subtitle"]] -[[!template text="""to ask ours in strawberry.""" start="00:11:48.206" video="qanda-private-ai" id="subtitle"]] -[[!template text="""All right, now she's going off the reservation. She's going""" start="00:12:09.571" video="qanda-private-ai" id="subtitle"]] -[[!template text="""in a different direction. Let me go ahead and reopen that""" start="00:12:12.376" video="qanda-private-ai" id="subtitle"]] -[[!template text="""again, because it's went down a bad hole there for a second.""" start="00:12:14.640" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Let me ask it to do write hello world in Emacs list.""" start="00:12:23.615" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Yep, that works. So the point being here, that was like two""" start="00:12:36.166" video="qanda-private-ai" id="subtitle"]] -[[!template text="""minutes of setup. And now we have a small AI embedded inside""" start="00:12:37.989" video="qanda-private-ai" id="subtitle"]] -[[!template text="""the system. So that gives you an idea just how easy it can be.""" start="00:12:41.695" video="qanda-private-ai" id="subtitle"]] -[[!template text="""And it's just running locally on the system. We also have the""" start="00:12:46.883" video="qanda-private-ai" id="subtitle"]] -[[!template text="""default system here as well. So not that bad.""" start="00:12:48.466" video="qanda-private-ai" id="subtitle"]] -[[!template text="""That's a basic solution, that's a basic setup that will get""" start="00:12:58.289" video="qanda-private-ai" id="subtitle"]] -[[!template text="""you to the point where you can go like, it's a party trick, but""" start="00:12:58.289" video="qanda-private-ai" id="subtitle"]] -[[!template text="""it's a very cool party trick. The way that GPTEL works is it""" start="00:12:58.289" video="qanda-private-ai" id="subtitle"]] -[[!template text="""puts it into buffers, it doesn't interfere with your flow""" start="00:13:06.422" video="qanda-private-ai" id="subtitle"]] -[[!template text="""that much, it's just an additional window you can pop open to""" start="00:13:06.422" video="qanda-private-ai" id="subtitle"]] -[[!template text="""ask questions and get information for, dump code into it and""" start="00:13:06.422" video="qanda-private-ai" id="subtitle"]] -[[!template text="""have it refactored. GPTEL has a lot of additional options""" start="00:13:06.422" video="qanda-private-ai" id="subtitle"]] -[[!template text="""for things that are really cool for that. But if you want a""" start="00:13:17.639" video="qanda-private-ai" id="subtitle"]] -[[!template text="""better solution, I recommend OLAM or LM Studio. They're""" start="00:13:21.886" video="qanda-private-ai" id="subtitle"]] -[[!template text="""both more capable than LAMA file. They can accept a lot of""" start="00:13:26.052" video="qanda-private-ai" id="subtitle"]] -[[!template text="""different models. You can do things like RAG. You can do""" start="00:13:28.355" video="qanda-private-ai" id="subtitle"]] -[[!template text="""loading of things onto the GPU more explicitly. It can speed""" start="00:13:32.240" video="qanda-private-ai" id="subtitle"]] -[[!template text="""stuff up. One of the things about the retrieval""" start="00:13:35.444" video="qanda-private-ai" id="subtitle"]] -[[!template text="""augmentation is it will let you put your data into the system""" start="00:13:36.686" video="qanda-private-ai" id="subtitle"]] -[[!template text="""so you can start uploading your code, your information, and""" start="00:13:36.686" video="qanda-private-ai" id="subtitle"]] -[[!template text="""actually being able to do analysis of it. OpenWebUI""" start="00:13:36.686" video="qanda-private-ai" id="subtitle"]] -[[!template text="""provides more capabilities. It provides an interface""" start="00:13:46.518" video="qanda-private-ai" id="subtitle"]] -[[!template text="""that's similar to what you're used to seeing for chat, GPT,""" start="00:13:49.562" video="qanda-private-ai" id="subtitle"]] -[[!template text="""and the other systems. It's really quite well done. And once""" start="00:13:49.562" video="qanda-private-ai" id="subtitle"]] -[[!template text="""again, GPTEL, I have to mention that because that's the one I""" start="00:13:56.149" video="qanda-private-ai" id="subtitle"]] -[[!template text="""really kind of like. And OlamaBuddy is also another really""" start="00:13:56.149" video="qanda-private-ai" id="subtitle"]] -[[!template text="""nice one. So what about the licensing of these models? Since""" start="00:14:00.454" video="qanda-private-ai" id="subtitle"]] -[[!template text="""I'm going out pulling down a model and doing this stuff.""" start="00:14:07.142" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Let's take a look at a couple of highlights from the""" start="00:14:11.307" video="qanda-private-ai" id="subtitle"]] -[[!template text="""MetaLlama 3 community license scale. If your service""" start="00:14:11.307" video="qanda-private-ai" id="subtitle"]] -[[!template text="""exceeds 700 million monthly users, you need additional""" start="00:14:15.576" video="qanda-private-ai" id="subtitle"]] -[[!template text="""licensing. Probably not going to be a problem for most of us.""" start="00:14:15.576" video="qanda-private-ai" id="subtitle"]] -[[!template text="""There's a competition restriction. You can't use this""" start="00:14:22.972" video="qanda-private-ai" id="subtitle"]] -[[!template text="""model to enhance competing models. And there's some""" start="00:14:24.576" video="qanda-private-ai" id="subtitle"]] -[[!template text="""limitations on using the meta trademarks. Not that big a""" start="00:14:27.622" video="qanda-private-ai" id="subtitle"]] -[[!template text="""deal. And the other ones are it's a permissive one designed""" start="00:14:30.629" video="qanda-private-ai" id="subtitle"]] -[[!template text="""to encourage innovation, open development, commercial""" start="00:14:32.854" video="qanda-private-ai" id="subtitle"]] -[[!template text="""use is allowed, but there are some restrictions on it. Yeah,""" start="00:14:32.854" video="qanda-private-ai" id="subtitle"]] -[[!template text="""you can modify the model, but you have to rely on the license""" start="00:14:42.172" video="qanda-private-ai" id="subtitle"]] -[[!template text="""terms. And you can distribute the model with derivatives.""" start="00:14:42.172" video="qanda-private-ai" id="subtitle"]] -[[!template text="""And there are some very cool ones out there. There's people""" start="00:14:48.542" video="qanda-private-ai" id="subtitle"]] -[[!template text="""who've done things to try and make the llama bee less, what's""" start="00:14:50.164" video="qanda-private-ai" id="subtitle"]] -[[!template text="""the phrase, ethical if you're doing penetration testing""" start="00:14:50.164" video="qanda-private-ai" id="subtitle"]] -[[!template text="""research and stuff like that. It has some very nice value""" start="00:14:50.164" video="qanda-private-ai" id="subtitle"]] -[[!template text="""there. Keep in mind licenses also vary depending on the""" start="00:14:58.517" video="qanda-private-ai" id="subtitle"]] -[[!template text="""model you're using. Mistral AI has the non-production""" start="00:15:01.021" video="qanda-private-ai" id="subtitle"]] -[[!template text="""license. It's designed to keep it to research and""" start="00:15:06.070" video="qanda-private-ai" id="subtitle"]] -[[!template text="""development. You can't use it commercially. So it's""" start="00:15:08.895" video="qanda-private-ai" id="subtitle"]] -[[!template text="""designed to clearly delineate between research and""" start="00:15:13.423" video="qanda-private-ai" id="subtitle"]] -[[!template text="""development and somebody trying to actually build""" start="00:15:13.423" video="qanda-private-ai" id="subtitle"]] -[[!template text="""something on top of it. And another question I get asked is,""" start="00:15:13.423" video="qanda-private-ai" id="subtitle"]] -[[!template text="""are there open source data model options? Yeah, but most of""" start="00:15:22.739" video="qanda-private-ai" id="subtitle"]] -[[!template text="""them are small or specialized currently. MoMo is a whole""" start="00:15:26.426" video="qanda-private-ai" id="subtitle"]] -[[!template text="""family of them, but there tend to be more specialized, but""" start="00:15:29.532" video="qanda-private-ai" id="subtitle"]] -[[!template text="""it's very cool to see where it's going. And it's another""" start="00:15:29.532" video="qanda-private-ai" id="subtitle"]] -[[!template text="""thing that's just going forward. It's under the MIT""" start="00:15:35.824" video="qanda-private-ai" id="subtitle"]] -[[!template text="""license. Some things to know to help you have a better""" start="00:15:37.548" video="qanda-private-ai" id="subtitle"]] -[[!template text="""experience with this. Get a LLAMA and OpenWebUI working by""" start="00:15:40.576" video="qanda-private-ai" id="subtitle"]] -[[!template text="""themselves, then set up your config file. I was fighting""" start="00:15:44.764" video="qanda-private-ai" id="subtitle"]] -[[!template text="""both at the same time, and it turned out I had a problem with my""" start="00:15:49.272" video="qanda-private-ai" id="subtitle"]] -[[!template text="""LLAMA. I had a conflict, so that was what my problem is. LLAMA""" start="00:15:49.272" video="qanda-private-ai" id="subtitle"]] -[[!template text="""file, GPTEL is a great way to start experimenting just to get""" start="00:15:55.725" video="qanda-private-ai" id="subtitle"]] -[[!template text="""you an idea of how it works and figure out how the interfaces""" start="00:15:55.725" video="qanda-private-ai" id="subtitle"]] -[[!template text="""work. Tremendous. RAG loading documents into it is really""" start="00:15:55.725" video="qanda-private-ai" id="subtitle"]] -[[!template text="""easy with open web UI. You can create models, you can put""" start="00:16:03.459" video="qanda-private-ai" id="subtitle"]] -[[!template text="""things like help desk developers and stuff like that,""" start="00:16:06.723" video="qanda-private-ai" id="subtitle"]] -[[!template text="""breaking it out. The Hacker News has a how to build a $300 AI""" start="00:16:06.723" video="qanda-private-ai" id="subtitle"]] -[[!template text="""computer. This is for March 2024, but it still has a lot of""" start="00:16:13.513" video="qanda-private-ai" id="subtitle"]] -[[!template text="""great information on how to benchmark the environments,""" start="00:16:17.199" video="qanda-private-ai" id="subtitle"]] -[[!template text="""what some values are like the Ryzen 5700U inside my Acer""" start="00:16:17.199" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Aspire, that's where I got the idea doing that. Make sure you""" start="00:16:27.674" video="qanda-private-ai" id="subtitle"]] -[[!template text="""do the RockM stuff correctly to get the GUI extensions. But""" start="00:16:30.399" video="qanda-private-ai" id="subtitle"]] -[[!template text="""it's just really good stuff. You don't need a great GPU or CPU""" start="00:16:34.886" video="qanda-private-ai" id="subtitle"]] -[[!template text="""to get started. Smaller models like Tiny Llama can run on""" start="00:16:36.870" video="qanda-private-ai" id="subtitle"]] -[[!template text="""very small systems. It gets you the ability to start playing""" start="00:16:39.334" video="qanda-private-ai" id="subtitle"]] -[[!template text="""with it and start experimenting and figure out if that's for""" start="00:16:43.521" video="qanda-private-ai" id="subtitle"]] -[[!template text="""you and to move forward with it. The AMD Ryzen AI Max 395 plus""" start="00:16:43.521" video="qanda-private-ai" id="subtitle"]] -[[!template text="""is a mini PC makes it really nice dedicated host. You used to""" start="00:16:51.232" video="qanda-private-ai" id="subtitle"]] -[[!template text="""be able to buy these for about $1200 now with the RAM price""" start="00:16:58.033" video="qanda-private-ai" id="subtitle"]] -[[!template text="""increase, you want to get 120 gig when you're pushing two""" start="00:16:58.033" video="qanda-private-ai" id="subtitle"]] -[[!template text="""brands so. It gets a little tighter. Macs work remarkably""" start="00:16:58.033" video="qanda-private-ai" id="subtitle"]] -[[!template text="""well with AI. My MacBook Air was one of my go-tos for a while,""" start="00:17:07.624" video="qanda-private-ai" id="subtitle"]] -[[!template text="""but once I started doing anything AI, I had a five-minute""" start="00:17:11.010" video="qanda-private-ai" id="subtitle"]] -[[!template text="""window before the thermal throttling became an issue. Keep""" start="00:17:11.010" video="qanda-private-ai" id="subtitle"]] -[[!template text="""in mind that's a MacBook Air, so it doesn't have the greatest""" start="00:17:19.123" video="qanda-private-ai" id="subtitle"]] -[[!template text="""ventilation. If you get the MacBook Pros and stuff, they""" start="00:17:19.123" video="qanda-private-ai" id="subtitle"]] -[[!template text="""tend to have more ventilation, but still you're going to be""" start="00:17:23.130" video="qanda-private-ai" id="subtitle"]] -[[!template text="""pushing against that. So Mac Minis and the Mac Ultras and""" start="00:17:23.130" video="qanda-private-ai" id="subtitle"]] -[[!template text="""stuff like that tend to work really well for that. Alex""" start="00:17:28.418" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Ziskin on YouTube has a channel. He does a lot of AI""" start="00:17:32.525" video="qanda-private-ai" id="subtitle"]] -[[!template text="""performance benchmarking, like I load a 70 billion""" start="00:17:36.372" video="qanda-private-ai" id="subtitle"]] -[[!template text="""parameter model on this mini PC and stuff like that. It's a""" start="00:17:36.372" video="qanda-private-ai" id="subtitle"]] -[[!template text="""lot of fun and interesting stuff there. And it's""" start="00:17:42.765" video="qanda-private-ai" id="subtitle"]] -[[!template text="""influencing my decision to buy my next AI style PC. Small""" start="00:17:45.690" video="qanda-private-ai" id="subtitle"]] -[[!template text="""domain specific LLMs are happening. An LLM that has all your""" start="00:17:50.648" video="qanda-private-ai" id="subtitle"]] -[[!template text="""code and information, it sounds like a really cool idea. It""" start="00:17:54.197" video="qanda-private-ai" id="subtitle"]] -[[!template text="""gives you capabilities to start training stuff that you""" start="00:17:57.666" video="qanda-private-ai" id="subtitle"]] -[[!template text="""couldn't do with like the big ones. Even with in terms of fine""" start="00:17:57.666" video="qanda-private-ai" id="subtitle"]] -[[!template text="""tuning and stuff, it's remarkable to see where that space is""" start="00:18:02.598" video="qanda-private-ai" id="subtitle"]] -[[!template text="""coming along in the next year or so. Hugging Face Co has""" start="00:18:02.598" video="qanda-private-ai" id="subtitle"]] -[[!template text="""pointers to tons of AI models. You'll find the one that works""" start="00:18:08.435" video="qanda-private-ai" id="subtitle"]] -[[!template text="""for you, hopefully there. If you're doing cybersecurity,""" start="00:18:12.801" video="qanda-private-ai" id="subtitle"]] -[[!template text="""there's a whole bunch out there for that, that have certain""" start="00:18:14.464" video="qanda-private-ai" id="subtitle"]] -[[!template text="""training on it, information. It's really good. One last""" start="00:18:14.464" video="qanda-private-ai" id="subtitle"]] -[[!template text="""thing to keep in mind is hallucinations are real. You will""" start="00:18:23.497" video="qanda-private-ai" id="subtitle"]] -[[!template text="""get BS back from the AI occasionally, so do validate""" start="00:18:26.762" video="qanda-private-ai" id="subtitle"]] -[[!template text="""everything you get from it. Don't be using it for court cases""" start="00:18:26.762" video="qanda-private-ai" id="subtitle"]] -[[!template text="""like some people have and run into those problems. So, That""" start="00:18:31.930" video="qanda-private-ai" id="subtitle"]] -[[!template text="""is my talk. What I would like you to get out of that is, if you""" start="00:18:38.945" video="qanda-private-ai" id="subtitle"]] -[[!template text="""haven't tried it, give GPTEL and LlamaFile a shot. Fire up a""" start="00:18:41.527" video="qanda-private-ai" id="subtitle"]] -[[!template text="""little small AI instance, play around with a little bit""" start="00:18:48.473" video="qanda-private-ai" id="subtitle"]] -[[!template text="""inside your Emacs, and see if it makes your life better.""" start="00:18:48.473" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Hopefully it will. And I really hope you guys learned""" start="00:18:55.499" video="qanda-private-ai" id="subtitle"]] -[[!template text="""something from this talk. And thanks for listening. And the""" start="00:18:56.821" video="qanda-private-ai" id="subtitle"]] -[[!template text="""links are at the end of the talk, if you have any questions.""" start="00:19:01.905" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Let me see if we got anything you want, Pat. You do. You've got""" start="00:19:05.908" video="qanda-private-ai" id="subtitle"]] -[[!template text="""a few questions. Hey, this is Corwin. Thank you so much.""" start="00:19:08.683" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Thank you, Aaron. What an awesome talk this was, actually.""" start="00:19:13.069" video="qanda-private-ai" id="subtitle"]] -[[!template text="""If you don't have a camera, I can get away with not having one""" start="00:19:17.255" video="qanda-private-ai" id="subtitle"]] -[[!template text="""too. I've got, I'll turn the camera on. Okay. All right. I'll""" start="00:19:17.255" video="qanda-private-ai" id="subtitle"]] -[[!template text="""turn mine back on. Here I come. Yeah, so there are a few""" start="00:19:23.683" video="qanda-private-ai" id="subtitle"]] -[[!template text="""questions, but first let me say thank you for a really""" start="00:19:27.089" video="qanda-private-ai" id="subtitle"]] -[[!template text="""captivating talk. I think a lot of people will be empowered""" start="00:19:27.089" video="qanda-private-ai" id="subtitle"]] -[[!template text="""from this to try to do more with less, especially locally.""" start="00:19:32.887" video="qanda-private-ai" id="subtitle"]] -[[!template text="""concerned about the data center footprint,""" start="00:19:44.538" video="qanda-private-ai" id="subtitle"]] -[[!template text="""environmentally concerned about the footprint of LLM""" start="00:19:44.538" video="qanda-private-ai" id="subtitle"]] -[[!template text="""inside data centers. So just thinking about how we can put""" start="00:19:44.538" video="qanda-private-ai" id="subtitle"]] -[[!template text="""infrastructure we have at home to use and get more done with""" start="00:19:52.918" video="qanda-private-ai" id="subtitle"]] -[[!template text="""less. Yeah, the data center impact's interesting because""" start="00:19:52.918" video="qanda-private-ai" id="subtitle"]] -[[!template text="""there was a study a while ago. Someone said every time you do a""" start="00:20:01.666" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Gemini query, it's like boiling a cup of water. Yeah, I've""" start="00:20:05.871" video="qanda-private-ai" id="subtitle"]] -[[!template text="""heard that one too. So do you want to, you know, I don't know""" start="00:20:12.399" video="qanda-private-ai" id="subtitle"]] -[[!template text="""how much direction you want. I'd be very happy to read out the""" start="00:20:14.021" video="qanda-private-ai" id="subtitle"]] -[[!template text="""questions for you. Yeah, that would be great. I'm having""" start="00:20:17.625" video="qanda-private-ai" id="subtitle"]] -[[!template text="""trouble getting to that tab. Okay, I'm there, so I'll put it""" start="00:20:21.510" video="qanda-private-ai" id="subtitle"]] -[[!template text="""into our chat too, so you can follow along if you'd like. The""" start="00:20:24.093" video="qanda-private-ai" id="subtitle"]] -[[!template text="""first question was, why is the David Bowie question a good""" start="00:20:32.106" video="qanda-private-ai" id="subtitle"]] -[[!template text="""one to start with? Does it have interesting failure""" start="00:20:32.106" video="qanda-private-ai" id="subtitle"]] -[[!template text="""conditions or what made you choose that? First off, huge fan""" start="00:20:38.436" video="qanda-private-ai" id="subtitle"]] -[[!template text="""of David Bowie. But I came down to it really taught me a few""" start="00:20:42.706" video="qanda-private-ai" id="subtitle"]] -[[!template text="""things about how old the models work in terms of things like""" start="00:20:46.070" video="qanda-private-ai" id="subtitle"]] -[[!template text="""how many kids he had, because deep seek, which is a very""" start="00:20:46.070" video="qanda-private-ai" id="subtitle"]] -[[!template text="""popular Chinese model that a lot of people are using now,""" start="00:20:46.070" video="qanda-private-ai" id="subtitle"]] -[[!template text="""misidentifies him having three daughters, and he has like""" start="00:20:46.070" video="qanda-private-ai" id="subtitle"]] -[[!template text="""one son and one, one, I think, two sons and a daughter or""" start="00:20:46.070" video="qanda-private-ai" id="subtitle"]] -[[!template text="""something like that. so there's differences on that and it""" start="00:20:46.070" video="qanda-private-ai" id="subtitle"]] -[[!template text="""just goes over there's a whole lot of stuff because his story""" start="00:21:06.955" video="qanda-private-ai" id="subtitle"]] -[[!template text="""spans like 60 years so it gives a good good feedback that's""" start="00:21:06.955" video="qanda-private-ai" id="subtitle"]] -[[!template text="""the real main reason I asked that question because I just""" start="00:21:06.955" video="qanda-private-ai" id="subtitle"]] -[[!template text="""needed one that sea monkeys I just picked because it was""" start="00:21:06.955" video="qanda-private-ai" id="subtitle"]] -[[!template text="""obscure and just always have right I used to have it right""" start="00:21:06.955" video="qanda-private-ai" id="subtitle"]] -[[!template text="""hello world and forth because I thought was an interesting""" start="00:21:06.955" video="qanda-private-ai" id="subtitle"]] -[[!template text="""one as well so It's just picking random ones like that. One""" start="00:21:06.955" video="qanda-private-ai" id="subtitle"]] -[[!template text="""question asked, sorry, a lot of models is, what is the""" start="00:21:30.265" video="qanda-private-ai" id="subtitle"]] -[[!template text="""closest star to the Earth? Because most of them will say""" start="00:21:30.265" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Alpha Centauri or Proxima Centauri and not the sun. And I""" start="00:21:35.911" video="qanda-private-ai" id="subtitle"]] -[[!template text="""have a whole nother talk where I just argue with the LLM""" start="00:21:40.376" video="qanda-private-ai" id="subtitle"]] -[[!template text="""trying to say, hey, the sun is a star. And he just wouldn't""" start="00:21:40.376" video="qanda-private-ai" id="subtitle"]] -[[!template text="""accept it, so.""" start="00:21:46.383" video="qanda-private-ai" id="subtitle"]] -[[!template text="""What? Oh, I can hear that. So what specific tasks do you like""" start="00:21:53.230" video="qanda-private-ai" id="subtitle"]] -[[!template text="""to use your local AI? I like to load a lot of my code into and""" start="00:21:56.956" video="qanda-private-ai" id="subtitle"]] -[[!template text="""actually have it do analysis of it. I was actually going""" start="00:22:01.883" video="qanda-private-ai" id="subtitle"]] -[[!template text="""through some code I have for some pen testing, and I was""" start="00:22:05.207" video="qanda-private-ai" id="subtitle"]] -[[!template text="""having it modified to update it for the newer version,""" start="00:22:05.207" video="qanda-private-ai" id="subtitle"]] -[[!template text="""because I hate to say this, but it was written for Python 2,""" start="00:22:05.207" video="qanda-private-ai" id="subtitle"]] -[[!template text="""and I needed to update it for Python 3. And the 2 to 3 tool did""" start="00:22:05.207" video="qanda-private-ai" id="subtitle"]] -[[!template text="""not do all of it, but the actual tool was able to do the""" start="00:22:17.982" video="qanda-private-ai" id="subtitle"]] -[[!template text="""refactoring. It's part of my laziness. But I use that for""" start="00:22:17.982" video="qanda-private-ai" id="subtitle"]] -[[!template text="""anything I don't want to hit the web. And that's a lot of stuff""" start="00:22:24.850" video="qanda-private-ai" id="subtitle"]] -[[!template text="""when you start thinking about if you're doing cyber""" start="00:22:27.654" video="qanda-private-ai" id="subtitle"]] -[[!template text="""security researching. and you have your white papers and""" start="00:22:27.654" video="qanda-private-ai" id="subtitle"]] -[[!template text="""stuff like that and stuff in there. I've got a lot of that""" start="00:22:31.398" video="qanda-private-ai" id="subtitle"]] -[[!template text="""loaded into RAG in one model on my OpenWebUI system. Neat.""" start="00:22:34.988" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Have you used have you used any small domain specific LLMs?""" start="00:22:44.272" video="qanda-private-ai" id="subtitle"]] -[[!template text="""What kind of tasks? If so, what kind of tasks that they""" start="00:22:50.459" video="qanda-private-ai" id="subtitle"]] -[[!template text="""specialize in? And you know, how? Not to be honest, but there""" start="00:22:51.540" video="qanda-private-ai" id="subtitle"]] -[[!template text="""are some out there like once again, for cybersecurity and""" start="00:22:58.428" video="qanda-private-ai" id="subtitle"]] -[[!template text="""stuff like that, that I really need to dig into that's on my to""" start="00:22:58.428" video="qanda-private-ai" id="subtitle"]] -[[!template text="""do list. I've got a couple weeks off at the end of the year. And""" start="00:22:58.428" video="qanda-private-ai" id="subtitle"]] -[[!template text="""that's a big part of my plan for that.""" start="00:23:07.778" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Are the various models updated pretty regularly? Can you""" start="00:23:12.539" video="qanda-private-ai" id="subtitle"]] -[[!template text="""add your own data to the pre-built models? Yes. The models""" start="00:23:15.495" video="qanda-private-ai" id="subtitle"]] -[[!template text="""are updated pretty reasonably. You can add data to a model in""" start="00:23:20.622" video="qanda-private-ai" id="subtitle"]] -[[!template text="""a couple of different ways. You can do something called""" start="00:23:23.184" video="qanda-private-ai" id="subtitle"]] -[[!template text="""fine-tuning, which requires a really nice GPU and a lot of""" start="00:23:25.827" video="qanda-private-ai" id="subtitle"]] -[[!template text="""CPU time. Probably not going to do that. You can do retrieval""" start="00:23:25.827" video="qanda-private-ai" id="subtitle"]] -[[!template text="""augmentation generation, which is you load your data on top""" start="00:23:31.694" video="qanda-private-ai" id="subtitle"]] -[[!template text="""of the system and puts inside a database and you can actually""" start="00:23:31.694" video="qanda-private-ai" id="subtitle"]] -[[!template text="""scan that and stuff. I have another talk where I go through""" start="00:23:31.694" video="qanda-private-ai" id="subtitle"]] -[[!template text="""and I start asking questions about, I load the talk into the""" start="00:23:39.422" video="qanda-private-ai" id="subtitle"]] -[[!template text="""engine and I ask questions against that. I would have one""" start="00:23:39.422" video="qanda-private-ai" id="subtitle"]] -[[!template text="""more time would have done that but it comes down to how many""" start="00:23:46.390" video="qanda-private-ai" id="subtitle"]] -[[!template text="""That's that's rag rag is pretty easy to do through open web UI""" start="00:23:46.390" video="qanda-private-ai" id="subtitle"]] -[[!template text="""or LM studio It's a great way you just like point a folder""" start="00:23:46.390" video="qanda-private-ai" id="subtitle"]] -[[!template text="""point it to a folder and it just sucks all that state into and""" start="00:23:46.390" video="qanda-private-ai" id="subtitle"]] -[[!template text="""it'll hit that data first you have like helpdesk and stuff""" start="00:23:46.390" video="qanda-private-ai" id="subtitle"]] -[[!template text="""and The other options there's vector databases, which is""" start="00:23:46.390" video="qanda-private-ai" id="subtitle"]] -[[!template text="""like if you use PostgreSQL. It has a PG vector I can do a lot of""" start="00:23:46.390" video="qanda-private-ai" id="subtitle"]] -[[!template text="""that stuff. I've not dug into that yet, but that is also on""" start="00:24:07.716" video="qanda-private-ai" id="subtitle"]] -[[!template text="""that to-do list I've got a lot of stuff planned for Cool. So""" start="00:24:09.679" video="qanda-private-ai" id="subtitle"]] -[[!template text="""what are your experience with rags? I don't even know what""" start="00:24:15.279" video="qanda-private-ai" id="subtitle"]] -[[!template text="""that means. Do you know what that means? Do you""" start="00:24:17.964" video="qanda-private-ai" id="subtitle"]] -[[!template text="""remember this question again? What is your experience with""" start="00:24:20.911" video="qanda-private-ai" id="subtitle"]] -[[!template text="""RAGS? RAGS is great. That's Retrieval Augmentation""" start="00:24:26.080" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Generation. That loads your data first, and it hits yours,""" start="00:24:31.387" video="qanda-private-ai" id="subtitle"]] -[[!template text="""and it'll actually cite it and stuff. There's a guy who wrote""" start="00:24:33.911" video="qanda-private-ai" id="subtitle"]] -[[!template text="""a RAG in 100 lines of Python, and it's an impressive piece of""" start="00:24:37.656" video="qanda-private-ai" id="subtitle"]] -[[!template text="""software. I think if you hit one of my site, I've got a private""" start="00:24:37.656" video="qanda-private-ai" id="subtitle"]] -[[!template text="""AI talk where I actually refer to that. But retrieval""" start="00:24:43.685" video="qanda-private-ai" id="subtitle"]] -[[!template text="""augmentation, it's easy, it's fast, it puts your data into""" start="00:24:48.411" video="qanda-private-ai" id="subtitle"]] -[[!template text="""the system, Yeah, start with that and go then iterate on top""" start="00:24:48.411" video="qanda-private-ai" id="subtitle"]] -[[!template text="""of that. That's one of the great things about AI, especially""" start="00:24:53.468" video="qanda-private-ai" id="subtitle"]] -[[!template text="""private AI, is you can do whatever you want to with it and""" start="00:24:57.454" video="qanda-private-ai" id="subtitle"]] -[[!template text="""build up with it as you get more experience. Any thoughts on""" start="00:24:57.454" video="qanda-private-ai" id="subtitle"]] -[[!template text="""running things on AWS, DigitalOcean, and so on? AWS is not""" start="00:25:06.067" video="qanda-private-ai" id="subtitle"]] -[[!template text="""bad. The DigitalOcean, they have some of their GPOs. I still""" start="00:25:14.140" video="qanda-private-ai" id="subtitle"]] -[[!template text="""don't like having the data leave my house, to be honest, or at""" start="00:25:18.868" video="qanda-private-ai" id="subtitle"]] -[[!template text="""work, because I tend to do some stuff that I don't want it even""" start="00:25:18.868" video="qanda-private-ai" id="subtitle"]] -[[!template text="""hitting that situation. But they have pretty good stuff.""" start="00:25:18.868" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Another one to consider is Oracle Cloud. Oracle has their AI""" start="00:25:30.106" video="qanda-private-ai" id="subtitle"]] -[[!template text="""infrastructure that's really well done. But I mean, once""" start="00:25:31.829" video="qanda-private-ai" id="subtitle"]] -[[!template text="""again, then you start looking at potential is saying your""" start="00:25:35.555" video="qanda-private-ai" id="subtitle"]] -[[!template text="""data is private, I don't necessarily trust it. But they do""" start="00:25:35.555" video="qanda-private-ai" id="subtitle"]] -[[!template text="""have good stuff, both DigitalOcean, AWS, Oracle Cloud has""" start="00:25:41.323" video="qanda-private-ai" id="subtitle"]] -[[!template text="""the free service, which isn't too bad, usually a certain""" start="00:25:41.323" video="qanda-private-ai" id="subtitle"]] -[[!template text="""number of stuff. And Google's also has it, but I still tend to""" start="00:25:41.323" video="qanda-private-ai" id="subtitle"]] -[[!template text="""keep more stuff on local PCs, because I just paranoid that""" start="00:25:48.051" video="qanda-private-ai" id="subtitle"]] -[[!template text="""way. Gotcha. What has your experience been using AI? Do you""" start="00:25:48.051" video="qanda-private-ai" id="subtitle"]] -[[!template text="""want to get into that, using AI for cybersecurity? You might""" start="00:26:02.366" video="qanda-private-ai" id="subtitle"]] -[[!template text="""have already touched on this. Yeah, really, for""" start="00:26:06.251" video="qanda-private-ai" id="subtitle"]] -[[!template text="""cybersecurity, what I've had to do is I've dumped logs to""" start="00:26:08.434" video="qanda-private-ai" id="subtitle"]] -[[!template text="""have a due correlation. Keep in mind, the size of that LLAMA""" start="00:26:08.434" video="qanda-private-ai" id="subtitle"]] -[[!template text="""file we were using for figuring out David Bowie, writing the""" start="00:26:13.702" video="qanda-private-ai" id="subtitle"]] -[[!template text="""hello world, all that stuff, is like six gig. How does it get""" start="00:26:13.702" video="qanda-private-ai" id="subtitle"]] -[[!template text="""the entire world in six gig? I still haven't figured that out""" start="00:26:20.712" video="qanda-private-ai" id="subtitle"]] -[[!template text="""in terms of quantization. So I'm really interested in""" start="00:26:23.476" video="qanda-private-ai" id="subtitle"]] -[[!template text="""seeing the ability to take all this stuff out of all my logs,""" start="00:26:26.240" video="qanda-private-ai" id="subtitle"]] -[[!template text="""dump it all in there, and actually be able to do intelligent""" start="00:26:26.240" video="qanda-private-ai" id="subtitle"]] -[[!template text="""queries against that. Microsoft has a project called""" start="00:26:26.240" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Security Copilot, which is trying to do that in the Cloud.""" start="00:26:34.572" video="qanda-private-ai" id="subtitle"]] -[[!template text="""But I want to work on something to do that more locally and be""" start="00:26:39.218" video="qanda-private-ai" id="subtitle"]] -[[!template text="""able to actually drive this stuff over that. That's one also""" start="00:26:39.218" video="qanda-private-ai" id="subtitle"]] -[[!template text="""on the long-term goals.""" start="00:26:44.726" video="qanda-private-ai" id="subtitle"]] -[[!template text="""So we got any other questions or? Those are the questions""" start="00:26:50.817" video="qanda-private-ai" id="subtitle"]] -[[!template text="""that I see. I want to just read out a couple of comments that I""" start="00:26:53.640" video="qanda-private-ai" id="subtitle"]] -[[!template text="""saw in IRC though. Jay Rutabaga says, it went very well from""" start="00:26:55.161" video="qanda-private-ai" id="subtitle"]] -[[!template text="""an audience perspective. And G Gundam says, respect your""" start="00:27:00.305" video="qanda-private-ai" id="subtitle"]] -[[!template text="""commitment to privacy. And then somebody is telling us we""" start="00:27:05.090" video="qanda-private-ai" id="subtitle"]] -[[!template text="""might have skipped a question. So I'm just going to run back""" start="00:27:10.635" video="qanda-private-ai" id="subtitle"]] -[[!template text="""to my list. Updated regularly experience. I just didn't""" start="00:27:13.037" video="qanda-private-ai" id="subtitle"]] -[[!template text="""type in the answer here's and there's a couple more""" start="00:27:20.855" video="qanda-private-ai" id="subtitle"]] -[[!template text="""questions coming in so Is there a disparity where you go to""" start="00:27:20.855" video="qanda-private-ai" id="subtitle"]] -[[!template text="""paid models because they are better and what problems? You""" start="00:27:20.855" video="qanda-private-ai" id="subtitle"]] -[[!template text="""know what would drive you to? That's a good question. Paid""" start="00:27:34.735" video="qanda-private-ai" id="subtitle"]] -[[!template text="""models, I don't mind them. I think they're good, but I don't""" start="00:27:40.687" video="qanda-private-ai" id="subtitle"]] -[[!template text="""think they're actually economically sustainable under""" start="00:27:42.970" video="qanda-private-ai" id="subtitle"]] -[[!template text="""their current system. Because right now, if you're paying""" start="00:27:42.970" video="qanda-private-ai" id="subtitle"]] -[[!template text="""20 bucks a month for Copilot and that goes up to 200 bucks, I'm""" start="00:27:49.200" video="qanda-private-ai" id="subtitle"]] -[[!template text="""not going to be as likely to use it. You know what I mean? But it""" start="00:27:49.200" video="qanda-private-ai" id="subtitle"]] -[[!template text="""does do some things in a way that I did not expect. For""" start="00:27:56.030" video="qanda-private-ai" id="subtitle"]] -[[!template text="""example, Grok was refactoring some of my code in the""" start="00:27:59.475" video="qanda-private-ai" id="subtitle"]] -[[!template text="""comments and dropped an F-bomb. which I did not see coming,""" start="00:27:59.475" video="qanda-private-ai" id="subtitle"]] -[[!template text="""but the other code before that I had gotten off GitHub had F""" start="00:28:04.966" video="qanda-private-ai" id="subtitle"]] -[[!template text="""bombs in it. So it was just emulating the style, but would""" start="00:28:04.966" video="qanda-private-ai" id="subtitle"]] -[[!template text="""that be something I'd want to turn in a pull request? I don't""" start="00:28:10.493" video="qanda-private-ai" id="subtitle"]] -[[!template text="""know. But, uh, there's, there's a lot of money going into""" start="00:28:15.619" video="qanda-private-ai" id="subtitle"]] -[[!template text="""these AIs and stuff, but in terms of the ability to get a""" start="00:28:16.180" video="qanda-private-ai" id="subtitle"]] -[[!template text="""decent one, like the llama, llama three, two, and load your""" start="00:28:16.180" video="qanda-private-ai" id="subtitle"]] -[[!template text="""data into it, you can be pretty competitive. You're not""" start="00:28:16.180" video="qanda-private-ai" id="subtitle"]] -[[!template text="""going to get all the benefits, but you have more control over""" start="00:28:27.534" video="qanda-private-ai" id="subtitle"]] -[[!template text="""it. So it's, it's a, this and that it's a, it's a balancing""" start="00:28:27.534" video="qanda-private-ai" id="subtitle"]] -[[!template text="""act.""" start="00:28:30.598" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Okay, and I think I see a couple more questions coming in.""" start="00:28:37.315" video="qanda-private-ai" id="subtitle"]] -[[!template text="""What is the largest parameter size for local models that""" start="00:28:40.821" video="qanda-private-ai" id="subtitle"]] -[[!template text="""you've been able to successfully run locally and do run into""" start="00:28:40.821" video="qanda-private-ai" id="subtitle"]] -[[!template text="""issues with limited context window size? The top eight""" start="00:28:40.821" video="qanda-private-ai" id="subtitle"]] -[[!template text="""models will tend to have a larger ceiling. Yes, yes, yes,""" start="00:28:52.560" video="qanda-private-ai" id="subtitle"]] -[[!template text="""yes, yes. By default, the context size is I think 1024. But""" start="00:28:57.650" video="qanda-private-ai" id="subtitle"]] -[[!template text="""I've upped it to 8192 on the on this box, the Pangolin because""" start="00:29:03.657" video="qanda-private-ai" id="subtitle"]] -[[!template text="""it seems to be some reason it's just a very working quite""" start="00:29:03.657" video="qanda-private-ai" id="subtitle"]] -[[!template text="""well. But the largest ones I've loaded have been in the have""" start="00:29:03.657" video="qanda-private-ai" id="subtitle"]] -[[!template text="""not been that huge. I've loaded this the last biggest one""" start="00:29:11.966" video="qanda-private-ai" id="subtitle"]] -[[!template text="""I've done. That's the reason why I'm planning on breaking""" start="00:29:17.573" video="qanda-private-ai" id="subtitle"]] -[[!template text="""down and buying a Ryzen. Actually, I'm going to buy an Intel""" start="00:29:22.118" video="qanda-private-ai" id="subtitle"]] -[[!template text="""i285H with 96 gig of RAM. Then I should be able to load a 70""" start="00:29:26.483" video="qanda-private-ai" id="subtitle"]] -[[!template text="""billion parameter model in that. How fast will it run? It's""" start="00:29:33.150" video="qanda-private-ai" id="subtitle"]] -[[!template text="""going to run slow as dog, but it's going to be cool to be able to""" start="00:29:38.176" video="qanda-private-ai" id="subtitle"]] -[[!template text="""do it. It's an AI bragging rights thing, but I mostly stick""" start="00:29:38.176" video="qanda-private-ai" id="subtitle"]] -[[!template text="""with the smaller size models and the ones that are more""" start="00:29:41.580" video="qanda-private-ai" id="subtitle"]] -[[!template text="""quantitized because it just tends to work better for me.""" start="00:29:41.580" video="qanda-private-ai" id="subtitle"]] -[[!template text="""We've still got over 10 minutes before we're cutting away,""" start="00:29:50.975" video="qanda-private-ai" id="subtitle"]] -[[!template text="""but I'm just anticipating that we're going to be going""" start="00:29:50.975" video="qanda-private-ai" id="subtitle"]] -[[!template text="""strong at the 10 minute mark. So I'm just, just letting, you""" start="00:29:50.975" video="qanda-private-ai" id="subtitle"]] -[[!template text="""know, we can go as long as we like here at a certain point. I may""" start="00:29:59.065" video="qanda-private-ai" id="subtitle"]] -[[!template text="""have to jump away and check in with the next speaker, but""" start="00:30:03.691" video="qanda-private-ai" id="subtitle"]] -[[!template text="""we'll post the entirety of this, even if we aren't able to""" start="00:30:03.691" video="qanda-private-ai" id="subtitle"]] -[[!template text="""stay with it all. Okay. And we've got 10 minutes where we're""" start="00:30:03.691" video="qanda-private-ai" id="subtitle"]] -[[!template text="""still going to stay live. So next question coming in, I see,""" start="00:30:14.633" video="qanda-private-ai" id="subtitle"]] -[[!template text="""are there free as in freedom, free as in FSF issues with the""" start="00:30:17.859" video="qanda-private-ai" id="subtitle"]] -[[!template text="""data? Yes, where's the data coming from is a huge question""" start="00:30:17.859" video="qanda-private-ai" id="subtitle"]] -[[!template text="""with AI. It's astonishing you can ask questions to models""" start="00:30:31.778" video="qanda-private-ai" id="subtitle"]] -[[!template text="""that you don't know where it's coming from. That is gonna be""" start="00:30:37.826" video="qanda-private-ai" id="subtitle"]] -[[!template text="""one of the big issues long-term. There are people who are""" start="00:30:42.532" video="qanda-private-ai" id="subtitle"]] -[[!template text="""working on trying to figure out that stuff, but it's, I mean,""" start="00:30:46.096" video="qanda-private-ai" id="subtitle"]] -[[!template text="""if you look at, God, I can't remember who it was. Somebody was""" start="00:30:46.096" video="qanda-private-ai" id="subtitle"]] -[[!template text="""actually out torrenting books just to be able to build into""" start="00:30:53.205" video="qanda-private-ai" id="subtitle"]] -[[!template text="""their AI system. I think it might've been Meta. So there's a""" start="00:30:53.205" video="qanda-private-ai" id="subtitle"]] -[[!template text="""lot of that going on. The open source of the stuff is going to""" start="00:30:58.272" video="qanda-private-ai" id="subtitle"]] -[[!template text="""be tough. There's going to be there's some models like the""" start="00:31:00.956" video="qanda-private-ai" id="subtitle"]] -[[!template text="""mobile guys have got their own license, but where they're""" start="00:31:04.240" video="qanda-private-ai" id="subtitle"]] -[[!template text="""getting their data from, I'm not sure on so that that's a huge""" start="00:31:04.240" video="qanda-private-ai" id="subtitle"]] -[[!template text="""question. That's a that's a talk in itself. But yeah, but you""" start="00:31:04.240" video="qanda-private-ai" id="subtitle"]] -[[!template text="""if you train on your rag and your data, you know what it's""" start="00:31:14.074" video="qanda-private-ai" id="subtitle"]] -[[!template text="""come, you know, you have a license that but the other stuff is""" start="00:31:14.074" video="qanda-private-ai" id="subtitle"]] -[[!template text="""just more lines of supplement if you're using a smaller""" start="00:31:14.074" video="qanda-private-ai" id="subtitle"]] -[[!template text="""model, but""" start="00:31:14.074" video="qanda-private-ai" id="subtitle"]] -[[!template text="""The comment online, I see a couple of them. I'll read them out""" start="00:31:27.449" video="qanda-private-ai" id="subtitle"]] -[[!template text="""in order here. Really interesting stuff. Thank you for your""" start="00:31:32.094" video="qanda-private-ai" id="subtitle"]] -[[!template text="""talk. Given that large AI companies are openly stealing""" start="00:31:34.496" video="qanda-private-ai" id="subtitle"]] -[[!template text="""intellectual property and copyright and therefore""" start="00:31:35.617" video="qanda-private-ai" id="subtitle"]] -[[!template text="""eroding the authority of such laws and maybe obscuring the""" start="00:31:35.617" video="qanda-private-ai" id="subtitle"]] -[[!template text="""truth itself, can you see a future where IP and copyright""" start="00:31:35.617" video="qanda-private-ai" id="subtitle"]] -[[!template text="""flaw become untenable? I think that's a great question. I'm""" start="00:31:35.617" video="qanda-private-ai" id="subtitle"]] -[[!template text="""not a lawyer, but it is really getting complicated. It is""" start="00:31:55.799" video="qanda-private-ai" id="subtitle"]] -[[!template text="""getting to the point, I asked a question from, I played with""" start="00:32:01.106" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Sora a little bit, and it generated someone, you can go like,""" start="00:32:01.106" video="qanda-private-ai" id="subtitle"]] -[[!template text="""oh, that's Jon Hamm, that's Christopher Walken, you start""" start="00:32:01.106" video="qanda-private-ai" id="subtitle"]] -[[!template text="""figuring out who the people they're modeling stuff after.""" start="00:32:01.106" video="qanda-private-ai" id="subtitle"]] -[[!template text="""There is an apocalypse, something going to happen right""" start="00:32:12.961" video="qanda-private-ai" id="subtitle"]] -[[!template text="""now. There is, but this is once again, my personal opinion,""" start="00:32:12.961" video="qanda-private-ai" id="subtitle"]] -[[!template text="""and I'm not a lawyer, and I do not have money. So don't sue me,""" start="00:32:17.466" video="qanda-private-ai" id="subtitle"]] -[[!template text="""is there's going to be the current administration tends is""" start="00:32:22.812" video="qanda-private-ai" id="subtitle"]] -[[!template text="""very AI pro AI. And there's very a great deal of lobbying by""" start="00:32:22.812" video="qanda-private-ai" id="subtitle"]] -[[!template text="""those groups. And it's on both sides. And it's going to be,""" start="00:32:29.019" video="qanda-private-ai" id="subtitle"]] -[[!template text="""it's gonna be interesting to see what happens to copyright""" start="00:32:33.423" video="qanda-private-ai" id="subtitle"]] -[[!template text="""the next 510 years. I just don't know how it keeps up without""" start="00:32:33.423" video="qanda-private-ai" id="subtitle"]] -[[!template text="""there being some adjustments and stuff. Okay, and then""" start="00:32:37.888" video="qanda-private-ai" id="subtitle"]] -[[!template text="""another comment I saw, file size is not going to be a""" start="00:32:44.180" video="qanda-private-ai" id="subtitle"]] -[[!template text="""bottleneck. RAM is. You'll need 16 gigabytes of RAM to run""" start="00:32:44.180" video="qanda-private-ai" id="subtitle"]] -[[!template text="""the smallest local models and 512 gigabytes of RAM to run the""" start="00:32:50.014" video="qanda-private-ai" id="subtitle"]] -[[!template text="""larger ones. You'll need a GPU with that much memory if you""" start="00:32:50.014" video="qanda-private-ai" id="subtitle"]] -[[!template text="""want it to run quickly. Yeah. Oh no. It also depends upon how""" start="00:32:57.912" video="qanda-private-ai" id="subtitle"]] -[[!template text="""your memory is laid out. Like example being the Ultra i285H I""" start="00:33:05.421" video="qanda-private-ai" id="subtitle"]] -[[!template text="""plan to buy, that has 96 gig of memory. It's unified between""" start="00:33:07.364" video="qanda-private-ai" id="subtitle"]] -[[!template text="""the GPU and the CPU share it, but they go over the same bus. So""" start="00:33:14.014" video="qanda-private-ai" id="subtitle"]] -[[!template text="""the overall bandwidth of it tends to be a bit less, but you're""" start="00:33:17.800" video="qanda-private-ai" id="subtitle"]] -[[!template text="""able to load more of it into memory. So it's able to do some""" start="00:33:17.800" video="qanda-private-ai" id="subtitle"]] -[[!template text="""additional stuff with it as opposed to come off disk. It's""" start="00:33:23.729" video="qanda-private-ai" id="subtitle"]] -[[!template text="""all balancing act. If you hit Zyskin's website, that guy's""" start="00:33:27.034" video="qanda-private-ai" id="subtitle"]] -[[!template text="""done some great work on it. I'm trying to figure out how big a""" start="00:33:28.516" video="qanda-private-ai" id="subtitle"]] -[[!template text="""model you can do, what you can do with it. And some of the stuff""" start="00:33:31.801" video="qanda-private-ai" id="subtitle"]] -[[!template text="""seems to be not obvious, because like example, being that""" start="00:33:34.826" video="qanda-private-ai" id="subtitle"]] -[[!template text="""MacBook Air, for the five minutes I can run the model, it runs""" start="00:33:34.826" video="qanda-private-ai" id="subtitle"]] -[[!template text="""it faster than a lot of other things that should be able to run""" start="00:33:34.826" video="qanda-private-ai" id="subtitle"]] -[[!template text="""it faster, just because of the way the ARM cores and the""" start="00:33:34.826" video="qanda-private-ai" id="subtitle"]] -[[!template text="""unified memory work on it. So it's a learning process. But if""" start="00:33:34.826" video="qanda-private-ai" id="subtitle"]] -[[!template text="""you want to, Network Chuck had a great video talking about""" start="00:33:52.151" video="qanda-private-ai" id="subtitle"]] -[[!template text="""building his own system with a couple really powerful""" start="00:33:52.151" video="qanda-private-ai" id="subtitle"]] -[[!template text="""NVIDIA cards and stuff like that in it. And just actually""" start="00:33:57.940" video="qanda-private-ai" id="subtitle"]] -[[!template text="""setting up on his system as a node and using a web UI on it. So""" start="00:34:01.864" video="qanda-private-ai" id="subtitle"]] -[[!template text="""there's a lot of stuff there, but it is a process of learning""" start="00:34:06.009" video="qanda-private-ai" id="subtitle"]] -[[!template text="""how big your data is, which models you want to use, how much""" start="00:34:06.009" video="qanda-private-ai" id="subtitle"]] -[[!template text="""information you need, but it's part of the learning. And you""" start="00:34:06.009" video="qanda-private-ai" id="subtitle"]] -[[!template text="""can run models, even as a Raspberry PI fives, if you want to,""" start="00:34:15.920" video="qanda-private-ai" id="subtitle"]] -[[!template text="""they'll run slow. Don't get me wrong, but they're possible.""" start="00:34:15.920" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Okay, and I think there's other questions coming in too, so""" start="00:34:25.497" video="qanda-private-ai" id="subtitle"]] -[[!template text="""I'll just bam for another second. We've got about five""" start="00:34:25.497" video="qanda-private-ai" id="subtitle"]] -[[!template text="""minutes before we'll, before we'll be cutting over, but I""" start="00:34:30.162" video="qanda-private-ai" id="subtitle"]] -[[!template text="""just want to say in case we get close for time here, how much I""" start="00:34:30.162" video="qanda-private-ai" id="subtitle"]] -[[!template text="""appreciate your talk. This is another one that I'm going to""" start="00:34:30.162" video="qanda-private-ai" id="subtitle"]] -[[!template text="""have to study after the conference. We greatly appreciate,""" start="00:34:40.992" video="qanda-private-ai" id="subtitle"]] -[[!template text="""all of us appreciate you guys putting on the conference.""" start="00:34:44.716" video="qanda-private-ai" id="subtitle"]] -[[!template text="""It's a great conference. It's well done. It's an honor to be""" start="00:34:48.279" video="qanda-private-ai" id="subtitle"]] -[[!template text="""on the stage with the brains of the project, which is you. So""" start="00:34:52.328" video="qanda-private-ai" id="subtitle"]] -[[!template text="""what else we got? Question wise. Okay, so just scanning""" start="00:34:59.440" video="qanda-private-ai" id="subtitle"]] -[[!template text="""here.""" start="00:35:01.785" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Have you used local models capable of tool calling? I'm, I'm""" start="00:35:13.746" video="qanda-private-ai" id="subtitle"]] -[[!template text="""scared of agentic. I, I am, I'm going to be a slow adopter of""" start="00:35:18.502" video="qanda-private-ai" id="subtitle"]] -[[!template text="""that. I want to do it, but I just don't have the, uh, four""" start="00:35:21.005" video="qanda-private-ai" id="subtitle"]] -[[!template text="""decimal fortitude right now to do it. I, I, I've had to give me""" start="00:35:25.490" video="qanda-private-ai" id="subtitle"]] -[[!template text="""the commands, but I still run the commands by hand. I'm""" start="00:35:30.897" video="qanda-private-ai" id="subtitle"]] -[[!template text="""looking into it and it's on once again, it's on that list, but""" start="00:35:34.922" video="qanda-private-ai" id="subtitle"]] -[[!template text="""I just, that's a big step for me. So.""" start="00:35:34.922" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Awesome. All right. Well, maybe it's, let me just scroll""" start="00:35:46.953" video="qanda-private-ai" id="subtitle"]] -[[!template text="""through because we might have missed one question.""" start="00:35:49.764" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Oh, I see. Here was the piggyback question. Now I see the""" start="00:36:00.908" video="qanda-private-ai" id="subtitle"]] -[[!template text="""question that I missed. So this was piggybacking on the""" start="00:36:03.033" video="qanda-private-ai" id="subtitle"]] -[[!template text="""question about model updates and adding data. And will""" start="00:36:04.997" video="qanda-private-ai" id="subtitle"]] -[[!template text="""models reach out to the web if they need more info? Or have you""" start="00:36:11.372" video="qanda-private-ai" id="subtitle"]] -[[!template text="""worked with any models that work that way? No, I've not seen""" start="00:36:16.524" video="qanda-private-ai" id="subtitle"]] -[[!template text="""any models to do that There's there was like a group working""" start="00:36:18.408" video="qanda-private-ai" id="subtitle"]] -[[!template text="""on something like a package updater that would do different""" start="00:36:18.408" video="qanda-private-ai" id="subtitle"]] -[[!template text="""diffs on it, but it's so Models change so much even who make""" start="00:36:18.408" video="qanda-private-ai" id="subtitle"]] -[[!template text="""minor changes and fine-tuning. It's hard just to update""" start="00:36:18.408" video="qanda-private-ai" id="subtitle"]] -[[!template text="""them in place So I haven't seen one, but that doesn't mean""" start="00:36:31.983" video="qanda-private-ai" id="subtitle"]] -[[!template text="""they're not out there. I'm curious topic though""" start="00:36:31.983" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Awesome Well, it's probably pretty good timing. Let me just""" start="00:36:37.249" video="qanda-private-ai" id="subtitle"]] -[[!template text="""scroll and make sure. And of course, before I can say that,""" start="00:36:45.728" video="qanda-private-ai" id="subtitle"]] -[[!template text="""there's one more question. So let's go ahead and have that. I""" start="00:36:47.049" video="qanda-private-ai" id="subtitle"]] -[[!template text="""want to make sure while we're still live, though, I give you a""" start="00:36:51.656" video="qanda-private-ai" id="subtitle"]] -[[!template text="""chance to offer any closing thoughts. So what scares you""" start="00:36:51.656" video="qanda-private-ai" id="subtitle"]] -[[!template text="""most about the eugenic tools? How would you think about""" start="00:36:57.503" video="qanda-private-ai" id="subtitle"]] -[[!template text="""putting a sandbox around that if you did adopt an eugenic""" start="00:37:01.889" video="qanda-private-ai" id="subtitle"]] -[[!template text="""workflow? That is a great question. In terms of that, I would""" start="00:37:01.889" video="qanda-private-ai" id="subtitle"]] -[[!template text="""just control what it's able to talk to, what machines, I""" start="00:37:09.670" video="qanda-private-ai" id="subtitle"]] -[[!template text="""would actually have it be air gap. I work for a defense""" start="00:37:09.670" video="qanda-private-ai" id="subtitle"]] -[[!template text="""contractor, and we spend a lot of time dealing with air gap""" start="00:37:16.320" video="qanda-private-ai" id="subtitle"]] -[[!template text="""systems, because that's just kind of the way it works out for""" start="00:37:16.320" video="qanda-private-ai" id="subtitle"]] -[[!template text="""us. So agentic, it's just going to take a while to get trust. I""" start="00:37:16.320" video="qanda-private-ai" id="subtitle"]] -[[!template text="""want to want to see more stuff happening. Humans screw up""" start="00:37:25.594" video="qanda-private-ai" id="subtitle"]] -[[!template text="""stuff enough. The last thing we need is to multiply that by""" start="00:37:27.757" video="qanda-private-ai" id="subtitle"]] -[[!template text="""1000. So in terms of that, I would be restricting what it can""" start="00:37:28.919" video="qanda-private-ai" id="subtitle"]] -[[!template text="""do. If you look at the capabilities, if I created a user and""" start="00:37:31.443" video="qanda-private-ai" id="subtitle"]] -[[!template text="""gave it permissions, I would have a lockdown through sudo,""" start="00:37:35.870" video="qanda-private-ai" id="subtitle"]] -[[!template text="""what it's able to do, what the account's able to do. I would do""" start="00:37:35.870" video="qanda-private-ai" id="subtitle"]] -[[!template text="""those kind of things, but it's going to be, it's happening.""" start="00:37:43.863" video="qanda-private-ai" id="subtitle"]] -[[!template text="""It's just, I'm going to be one of the laggards on that one. So""" start="00:37:47.068" video="qanda-private-ai" id="subtitle"]] -[[!template text="""airgab, jail, extremely locked down environments, like""" start="00:37:49.171" video="qanda-private-ai" id="subtitle"]] -[[!template text="""we're talking about separate physicals, not Docker. Yeah,""" start="00:37:49.171" video="qanda-private-ai" id="subtitle"]] -[[!template text="""hopefully. Right, fair. So tool calling can be read-only,""" start="00:37:59.152" video="qanda-private-ai" id="subtitle"]] -[[!template text="""such as giving models the ability to search the web before""" start="00:38:04.060" video="qanda-private-ai" id="subtitle"]] -[[!template text="""answering your question, you know, write access, execute""" start="00:38:04.060" video="qanda-private-ai" id="subtitle"]] -[[!template text="""access. I'm interested to know if local models are any good""" start="00:38:04.060" video="qanda-private-ai" id="subtitle"]] -[[!template text="""at that. Yes, local models can do a lot of that stuff. It's""" start="00:38:12.253" video="qanda-private-ai" id="subtitle"]] -[[!template text="""their capabilities. If you load LM studio, you can do a lot of""" start="00:38:21.052" video="qanda-private-ai" id="subtitle"]] -[[!template text="""wonderful stuff with that or with open web UI with a llama.""" start="00:38:22.473" video="qanda-private-ai" id="subtitle"]] -[[!template text="""It's a lot of capabilities. It's amazing. Open web UI is""" start="00:38:28.561" video="qanda-private-ai" id="subtitle"]] -[[!template text="""actually what a lot of companies are using now to put their""" start="00:38:31.625" video="qanda-private-ai" id="subtitle"]] -[[!template text="""data behind that. They're curated data and stuff like that.""" start="00:38:31.625" video="qanda-private-ai" id="subtitle"]] -[[!template text="""So works well. I can confirm that from my own professional""" start="00:38:37.893" video="qanda-private-ai" id="subtitle"]] -[[!template text="""experience. Excellent. Okay, well, our timing should be""" start="00:38:38.894" video="qanda-private-ai" id="subtitle"]] -[[!template text="""just perfect if you want to give us like a 30-second,""" start="00:38:42.976" video="qanda-private-ai" id="subtitle"]] -[[!template text="""45-second wrap-up. Aaron, let me squeeze in mine. Thank you""" start="00:38:42.976" video="qanda-private-ai" id="subtitle"]] -[[!template text="""again so much for preparing this talk and for entertaining""" start="00:38:50.320" video="qanda-private-ai" id="subtitle"]] -[[!template text="""all of our questions. Yeah, let me just thank you guys for the""" start="00:38:50.320" video="qanda-private-ai" id="subtitle"]] -[[!template text="""conference again. This is a great one. I've enjoyed a lot of""" start="00:38:56.527" video="qanda-private-ai" id="subtitle"]] -[[!template text="""it. I've only had a couple of talks so far, but I'm looking""" start="00:39:00.232" video="qanda-private-ai" id="subtitle"]] -[[!template text="""forward to hitting the ones after this and tomorrow. But the""" start="00:39:01.333" video="qanda-private-ai" id="subtitle"]] -[[!template text="""AI stuff is coming. Get on board. Definitely recommend it.""" start="00:39:06.681" video="qanda-private-ai" id="subtitle"]] -[[!template text="""If you want to just try it out and get a little taste of it, what""" start="00:39:12.028" video="qanda-private-ai" id="subtitle"]] -[[!template text="""my minimal viable product with just LlamaFile and GPTEL""" start="00:39:12.028" video="qanda-private-ai" id="subtitle"]] -[[!template text="""will get you to the point where you start figuring out. GPTEL""" start="00:39:12.028" video="qanda-private-ai" id="subtitle"]] -[[!template text="""is an amazing thing. It just gets out of your way, but it works""" start="00:39:19.057" video="qanda-private-ai" id="subtitle"]] -[[!template text="""solo with Emacs. Design because it takes doesn't take your""" start="00:39:20.639" video="qanda-private-ai" id="subtitle"]] -[[!template text="""hands off the keyboard. It's just another buffer and you""" start="00:39:24.725" video="qanda-private-ai" id="subtitle"]] -[[!template text="""just put information in there. It's quite quite a wonderful""" start="00:39:27.789" video="qanda-private-ai" id="subtitle"]] -[[!template text="""It's a wonderful time. Let's put that way That's all I got""" start="00:39:30.193" video="qanda-private-ai" id="subtitle"]] -[[!template text="""Thank you so much for once again, and we're we're just cut""" start="00:39:33.057" video="qanda-private-ai" id="subtitle"]] -[[!template text="""away. So I'll stop the recording and you're on your own""" start="00:39:33.057" video="qanda-private-ai" id="subtitle"]] -[[!template text="""recognizance Well, I'm gonna punch out if anybody has any""" start="00:39:40.447" video="qanda-private-ai" id="subtitle"]] -[[!template text="""questions or anything my email address is AJ growthy at""" start="00:39:40.447" video="qanda-private-ai" id="subtitle"]] -[[!template text="""yahoo.com or at gmail and Thank you all for attending and""" start="00:39:40.447" video="qanda-private-ai" id="subtitle"]] -[[!template text="""thanks again for the conference Okay, I'm gonna go ahead and""" start="00:39:40.447" video="qanda-private-ai" id="subtitle"]] -[[!template text="""end the room there, thank you. Excellent, thanks, bye.""" start="00:39:55.994" video="qanda-private-ai" id="subtitle"]] - </div>Questions or comments? Please e-mail [ajgrothe@yahoo.com](mailto:ajgrothe@yahoo.com?subject=Comment%20for%20EmacsConf%202023%20private-ai%3A%20Emacs%20and%20private%20AI%3A%20a%20great%20match) |