Transcript (unedited)

[[!template text="""Hey, everybody. Welcome from frigid Omaha, Nebraska.""" start="00:00:00.000" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I'm just going to kick off my talk here,""" start="00:00:04.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and we'll see how it all goes. Thanks for attending.""" start="00:00:06.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So the slides will be available on my site, https://grothe.us,""" start="00:00:23.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""in the presentation section tonight or tomorrow.""" start="00:00:26.940" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""This is a quick intro to one way to do private AI in Emacs.""" start="00:00:29.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""There are a lot of other ways to do it.""" start="00:00:33.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""This one is really just more or less the easiest way to do it.""" start="00:00:35.300" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It's a minimal viable product""" start="00:00:38.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""to get you an idea of how to get started with it""" start="00:00:40.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and how to give it a spin.""" start="00:00:42.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Really hope some of you give it a shot""" start="00:00:43.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and learn something along the way.""" start="00:00:45.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So the overview of the talk""" start="00:00:48.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""broke down these basic bullet points of why private AI,""" start="00:00:50.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""what do I need to do private AI, Emacs and private AI,""" start="00:00:54.940" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""pieces for an AI Emacs solution,""" start="00:00:58.940" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""a demo of a minimal viable product, and the summary.""" start="00:01:02.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Why private AI? This is pretty simple.""" start="00:01:08.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Just read the terms and conditions""" start="00:01:10.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""for any AI system you're currently using.""" start="00:01:12.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""If you're using the free tiers, your queries,""" start="00:01:14.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""code, uploaded information""" start="00:01:17.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""is being used to train the models.""" start="00:01:18.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""In some cases, you are giving the company""" start="00:01:20.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""a perpetual license to your data.""" start="00:01:22.940" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""You have no control over this,""" start="00:01:25.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""except for not using the engine.""" start="00:01:27.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And keep in mind, the terms""" start="00:01:29.220" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""are changing all the time on that,""" start="00:01:30.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and they're not normally changing for our benefit.""" start="00:01:32.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So that's not necessarily a good thing.""" start="00:01:34.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""If you're using the paid tiers,""" start="00:01:38.260" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""you may be able to opt out of the data collection.""" start="00:01:40.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""But keep in mind, this can change,""" start="00:01:43.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""or they may start charging for that option.""" start="00:01:45.540" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Every AI company wants more and more data.""" start="00:01:48.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""They need more and more data to train their models.""" start="00:01:51.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It is just the way it is.""" start="00:01:53.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""They need more and more information""" start="00:01:56.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""to get it more and more accurate to keep it up to date.""" start="00:01:57.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""There's been a story about Stack Overflow.""" start="00:02:00.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It has like half the number of queries they had a year ago""" start="00:02:03.220" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""because people are using AI.""" start="00:02:05.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""The problem with that is now""" start="00:02:07.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""there's less data going to Stack Overflow""" start="00:02:08.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""for the AI to get. Vicious cycle,""" start="00:02:10.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""especially when you start looking at""" start="00:02:12.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""newer language like Ruby and stuff like that.""" start="00:02:14.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So it comes down to being an interesting time.""" start="00:02:16.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Another reason why to go private AI is your costs are going to vary.""" start="00:02:21.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Right now, these services are being heavily subsidized.""" start="00:02:24.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""If you're paying Claude $20 a month,""" start="00:02:27.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""it is not costing Claude, those guys, $20 a month""" start="00:02:29.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""to host all the infrastructure""" start="00:02:32.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""to build all these data centers.""" start="00:02:34.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""They are severely subsidizing that""" start="00:02:35.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""at a very much a loss right now.""" start="00:02:38.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""When they start charging the real costs plus a profit,""" start="00:02:41.260" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""it's going to change.""" start="00:02:43.660" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Right now, I use a bunch of different services.""" start="00:02:45.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I've played with Grok and a bunch of other ones.""" start="00:02:48.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""But Grok right now is like $30 a month""" start="00:02:50.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""for a regular Super Grok.""" start="00:02:52.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""When they start charging the real cost of that,""" start="00:02:54.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""it's going to go from $30 to something a great deal more,""" start="00:02:56.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""perhaps, I think, $100 or $200""" start="00:02:59.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""or whatever really turns out to be the cost""" start="00:03:02.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""when you figure everything into it.""" start="00:03:04.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""When you start adding that cost into that,""" start="00:03:06.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""a lot of people are using public AI right now""" start="00:03:07.540" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""are going to have no option but to move to private AI""" start="00:03:10.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""or give up on AI overall.""" start="00:03:11.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""What do you need to be able to do private AI?""" start="00:03:16.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""If you're going to run your own AI,""" start="00:03:18.660" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""you're going to need a system with either some cores,""" start="00:03:21.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""a graphics processor unit,""" start="00:03:23.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""or a neural processing unit, a GPU or an NPU.""" start="00:03:25.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I currently have four systems""" start="00:03:28.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I'm experimenting with and playing around with on a daily basis.""" start="00:03:29.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I have a System76 Pangolin AMD Ryzen 7 78040U""" start="00:03:32.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""with a Radeon 7080M integrated graphics card.""" start="00:03:37.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It's got 32 gigs of RAM.""" start="00:03:41.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It's a beautiful piece of hardware. I really do like it.""" start="00:03:42.540" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I have my main workstation,""" start="00:03:45.260" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""it's an HP Z620 with dual Intel Xeons""" start="00:03:46.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""with four NVIDIA K2200 graphics cards in it.""" start="00:03:50.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Why the four NVIDIA K2200 graphics card on it?""" start="00:03:53.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Because I could buy four of them on eBay for $100""" start="00:03:56.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and it was still supported by the NVIDIA drivers for Debian.""" start="00:03:59.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So that's why that is. A MacBook Air with an M1 processor,""" start="00:04:02.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""a very nice piece of kit I picked up a couple years ago,""" start="00:04:08.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""very cheap, but it runs AI surprisingly well,""" start="00:04:10.940" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and an Acer Aspire 1 with an AMD Ryzen 5700H in it.""" start="00:04:14.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""This was my old laptop. It was a sturdy beast.""" start="00:04:18.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It was able to do enough AI to do demos and stuff,""" start="00:04:22.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and I liked it quite a bit for that.""" start="00:04:24.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I'm using the Pangolin for this demonstration""" start="00:04:25.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""because it's just better.""" start="00:04:28.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Apple's M4 chip has 38 teraflops of MPU performance.""" start="00:04:30.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""The Microsoft co-pilots are now requiring""" start="00:04:37.220" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""45 teraflops of MPU""" start="00:04:40.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""to be able to have the co-pilot badge on it.""" start="00:04:41.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And Raspberry Pi's new AI top is about 18 teraflops""" start="00:04:43.940" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and is $70 on top of the cost of Raspberry Pi 5.""" start="00:04:48.300" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Keep in mind, Raspberry recently""" start="00:04:51.220" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""raised the cost of their Pi 5s because of RAM pricing,""" start="00:04:56.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""which is going to be affecting""" start="00:04:59.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""a lot of these types of solutions in the near future.""" start="00:05:00.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""But there's going to be a lot of""" start="00:05:02.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""local power available in the future.""" start="00:05:05.300" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""That's what it really comes down to.""" start="00:05:06.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""A lot of people are going to have PCs on their desks.""" start="00:05:08.220" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""They're going to run a decent private AI""" start="00:05:11.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""without much issue.""" start="00:05:13.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So for Emacs and private AI,""" start="00:05:16.348" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""there's a couple popular solutions.""" start="00:05:18.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Gptel, which is the one we're going to talk about.""" start="00:05:20.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It's a simple interface. It's a minimal interface.""" start="00:05:22.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It integrates easily into your workflow.""" start="00:05:24.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It's just, quite honestly, chef's kiss,""" start="00:05:26.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""just a beautifully well-done piece of software.""" start="00:05:29.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Ollama Buddy has more features,""" start="00:05:31.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""a menu interface, has quick access""" start="00:05:33.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""for things like code refactoring,""" start="00:05:36.260" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""text-free formatting, et cetera.""" start="00:05:37.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""This is the one that you spend a little more time with,""" start="00:05:38.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but you also get a little bit more back from it.""" start="00:05:41.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Ellama is another one, has some really good features to it,""" start="00:05:43.940" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""more different capabilities,""" start="00:05:49.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but it's a different set of rules and capabilities to it.""" start="00:05:51.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Aidermac, which is programming with your AI and Emacs.""" start="00:05:54.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""The closest thing I can come up""" start="00:05:59.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""to comparing this to is Cursor, except it's in Emacs.""" start="00:06:01.220" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It's really quite well done.""" start="00:06:04.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""These are all really quite well done.""" start="00:06:05.660" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""There's a bunch of other projects out there.""" start="00:06:07.300" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""If you go out to GitHub, type Emacs AI,""" start="00:06:08.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""you'll find a lot of different options.""" start="00:06:10.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So what is a minimal viable product that can be done?""" start="00:06:13.220" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""A minimal viable product to show what an AI Emacs solution is""" start="00:06:18.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""can be done with only needing two pieces of software.""" start="00:06:23.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Llamafile, this is an amazing piece of software.""" start="00:06:27.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""This is a whole LLM contained in one file.""" start="00:06:31.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And the same file runs on Mac OS X,""" start="00:06:32.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Linux, Windows, and the BSDs.""" start="00:06:36.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It's a wonderful piece of kit""" start="00:06:39.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""based on these people who created""" start="00:06:42.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""this thing called Cosmopolitan""" start="00:06:44.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""that lets you create and execute""" start="00:06:45.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""while it runs on a bunch of different systems.""" start="00:06:46.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And Gptel, which is an easy plug-in for Emacs,""" start="00:06:48.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""which we talked about in the last slide a bit.""" start="00:06:51.300" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So setting up the LLM, you have to just go out""" start="00:06:56.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and just hit a page for it""" start="00:07:00.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and go out and do a wget of it.""" start="00:07:03.543" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""That's all it takes there.""" start="00:07:05.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Chmodding it so you can actually execute the executable.""" start="00:07:07.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And then just go ahead and actually running it.""" start="00:07:10.260" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And let's go ahead and do that.""" start="00:07:12.940" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I've already downloaded it because I don't want to wait.""" start="00:07:16.940" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And let's just take a look at it.""" start="00:07:18.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I've actually downloaded several of them,""" start="00:07:21.260" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but let's go ahead and just run llama 3.2-1b""" start="00:07:22.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""with the 3 billion instructions. And that's it firing up.""" start="00:07:25.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And it is nice enough to actually be listening in port 8080,""" start="00:07:31.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""which we'll need in a minute.""" start="00:07:33.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So once you do that, you have to install gptel and emacs.""" start="00:07:35.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""That's as simple as firing up emacs,""" start="00:07:43.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""doing the M-x install-package,""" start="00:07:45.660" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and then just typing gptel,""" start="00:07:48.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""if you have your repository set up right,""" start="00:07:49.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""which hopefully you do.""" start="00:07:51.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And then you just go ahead and have it.""" start="00:07:52.300" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""You also have to set up a config file.""" start="00:07:56.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Here's my example config file as it currently set up,""" start="00:07:58.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""requiring, ensuring Gptel is loaded,""" start="00:08:01.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""defining the Llamafile backend.""" start="00:08:04.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""You can put multiple backends into it,""" start="00:08:05.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but I just have the one defined on this example.""" start="00:08:07.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""But it's pretty straightforward.""" start="00:08:09.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Llama local file, name for it, stream, protocol HTTP.""" start="00:08:12.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""If you have HTTPS set up, that's obviously preferable,""" start="00:08:16.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but a lot of people don't for their home labs.""" start="00:08:20.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Host is just 127.0.0.1 port 8080.""" start="00:08:22.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Keep in mind, some of the AIs run on a different port,""" start="00:08:26.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""so you may be 8081""" start="00:08:30.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""if you're running OpenWebView at the same time. The key,""" start="00:08:31.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""we don't need an API key because it's a local server.""" start="00:08:34.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And the models just, uh, we can put multiple models""" start="00:08:37.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""on there if we want to.""" start="00:08:40.260" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So if we create one with additional stuff""" start="00:08:41.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""or like rag and stuff like that,""" start="00:08:43.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""we can actually name those models by their domain,""" start="00:08:45.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""which is really kind of cool.""" start="00:08:47.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""But, uh, that's all that takes.""" start="00:08:48.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So let's go ahead and go to a quick test of it.""" start="00:08:52.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Oops. Alt-X, gptel. And we're going to just choose""" start="00:09:03.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""the default buffer to make things easier.""" start="00:09:11.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Going to resize it up a bit.""" start="00:09:12.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And usually the go-to question I go to is, who was David Bowie?""" start="00:09:15.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""This one is actually a question""" start="00:09:19.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""that's turned out to be really good""" start="00:09:24.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""for figuring out whether or not AI is complete.""" start="00:09:26.220" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""This is one that some engines do well on, other ones don't.""" start="00:09:28.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And we can just do, we can either do""" start="00:09:31.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""the alt X and send the gptel-send,""" start="00:09:33.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""or we can just do C-c and hit enter.""" start="00:09:36.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""We'll just do C-c and enter.""" start="00:09:37.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And now it's going ahead and hitting our local AI system""" start="00:09:39.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""running on port 8080. And that looks pretty good,""" start="00:09:43.660" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but let's go ahead and say, hey, it's set to terse mode right now.""" start="00:09:46.660" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Please expand upon this. And there we go.""" start="00:09:50.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""We're getting a full description""" start="00:10:03.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""of the majority of, uh, about David Bowie's life""" start="00:10:05.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and other information about him.""" start="00:10:08.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So very, very happy with that.""" start="00:10:10.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""One thing to keep in mind is you look at things""" start="00:10:21.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""when you're looking for hallucinations,""" start="00:10:23.540" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""how accurate AI is, how it's compressed""" start="00:10:24.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""is it will tend to screw up on things like""" start="00:10:26.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""how many children he had and stuff like that.""" start="00:10:29.260" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Let me see if it gets to that real quick.""" start="00:10:30.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Is it not actually on this one?""" start="00:10:32.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Alright, so that's the first question I always ask one.""" start="00:10:39.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""The next one is what are sea monkeys?""" start="00:10:42.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It gives you an idea of the breadth of the system.""" start="00:10:44.660" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It's querying right now. Pulls it back correctly. Yes.""" start="00:10:48.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And it's smart enough to actually detect David Bowie""" start="00:11:10.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""even referenced see monkeys in the song sea of love,""" start="00:11:12.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""which came at hit single.""" start="00:11:15.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So it's actually keeping the context alive""" start="00:11:16.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and that which is very cool feature.""" start="00:11:18.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I did not see that coming.""" start="00:11:20.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Here's one that some people say is a really good one""" start="00:11:21.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""to ask. Rs in "strawberry."""" start="00:11:24.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""All right, now she's going off the reservation.""" start="00:11:42.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""She's going in a different direction.""" start="00:11:46.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Let me go ahead and reopen that again,""" start="00:11:48.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""because it went down a bad hole there for a second.""" start="00:11:49.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Let me ask it to write hello world in Emacs Lisp.""" start="00:11:57.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Yep, that works. So the point being here,""" start="00:11:58.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""that was like two minutes of setup.""" start="00:12:10.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And now we have a small AI embedded inside the system.""" start="00:12:14.940" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So that gives you an idea just how easy it can be.""" start="00:12:18.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And it's just running locally on the system.""" start="00:12:20.540" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""We also have the default system here as well.""" start="00:12:22.300" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So not that bad.""" start="00:12:25.260" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""That's a basic solution, that's a basic setup""" start="00:12:32.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""that will get you to the point where you can go like,""" start="00:12:35.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""it's a party trick, but it's a very cool party trick.""" start="00:12:37.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""The way that Gptel works is it puts it into buffers,""" start="00:12:39.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""it doesn't interfere with your flow that much,""" start="00:12:42.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""it's just an additional window you can pop open""" start="00:12:45.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""to ask questions and get information for,""" start="00:12:47.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""dump code into it and have it refactored.""" start="00:12:49.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Gptel has a lot of additional options""" start="00:12:51.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""for things that are really cool for that.""" start="00:12:53.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""But if you want a better solution,""" start="00:12:55.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I recommend Ollama or LM Studio.""" start="00:12:57.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""They're both more capable than Llamafile.""" start="00:12:59.940" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""They can accept a lot of different models.""" start="00:13:01.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""You can do things like RAG.""" start="00:13:03.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""You can do loading of things onto the GPU more explicitly.""" start="00:13:05.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It can speed stuff up.""" start="00:13:09.220" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""One of the things about the retrieval augmentation is""" start="00:13:10.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""it will let you put your data into the system""" start="00:13:13.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""so you can start uploading your code, your information,""" start="00:13:15.540" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and actually being able to do analysis of it.""" start="00:13:17.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Open WebUI provides more capabilities.""" start="00:13:20.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It provides an interface that's similar""" start="00:13:23.540" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""to what you're used to seeing""" start="00:13:24.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""for ChatGPT and the other systems.""" start="00:13:25.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It's really quite well done.""" start="00:13:28.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And once again, gptel, I have to mention that""" start="00:13:29.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""because that's the one I really kind of like.""" start="00:13:32.540" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And Ollama Buddy is also another really nice one.""" start="00:13:34.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So what about the licensing of these models?""" start="00:13:36.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Since I'm going out pulling down""" start="00:13:41.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""a model and doing this stuff.""" start="00:13:42.300" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Let's take a look at a couple of highlights""" start="00:13:43.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""from the Meta Llama 3 community license scale.""" start="00:13:46.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""If your service exceeds 700 million monthly users,""" start="00:13:49.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""you need additional licensing.""" start="00:13:52.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Probably not going to be a problem for most of us.""" start="00:13:54.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""There's a competition restriction.""" start="00:13:56.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""You can't use this model to enhance competing models.""" start="00:13:58.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And there's some limitations on using the Meta trademarks.""" start="00:14:00.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Not that big a deal.""" start="00:14:04.220" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And the other ones are it's a permissive one""" start="00:14:05.940" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""designed to encourage innovation,""" start="00:14:09.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""open development, commercial use is allowed,""" start="00:14:10.940" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but there are some restrictions on it.""" start="00:14:13.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Yeah, you can modify the model,""" start="00:14:15.220" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but you have to rely on the license terms.""" start="00:14:17.260" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And you can distribute the model with derivatives.""" start="00:14:20.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And there are some very cool ones out there.""" start="00:14:22.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""There's people who've done things""" start="00:14:24.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""to try and make the Llama be less, what's the phrase,""" start="00:14:25.260" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""ethical if you're doing penetration testing research""" start="00:14:29.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and stuff like that.""" start="00:14:31.940" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It has some very nice value there.""" start="00:14:32.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Keep in mind licenses also vary""" start="00:14:34.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""depending on the model you're using.""" start="00:14:37.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Mistral AI has the non-production license.""" start="00:14:39.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It's designed to keep it to research and development.""" start="00:14:42.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""You can't use it commercially.""" start="00:14:45.220" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So it's designed to clearly delineate""" start="00:14:46.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""between research and development""" start="00:14:51.793" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and somebody trying to actually build""" start="00:14:52.940" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""something on top of it.""" start="00:14:54.260" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And another question I get asked is,""" start="00:14:56.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""are there open source data model options?""" start="00:14:57.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Yeah, but most of them are small or specialized currently.""" start="00:14:59.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""MoMo is a whole family of them,""" start="00:15:02.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but there tend to be more specialized,""" start="00:15:05.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but it's very cool to see where it's going.""" start="00:15:07.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And it's another thing that's just going forward.""" start="00:15:09.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It's under the MIT license.""" start="00:15:11.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Some things to know to help you""" start="00:15:14.520" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""have a better experience with this.""" start="00:15:15.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Get ollama and Open WebUI working by themselves,""" start="00:15:17.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""then set up your config file.""" start="00:15:21.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I was fighting both at the same time,""" start="00:15:22.660" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and it turned out I had a problem with my ollama.""" start="00:15:24.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I had a conflict, so that was what my problem is.""" start="00:15:26.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Llamafile, gptel is a great way to start experimenting""" start="00:15:28.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""just to get you an idea of how it works""" start="00:15:32.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and figure out how the interfaces work. Tremendous.""" start="00:15:34.300" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""RAG loading documents into it is really easy with open web UI.""" start="00:15:36.940" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""You can create models, you can put things like""" start="00:15:40.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""help desk developers and stuff like that, breaking it out.""" start="00:15:43.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""The Hacker Noon has a how to build a $300 AI computer.""" start="00:15:46.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""This is for March 2024,""" start="00:15:51.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but it still has a lot of great information""" start="00:15:52.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""on how to benchmark the environments,""" start="00:15:55.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""what some values are like the Ryzen 5700U""" start="00:15:56.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""inside my Acer Aspire,""" start="00:16:01.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""that's where I got the idea doing that.""" start="00:16:02.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Make sure you do the ROCm stuff correctly""" start="00:16:04.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""to get the GUI extensions. But it's just really good stuff.""" start="00:16:06.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""You don't need a great GPU or CPU to get started.""" start="00:16:09.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Smaller models like tinyllama""" start="00:16:13.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""can run on very small systems.""" start="00:16:14.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It gets you the ability to start playing with it""" start="00:16:16.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and start experimenting and figure out if that's for you""" start="00:16:19.043" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and to move forward with it.""" start="00:16:21.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""The AMD Ryzen AI Max+ 395 is a mini PC""" start="00:16:23.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""makes it really nice dedicated host.""" start="00:16:29.220" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""You used to be able to buy these for about $1200.""" start="00:16:31.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Now with the RAM price increase,""" start="00:16:34.079" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""you want to get 120 gig when you're pushing two brands,""" start="00:16:35.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""so it gets a little tighter.""" start="00:16:38.459" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Macs work remarkably well with AI.""" start="00:16:40.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""My MacBook Air was one of my go-tos for a while,""" start="00:16:44.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but once I started doing anything AI,""" start="00:16:47.660" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I had a five-minute window""" start="00:16:49.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""before the thermal throttling became an issue.""" start="00:16:50.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Keep in mind that's a MacBook Air,""" start="00:16:52.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""so it doesn't have the greatest ventilation.""" start="00:16:54.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""If you get the MacBook Pros and stuff,""" start="00:16:56.660" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""they tend to have more ventilation,""" start="00:16:58.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but still you're going to be pushing against that.""" start="00:17:00.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So Mac Minis and the Mac Ultras and stuff like that""" start="00:17:02.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""tend to work really well for that.""" start="00:17:04.940" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Alex Ziskind on YouTube has a channel.""" start="00:17:06.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""He does a lot of AI performance benchmarking,""" start="00:17:09.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""like "I load a 70 billion parameter model""" start="00:17:11.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""on this mini PC" and stuff like that.""" start="00:17:14.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It's a lot of fun and interesting stuff there.""" start="00:17:16.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And it's influencing my decision""" start="00:17:19.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""to buy my next AI style PC.""" start="00:17:21.220" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Small domain specific LLMs are happening.""" start="00:17:22.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""An LLM that has all your code and information,""" start="00:17:27.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""it sounds like a really cool idea.""" start="00:17:29.940" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It gives you capabilities to start training stuff""" start="00:17:31.660" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""that you couldn't do with like the big ones.""" start="00:17:34.300" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Even with in terms of fine-tuning and stuff,""" start="00:17:35.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""it's remarkable to see where that space is coming along""" start="00:17:38.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""in the next year or so.""" start="00:17:40.540" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""HuggingFace.co has pointers to tons of AI models.""" start="00:17:41.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""You'll find the one that works for you, hopefully there.""" start="00:17:46.220" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""If you're doing cybersecurity,""" start="00:17:48.418" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""there's a whole bunch out there for that,""" start="00:17:50.540" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""that have certain training on it, information.""" start="00:17:52.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It's really good.""" start="00:17:54.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""One last thing to keep in mind is hallucinations are real.""" start="00:17:56.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""You will get BS back from the AI occasionally,""" start="00:18:00.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""so do validate everything you get from it.""" start="00:18:02.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Don't be using it for court cases like some people have""" start="00:18:05.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and run into those problems. So, That is my talk.""" start="00:18:08.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""What I would like you to get out of that is,""" start="00:18:14.540" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""if you haven't tried it, give Gptel and LlamaFile a shot.""" start="00:18:17.220" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Fire up a little small AI instance,""" start="00:18:21.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""play around with a little bit inside your Emacs,""" start="00:18:23.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and see if it makes your life better. Hopefully it will.""" start="00:18:27.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And I really hope you guys""" start="00:18:30.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""learned something from this talk. And thanks for listening.""" start="00:18:32.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And the links are at the end of the talk, if you have any questions.""" start="00:18:34.660" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Let me see if we got anything you want, Pat. You do.""" start="00:18:38.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""You've got a few questions.""" start="00:18:42.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Corwin]: Hey, this is Corwin. Thank you so much. Thank you, Aaron.""" start="00:18:43.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""What an awesome talk this was, actually.""" start="00:18:48.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""If you don't have a camera,""" start="00:18:50.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I can get away with not having one too.""" start="00:18:52.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Aaron]: I've got, I'll turn the camera on.""" start="00:18:54.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Corwin]: Okay. All right. I'll turn mine back on. Here I come.""" start="00:18:56.300" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Yeah, so there are a few questions,""" start="00:18:59.834" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but first let me say thank you""" start="00:19:03.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""for a really captivating talk.""" start="00:19:04.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I think a lot of people will be empowered from this""" start="00:19:06.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""to try to do more with less, especially locally.""" start="00:19:10.940" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""concerned about the data center footprint,""" start="00:19:15.260" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""environmentally concerned""" start="00:19:20.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""about the footprint of LLM inside data centers.""" start="00:19:23.660" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So just thinking about how we can""" start="00:19:26.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""put infrastructure we have at home to use""" start="00:19:28.220" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and get more done with less.""" start="00:19:32.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Aaron]: Yeah, the data center impact's interesting""" start="00:19:34.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""because there was a study a while ago.""" start="00:19:37.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Someone said every time you do a Gemini query,""" start="00:19:39.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""it's like boiling a cup of water.""" start="00:19:42.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Corwin]: Yeah, I've heard that one too. So do you want to, you know,""" start="00:19:45.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I don't know how much direction you want.""" start="00:19:48.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I'd be very happy to read out the questions for you.""" start="00:19:51.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Aaron]: Yeah, that would be great.""" start="00:19:53.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I'm having trouble getting to that tab.""" start="00:19:55.220" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Corwin]: Okay, I'm there, so I'll put it into our chat too,""" start="00:19:57.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""so you can follow along if you'd like.""" start="00:20:02.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Corwin]: The first question was, why is the David Bowie question""" start="00:20:07.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""a good one to start with?""" start="00:20:11.220" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Does it have interesting failure conditions""" start="00:20:12.220" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""or what made you choose that?""" start="00:20:14.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Aaron]: First off, huge fan of David Bowie.""" start="00:20:16.640" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""But I came down to it really taught me a few things""" start="00:20:21.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""about how the models work""" start="00:20:24.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""in terms of things like how many kids he had,""" start="00:20:26.300" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""because Deepseek, which is a very popular Chinese model""" start="00:20:28.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""that a lot of people are using now,""" start="00:20:31.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""misidentifies him having three daughters,""" start="00:20:33.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and he has like one son and one, one, I think,""" start="00:20:35.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""two sons and a daughter or something like that.""" start="00:20:38.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""so there's differences on that, and it just goes over...""" start="00:20:40.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""there's a whole lot of stuff""" start="00:20:43.660" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""because his story spans like 60 years,""" start="00:20:45.300" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""so it gives good feedback.""" start="00:20:47.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""That's the real main reason I asked that question""" start="00:20:49.660" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""because I just needed one... That sea monkeys, I just picked""" start="00:20:51.540" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""because it was obscure, and just always have, write,""" start="00:20:53.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I used to have it write hello world in forth""" start="00:20:56.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""because I thought was an interesting one as well.""" start="00:20:58.940" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It's just picking random ones like that.""" start="00:21:01.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""One question I ask a lot of models is,""" start="00:21:03.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""what is the closest star to the Earth?""" start="00:21:06.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Because most of them will say Alpha Centauri""" start="00:21:09.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""or Proxima Centauri and not the sun.""" start="00:21:12.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And I have a whole 'nother talk""" start="00:21:13.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""where I just argue with the LLM""" start="00:21:15.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""trying to say, hey, the sun is a star.""" start="00:21:17.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And he just wouldn't accept it, so. What?""" start="00:21:20.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Oh, I can... You're there.""" start="00:21:26.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Corwin]: So what specific tasks do you like to use your local AI?""" start="00:21:30.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Aaron]: I like to load a lot of my code into""" start="00:21:34.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and actually have it do analysis of it.""" start="00:21:37.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I was actually going through some code""" start="00:21:39.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I have for some pen testing, and I was having it modified""" start="00:21:42.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""to update it for the newer version,""" start="00:21:45.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""because I hate to say this,""" start="00:21:47.260" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but it was written for Python 2,""" start="00:21:48.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and I needed to update it for Python 3.""" start="00:21:49.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And the 2 to 3 tool did not do all of it,""" start="00:21:51.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but the actual tool was able to do the refactoring.""" start="00:21:53.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It's part of my laziness.""" start="00:21:56.660" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""But I use that for anything I don't want to hit the web.""" start="00:21:58.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And that's a lot of stuff when you start thinking about""" start="00:22:01.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""if you're doing cyber security researching.""" start="00:22:03.260" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and you have your white papers""" start="00:22:04.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and stuff like that and stuff in there.""" start="00:22:06.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I've got a lot of that loaded into RAG""" start="00:22:08.418" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""in one model on my Open WebUI system.""" start="00:22:10.626" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Corwin]: Neat. Have you used have you used""" start="00:22:16.880" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""any small domain specific LLMs? What kind of tasks?""" start="00:22:21.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""If so, what kind of tasks that they specialize in?""" start="00:22:25.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And you know, how?""" start="00:22:30.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Aaron]: Not to be honest, but there are some out there like once again,""" start="00:22:32.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""for cybersecurity and stuff like that,""" start="00:22:34.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""that I really need to dig into that's on my to do list.""" start="00:22:36.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I've got a couple weeks off at the end of the year.""" start="00:22:39.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And that's a big part of my plan for that.""" start="00:22:41.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Corwin]: Are the various models updated pretty regularly?""" start="00:22:46.540" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Can you add your own data to the pre-built models?""" start="00:22:49.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Aaron]: Yes. The models are updated pretty reasonably.""" start="00:22:52.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""You can add data to a model in a couple of different ways.""" start="00:22:56.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""You can do something called fine-tuning,""" start="00:22:59.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""which requires a really nice GPU and a lot of CPU time.""" start="00:23:01.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Probably not going to do that.""" start="00:23:03.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""You can do retrieval augmentation generation,""" start="00:23:05.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""which is you load your data on top of the system""" start="00:23:07.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and put inside a database,""" start="00:23:09.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and you can actually scan that and stuff.""" start="00:23:11.300" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I have another talk where I go through""" start="00:23:12.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and I start asking questions about,""" start="00:23:14.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I load the talk into the engine""" start="00:23:16.220" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and I ask questions against that.""" start="00:23:18.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""If I would have had time, I would have done that,""" start="00:23:20.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but it comes down to how many... That's RAG.""" start="00:23:22.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""RAG is pretty easy to do through Open WebUI or LM studio.""" start="00:23:25.797" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It's a great way, you just, like,""" start="00:23:29.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""point it to a folder and it just sucks all that state into...""" start="00:23:31.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and it'll hit that data first.""" start="00:23:34.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""You have like helpdesk and stuff and...""" start="00:23:35.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""The other options: there's vector databases,""" start="00:23:36.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""which is, like, if you use PostgreSQL,""" start="00:23:39.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""it has a pg vector that can do a lot of that stuff.""" start="00:23:41.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I've not dug into that yet,""" start="00:23:43.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but that is also on that to-do list""" start="00:23:44.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I've got a lot of stuff planned for...""" start="00:23:46.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Corwin]: Cool. So what are your experience with RAGs?""" start="00:23:48.056" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I don't even know what that means.""" start="00:23:51.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Do you know what that means?""" start="00:23:54.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Do you remember this question again?""" start="00:23:57.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""What is your experience with RAGs?""" start="00:23:59.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Aaron]: RAGs is great. That's Retrieval Augmentation Generation.""" start="00:24:03.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""That loads your data first, and it hits yours,""" start="00:24:07.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and it'll actually cite it and stuff.""" start="00:24:09.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""There's a guy who wrote a RAG in 100 lines of Python,""" start="00:24:11.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and it's an impressive piece of software.""" start="00:24:14.660" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I think if you hit one of my sites,""" start="00:24:16.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I've got a private AI talk where I actually refer to that.""" start="00:24:18.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""But retrieval augmentation, it's easy, it's fast,""" start="00:24:22.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""it puts your data into the system,""" start="00:24:25.220" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Yeah, start with that and go then iterate on top of that.""" start="00:24:26.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""That's one of the great things about AI,""" start="00:24:31.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""especially private AI,""" start="00:24:32.660" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""is you can do whatever you want to with it""" start="00:24:33.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and build up with it as you get more experience.""" start="00:24:35.626" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Corwin]: Any thoughts on running things""" start="00:24:38.834" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""on AWS, DigitalOcean, and so on?""" start="00:24:44.220" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Aaron]: AWS is not bad.""" start="00:24:49.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""The DigitalOcean, they have some of their GPUs.""" start="00:24:50.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I still don't like having the data""" start="00:24:52.660" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""leave my house, to be honest, or at work,""" start="00:24:54.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""because I tend to do some stuff""" start="00:24:57.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""that I don't want it even hitting that situation.""" start="00:24:59.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""But they have pretty good stuff.""" start="00:25:01.260" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Another one to consider is Oracle Cloud.""" start="00:25:03.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Oracle has their AI infrastructure that's really well done.""" start="00:25:05.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""But I mean, once again, then you start looking at potential""" start="00:25:09.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""is saying your data is private,""" start="00:25:12.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I don't necessarily trust it.""" start="00:25:13.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""But they do have good stuff, both DigitalOcean, AWS,""" start="00:25:14.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Oracle Cloud has the free service, which isn't too bad,""" start="00:25:17.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""usually a certain number of stuff.""" start="00:25:20.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And Google's also has it,""" start="00:25:21.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but I still tend to keep more stuff on local PCs,""" start="00:25:23.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""because I'm just paranoid that way.""" start="00:25:26.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Corwin]: Gotcha. What has your experience been using AI?""" start="00:25:31.078" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Do you want to get into that, using AI for cybersecurity?""" start="00:25:35.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""You might have already touched on this.""" start="00:25:40.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Aaron]: Yeah, really, for cybersecurity,""" start="00:25:42.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""what I've had to do is I've dumped logs""" start="00:25:44.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""to have it do correlation.""" start="00:25:46.260" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Keep in mind, the size of that Llama file we were using""" start="00:25:47.300" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""for figuring out David Bowie, writing the hello world,""" start="00:25:49.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""all that stuff, is like six gig.""" start="00:25:52.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""How does it get the entire world in six gig?""" start="00:25:54.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I still haven't figured that out in terms of quantization.""" start="00:25:56.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So I'm really interested in seeing the ability""" start="00:25:59.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""to take all this stuff out of all my logs,""" start="00:26:02.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""dump it all in there,""" start="00:26:05.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and actually be able to do intelligent queries against that.""" start="00:26:06.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Microsoft has a project called Security Copilot,""" start="00:26:08.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""which is trying to do that in the Cloud.""" start="00:26:10.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""But I want to work on something to do that more locally""" start="00:26:12.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and be able to actually drive this stuff over that.""" start="00:26:15.300" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""That's one also on the long-term goals.""" start="00:26:19.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Corwin]: So we got any other questions or?""" start="00:26:24.660" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Those are the questions that I see.""" start="00:26:26.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I want to just read out a couple of comments""" start="00:26:29.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""that I saw in IRC though.""" start="00:26:31.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""jrootabaga says, it went very well""" start="00:26:33.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""from an audience perspective.""" start="00:26:36.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And GGundam says, respect your commitment to privacy.""" start="00:26:39.260" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And then somebody is telling us""" start="00:26:43.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""we might have skipped a question.""" start="00:26:45.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So I'm just going to run back to my list.""" start="00:26:46.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Updated regularly experience.""" start="00:26:50.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I just didn't type in the answer here's""" start="00:26:52.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and there's a couple more questions coming in so""" start="00:26:57.660" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Is there a disparity where you go to paid models""" start="00:26:59.660" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""because they are better and what problems?""" start="00:27:04.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""You know what would drive you to? That's a good question.""" start="00:27:08.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Paid models, I don't mind them. I think they're good,""" start="00:27:14.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but I don't think they're actually economically sustainable""" start="00:27:17.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""under their current system.""" start="00:27:21.300" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Because right now, if you're paying""" start="00:27:22.660" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""20 bucks a month for Copilot and that goes up to 200 bucks,""" start="00:27:24.300" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I'm not going to be as likely to use it.""" start="00:27:26.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""You know what I mean?""" start="00:27:28.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""But it does do some things in a way that I did not expect.""" start="00:27:29.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""For example, Grok was refactoring""" start="00:27:33.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""some of my code in the comments and dropped an F-bomb.""" start="00:27:35.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""which I did not see coming,""" start="00:27:38.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but the other code before""" start="00:27:39.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""that I had gotten off GitHub""" start="00:27:41.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""had F bombs in it.""" start="00:27:43.220" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So it was just emulating the style,""" start="00:27:44.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but would that be something""" start="00:27:45.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I'd want to turn in a pull request? I don't know.""" start="00:27:47.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""But, uh, there's, there's a lot of money""" start="00:27:49.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""going into these AIs and stuff,""" start="00:27:52.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but in terms of the ability to get a decent one,""" start="00:27:53.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""like the llama, llama 3.2,""" start="00:27:56.220" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and load your data into it, you can be pretty competitive.""" start="00:27:57.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""You're not going to get all the benefits,""" start="00:28:01.240" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but you have more control over it.""" start="00:28:02.793" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So it's a balancing act.""" start="00:28:04.334" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Corwin]: Okay, and I think I see a couple more questions coming in.""" start="00:28:11.001" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""What is the largest parameter size for local models""" start="00:28:14.126" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""that you've been able to successfully run locally""" start="00:28:19.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and do you run into issues with limited context window size?""" start="00:28:22.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""The top paid models will tend to have a larger ceiling.""" start="00:28:26.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Aaron]: Yes, yes, yes, yes, yes.""" start="00:28:29.660" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""By default, the context size is I think 1024.""" start="00:28:32.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""But I've upped it to 8192 on this box, the Pangolin,""" start="00:28:37.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""because it seems to be, for some reason,""" start="00:28:41.161" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""it's just a very... working quite well.""" start="00:28:43.543" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""But the largest ones I've loaded have been in the...""" start="00:28:45.209" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""have not been that huge.""" start="00:28:49.751" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I've loaded this... the last biggest one I've done...""" start="00:28:51.334" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""That's the reason why I'm planning""" start="00:28:55.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""on breaking down and buying a Ryzen.""" start="00:28:57.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Actually, I'm going to buy""" start="00:29:01.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""an Intel i285H with 96 gig of RAM.""" start="00:29:03.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Then I should be able to load""" start="00:29:06.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""a 70 billion parameter model in that. How fast will it run?""" start="00:29:08.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It's going to run slow as dog,""" start="00:29:12.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but it's going to be cool to be able to do it.""" start="00:29:13.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It's an AI bragging rights thing,""" start="00:29:15.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but I mostly stick with the smaller size models""" start="00:29:17.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and the ones that are more quantitized""" start="00:29:20.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""because it just tends to work better for me.""" start="00:29:22.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Corwin]: We've still got over 10 minutes before we're cutting away,""" start="00:29:26.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but I'm just anticipating""" start="00:29:29.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""that we're going to be going strong at the 10 minute mark.""" start="00:29:30.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So I'm just, just letting, you know,""" start="00:29:32.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""we can go as long as we like here at a certain point.""" start="00:29:34.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I may have to jump away and check in with the next speaker,""" start="00:29:37.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but we'll post the entirety of this,""" start="00:29:41.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""even if we aren't able to stay with it all.""" start="00:29:44.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Okay. And we've got 10 minutes""" start="00:29:47.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""where we're still going to stay live.""" start="00:29:49.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So next question coming in, I see, are there free as in freedom,""" start="00:29:52.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""free as in FSF issues with the data?""" start="00:30:00.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Aaron]: Yes, where's the data coming from is a huge question with AI.""" start="00:30:05.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It's astonishing you can ask questions""" start="00:30:11.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""to models that you don't know where it's coming from.""" start="00:30:13.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""That is gonna be one of the big issues long-term.""" start="00:30:16.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""There are people who are working""" start="00:30:19.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""on trying to figure out that stuff,""" start="00:30:21.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but it's, I mean, if you look at, God,""" start="00:30:22.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I can't remember who it was.""" start="00:30:25.260" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Somebody was actually out torrenting books""" start="00:30:27.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""just to be able to build it into their AI system.""" start="00:30:28.660" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I think it might've been Meta.""" start="00:30:30.940" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So there's a lot of that going on.""" start="00:30:32.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""The open source of the stuff is going to be tough.""" start="00:30:34.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""There's going to be there's some models""" start="00:30:38.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""like the mobile guys have got their own license,""" start="00:30:39.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but where they're getting their data from,""" start="00:30:41.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I'm not sure, so that's a huge question.""" start="00:30:42.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""That's a talk in itself.""" start="00:30:45.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""But yeah, if you train on your RAG and your data,""" start="00:30:47.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""you know what it's come, you know,""" start="00:30:51.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""you have a license that""" start="00:30:53.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but the other stuff is just""" start="00:30:54.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""more lines of supplement""" start="00:30:55.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""if you're using a smaller model.""" start="00:30:56.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Corwin]: The comments online, I see a couple of them.""" start="00:31:01.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I'll read them out in order here. Really interesting stuff.""" start="00:31:05.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Thank you for your talk.""" start="00:31:08.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Given that large AI companies""" start="00:31:09.557" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""are openly stealing intellectual property and copyright""" start="00:31:11.660" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and therefore eroding the authority of such laws""" start="00:31:14.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and maybe obscuring the truth itself,""" start="00:31:18.940" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""can you see a future where IP and copyright flaw become untenable?""" start="00:31:21.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I think that's a great question.""" start="00:31:26.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I'm not a lawyer, but it is really getting complicated.""" start="00:31:29.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It is getting to the point, I asked a question from,""" start="00:31:34.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I played with Sora a little bit, and it generated someone,""" start="00:31:37.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""you can go like, oh, that's Jon Hamm,""" start="00:31:41.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""that's Christopher Walken,""" start="00:31:42.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""you start figuring out who the people""" start="00:31:44.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""they're modeling stuff after.""" start="00:31:45.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""There is an apocalypse, something""" start="00:31:47.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""going to happen right now.""" start="00:31:48.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""There is, but this is once again,""" start="00:31:52.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""my personal opinion, and I'm not a lawyer,""" start="00:31:53.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and I do not have money.""" start="00:31:56.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So don't sue me, is there's going to be""" start="00:31:57.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""the current administration tends is very AI, pro AI.""" start="00:31:58.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And there's very a great deal of lobbying by those groups.""" start="00:32:02.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And it's on both sides.""" start="00:32:05.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And it's going to be, it's gonna be interesting to see""" start="00:32:07.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""what happens to copyright the next 510 years.""" start="00:32:09.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I just don't know how it keeps up""" start="00:32:11.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""without there being some adjustments and stuff.""" start="00:32:13.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Corwin]: Okay, and then another comment I saw,""" start="00:32:18.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""file size is not going to be a bottleneck.""" start="00:32:20.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""RAM is. You'll need 16 gigabytes of RAM""" start="00:32:23.220" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""to run the smallest local models""" start="00:32:25.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and 512 gigabytes of RAM to run the larger ones.""" start="00:32:28.260" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""You'll need a GPU with that much memory""" start="00:32:31.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""if you want it to run quickly.""" start="00:32:35.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Aaron]: Yeah. Oh no. It also depends upon how your memory is laid out.""" start="00:32:38.319" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Like example being the Ultra i285H""" start="00:32:41.260" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I plan to buy, that has 96 gig of memory.""" start="00:32:45.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It's unified between the GPU and the CPU share it,""" start="00:32:47.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but they go over the same bus.""" start="00:32:50.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So the overall bandwidth of it tends to be a bit less,""" start="00:32:52.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but you're able to load more of it into memory.""" start="00:32:55.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So it's able to do some additional stuff with it""" start="00:32:57.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""as opposed to come off disk.""" start="00:32:59.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It's all balancing act. If you hit Ziskind's website,""" start="00:33:00.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""that guy's done some great work on it.""" start="00:33:03.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I'm trying to figure out how big a model you can do,""" start="00:33:05.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""what you can do with it.""" start="00:33:07.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And some of the stuff seems to be not obvious,""" start="00:33:08.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""because like example, being that MacBook Air,""" start="00:33:12.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""for the five minutes I can run the model,""" start="00:33:15.300" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""it runs it faster than a lot of other things""" start="00:33:17.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""that should be able to run it faster,""" start="00:33:19.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""just because of the way the ARM cores and the unified memory work on it.""" start="00:33:21.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So it's a learning process.""" start="00:33:24.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""But if you want to, Network Chuck had a great video""" start="00:33:26.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""talking about building his own system""" start="00:33:29.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""with a couple really powerful Nvidia cards""" start="00:33:30.940" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and stuff like that in it.""" start="00:33:34.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And just actually setting up on his system as a node""" start="00:33:35.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and using a web UI on it. So there's a lot of stuff there,""" start="00:33:38.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but it is a process of learning how big your data is,""" start="00:33:41.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""which models you want to use,""" start="00:33:43.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""how much information you need,""" start="00:33:44.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but it's part of the learning.""" start="00:33:46.220" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And you can run models, even on Raspberry Pi 5s,""" start="00:33:49.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""if you want to, they'll run slow.""" start="00:33:52.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Don't get me wrong, but they're possible.""" start="00:33:54.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Corwin]: Okay, and I think there's other questions coming in too,""" start="00:33:59.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""so I'll just bam for another second.""" start="00:34:02.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""We've got about five minutes before we'll,""" start="00:34:04.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""before we'll be cutting over,""" start="00:34:06.300" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but I just want to say in case we get close for time here,""" start="00:34:09.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""how much I appreciate your talk.""" start="00:34:13.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""This is another one that I'm going to""" start="00:34:14.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""have to study after the conference.""" start="00:34:15.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Aaron]: We greatly appreciate, all of us appreciate""" start="00:34:18.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""you guys putting on the conference.""" start="00:34:21.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It's a great conference. It's well done.""" start="00:34:22.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Corwin]: It's an honor to be on the stage""" start="00:34:26.300" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""with the brains of the project, which is you.""" start="00:34:28.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Aaron]: So what else we got? Question wise.""" start="00:34:33.125" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Corwin]: Okay, so just scanning here.""" start="00:34:34.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Have you used local models capable of tool calling?""" start="00:34:46.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I'm scared of agentic.""" start="00:34:50.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I'm going to be a slow adopter of that.""" start="00:34:54.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I want to do it, but I just don't have the, uh,""" start="00:34:58.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""four decimal fortitude right now to do it.""" start="00:35:02.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I've had to give me the commands,""" start="00:35:04.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but I still run the commands by hand.""" start="00:35:07.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I'm looking into it and it's on once again,""" start="00:35:08.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""it's on that list, but I just, that's a big step for me.""" start="00:35:10.540" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Corwin]: So. Awesome. All right.""" start="00:35:20.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Well, maybe it's, let me just scroll through""" start="00:35:23.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""because we might have missed one question. Oh, I see.""" start="00:35:27.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Here was the piggyback question.""" start="00:35:31.540" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Now I see the question that I missed.""" start="00:35:36.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So this was piggybacking on the question""" start="00:35:38.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""about model updates and adding data.""" start="00:35:41.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And will models reach out to the web""" start="00:35:44.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""if they need more info?""" start="00:35:46.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Or have you worked with any models that work that way?""" start="00:35:47.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Aaron]: No, I've not seen any models to do that""" start="00:35:52.480" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""There's there was like a group""" start="00:35:55.260" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""working on something like a package updater""" start="00:35:57.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""that would do different diffs on it,""" start="00:35:59.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but it's so... Models change so much,""" start="00:36:02.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""even who make minor changes and fine-tuning,""" start="00:36:03.940" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It's hard just to update them in place.""" start="00:36:05.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So I haven't seen one, but that doesn't mean""" start="00:36:07.660" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""they're not out there. Curious topic though.""" start="00:36:10.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Corwin]: Awesome.""" start="00:36:15.714" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Well, it's probably pretty good timing.""" start="00:36:16.260" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Let me just scroll and make sure.""" start="00:36:19.540" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""And of course, before I can say that,""" start="00:36:21.300" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""there's one more question. So let's go ahead and have that.""" start="00:36:23.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I want to make sure while we're still live, though,""" start="00:36:25.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I give you a chance to offer any closing thoughts.""" start="00:36:28.300" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So what scares you most about the agentic tools?""" start="00:36:31.300" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""How would you think about putting a sandbox around that""" start="00:36:35.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""if you did adopt an agentic workflow?""" start="00:36:38.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Aaron]: That is a great question.""" start="00:36:41.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""In terms of that, I would just control""" start="00:36:42.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""what it's able to talk to, what machines,""" start="00:36:45.940" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I would actually have it be air gap.""" start="00:36:48.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I work for a defense contractor,""" start="00:36:50.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and we spend a lot of time dealing with air gap systems,""" start="00:36:52.100" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""because that's just kind of the way it works out for us.""" start="00:36:53.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So agentic, it's just going to take a while to get trust.""" start="00:36:55.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I want to see more stuff happening.""" start="00:36:58.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Humans screw up stuff enough.""" start="00:37:01.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""The last thing we need is to multiply that by 1000.""" start="00:37:02.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So in terms of that, I would be restricting what it can do.""" start="00:37:04.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""If you look at the capabilities,""" start="00:37:09.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""if I created a user and gave it permissions,""" start="00:37:10.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I would have a lockdown through sudo,""" start="00:37:13.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""what it's able to do, what the account's able to do.""" start="00:37:15.300" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I would do those kind of things,""" start="00:37:17.380" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but it's going to be, it's happening.""" start="00:37:18.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It's just, I'm going to be one of the laggards on that one.""" start="00:37:20.860" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So air gap, jail, extremely locked down environments,""" start="00:37:25.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""like we're talking about separate physicals, not Docker.""" start="00:37:29.260" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Yeah, hopefully.""" start="00:37:34.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Corwin]: Right, fair. So tool calling can be read-only,""" start="00:37:36.578" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""such as giving models the ability to search the web""" start="00:37:39.900" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""before answering your question,""" start="00:37:42.540" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""you know, write access, execute access.""" start="00:37:43.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I'm interested to know if local models""" start="00:37:46.220" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""are any good at that.""" start="00:37:49.220" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Aaron]: Yes, local models can do a lot of that stuff.""" start="00:37:51.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It's their capabilities.""" start="00:37:55.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""If you load LM studio, you can do a lot of wonderful stuff""" start="00:37:56.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""with that or with Open Web UI with ollama.""" start="00:37:59.020" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It's a lot of capabilities. It's amazing.""" start="00:38:02.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Open Web UI is actually what a lot of companies are using now""" start="00:38:05.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""to put their data behind that.""" start="00:38:08.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""They're curated data and stuff like that. So works well.""" start="00:38:10.260" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I can confirm that from my own professional experience.""" start="00:38:12.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Excellent.""" start="00:38:15.820" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Corwin]: Okay, well, our timing should be just perfect""" start="00:38:16.916" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""if you want to give us like a 30-second, 45-second wrap-up.""" start="00:38:19.660" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Aaron, let me squeeze in mine.""" start="00:38:22.660" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Thank you again so much for preparing this talk""" start="00:38:24.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and for entertaining all of our questions.""" start="00:38:26.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Aaron]: Yeah, let me just thank you guys for the conference again.""" start="00:38:30.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""This is a great one. I've enjoyed a lot of it.""" start="00:38:33.300" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""I've only had a couple of talks so far,""" start="00:38:35.180" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but I'm looking forward to hitting the ones after this and tomorrow.""" start="00:38:37.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""But the AI stuff is coming. Get on board.""" start="00:38:41.660" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Definitely recommend it. If you want to just try it out""" start="00:38:44.740" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and get a little taste of it,""" start="00:38:46.940" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""what my minimal viable product""" start="00:38:48.420" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""with just Llamafile and gptel""" start="00:38:49.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""will get you to the point where you start figuring out.""" start="00:38:51.620" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Gptel is an amazing thing. It just gets out of your way,""" start="00:38:53.140" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""but it works so well with Emacs's design because""" start="00:38:55.580" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""it doesn't take your hands off the keyboard.""" start="00:39:00.460" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It's just another buffer,""" start="00:39:01.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and you just put information in there.""" start="00:39:02.500" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""It's quite a wonderful time.""" start="00:39:04.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Let's put that way. That's all I got.""" start="00:39:06.980" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Corwin]: Thank you so much for once again, and we've just cut away.""" start="00:39:10.502" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""So I'll stop the recording""" start="00:39:14.340" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and you're on your own recognizance.""" start="00:39:15.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""[Aaron]: Well, I'm gonna punch out""" start="00:39:18.260" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""if anybody has any questions or anything""" start="00:39:19.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""my email address is ajgrothe@yahoo.com or at gmail and""" start="00:39:21.060" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""thank you all for attending,""" start="00:39:24.700" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""and thanks again for the conference""" start="00:39:26.780" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Okay, I'm gonna go ahead and end the room there, thank you.""" start="00:39:29.940" video="mainVideo-private-ai" id="subtitle"]] [[!template text="""Excellent, thanks, bye.""" start="00:39:32.580" video="mainVideo-private-ai" id="subtitle"]]