Hey, everybody. Welcome from frigid Omaha, Nebraska. @@ -10,7 +12,7 @@ I'm just going to kick off my talk here, and we'll see how it all goes. Thanks for attending.
-So the slides will be available on my site, growthy.us, +So the slides will be available on my site, https://grothe.us,
in the presentation section tonight or tomorrow. @@ -39,8 +41,10 @@ Really hope some of you give it a shot
and learn something along the way. +NOTE Overview of talk +
-So the overview of the talk. +So the overview of the talk
broke down these basic bullet points of why private AI, @@ -54,6 +58,8 @@ pieces for an AI Emacs solution,
a demo of a minimal viable product, and the summary. +NOTE Why private AI? +
Why private AI? This is pretty simple. @@ -67,7 +73,7 @@ for any AI system you're currently using. If you're using the free tiers, your queries,
-code uploaded information +code, uploaded information
is being used to train the models. @@ -139,7 +145,7 @@ The problem with that is now there's less data going to Stack Overflow
-for the AI to get. vicious cycle, +for the AI to get. Vicious cycle,
especially when you start looking at @@ -160,7 +166,7 @@ Right now, these services are being heavily subsidized. If you're paying Claude $20 a month,
-it is not costing Claude, those guys $20 a month +it is not costing Claude, those guys, $20 a month
to host all the infrastructure @@ -219,6 +225,8 @@ are going to have no option but to move to private AI
or give up on AI overall. +NOTE What do I need for private AI? +
What do you need to be able to do private AI? @@ -316,7 +324,7 @@ And Raspberry Pi's new AI top is about 18 teraflops and is $70 on top of the cost of Raspberry Pi 5.
-Keep in mind Raspberry recently +Keep in mind, Raspberry recently
raised the cost of their Pi 5s because of RAM pricing, @@ -342,8 +350,13 @@ A lot of people are going to have PCs on their desks.
They're going to run a decent private AI
-without much issue. So for Emacs and private AI, +00:05:13.460 --> 00:05:16.347 +without much issue. + +NOTE Emacs and private AI + +00:05:16.348 --> 00:05:18.059 +So for Emacs and private AI,
there's a couple popular solutions. @@ -364,7 +377,7 @@ It's just, quite honestly, chef's kiss, just a beautifully well-done piece of software.
-OllamaBuddy has more features, +Ollama Buddy has more features,
a menu interface, has quick access @@ -382,7 +395,7 @@ This is the one that you spend a little more time with, but you also get a little bit more back from it.
-Elama is another one, has some really good features to it, +Ellama is another one, has some really good features to it,
more different capabilities, @@ -391,13 +404,13 @@ more different capabilities, but it's a different set of rules and capabilities to it.
-Itermac, which is programming with your AI and Emacs. +Aidermac, which is programming with your AI and Emacs.
The closest thing I can come up
-to comparing this to is Cursor, except it's an Emacs. +to comparing this to is Cursor, except it's in Emacs.
It's really quite well done. @@ -414,6 +427,8 @@ If you go out to GitHub, type Emacs AI,
you'll find a lot of different options. +NOTE Pieces for an AI Emacs solution +
So what is a minimal viable product that can be done? @@ -453,16 +468,16 @@ while it runs on a bunch of different systems.
And Gptel, which is an easy plug-in for Emacs,
+00:06:51.300 --> 00:06:56.339 which we talked about in the last slide a bit.
+00:06:56.340 --> 00:07:00.179 So setting up the LLM, you have to just go out
-and just hit the a page for it +00:07:00.180 --> 00:07:03.542 +and just hit a page for it
+00:07:03.543 --> 00:07:05.099 and go out and do a wget of it.
@@ -487,7 +502,7 @@ And let's just take a look at it. I've actually downloaded several of them,
-but let's go ahead and just run lava 3.2b +but let's go ahead and just run llama 3.2-1b
with the 3 billion instructions. And that's it firing up. @@ -505,10 +520,10 @@ So once you do that, you have to install gptel and emacs. That's as simple as firing up emacs,
-doing the meta x install package, +doing the M-x install-package,
-and then just typing gptel +and then just typing gptel,
if you have your repository set up right, @@ -516,17 +531,19 @@ if you have your repository set up right,
which hopefully you do.
+00:07:52.300 --> 00:07:56.339 And then you just go ahead and have it.
+NOTE Config file + +00:07:56.340 --> 00:07:58.139 You also have to set up a config file.
Here's my example config file as it currently set up,
-requiring ensuring Gptel is loaded, +requiring, ensuring Gptel is loaded,
defining the Llamafile backend. @@ -585,6 +602,8 @@ which is really kind of cool.
But, uh, that's all that takes. +NOTE Demo: Who was David Bowie? +
So let's go ahead and go to a quick test of it. @@ -619,10 +638,10 @@ And we can just do, we can either do the alt X and send the gptel-send,
-or we can just do control C and hit enter. +or we can just do C-c and hit enter.
-We'll just do control C and enter. +We'll just do C-c and enter.
And now it's going ahead and hitting our local AI system @@ -648,6 +667,8 @@ and other information about him.
So very, very happy with that. +NOTE Hallucinations +
One thing to keep in mind is you look at things @@ -672,6 +693,8 @@ Is it not actually on this one?
Alright, so that's the first question I always ask one. +NOTE Next question: What are sea monkeys? +
The next one is what are sea monkeys? @@ -702,10 +725,10 @@ I did not see that coming.
Here's one that some people say is a really good one
-to ask ours in strawberry. +00:11:24.140 --> 00:11:42.779 +to ask. Rs in "strawberry."
+00:11:42.780 --> 00:11:46.179 All right, now she's going off the reservation.
@@ -714,11 +737,13 @@ She's going in a different direction.
Let me go ahead and reopen that again,
-because it's went down a bad hole there for a second. +00:11:49.980 --> 00:11:57.179 +because it went down a bad hole there for a second. + +NOTE Writing Hello World in Emacs Lisp
-Let me ask it to do write hello world in Emacs list. +00:11:57.180 --> 00:11:58.419 +Let me ask it to write hello world in Emacs Lisp.
Yep, that works. So the point being here, @@ -741,6 +766,8 @@ We also have the default system here as well.
So not that bad. +NOTE Pieces for a better solution +
That's a basic solution, that's a basic setup @@ -778,7 +805,7 @@ But if you want a better solution, I recommend Ollama or LM Studio.
-They're both more capable than llama file. +They're both more capable than Llamafile.
They can accept a lot of different models. @@ -805,7 +832,7 @@ so you can start uploading your code, your information, and actually being able to do analysis of it.
-OpenWebUI provides more capabilities. +Open WebUI provides more capabilities.
It provides an interface that's similar @@ -814,7 +841,7 @@ It provides an interface that's similar to what you're used to seeing
-for chat, GPT, and the other systems. +for ChatGPT and the other systems.
It's really quite well done. @@ -826,7 +853,9 @@ And once again, gptel, I have to mention that because that's the one I really kind of like.
-And OlamaBuddy is also another really nice one. +And Ollama Buddy is also another really nice one. + +NOTE What about the license?
So what about the licensing of these models? @@ -841,7 +870,7 @@ a model and doing this stuff. Let's take a look at a couple of highlights
-from the MetaLlama 3 community license scale. +from the Meta Llama 3 community license scale.
If your service exceeds 700 million monthly users, @@ -892,7 +921,7 @@ And there are some very cool ones out there. There's people who've done things
-to try and make the llama bee less, what's the phrase, +to try and make the Llama be less, what's the phrase,
ethical if you're doing penetration testing research @@ -918,19 +947,21 @@ It's designed to keep it to research and development.
You can't use it commercially.
+00:14:46.740 --> 00:14:51.792 So it's designed to clearly delineate
+00:14:51.793 --> 00:14:52.939 between research and development
and somebody trying to actually build
+00:14:54.260 --> 00:14:56.579 something on top of it.
+NOTE Are there open source data model options? + +00:14:56.580 --> 00:14:57.979 And another question I get asked is,
@@ -951,17 +982,19 @@ but it's very cool to see where it's going.
And it's another thing that's just going forward.
+00:15:11.340 --> 00:15:14.519 It's under the MIT license.
+NOTE Things to know + +00:15:14.520 --> 00:15:15.819 Some things to know to help you
have a better experience with this.
-Get a Llama and OpenWebUI working by themselves, +Get ollama and Open WebUI working by themselves,
then set up your config file. @@ -970,7 +1003,7 @@ then set up your config file. I was fighting both at the same time,
-and it turned out I had a problem with my LLAMA. +and it turned out I had a problem with my ollama.
I had a conflict, so that was what my problem is. @@ -994,7 +1027,7 @@ You can create models, you can put things like help desk developers and stuff like that, breaking it out.
-The Hacker News has a how to build a $300 AI computer. +The Hacker Noon has a how to build a $300 AI computer.
This is for March 2024, @@ -1015,7 +1048,7 @@ inside my Acer Aspire, that's where I got the idea doing that.
-Make sure you do the RockM stuff correctly +Make sure you do the ROCm stuff correctly
to get the GUI extensions. But it's just really good stuff. @@ -1024,37 +1057,37 @@ to get the GUI extensions. But it's just really good stuff. You don't need a great GPU or CPU to get started.
-Smaller models like Tiny Llama +Smaller models like tinyllama
+00:16:14.820 --> 00:16:16.819 can run on very small systems.
+00:16:16.820 --> 00:16:19.042 It gets you the ability to start playing with it
+00:16:19.043 --> 00:16:21.619 and start experimenting and figure out if that's for you
and to move forward with it.
-The AMD Ryzen AI Max 395 plus is a mini PC +The AMD Ryzen AI Max+ 395 is a mini PC
makes it really nice dedicated host.
-You used to be able to buy these for about $1200 now +00:16:31.180 --> 00:16:34.078 +You used to be able to buy these for about $1200.
-with the RAM price increase, +00:16:34.079 --> 00:16:35.579 +Now with the RAM price increase,
-you want to get 120 gig when you're pushing two brands so. +00:16:35.580 --> 00:16:38.458 +you want to get 120 gig when you're pushing two brands,
-It gets a little tighter. +00:16:38.459 --> 00:16:40.739 +so it gets a little tighter.
Macs work remarkably well with AI. @@ -1093,16 +1126,16 @@ So Mac Minis and the Mac Ultras and stuff like that tend to work really well for that.
-Alex Ziskin on YouTube has a channel. +Alex Ziskind on YouTube has a channel.
He does a lot of AI performance benchmarking,
-like I load a 70 billion parameter model +like "I load a 70 billion parameter model
-on this mini PC and stuff like that. +on this mini PC" and stuff like that.
It's a lot of fun and interesting stuff there. @@ -1129,7 +1162,7 @@ It gives you capabilities to start training stuff that you couldn't do with like the big ones.
-Even with in terms of fine tuning and stuff, +Even with in terms of fine-tuning and stuff,
it's remarkable to see where that space is coming along @@ -1138,12 +1171,12 @@ it's remarkable to see where that space is coming along in the next year or so.
-Hugging Face Co has pointers to tons of AI models. +HuggingFace.co has pointers to tons of AI models.
+00:17:46.220 --> 00:17:48.417 You'll find the one that works for you, hopefully there.
+00:17:48.418 --> 00:17:50.539 If you're doing cybersecurity,
@@ -1174,7 +1207,7 @@ and run into those problems. So, That is my talk. What I would like you to get out of that is,
-if you haven't tried it, give GPTEL and LlamaFile a shot. +if you haven't tried it, give Gptel and LlamaFile a shot.
Fire up a little small AI instance, @@ -1201,7 +1234,7 @@ Let me see if we got anything you want, Pat. You do. You've got a few questions.
-Hey, this is Corwin. Thank you so much. Thank you, Aaron. +[Corwin]: Hey, this is Corwin. Thank you so much. Thank you, Aaron.
What an awesome talk this was, actually. @@ -1213,12 +1246,12 @@ If you don't have a camera, I can get away with not having one too.
-I've got, I'll turn the camera on. +[Aaron]: I've got, I'll turn the camera on.
-Okay. All right. I'll turn mine back on. Here I come. +00:18:56.300 --> 00:18:59.833 +[Corwin]: Okay. All right. I'll turn mine back on. Here I come.
+00:18:59.834 --> 00:19:03.139 Yeah, so there are a few questions,
@@ -1252,7 +1285,7 @@ put infrastructure we have at home to use and get more done with less.
-Yeah, the data center impact's interesting +[Aaron]: Yeah, the data center impact's interesting
because there was a study a while ago. @@ -1264,7 +1297,7 @@ Someone said every time you do a Gemini query, it's like boiling a cup of water.
-Yeah, I've heard that one too. So do you want to, you know, +[Corwin]: Yeah, I've heard that one too. So do you want to, you know,
I don't know how much direction you want. @@ -1273,19 +1306,21 @@ I don't know how much direction you want. I'd be very happy to read out the questions for you.
-Yeah, that would be great. +[Aaron]: Yeah, that would be great.
I'm having trouble getting to that tab.
-Okay, I'm there, so I'll put it into our chat too, +[Corwin]: Okay, I'm there, so I'll put it into our chat too,
so you can follow along if you'd like. +NOTE Q: Why is the David Bowie question a good one for testing a model? e.g. does it fail in interesting ways? +
-The first question was, why is the David Bowie question +[Corwin]: The first question was, why is the David Bowie question
a good one to start with? @@ -1293,23 +1328,23 @@ a good one to start with?
Does it have interesting failure conditions
+00:20:14.420 --> 00:20:16.639 or what made you choose that?
-First off, huge fan of David Bowie. +00:20:16.640 --> 00:20:21.979 +[Aaron]: First off, huge fan of David Bowie.
But I came down to it really taught me a few things
-about how old the models work +about how the models work
in terms of things like how many kids he had,
-because deep seek, which is a very popular Chinese model +because Deepseek, which is a very popular Chinese model
that a lot of people are using now, @@ -1324,37 +1359,37 @@ and he has like one son and one, one, I think, two sons and a daughter or something like that.
-so there's differences on that and it just goes over +so there's differences on that, and it just goes over...
there's a whole lot of stuff
-because his story spans like 60 years +because his story spans like 60 years,
-so it gives a good good feedback +so it gives good feedback.
-that's the real main reason I asked that question +That's the real main reason I asked that question
-because I just needed one that sea monkeys I just picked +because I just needed one... That sea monkeys, I just picked
-because it was obscure and just always have right +because it was obscure, and just always have, write,
-I used to have it right hello world and forth +I used to have it write hello world in forth
-because I thought was an interesting one as well so +because I thought was an interesting one as well.
It's just picking random ones like that.
-One question asked, sorry, a lot of models is, +One question I ask a lot of models is,
what is the closest star to the Earth? @@ -1366,7 +1401,7 @@ Because most of them will say Alpha Centauri or Proxima Centauri and not the sun.
-And I have a whole nother talk +And I have a whole 'nother talk
where I just argue with the LLM @@ -1377,19 +1412,21 @@ trying to say, hey, the sun is a star.
And he just wouldn't accept it, so. What?
-Oh, I can hear that. +00:21:26.580 --> 00:21:30.739 +Oh, I can... You're there. + +NOTE Q: What specific tasks do you use local AI for?
-So what specific tasks do you like to use your local AI? +00:21:30.740 --> 00:21:34.379 +[Corwin]: So what specific tasks do you like to use your local AI?
-I like to load a lot of my code into +[Aaron]: I like to load a lot of my code into
+00:21:37.460 --> 00:21:39.099 and actually have it do analysis of it.
+00:21:39.100 --> 00:21:42.339 I was actually going through some code
@@ -1428,17 +1465,19 @@ if you're doing cyber security researching.
and you have your white papers
+00:22:06.820 --> 00:22:08.417 and stuff like that and stuff in there.
+00:22:08.418 --> 00:22:10.625 I've got a lot of that loaded into RAG
-in one model on my OpenWebUI system. +00:22:10.626 --> 00:22:16.879 +in one model on my Open WebUI system.
-Neat. Have you used have you used +NOTE Q: Have you used any small domain-specific LLMs? What are the kinds of tasks they specialize in, and how do I find and use them? + +00:22:16.880 --> 00:22:21.059 +[Corwin]: Neat. Have you used have you used
any small domain specific LLMs? What kind of tasks? @@ -1450,7 +1489,7 @@ If so, what kind of tasks that they specialize in? And you know, how?
-Not to be honest, but there are some out there like once again, +[Aaron]: Not to be honest, but there are some out there like once again,
for cybersecurity and stuff like that, @@ -1461,17 +1500,19 @@ that I really need to dig into that's on my to do list.
I've got a couple weeks off at the end of the year.
+00:22:41.700 --> 00:22:46.539 And that's a big part of my plan for that.
-Are the various models updated pretty regularly? +NOTE Q: Are the various models updated regularly? Can you add your own data to pre-built models? + +00:22:46.540 --> 00:22:49.379 +[Corwin]: Are the various models updated pretty regularly?
Can you add your own data to the pre-built models?
-Yes. The models are updated pretty reasonably. +[Aaron]: Yes. The models are updated pretty reasonably.
You can add data to a model in a couple of different ways. @@ -1492,7 +1533,7 @@ You can do retrieval augmentation generation, which is you load your data on top of the system
-and puts inside a database +and put inside a database,
and you can actually scan that and stuff. @@ -1510,34 +1551,34 @@ I load the talk into the engine and I ask questions against that.
-I would have one more time would have done that +If I would have had time, I would have done that,
-but it comes down to how many That's that's rag rag +00:23:22.180 --> 00:23:25.796 +but it comes down to how many... That's RAG.
-is pretty easy to do through open web UI or LM studio +00:23:25.797 --> 00:23:29.419 +RAG is pretty easy to do through Open WebUI or LM studio.
-It's a great way you just like point a folder +It's a great way, you just, like,
-point it to a folder and it just sucks all that state into +point it to a folder and it just sucks all that state into...
-and it'll hit that data first +and it'll hit that data first.
-you have like helpdesk and stuff and +You have like helpdesk and stuff and...
-The other options there's vector databases, +The other options: there's vector databases,
-which is like if you use PostgreSQL. +which is, like, if you use PostgreSQL,
-It has a PG vector I can do a lot of that stuff. +it has a pg vector that can do a lot of that stuff.
I've not dug into that yet, @@ -1545,11 +1586,13 @@ I've not dug into that yet,
but that is also on that to-do list
-I've got a lot of stuff planned for Cool. +00:23:46.100 --> 00:23:48.055 +I've got a lot of stuff planned for... + +NOTE Q: What is your experience with RAG? Are you using them and how have they helped?
-So what are your experience with rags? +00:23:48.056 --> 00:23:51.819 +[Corwin]: Cool. So what are your experience with RAGs?
I don't even know what that means. @@ -1561,10 +1604,10 @@ Do you know what that means? Do you remember this question again?
-What is your experience with RAGs? RAGs is great. +What is your experience with RAGs?
-That's Retrieval Augmentation Generation. +[Aaron]: RAGs is great. That's Retrieval Augmentation Generation.
That loads your data first, and it hits yours, @@ -1579,7 +1622,7 @@ There's a guy who wrote a RAG in 100 lines of Python, and it's an impressive piece of software.
-I think if you hit one of my site, +I think if you hit one of my sites,
I've got a private AI talk where I actually refer to that. @@ -1599,20 +1642,22 @@ That's one of the great things about AI,
especially private AI,
+00:24:33.620 --> 00:24:35.625 is you can do whatever you want to with it
+00:24:35.626 --> 00:24:38.833 and build up with it as you get more experience.
-Any thoughts on running things +NOTE Q: Thoughts on running things on AWS/digital ocean instances, etc? + +00:24:38.834 --> 00:24:44.219 +[Corwin]: Any thoughts on running things
on AWS, DigitalOcean, and so on?
-AWS is not bad. +[Aaron]: AWS is not bad.
The DigitalOcean, they have some of their GPUs. @@ -1662,11 +1707,13 @@ And Google's also has it,
but I still tend to keep more stuff on local PCs,
-because I just paranoid that way. Gotcha. +00:25:26.740 --> 00:25:31.077 +because I'm just paranoid that way. + +NOTE Q: What has your experience been using AI for cyber security applications? What do you usually use it for?
-What has your experience been using AI? +00:25:31.078 --> 00:25:35.579 +[Corwin]: Gotcha. What has your experience been using AI?
Do you want to get into that, using AI for cybersecurity? @@ -1675,16 +1722,16 @@ Do you want to get into that, using AI for cybersecurity? You might have already touched on this.
-Yeah, really, for cybersecurity, +[Aaron]: Yeah, really, for cybersecurity,
what I've had to do is I've dumped logs
-to have a due correlation. +to have it do correlation.
-Keep in mind, the size of that LLAMA file we were using +Keep in mind, the size of that Llama file we were using
for figuring out David Bowie, writing the hello world, @@ -1722,11 +1769,11 @@ But I want to work on something to do that more locally
and be able to actually drive this stuff over that.
+00:26:19.020 --> 00:26:24.659 That's one also on the long-term goals.
-So we got any other questions or? +00:26:24.660 --> 00:26:26.059 +[Corwin]: So we got any other questions or?
Those are the questions that I see. @@ -1738,13 +1785,13 @@ I want to just read out a couple of comments that I saw in IRC though.
-Jay Rutabaga says, it went very well +jrootabaga says, it went very well
from an audience perspective.
-And G Gundam says, respect your commitment to privacy. +And GGundam says, respect your commitment to privacy.
And then somebody is telling us @@ -1764,6 +1811,8 @@ I just didn't type in the answer here's
and there's a couple more questions coming in so +NOTE Q: Is there a disparity where you go to paid models becouse they are better and what problems would those be? +
Is there a disparity where you go to paid models @@ -1834,61 +1883,60 @@ going into these AIs and stuff, but in terms of the ability to get a decent one,
-like the llama, llama three, two, +like the llama, llama 3.2,
+00:27:57.980 --> 00:28:01.239 and load your data into it, you can be pretty competitive.
+00:28:01.240 --> 00:28:02.792 You're not going to get all the benefits,
+00:28:02.793 --> 00:28:04.333 but you have more control over it.
-So it's, it's a, this and that it's a, +00:28:04.334 --> 00:28:11.000 +So it's a balancing act.
-it's a balancing act. +00:28:11.001 --> 00:28:14.125 +[Corwin]: Okay, and I think I see a couple more questions coming in.
-Okay, and I think I see a couple more questions coming in. +NOTE Q: What's the largest (in parameter size) local model you've been able to successfully run locally, and do you run into issues with limited context window size?
+00:28:14.126 --> 00:28:19.619 What is the largest parameter size for local models
that you've been able to successfully run locally
-and do run into issues with limited context window size? +and do you run into issues with limited context window size?
-The top eight models will tend to have a larger ceiling. +The top paid models will tend to have a larger ceiling.
-Yes, yes, yes, yes, yes. +[Aaron]: Yes, yes, yes, yes, yes.
By default, the context size is I think 1024.
-But I've upped it to 8192 on the on this box, the Pangolin +00:28:37.020 --> 00:28:41.160 +But I've upped it to 8192 on this box, the Pangolin,
-because it seems to be some reason +00:28:41.161 --> 00:28:43.542 +because it seems to be, for some reason,
-it's just a very working quite well. +00:28:43.543 --> 00:28:45.208 +it's just a very... working quite well.
-But the largest ones I've loaded have been in +00:28:45.209 --> 00:28:49.750 +But the largest ones I've loaded have been in the...
-the have not been that huge. +00:28:49.751 --> 00:28:51.333 +have not been that huge.
-I've loaded this the last biggest one I've done. +00:28:51.334 --> 00:28:55.699 +I've loaded this... the last biggest one I've done...
That's the reason why I'm planning @@ -1927,7 +1975,7 @@ and the ones that are more quantitized because it just tends to work better for me.
-We've still got over 10 minutes before we're cutting away, +[Corwin]: We've still got over 10 minutes before we're cutting away,
but I'm just anticipating @@ -1956,6 +2004,8 @@ Okay. And we've got 10 minutes
where we're still going to stay live. +NOTE Q: Are there "Free" as in FSF/open source issues with the data? +
So next question coming in, I see, are there free as in freedom, @@ -1963,7 +2013,7 @@ So next question coming in, I see, are there free as in freedom, free as in FSF issues with the data?
-Yes, where's the data coming from is a huge question with AI. +[Aaron]: Yes, where's the data coming from is a huge question with AI.
It's astonishing you can ask questions @@ -1990,7 +2040,7 @@ I can't remember who it was. Somebody was actually out torrenting books
-just to be able to build into their AI system. +just to be able to build it into their AI system.
I think it might've been Meta. @@ -2011,13 +2061,13 @@ like the mobile guys have got their own license, but where they're getting their data from,
-I'm not sure on so that that's a huge question. +I'm not sure, so that's a huge question.
-That's a that's a talk in itself. +That's a talk in itself.
-But yeah, but you if you train on your RAG and your data, +But yeah, if you train on your RAG and your data,
you know what it's come, you know, @@ -2032,16 +2082,21 @@ but the other stuff is just more lines of supplement
-if you're using a smaller model, +if you're using a smaller model.
-but the comment online, I see a couple of them. +[Corwin]: The comments online, I see a couple of them.
I'll read them out in order here. Really interesting stuff.
-Thank you for your talk. Given that large AI companies +00:31:08.340 --> 00:31:09.556 +Thank you for your talk. + +NOTE Q: Given that large AI companies are openly stealing IP and copyright, thereby eroding the authority of such law (and eroding truth itself as well), can you see a future where IP & copyright flaw become untenable and what sort of onwards effect might that have? + +00:31:09.557 --> 00:31:11.659 +Given that large AI companies
are openly stealing intellectual property and copyright @@ -2098,7 +2153,7 @@ and I do not have money. So don't sue me, is there's going to be
-the current administration tends is very AI pro AI. +the current administration tends is very AI, pro AI.
And there's very a great deal of lobbying by those groups. @@ -2115,11 +2170,13 @@ what happens to copyright the next 510 years.
I just don't know how it keeps up
+00:32:13.340 --> 00:32:18.059 without there being some adjustments and stuff.
-Okay, and then another comment I saw, +NOTE Comment: File size is not going to be the bottleneck, your RAM is. + +00:32:18.060 --> 00:32:20.419 +[Corwin]: Okay, and then another comment I saw,
file size is not going to be a bottleneck. @@ -2136,11 +2193,11 @@ and 512 gigabytes of RAM to run the larger ones.
You'll need a GPU with that much memory
-if you want it to run quickly. Yeah. Oh no. +00:32:35.060 --> 00:32:38.318 +if you want it to run quickly.
-It also depends upon how your memory is laid out. +00:32:38.319 --> 00:32:41.259 +[Aaron]: Yeah. Oh no. It also depends upon how your memory is laid out.
Like example being the Ultra i285H @@ -2167,7 +2224,7 @@ So it's able to do some additional stuff with it as opposed to come off disk.
-It's all balancing act. If you hit Zyskin's website, +It's all balancing act. If you hit Ziskind's website,
that guy's done some great work on it. @@ -2206,7 +2263,7 @@ But if you want to, Network Chuck had a great video talking about building his own system
-with a couple really powerful NVIDIA cards +with a couple really powerful Nvidia cards
and stuff like that in it. @@ -2226,20 +2283,20 @@ which models you want to use,
how much information you need,
+00:33:46.220 --> 00:33:49.579 but it's part of the learning.
-And you can run models, even as a Raspberry PI fives, +00:33:49.580 --> 00:33:52.899 +And you can run models, even on Raspberry Pi 5s,
if you want to, they'll run slow.
+00:33:54.500 --> 00:33:59.339 Don't get me wrong, but they're possible.
-Okay, and I think there's other questions coming in too, +00:33:59.340 --> 00:34:02.179 +[Corwin]: Okay, and I think there's other questions coming in too,
so I'll just bam for another second. @@ -2263,7 +2320,7 @@ This is another one that I'm going to have to study after the conference.
-We greatly appreciate, all of us appreciate +[Aaron]: We greatly appreciate, all of us appreciate
you guys putting on the conference. @@ -2272,25 +2329,27 @@ you guys putting on the conference. It's a great conference. It's well done.
-It's an honor to be on the stage +[Corwin]: It's an honor to be on the stage
+00:34:28.020 --> 00:34:33.124 with the brains of the project, which is you.
-So what else we got? Question wise. +00:34:33.125 --> 00:34:34.699 +[Aaron]: So what else we got? Question wise. + +00:34:34.700 --> 00:34:46.899 +[Corwin]: Okay, so just scanning here.
-Okay, so just scanning here. +NOTE Q: Have you used local models capable of tool-calling?
+00:34:46.900 --> 00:34:50.699 Have you used local models capable of tool calling?
-I'm, I'm scared of agentic. +I'm scared of agentic.
-I, I am, I'm going to be a slow adopter of that. +I'm going to be a slow adopter of that.
I want to do it, but I just don't have the, uh, @@ -2299,7 +2358,7 @@ I want to do it, but I just don't have the, uh, four decimal fortitude right now to do it.
-I, I, I've had to give me the commands, +I've had to give me the commands,
but I still run the commands by hand. @@ -2307,11 +2366,11 @@ but I still run the commands by hand.
I'm looking into it and it's on once again,
+00:35:10.540 --> 00:35:20.899 it's on that list, but I just, that's a big step for me.
-So. Awesome. All right. +00:35:20.900 --> 00:35:23.139 +[Corwin]: So. Awesome. All right.
Well, maybe it's, let me just scroll through @@ -2331,17 +2390,19 @@ So this was piggybacking on the question
about model updates and adding data. +NOTE Q: Will the models reach out to the web if they need to for more info? +
And will models reach out to the web
if they need more info?
+00:35:47.820 --> 00:35:52.479 Or have you worked with any models that work that way?
-No, I've not seen any models to do that +00:35:52.480 --> 00:35:55.259 +[Aaron]: No, I've not seen any models to do that
There's there was like a group @@ -2353,19 +2414,22 @@ working on something like a package updater that would do different diffs on it,
-but it's so Models change so much +but it's so... Models change so much,
-even who make minor changes and fine-tuning. +even who make minor changes and fine-tuning,
-It's hard just to update them in place +It's hard just to update them in place.
So I haven't seen one, but that doesn't mean
-they're not out there. I'm curious topic though Awesome +00:36:10.100 --> 00:36:15.713 +they're not out there. Curious topic though. + +00:36:15.714 --> 00:36:16.259 +[Corwin]: Awesome.
Well, it's probably pretty good timing. @@ -2385,17 +2449,19 @@ I want to make sure while we're still live, though,
I give you a chance to offer any closing thoughts. +NOTE Q: What scares you most about agentic tools? How would you think about putting a sandbox around it if you adopt an agentic workflow? +
So what scares you most about the agentic tools?
How would you think about putting a sandbox around that
+00:36:38.420 --> 00:36:41.619 if you did adopt an agentic workflow?
-That is a great question. +00:36:41.620 --> 00:36:42.899 +[Aaron]: That is a great question.
In terms of that, I would just control @@ -2419,7 +2485,7 @@ because that's just kind of the way it works out for us. So agentic, it's just going to take a while to get trust.
-I want to want to see more stuff happening. +I want to see more stuff happening.
Humans screw up stuff enough. @@ -2452,16 +2518,18 @@ but it's going to be, it's happening. It's just, I'm going to be one of the laggards on that one.
-So airgab, jail, extremely locked down environments, +So air gap, jail, extremely locked down environments,
like we're talking about separate physicals, not Docker.
-Yeah, hopefully. Right, fair. +00:37:34.900 --> 00:37:36.577 +Yeah, hopefully. + +NOTE Q: Tool calling can be read-only, such as giving models the ability to search the web before answersing your question. (No write access or execute access) I'm interested to know if local models are any good at calling tools, though.
-So tool calling can be read-only, +00:37:36.578 --> 00:37:39.899 +[Corwin]: Right, fair. So tool calling can be read-only,
such as giving models the ability to search the web @@ -2479,7 +2547,7 @@ I'm interested to know if local models are any good at that.
-Yes, local models can do a lot of that stuff. +[Aaron]: Yes, local models can do a lot of that stuff.
It's their capabilities. @@ -2488,13 +2556,13 @@ It's their capabilities. If you load LM studio, you can do a lot of wonderful stuff
-with that or with open web UI with a llama. +with that or with Open Web UI with ollama.
It's a lot of capabilities. It's amazing.
-Open web UI is actually what a lot of companies are using now +Open Web UI is actually what a lot of companies are using now
to put their data behind that. @@ -2505,8 +2573,11 @@ They're curated data and stuff like that. So works well.
I can confirm that from my own professional experience.
-Excellent. Okay, well, our timing should be just perfect +00:38:15.820 --> 00:38:16.915 +Excellent. + +00:38:16.916 --> 00:38:19.659 +[Corwin]: Okay, well, our timing should be just perfect
if you want to give us like a 30-second, 45-second wrap-up. @@ -2521,7 +2592,7 @@ Thank you again so much for preparing this talk and for entertaining all of our questions.
-Yeah, let me just thank you guys for the conference again. +[Aaron]: Yeah, let me just thank you guys for the conference again.
This is a great one. I've enjoyed a lot of it. @@ -2532,6 +2603,8 @@ I've only had a couple of talks so far,
but I'm looking forward to hitting the ones after this and tomorrow. +NOTE Wrapping up +
But the AI stuff is coming. Get on board. @@ -2545,7 +2618,7 @@ and get a little taste of it, what my minimal viable product
-with just LlamaFile and GPTEL +with just Llamafile and gptel
will get you to the point where you start figuring out. @@ -2554,34 +2627,34 @@ will get you to the point where you start figuring out. Gptel is an amazing thing. It just gets out of your way,
-but it works solo with Emacs. Design because it takes +but it works so well with Emacs's design because
-doesn't take your hands off the keyboard. +it doesn't take your hands off the keyboard.
-It's just another buffer +It's just another buffer,
and you just put information in there.
-It's quite quite a wonderful It's a wonderful time. +It's quite a wonderful time.
-Let's put that way That's all I got Thank you +00:39:06.980 --> 00:39:10.501 +Let's put that way. That's all I got.
-so much for once again, and we're we're just cut away. +00:39:10.502 --> 00:39:14.339 +[Corwin]: Thank you so much for once again, and we've just cut away.
So I'll stop the recording
-and you're on your own recognizance +and you're on your own recognizance.
-Well, I'm gonna punch out +[Aaron]: Well, I'm gonna punch out
if anybody has any questions or anything @@ -2590,7 +2663,7 @@ if anybody has any questions or anything my email address is ajgrothe@yahoo.com or at gmail and
-Thank you all for attending +thank you all for attending,
and thanks again for the conference -- cgit v1.2.3