summaryrefslogtreecommitdiffstats
path: root/2026/info/hutch-after.md
diff options
context:
space:
mode:
Diffstat (limited to '2026/info/hutch-after.md')
-rw-r--r--2026/info/hutch-after.md260
1 files changed, 260 insertions, 0 deletions
diff --git a/2026/info/hutch-after.md b/2026/info/hutch-after.md
index 38d008cc..be7cd140 100644
--- a/2026/info/hutch-after.md
+++ b/2026/info/hutch-after.md
@@ -1,6 +1,266 @@
<!-- Automatically generated by emacsconf-publish-after-page -->
+<div class="transcript transcript-mainVideo"><a name="hutch-mainVideo-transcript"></a><h1>Transcript</h1>
+
+
+<div class="transcript-heading">[[!template new="1" text="""Introduction""" start="00:00:00.071" video="mainVideo-hutch" id="subtitle"]]</div>
+[[!template text="""I'm going to talk about a local code review tool""" start="00:00:01.112" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""I've been building for Emacs for the past five""" start="00:00:03.774" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""months or so.""" start="00:00:06.557" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""I've previously been an Emacs user for about 10""" start="00:00:08.239" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""years, and I use it as a daily driver for most of""" start="00:00:10.701" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""my code and prose editing.""" start="00:00:13.724" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""When I say a local review tool, I mean local""" start="00:00:16.306" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""to Emacs and not necessarily local LLMs on""" start="00:00:19.529" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""device, even though I think that's theoretically""" start="00:00:22.292" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""possible today.""" start="00:00:24.374" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""By code reviews, I specifically mean the""" start="00:00:26.475" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""thing we do on pull requests and on tools like""" start="00:00:28.837" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""Gerrit and Phabricator in the past.""" start="00:00:32.378" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""In other words, a system for gating code before""" start="00:00:36.242" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""it's merged to be released.""" start="00:00:38.984" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""I should say that there have been many good""" start="00:00:42.778" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""talks about LLMs in AI and what that means for""" start="00:00:45.381" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""Emacs, so I'm not going to relitigate any of""" start="00:00:48.564" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""those arguments again.""" start="00:00:51.307" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""I approach this, as many others in the community,""" start="00:00:52.728" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""by letting go of semantic boundaries under which""" start="00:00:55.611" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""Emacs is considered an editor, but just rather a""" start="00:00:58.934" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""mutable productivity environment instead.""" start="00:01:03.158" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""But I do want to briefly show you my experience""" start="00:01:07.072" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""with existing code review tools that are around""" start="00:01:09.114" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""because I have a bit of a love-hate relationship""" start="00:01:12.296" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""with them.""" start="00:01:15.759" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""It can be pretty overwhelming to keep up with the""" start="00:01:16.819" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""speed at which the amount of code gets produced""" start="00:01:19.841" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""daily these days, especially if you work in a""" start="00:01:23.384" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""team, but also the amount of review text that now""" start="00:01:26.426" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""gets attached to that code.""" start="00:01:31.169" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""In my experience, even though the noise to""" start="00:01:34.639" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""signal for these tools worst case""" start="00:01:37.962" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""is like 10 to 1, I don't feel like""" start="00:01:39.443" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""we can get rid of them entirely""" start="00:01:42.124" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""because those ones can sometimes be""" start="00:01:44.346" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""pretty sharp and important.""" start="00:01:45.827" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""But I do think there's value in sort of shifting""" start="00:01:49.330" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""the process to the left to borrow a term from the""" start="00:01:53.613" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""DevRel industry.""" start="00:01:56.555" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""Because it's useful to see the work you did""" start="00:01:59.997" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""before you create more work for others, for your""" start="00:02:02.699" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""colleagues and your peers.""" start="00:02:06.020" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""Many of these flows can now be easily""" start="00:02:07.821" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""unlocked in Emacs given the excellent building""" start="00:02:10.002" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""blocks we have today.""" start="00:02:12.784" video="mainVideo-hutch" id="subtitle"]]
+
+<div class="transcript-heading">[[!template new="1" text="""Typical coding agent review""" start="00:02:14.825" video="mainVideo-hutch" id="subtitle"]]</div>
+[[!template text="""workflow works, I want to show you what a typical""" start="00:02:17.627" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""coding agent review looks like today.""" start="00:02:20.408" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""These examples are from a real open source""" start="00:02:23.490" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""product that I've been building for the past""" start="00:02:25.951" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""three years.""" start="00:02:28.092" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""This is one pull request from the project.""" start="00:02:29.671" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""It's a single review pass by a review agent.""" start="00:02:32.432" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""Has 23 comments.""" start="00:02:35.834" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""Every single one of them flagged as a potential issue.""" start="00:02:37.775" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""Five of them are marked critical.""" start="00:02:42.397" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""This is technically a review and it's probably a""" start="00:02:45.359" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""decent one.""" start="00:02:48.661" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""But it's also a bit of a DOS attack on my attention.""" start="00:02:50.041" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""If we look behind the scenes on one of those""" start="00:02:54.982" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""critical flags, it ran 40-something shell scripts.""" start="00:02:57.145" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""It ran web searches.""" start="00:03:00.930" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""It wrote a little prompt for another AI to go and""" start="00:03:02.452" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""fix the thing.""" start="00:03:04.955" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""All of it to tell me that one method was""" start="00:03:06.297" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""undefined and would crash at runtime.""" start="00:03:08.600" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""But at the end of all of it, it was still incorrect.""" start="00:03:11.687" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""The method comes from a Ruby gem, and this was""" start="00:03:14.730" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""simply my reply in six words, which to its credit""" start="00:03:17.413" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""it apologized and updated its memory for.""" start="00:03:20.336" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""This is not a one-off thing.""" start="00:03:23.960" video="mainVideo-hutch" id="subtitle"]]
+
+<div class="transcript-heading">[[!template new="1" text="""Hutch""" start="00:03:28.261" video="mainVideo-hutch" id="subtitle"]]</div>
+[[!template text="""It's up here.""" start="00:03:31.022" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""It lives in Emacs and on top of Magit.""" start="00:03:31.742" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""You decide when you want to trigger it and any""" start="00:03:35.403" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""obvious to relatively bigger issues get caught up""" start="00:03:38.303" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""front before you push your code up.""" start="00:03:41.684" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""When it does have something to say, it's usually""" start="00:03:45.305" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""a full-size diff you can directly apply.""" start="00:03:47.806" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""But even if it doesn't, it's prose for you to""" start="00:03:51.107" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""read and not five other people collectively.""" start="00:03:53.987" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""So here's how you kick it off.""" start="00:03:58.338" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""When you're ready to commit, you open up the""" start="00:04:00.519" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""change diffs in Magit, you stage your changes as""" start="00:04:02.701" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""usual, and then select the hutch action and""" start="00:04:06.503" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""trigger a new review.""" start="00:04:09.345" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""Once this is done, you'll see a new hutch buffer""" start="00:04:11.666" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""show up.""" start="00:04:13.667" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""I also have a debug log buffer that can be""" start="00:04:14.628" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""optionally enabled.""" start="00:04:17.309" video="mainVideo-hutch" id="subtitle"]]
+
+<div class="transcript-heading">[[!template new="1" text="""Hutch tools""" start="00:04:19.220" video="mainVideo-hutch" id="subtitle"]]</div>
+[[!template text="""itself and the agent is sifting through the code""" start="00:04:21.682" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""and you can see a bunch of tools being called""" start="00:04:24.404" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""like `read_file` and `search_codebase` and stuff.""" start="00:04:26.206" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""There's around five tools that Hutch exposes, one""" start="00:04:29.649" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""of which is called `surrounding_context` that uses""" start="00:04:32.812" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""TreeSitter if you have that installed to""" start="00:04:35.694" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""increasingly fetch more context around certain""" start="00:04:37.836" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""suspicious lines without needing to read whole""" start="00:04:40.718" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""files and waste tokens.""" start="00:04:43.781" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""You'll see that it took around five rounds to""" start="00:04:45.927" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""finish this review this time.""" start="00:04:48.229" video="mainVideo-hutch" id="subtitle"]]
+
+<div class="transcript-heading">[[!template new="1" text="""Hutch walkthrough""" start="00:04:51.291" video="mainVideo-hutch" id="subtitle"]]</div>
+[[!template text="""something like this.""" start="00:04:53.572" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""It'll report the total tokens at the top.""" start="00:04:54.553" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""You'll see the findings organized under""" start="00:04:57.055" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""suggestions, comments, and LGTM""" start="00:04:59.276" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""(which is &quot;looks good to me&quot;).""" start="00:05:01.998" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""Just like Magit, you can open the submenus up.""" start="00:05:03.759" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""If you agree with the suggestion and the""" start="00:05:06.961" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""attached patch, you can hit `m` to queue it up.""" start="00:05:09.343" video="mainVideo-hutch" id="subtitle"]]
+
+<div class="transcript-heading">[[!template new="1" text="""Applying suggestions""" start="00:05:14.317" video="mainVideo-hutch" id="subtitle"]]</div>
+[[!template text="""determine if they make sense.""" start="00:05:16.159" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""These can also be copied and sent off to someone""" start="00:05:17.601" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""else to take a look or another agent that you""" start="00:05:19.763" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""might have running.""" start="00:05:22.306" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""Once all the suggestions are queued up, you""" start="00:05:24.568" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""can apply them all with the key binding `A`""" start="00:05:27.131" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""and the patches get applied to the staged files""" start="00:05:31.236" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""directly, bottom-up.""" start="00:05:33.679" video="mainVideo-hutch" id="subtitle"]]
+
+<div class="transcript-heading">[[!template new="1" text="""Additional reviews""" start="00:05:37.385" video="mainVideo-hutch" id="subtitle"]]</div>
+[[!template text="""I'd like to run another review on these corrected""" start="00:05:40.887" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""changes, and I can do so.""" start="00:05:43.368" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""So here we get another suggestion to fix that""" start="00:05:52.573" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""wasn't caught the last time.""" start="00:05:55.335" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""We queue it up, apply it, and verify that it's""" start="00:05:57.636" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""lodged in, and then run one more final review""" start="00:06:00.938" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""before we commit.""" start="00:06:04.059" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""So at this point, I'm happy with the changes.""" start="00:06:07.010" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""The comments are not actionable, so I go ahead""" start="00:06:08.711" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""and write my commit.""" start="00:06:11.112" video="mainVideo-hutch" id="subtitle"]]
+
+<div class="transcript-heading">[[!template new="1" text="""Storing review notes""" start="00:06:13.194" video="mainVideo-hutch" id="subtitle"]]</div>
+[[!template text="""post-commit hook is having your review notes be""" start="00:06:15.235" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""stored within Git as a separate ref.""" start="00:06:17.937" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""In the terminal pane on the right, you can see""" start="00:06:21.219" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""our last review was durably kept for posterity.""" start="00:06:22.900" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""This can be useful for maintaining a related""" start="00:06:26.522" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""history of reviews along with your commits and""" start="00:06:29.043" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""for sharing the reviews with your team as well.""" start="00:06:31.465" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""That's it.""" start="00:06:35.874" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""It's hopefully pretty simple and intuitive to get""" start="00:06:36.554" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""started with.""" start="00:06:40.116" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""There's a lot more things to configure that you""" start="00:06:41.396" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""can find in the README.""" start="00:06:43.277" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""If you're the kind that never wants to go back to""" start="00:06:45.177" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""writing code in editors by hand again, but are""" start="00:06:47.718" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""still the kind that enjoys programming on their""" start="00:06:50.779" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""own projects, this might be a nice thin layer to""" start="00:06:53.440" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""get your work verified without over-relying on""" start="00:06:57.121" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""LLMs for everything.""" start="00:07:00.003" video="mainVideo-hutch" id="subtitle"]]
+
+<div class="transcript-heading">[[!template new="1" text="""Formal eval and Perfetto""" start="00:07:03.876" video="mainVideo-hutch" id="subtitle"]]</div>
+[[!template text="""a formal eval pipeline.""" start="00:07:06.499" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""If you set `hutch-trace-dir` and give it a""" start="00:07:09.302" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""path, it can emit Perfetto traces that can then""" start="00:07:12.245" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""be piped into some of the eval scripts that we""" start="00:07:15.789" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""have to get fresh data about agent quality.""" start="00:07:18.632" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""We can also use the traces to visualize the agent itself.""" start="00:07:23.812" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""They're useful for both post runtime observability""" start="00:07:29.116" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""and to fetch data for verifying its correctness.""" start="00:07:33.460" video="mainVideo-hutch" id="subtitle"]]
+
+<div class="transcript-heading">[[!template new="1" text="""Tracing performance with Perfetto""" start="00:07:38.342" video="mainVideo-hutch" id="subtitle"]]</div>
+[[!template text="""that comes with a bundled SQL query engine as well.""" start="00:07:41.604" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""You see this typically being used in Android""" start="00:07:46.087" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""and on Chrome for performance profiling and things.""" start="00:07:49.209" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""But it turns out it works pretty well as a simple""" start="00:07:53.251" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""agent tracing tool because it's entirely""" start="00:07:56.553" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""infrastructure-free.""" start="00:07:59.875" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""You just emit JSON in its own trace format and""" start="00:08:01.476" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""you can load it up on ui.perfetto.dev.""" start="00:08:04.698" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""So once you load it up, this is what the""" start="00:08:08.823" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""visualization looks like.""" start="00:08:10.766" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""There's a timeline of what the agent did, what""" start="00:08:12.228" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""tools it called, the slices of LLM calls, and""" start="00:08:15.913" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""down below the tokens that were consumed.""" start="00:08:19.558" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""We can zoom into one of these tool calls and we""" start="00:08:24.710" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""can check out what all the agent did.""" start="00:08:26.791" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""So here we see a `search_codebase` tool call with""" start="00:08:30.013" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""that search string and it looks like it took""" start="00:08:34.515" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""about 400 milliseconds to finish.""" start="00:08:37.196" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""If we move further along, we'll see another""" start="00:08:40.897" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""`search_codebase` call being called with slightly""" start="00:08:43.779" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""different arguments.""" start="00:08:48.160" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""It took a little longer to finish.""" start="00:08:49.541" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""As you can see, as we move along the trace,""" start="00:08:53.357" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""the number of tokens consumed are increasing over time.""" start="00:08:57.361" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""If we go all the way to the end, we'll see two markers.""" start="00:09:01.164" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""The first one is where the LLM calls end.""" start="00:09:06.028" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""This is where the agent is done with all its findings.""" start="00:09:10.672" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""At this point, no models are being called and""" start="00:09:13.715" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""everything that we got back is being now""" start="00:09:16.517" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""processed by Hutch.""" start="00:09:18.859" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""A few milliseconds later, we see the final""" start="00:09:21.081" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""findings that are eventually rendered onto the""" start="00:09:22.963" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""Hutch buffer.""" start="00:09:25.426" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""These are verified and are sanitized for correctness.""" start="00:09:26.286" video="mainVideo-hutch" id="subtitle"]]
+
+<div class="transcript-heading">[[!template new="1" text="""Perfetto SQL queries""" start="00:09:32.417" video="mainVideo-hutch" id="subtitle"]]</div>
+[[!template text="""Here we have a very simple query to fetch all the""" start="00:09:37.521" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""tool call durations.""" start="00:09:40.484" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""Here we can see the `search_codebase`""" start="00:09:42.185" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""tool call seems to have taken the most amount of time""" start="00:09:45.107" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""in this particular trace.""" start="00:09:48.510" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""`read_file` seems to have been called the most.""" start="00:09:50.071" video="mainVideo-hutch" id="subtitle"]]
+
+<div class="transcript-heading">[[!template new="1" text="""Comparison of model findings""" start="00:09:55.100" video="mainVideo-hutch" id="subtitle"]]</div>
+[[!template text="""a couple of quick findings from slightly older""" start="00:09:57.382" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""models run on Martian's code review dataset""" start="00:10:00.124" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""of 40 PRs across different difficulty levels.""" start="00:10:03.387" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""You can find more detailed benchmarks and eval""" start="00:10:07.911" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""runs in the README if anyone's curious.""" start="00:10:10.213" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""This first graph shows that different models tend""" start="00:10:14.137" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""to find different issues.""" start="00:10:16.499" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""So not all of them necessarily agree on what a""" start="00:10:18.445" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""real bug is.""" start="00:10:21.367" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""But if you theoretically combine their results,""" start="00:10:23.188" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""you would get about a 50% jump in the bug hit rate.""" start="00:10:25.970" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""The second graph shows how many rounds each model""" start="00:10:30.853" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""took to grind through the dataset.""" start="00:10:34.076" video="mainVideo-hutch" id="subtitle"]]
+
+<div class="transcript-heading">[[!template new="1" text="""Rounds vs\. token cost""" start="00:10:36.697" video="mainVideo-hutch" id="subtitle"]]</div>
+[[!template text="""But rounds can be misleading because they don't""" start="00:10:41.881" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""track cost.""" start="00:10:43.962" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""Opus finishes in the fewest rounds but burns the""" start="00:10:46.221" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""most input tokens by far, likely because of the""" start="00:10:49.182" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""growing conversation context.""" start="00:10:52.644" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""GLM does it in under 2 million.""" start="00:10:55.065" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""So on the axis that actually costs money, GLM is""" start="00:10:57.627" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""the cheapest of the three, which is the reason""" start="00:10:59.988" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""why it's my default model of choice.""" start="00:11:02.209" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""That's it from me for today.""" start="00:11:05.891" video="mainVideo-hutch" id="subtitle"]]
+
+<div class="transcript-heading">[[!template new="1" text="""Summary""" start="00:11:09.356" video="mainVideo-hutch" id="subtitle"]]</div>
+[[!template text="""aren't tools, they're automation.""" start="00:11:12.741" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""They do the work and they take the thinking with it.""" start="00:11:15.985" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""Hutch is my attempt at keeping as much of this""" start="00:11:20.961" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""process under your control as possible.""" start="00:11:23.804" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""You choose when to run it, you read what it says,""" start="00:11:27.327" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""and then you apply the fixes yourself.""" start="00:11:30.950" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""It's a small local &quot;tool-enough&quot; for people still""" start="00:11:33.993" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""reading what they write, and it's available to""" start="00:11:38.718" video="mainVideo-hutch" id="subtitle"]]
+[[!template text="""download on MELPA and on GitHub.""" start="00:11:41.180" video="mainVideo-hutch" id="subtitle"]]
+
+</div>
+
+Captioner: sachac
+
Questions or comments? Please e-mail [mail@kitallis.in](mailto:mail@kitallis.in?subject=Comment%20for%20EmacsConf%202023%20hutch%3A%20hutch%3A%20code%20reviews%20for%20the%20mildly%20disenfranchised)