path: root/2025/captions/emacsconf-2025-python--interactive-python-programming-in-emacs--david-vujic--main.vtt

WEBVTT captioned by sachac

00:00:00.000 --> 00:00:04.439
Okay, so welcome to this session about interactive Python

00:00:04.440 --> 00:00:09.679
programming. My name is David Vujic and I live and work in

00:00:09.680 --> 00:00:15.319
Stockholm, Sweden.  a developer and today I focus

00:00:15.320 --> 00:00:20.439
mainly on Python software development. So I do this at work

00:00:20.440 --> 00:00:25.999
and I also do this on my spare time in my open source projects.

00:00:26.000 --> 00:00:30.479
Before that, I've been part of the Lisp community. I've

00:00:30.480 --> 00:00:33.700
been a Clojure developer, and also, like, way back,

00:00:33.701 --> 00:00:40.279
I was in the Microsoft world and developed C# and .NET stuff.

00:00:40.280 --> 00:00:45.999
What I've been doing lately is to try to improve the

00:00:46.000 --> 00:00:52.399
developer experience when you write Python code. So what I

00:00:52.400 --> 00:00:56.159
want to talk about is this, but also I want to begin with

00:00:56.160 --> 00:01:00.839
feedback loops because I think it's very related to this

00:01:00.840 --> 00:01:05.359
interactive programming style, like having this nice

00:01:05.360 --> 00:01:07.067
feedback when you write code.

00:01:07.068 --> 00:01:10.533
So I'm going to begin with that.

NOTE Feedback loops

00:01:10.534 --> 00:01:14.199
So this image, you know, this circle is supposed to be a

00:01:14.200 --> 00:01:19.879
visualization of a feedback loop. Let's say we write our

00:01:19.880 --> 00:01:25.239
code and then we deploy it to production. Then when it's

00:01:25.240 --> 00:01:29.639
running there, we can check if things work, or if maybe someone

00:01:29.640 --> 00:01:35.319
else will let us know. Maybe our customers will let us know.

00:01:35.320 --> 00:01:39.639
That's a pretty slow feedback loop with potential risks of

00:01:39.640 --> 00:01:41.867
damaging your business or whatever.

00:01:41.868 --> 00:01:44.167
This is obvious, of course.

00:01:44.168 --> 00:01:50.000
So a faster feedback loop probably is to have

00:01:50.001 --> 00:01:54.066
some kind of automation when you do commits

00:01:54.067 --> 00:01:59.733
or maybe you have this pull request things and even reviews.

00:01:59.734 --> 00:02:02.933
So maybe not always as fast as deploy,

00:02:02.934 --> 00:02:05.839
don't deploy directly to production, but

00:02:05.840 --> 00:02:10.539
it's probably safer and often you get this automated

00:02:10.540 --> 00:02:16.199
feedback faster anyway. But it's still kind of slow. You

00:02:16.200 --> 00:02:20.239
have to wait. You have to push things to GitHub maybe and

00:02:20.240 --> 00:02:24.279
wait. So there's faster ways for sure to get feedback.

00:02:24.280 --> 00:02:27.967
So a much faster way is to write code,

00:02:27.968 --> 00:02:31.367
and write some unit tests, and run those unit tests.

00:02:31.368 --> 00:02:33.467
So then you do everything on your local machine

00:02:33.468 --> 00:02:39.039
and you will fairly quickly learn if your code does

00:02:39.040 --> 00:02:47.159
what you think it does or if it doesn't. I want to zoom in to

00:02:47.160 --> 00:02:55.999
this test write code and test flow a bit. Let's do that.

NOTE Test-driven development

00:02:56.000 --> 00:02:59.759
As a developer, I have used a thing called test-driven

00:02:59.760 --> 00:03:05.999
development for quite some time. I find that this way of

00:03:06.000 --> 00:03:11.259
working is very fast when it comes to getting feedback on

00:03:11.260 --> 00:03:14.519
what your code does and how you should continue the

00:03:14.520 --> 00:03:19.980
development. So, test-driven development,

00:03:19.981 --> 00:03:24.220
basically that you start writing a test for

00:03:24.221 --> 00:03:27.020
something that you want to develop, and then you continue

00:03:27.021 --> 00:03:31.019
developing that, and then you go back to the test, and modify

00:03:31.020 --> 00:03:35.079
and modify the code, and you go back and forth between the

00:03:35.080 --> 00:03:36.959
tests and the code.

00:03:36.960 --> 00:03:44.419
It's sort of like a ping-pong game. I find this very

00:03:44.420 --> 00:03:50.519
effective when you want to get feedback and to know how to

00:03:50.520 --> 00:03:57.233
continue the development. The most important thing

00:03:57.234 --> 00:04:01.700
that I feel is that you know what the code does.

00:04:01.701 --> 00:04:05.559
You learn very quickly.

NOTE REPL-driven development

00:04:05.560 --> 00:04:12.199
Let's zoom into this TDD flow a little bit. The last couple of

00:04:12.200 --> 00:04:17.379
years, I've been doing a slightly different thing which is

00:04:17.380 --> 00:04:21.979
called REPL-driven development. REPL-driven

00:04:21.980 --> 00:04:25.719
development is very similar to test-driven development,

00:04:25.720 --> 00:04:31.159
but I find it even quicker. You get feedback even quicker

00:04:31.160 --> 00:04:34.979
than with a regular TDD setup. So REPL-driven development

00:04:34.980 --> 00:04:41.199
is about writing and evaluating code in a REPL basically.

00:04:41.200 --> 00:04:46.839
And you can do experiments and you can refactor and

00:04:46.840 --> 00:04:51.699
re-evaluate and you get instant feedback on what the code

00:04:51.700 --> 00:04:54.799
does and what you need to change. So I think that's even

00:04:54.800 --> 00:04:59.519
faster than test-driven development.

00:04:59.520 --> 00:05:02.899
Okay, REPL driven development. Let's go back. What's the

00:05:02.900 --> 00:05:10.759
REPL? Most of developers know what a REPL is. The most common

00:05:10.760 --> 00:05:16.399
setup is you open this shell and you use the REPL for your

00:05:16.400 --> 00:05:19.359
programming language. In this case I'm using the Python

00:05:19.360 --> 00:05:25.619
REPL or the IPython REPL which is an enhanced REPL for Python

00:05:25.620 --> 00:05:30.679
development. So what happens here is that we start a REPL

00:05:30.680 --> 00:05:34.919
session in isolation. So this session knows about the

00:05:34.920 --> 00:05:38.119
Python environment. So it knows about the Python language

00:05:38.120 --> 00:05:42.359
basically. So as soon as we start writing things, adding

00:05:42.360 --> 00:05:47.359
variables or creating writing functions or even doing

00:05:47.360 --> 00:05:51.679
imports. Then the session will be more and more aware of the

00:05:51.680 --> 00:05:55.819
code so we will add things to the to the session and then that

00:05:55.820 --> 00:06:00.519
means that we can run functions we can print out these

00:06:00.520 --> 00:06:05.859
variables and things like that. But with REPL driven

00:06:05.860 --> 00:06:09.839
development it's not really that well at least not what I

00:06:09.840 --> 00:06:14.039
mean with REPL driven development. So what I'm thinking of

00:06:14.040 --> 00:06:19.639
is that you are in your code editor where you have your

00:06:19.640 --> 00:06:22.799
autocomplete, and you have your syntax highlighting and

00:06:22.800 --> 00:06:30.459
your favorite theme, color theme, and all of those things. But

00:06:30.460 --> 00:06:34.979
instead, you have this running REPL in the background or in a

00:06:34.980 --> 00:06:41.139
smaller window or buffer. So that means that you write code

00:06:41.140 --> 00:06:45.319
and you can send that code to the running REPL, to the REPL

00:06:45.320 --> 00:06:50.399
session. You write and do everything as you would do when

00:06:50.400 --> 00:06:55.219
writing your code basically. In this case, in this

00:06:55.220 --> 00:07:00.599
example, I have evaluated these two functions. I've sent

00:07:00.600 --> 00:07:05.819
them to the REPL session so it's aware of these functions.

00:07:05.820 --> 00:07:10.399
Then I switched to a separate different module and

00:07:10.400 --> 00:07:14.039
evaluated that one. So the REPL session now knows about

00:07:14.040 --> 00:07:19.039
these two functions and also these two variables. That

00:07:19.040 --> 00:07:23.999
means that I can evaluate the state of those variables and

00:07:24.000 --> 00:07:28.999
change code and re-evaluate and things like that. So in this

00:07:29.000 --> 00:07:33.639
example if you look in the smaller area there you see that I

00:07:33.640 --> 00:07:39.639
have evaluated this res variable on line 6 and the output was

00:07:39.640 --> 00:07:42.399
that it's a dictionary with two keys and two values

00:07:42.400 --> 00:07:51.219
basically. So this setup works in basically any of your

00:07:51.220 --> 00:07:54.079
favorite code editors. So you can do this in Visual Studio

00:07:54.080 --> 00:08:01.239
Code, you can do this in PyCharm or Vim. But what I have done is

00:08:01.240 --> 00:08:07.119
that... More like what I have missed is that when I write code

00:08:07.120 --> 00:08:10.239
and do this evaluation, this is really cool, but then I need

00:08:10.240 --> 00:08:15.459
to switch context if I want to see the result. I have to switch

00:08:15.460 --> 00:08:21.979
context to this other window. I

00:08:21.980 --> 00:08:25.759
have my focus on the code and then I have to look in a different

00:08:25.760 --> 00:08:31.799
place to know the results. And if it's a larger output, then

00:08:31.800 --> 00:08:37.479
maybe I need to scroll. So I wanted to find out if it was

00:08:37.480 --> 00:08:43.479
possible to make this even smoother and faster, this

00:08:43.480 --> 00:08:45.479
feedback loop even faster, so I don't have to switch

00:08:45.480 --> 00:08:52.119
context. What I've done here is that... I can select a row or a

00:08:52.120 --> 00:08:58.079
region and I can evaluate and then an overlay, a small pop-up

00:08:58.080 --> 00:09:03.119
shows up with the evaluated result right next to it. So I can

00:09:03.120 --> 00:09:07.519
change code and re-evaluate and quickly see the result of it

00:09:07.520 --> 00:09:12.640
without doing this context switching. So the way I've done

00:09:12.641 --> 00:09:20.679
it is that I wanted to reuse the existing tooling that I

00:09:20.680 --> 00:09:27.739
already had. I know that my in-editor REPL, the IPython

00:09:27.740 --> 00:09:31.559
REPL, already does this evaluation. So I figured maybe I can

00:09:31.560 --> 00:09:35.359
extract the data and do this visualization as a separate

00:09:35.360 --> 00:09:40.839
thing. That's how I've done it. What I've done is that

00:09:40.840 --> 00:09:47.199
I've created this overlay and placed it where my cursor

00:09:47.200 --> 00:09:50.859
currently is, right next to the code. Then I've

00:09:50.860 --> 00:09:55.719
extracted the evaluated result and put it in this overlay.

00:09:55.720 --> 00:10:01.039
I also want this overlay to have this nice looking syntax,

00:10:01.040 --> 00:10:04.759
so I've set it to this Python mode, so we get this syntax

00:10:04.760 --> 00:10:10.559
highlighting. Make it look very readable. And as a nice

00:10:10.560 --> 00:10:16.879
developer experience thing,

00:10:16.880 --> 00:10:20.379
when you move the cursor, of course you don't want the

00:10:20.380 --> 00:10:25.679
overlay to be there. You want it to disappear. So those kinds

00:10:25.680 --> 00:10:28.999
of things I've added. So putting the overlay at the right

00:10:29.000 --> 00:10:33.279
place and feed it with the evaluated data and then make it

00:10:33.280 --> 00:10:39.839
disappear when it's not interesting to look at anymore.

00:10:39.840 --> 00:10:44.639
What I've described so far is something that I use on a

00:10:44.640 --> 00:10:50.639
daily basis, and it covers most of my needs while doing Python

00:10:50.640 --> 00:10:56.119
development. But one thing I still miss, and I miss it from my

00:10:56.120 --> 00:11:03.479
days as a Clojure developer, because over there we could

00:11:03.480 --> 00:11:07.919
have a running app on our local machine and we can have our

00:11:07.920 --> 00:11:12.719
editor, and the app and the editor were connected. So when I

00:11:12.720 --> 00:11:17.199
did some changes in the code, the app would change without

00:11:17.200 --> 00:11:20.559
any restarts or anything like that. And the same if I would

00:11:20.560 --> 00:11:24.679
change the state of the app, I can inspect the state from the

00:11:24.680 --> 00:11:28.919
code. So they were connected. They are connected. So I was

00:11:28.920 --> 00:11:32.839
thinking, hey, this would be really cool if we could have

00:11:32.840 --> 00:11:39.199
something like this in Python. And that reminded me of

00:11:39.200 --> 00:11:43.839
Jupyter and Jupyter notebooks because I think notebooks,

00:11:43.840 --> 00:11:49.659
the way you do things there, is very similar to what I was

00:11:49.660 --> 00:11:56.879
trying to achieve. So I was reading up a little bit on how this

00:11:56.880 --> 00:12:00.919
notebook thing works. It turns out that a notebook is a

00:12:00.920 --> 00:12:05.279
client that talks to a server, that communicates with a

00:12:05.280 --> 00:12:08.799
server. It's on the server that all this Python

00:12:08.800 --> 00:12:14.159
evaluation and all this thing happens. Then what I've

00:12:14.160 --> 00:12:19.659
done is that instead of starting up IPython in my editor, I

00:12:19.660 --> 00:12:23.519
start the Jupyter console instead. And then I can give it

00:12:23.520 --> 00:12:27.159
that unique ID and it will be connected to that running

00:12:27.160 --> 00:12:30.919
kernel.

NOTE FastAPI CRUD

00:12:30.920 --> 00:12:37.199
In this example, I've created this FastAPI CRUD app that

00:12:37.200 --> 00:12:41.919
has this create, read, update, and delete endpoints. It

00:12:41.920 --> 00:12:46.399
has this, it's locally running, it has this database where

00:12:46.400 --> 00:12:51.639
you can do all these things. I'm running this FastAPI app

00:12:51.640 --> 00:12:58.059
in the kernel and then I've connected to, I've connected to

00:12:58.060 --> 00:13:03.239
the kernel in my editor too. Both of them are connected to

00:13:03.240 --> 00:13:09.719
the kernel. What I do now is that I want to initially create

00:13:09.720 --> 00:13:15.239
some data. I'm going to add this, creating this message.

00:13:15.240 --> 00:13:19.899
What I get back is a message ID. I want to experiment in

00:13:19.900 --> 00:13:24.359
my browser. What do I get with that message ID? I'm

00:13:24.360 --> 00:13:30.239
evaluating the read function. I instantly get this

00:13:30.240 --> 00:13:34.779
evaluated result, which was this hello world text. So what

00:13:34.780 --> 00:13:39.919
happens if I do some changes in this app? I'm going to grab

00:13:39.920 --> 00:13:49.659
this message ID and write something else.

00:13:49.660 --> 00:13:53.759
Now I can evaluate the same thing again, and you can see that

00:13:53.760 --> 00:14:02.399
the content has changed to this new value. My editor isn't

00:14:02.400 --> 00:14:07.719
in any debug mode or something like that. It doesn't know

00:14:07.720 --> 00:14:11.239
what database it is. It doesn't have any environment

00:14:11.240 --> 00:14:14.479
variables set up or something like that. It is only

00:14:14.480 --> 00:14:17.599
connected to the kernel, and the kernel is aware of that. It's

00:14:17.600 --> 00:14:20.479
running the app. It has the connection strings and

00:14:20.480 --> 00:14:28.799
everything that is needed. So that's how this thing works.

00:14:28.800 --> 00:14:34.199
Now I want to do some inline hacking because I want to store

00:14:34.200 --> 00:14:37.799
this input that is sent from this app because I want to work

00:14:37.800 --> 00:14:42.039
with it afterwards. I can add this dictionary that stores

00:14:42.040 --> 00:14:48.759
this message. I'm updating the source code of this app, and

00:14:48.760 --> 00:15:03.079
when I run any of these endpoints again, you will see that

00:15:03.080 --> 00:15:08.759
the state changes, and the new inputs, I can grab and I can use

00:15:08.760 --> 00:15:14.399
them for quick evaluation or testing. This example is

00:15:14.400 --> 00:15:18.519
really simple. It was just an integer. For example, if you

00:15:18.520 --> 00:15:23.519
are sending a more complex object, maybe a pydantic schema

00:15:23.520 --> 00:15:28.199
or something, and you want to inspect what's coming in, and if

00:15:28.200 --> 00:15:34.199
you have some sort of validation that you want to test out.

00:15:34.200 --> 00:15:38.399
The configuration or the code that I wrote to make this work

00:15:38.400 --> 00:15:44.159
is a little bit different than just adding an overlay. I'm

00:15:44.160 --> 00:15:50.999
using this overlay just like with the IPython example, but in

00:15:51.000 --> 00:15:57.839
this case, when I change code, I have to think about where that

00:15:57.840 --> 00:16:02.159
code lives, because it's the app that runs the code. So it's

00:16:02.160 --> 00:16:07.039
in the app context I need to manipulate with the data. If you

00:16:07.040 --> 00:16:11.919
have started the app from maybe a main function and that

00:16:11.920 --> 00:16:17.879
module imports namespaces, then you need to, if you want to

00:16:17.880 --> 00:16:22.359
update a function or something like that, you need to update

00:16:22.360 --> 00:16:26.679
it in the correct namespace. What I did before in IPython

00:16:26.680 --> 00:16:29.919
by adding and changing things, everything ends up in the

00:16:29.920 --> 00:16:34.439
global namespace. But here, if you want the app to actually

00:16:34.440 --> 00:16:38.479
react to the changes, you need to put it in the right

00:16:38.480 --> 00:16:43.479
namespace. So that's what I do here. I do some lookups, where

00:16:43.480 --> 00:16:49.139
is this function, and then I do this reload of this function or

00:16:49.140 --> 00:16:54.799
module. And when I was developing this, I was thinking, hey,

00:16:54.800 --> 00:16:59.319
this is really ugly. I'm in this REPL and do some

00:16:59.320 --> 00:17:03.559
manipulation of the imports and things like that. That

00:17:03.560 --> 00:17:09.759
didn't feel good. Then I was reminded of the IPython. And

00:17:09.760 --> 00:17:15.519
IPython has this feature to reload any updated

00:17:15.520 --> 00:17:19.119
submodules. I was curious how do they do it. I looked in the

00:17:19.120 --> 00:17:24.079
IPython source code and saw that they also use importlib and

00:17:24.080 --> 00:17:28.359
reloading of this module. Once I've learned that, then I

00:17:28.360 --> 00:17:32.599
stopped thinking that my code was hacky. I thought it was

00:17:32.600 --> 00:17:37.159
good enough at least.

NOTE Testing with an LLM

00:17:37.160 --> 00:17:45.059
But one thing that has bothered me for a long time is I quite

00:17:45.060 --> 00:17:50.199
often want to test out and evaluate individual rows that

00:17:50.200 --> 00:17:58.559
lives in a function. Quite often, this code uses the input

00:17:58.560 --> 00:18:02.639
to that function like the input parameters. To be able to

00:18:02.640 --> 00:18:07.719
do that, I need to manually type some fake data and set it to

00:18:07.720 --> 00:18:12.279
this variable, and then I can evaluate the code. But I think

00:18:12.280 --> 00:18:17.779
that takes... That slows me down. I was thinking, maybe I can

00:18:17.780 --> 00:18:23.439
do this in a quicker way, so I have this quicker feedback, so I

00:18:23.440 --> 00:18:27.933
can run this or evaluate this code much quicker.

00:18:27.934 --> 00:18:29.439
So my idea was maybe I

00:18:29.440 --> 00:18:35.239
can use an LLM for this. If I give it the parameters, maybe it

00:18:35.240 --> 00:18:41.119
can return some random data so I don't have to write it

00:18:41.120 --> 00:18:44.119
myself. I ended up doing that. I have this source code.

00:18:44.120 --> 00:18:50.399
I'm loading the REPL with the code. Then I select this

00:18:50.400 --> 00:18:56.719
function name and the parameters with its data type. I

00:18:56.720 --> 00:19:02.839
have this prompt that instructs the LLM to come up with fake

00:19:02.840 --> 00:19:06.239
data based on the tag name and on the data type. And then I can

00:19:06.240 --> 00:19:10.099
send that to the REPL. I do that with a key command. Then

00:19:10.100 --> 00:19:16.019
I can proceed by running the code within the function that

00:19:16.020 --> 00:19:21.719
uses these inputs. This works for all the data types. If

00:19:21.720 --> 00:19:26.279
there's a custom data type, you need to give the LLM extra

00:19:26.280 --> 00:19:30.399
context. So that's something to think about. Once it knows

00:19:30.400 --> 00:19:35.679
the context, it can generate this fake data that very often is

00:19:35.680 --> 00:19:39.839
good enough just to test out, you know, like I've done here, like

00:19:39.840 --> 00:19:45.399
string... sorry, list destructuring and parsing and things

00:19:45.400 --> 00:19:51.879
like that. I think that was all I had, and thank you for

00:19:51.880 --> 00:19:52.920
listening!