path: root/2024/captions/emacsconf-2024-literate--literate-programming-for-the-21st-century--howard-abrams--main.vtt

WEBVTT captioned by sachac

NOTE Introduction

00:00:00.000 --> 00:00:09.359
Can you believe it's been a decade since I started

00:00:09.360 --> 00:00:12.358
pontificating on literate programming?

00:00:12.359 --> 00:00:17.542
I am Howard Abrams. In 2015, I spoke at this EmacsConf

00:00:17.543 --> 00:00:21.705
where I described my challenges I called Literate DevOps.

00:00:21.706 --> 00:00:25.634
The conference wasn't completely virtual, even though I was.

00:00:25.635 --> 00:00:29.317
My city of Portland was suffering a citywide electrical outage

00:00:29.318 --> 00:00:33.479
and I was without power, so I gave the talk in a corner of my

00:00:33.480 --> 00:00:37.439
friend's living room. People online asking questions and

00:00:37.440 --> 00:00:41.439
wondering about literate programming... I also see comments

00:00:41.440 --> 00:00:44.599
explaining why literate programming hasn't caught on in

00:00:44.600 --> 00:00:49.079
corporate practice. I often don't engage. I mean, is the

00:00:49.080 --> 00:00:51.599
online arguments and chatter over ignorance or

00:00:51.600 --> 00:00:56.719
preference? Sure, we're wired differently. I mean, my

00:00:56.720 --> 00:00:59.559
favorite programming languages put the parentheses

00:00:59.560 --> 00:01:01.939
before the function name.

00:01:01.940 --> 00:01:03.800
Literate programming has come a long way

00:01:03.801 --> 00:01:08.519
since Knuth proposed it in the 19th century. I feel

00:01:08.520 --> 00:01:12.999
it's come a long way just in the last 10 years. Obviously,

00:01:13.000 --> 00:01:16.399
this interest is due to Org. I don't think I would bother if

00:01:16.400 --> 00:01:21.359
all I had was Knuth's original preprocessor. But since I'm

00:01:21.360 --> 00:01:24.839
talking to fellow nerds about an open source project

00:01:24.840 --> 00:01:27.919
without corporate backing, let me change the title of my

00:01:27.920 --> 00:01:32.919
talk and re-pitch Literate Programming in the 24th and a

00:01:32.920 --> 00:01:35.252
Half Century!

NOTE Do I still literate?

00:01:35.253 --> 00:01:36.653
People often ask if I still program that way.

00:01:36.654 --> 00:01:42.759
I guess they want to know if there's any long-term benefits,

00:01:42.760 --> 00:01:45.919
for many of our tools and our workflows, while initially

00:01:45.920 --> 00:01:51.079
tantalizing, often don't last. But yes, when I sit down to

00:01:51.080 --> 00:01:57.759
write a program, I create a file with an extension of .org.

00:01:57.760 --> 00:02:03.799
I guess you can say I program literally.

00:02:03.800 --> 00:02:07.359
Let me be transparent. Do I use literate programming during

00:02:07.360 --> 00:02:12.599
my day job? Yes, but only for personal tools or for initial

00:02:12.600 --> 00:02:16.759
investigation. At the end of the sprint, I tangle the file

00:02:16.760 --> 00:02:21.079
and git commit that. My personal projects, on the other

00:02:21.080 --> 00:02:25.679
hand, are Org files. Since I can't show you the code from

00:02:25.680 --> 00:02:27.839
my day job, I'm afraid my example code will have a lot of

00:02:27.840 --> 00:02:31.159
parentheses.

00:02:31.160 --> 00:02:33.955
I'm sure you won't mind.

00:02:33.956 --> 00:02:37.356
I like having my Emacs configuration in Org.

00:02:37.357 --> 00:02:40.359
It's pretty bling. It has over 8,000

00:02:40.360 --> 00:02:44.559
lines of code. I know, I can hear the screams and gasps over

00:02:44.560 --> 00:02:49.439
the network. However, the surrounding prose in Org adds

00:02:49.440 --> 00:02:53.410
10,000 lines, and those lines are non-wrapped paragraphs.

00:02:53.411 --> 00:02:58.119
I mean, is that large? Sure, we've all worked on

00:02:58.120 --> 00:03:03.639
larger, so I guess it's not huge. Come on, it's still

00:03:03.640 --> 00:03:06.331
significant.

NOTE Advantages

00:03:06.332 --> 00:03:09.799
Advantages? Look who I'm talking to. I'm sure

00:03:09.800 --> 00:03:14.279
you know the advantages, but indulge me. I feel that one

00:03:14.280 --> 00:03:16.799
advantage of literate programming, especially with large

00:03:16.800 --> 00:03:20.279
code bases, is how you can organize and manage the

00:03:20.280 --> 00:03:24.839
complexity. Most programming languages tame large bases

00:03:24.840 --> 00:03:29.119
by putting code in separate files. While Org can too, with

00:03:29.120 --> 00:03:32.279
Org, we can group related functions together under

00:03:32.280 --> 00:03:35.043
expandable headlines.

00:03:35.044 --> 00:03:37.279
Here's one. You can see that

00:03:37.280 --> 00:03:40.706
I've got different sections grouped together.

00:03:40.707 --> 00:03:43.759
In my original talk, I mentioned how I would attempt to organize

00:03:43.760 --> 00:03:47.839
my thoughts before coding. I appreciate how I can look back

00:03:47.840 --> 00:03:53.599
at my notes. In my Emacs configuration, I review the prose to

00:03:53.600 --> 00:03:57.799
help memorize key bindings.

00:03:57.800 --> 00:04:01.039
My section on getting email working with Emacs using

00:04:01.040 --> 00:04:04.079
notmuch means creating small collections of scripts and

00:04:04.080 --> 00:04:08.199
configuration files. I can tangle them all from one Org

00:04:08.200 --> 00:04:16.799
file. I like that I can explain each part separately.

00:04:16.800 --> 00:04:20.879
You just can't beat having links back to Stack Overflow or

00:04:20.880 --> 00:04:25.519
that GitHub repo where you stole, I mean, became inspired to

00:04:25.520 --> 00:04:28.719
write your code.

NOTE Disadvantages

00:04:28.720 --> 00:04:34.279
Literate programming may push the boundaries of our

00:04:34.280 --> 00:04:38.119
workflows and revealing some abrasion, but we aren't

00:04:38.120 --> 00:04:41.239
solely working with Org. We have the flexibility of a Lisp

00:04:41.240 --> 00:04:45.119
engine to file down those rough parts. You may have your

00:04:45.120 --> 00:04:48.159
concerns. Perhaps you could reach out to me, and with

00:04:48.160 --> 00:04:54.239
particular issues, maybe we can figure something out.

00:04:54.240 --> 00:04:57.439
Here is my list of frictions, and the rest of my talk

00:04:57.440 --> 00:05:02.159
demonstrates my answers and my hacks. The goal in literate

00:05:02.160 --> 00:05:05.039
programming with Org is that it should not require more

00:05:05.040 --> 00:05:08.679
effort than non-literate programming. For instance, I

00:05:08.680 --> 00:05:12.119
shouldn't have to type much more than regular programming

00:05:12.120 --> 00:05:15.719
to get my code literate. I also shouldn't have to worry about

00:05:15.720 --> 00:05:20.799
the state between my Org file and the source code. I want

00:05:20.800 --> 00:05:24.132
to be able to jump around my code just as easily.

NOTE Ease of typing

00:05:24.133 --> 00:05:28.654
Let me explain more. I've created some templates using

00:05:28.655 --> 00:05:34.679
yasnippet. Since I was used to the old org-tempo feature,

00:05:34.680 --> 00:05:37.145
my habit has all the snippets starting with a

00:05:37.146 --> 00:05:40.759
< character. I'm not sure if I should demonstrate all of them

00:05:40.760 --> 00:05:45.999
as you may be doing something similar. I like to build on top

00:05:46.000 --> 00:05:49.999
of characters to remind me that if I just enter a <s, I

00:05:50.000 --> 00:05:53.519
need to put in the language. But if I append a mnemonic, I can

00:05:53.520 --> 00:05:56.839
get a full language. Why not do that with a full function

00:05:56.840 --> 00:06:01.199
definition? In this case, I'm smooshing one yasnippet

00:06:01.200 --> 00:06:11.679
inside another one in order to save myself some typing.

00:06:11.680 --> 00:06:15.159
My point here is to pay attention to what slows you down or

00:06:15.160 --> 00:06:24.719
hinders you from getting the advantages you want.

NOTE Keep tangled code sync'd

00:06:24.720 --> 00:06:28.399
Do you ever forget to tangle your code? You can append this

00:06:28.400 --> 00:06:31.519
code to the bottom of your Org file so that it gets tangled

00:06:31.520 --> 00:06:36.159
every time you save. I've written a function so I can visit

00:06:36.160 --> 00:06:40.559
that tangled file and then return. I've grouped all my

00:06:40.560 --> 00:06:45.119
functions together. I've taken a cue from Charles Choi, you

00:06:45.120 --> 00:06:48.639
know, kickingvegas, and his Casual feature set. But

00:06:48.640 --> 00:06:52.374
instead of Transient, I've just made a hydra using

00:06:52.375 --> 00:06:57.399
the major-mode-hydra package. Anyway, this allows me to use and

00:06:57.400 --> 00:07:00.136
remember my micro-optimizations.

00:07:00.137 --> 00:07:03.697
If you set the :comments property to link,

00:07:03.698 --> 00:07:06.999
the tangled output is back-connected.

00:07:07.000 --> 00:07:11.479
This allows us to edit the tangled code and have it update the

00:07:11.480 --> 00:07:16.879
Org file. Personally, I don't like this. My source of truth

00:07:16.880 --> 00:07:22.500
is the Org file, and I tangle as a one-way diode.

NOTE Code evaluation

00:07:22.501 --> 00:07:25.603
Often a block of code will reference a variable

00:07:25.604 --> 00:07:29.046
or call a function to find in another block of code.

00:07:29.047 --> 00:07:31.508
In my original literate DevOps talk,

00:07:31.509 --> 00:07:34.519
I discussed how to use the output from one block into

00:07:34.520 --> 00:07:37.799
another block by naming the first block and referencing it

00:07:37.800 --> 00:07:42.159
with a :var for the second. However, if all the blocks use the

00:07:42.160 --> 00:07:46.039
same language, you can use sessions, which create a

00:07:46.040 --> 00:07:51.479
persistent REPL behind the scenes. Let's evaluate the

00:07:51.480 --> 00:07:53.199
blocks of Python code in this file.

00:07:53.200 --> 00:08:00.119
The evaluation created a Python REPL. It's available in

00:08:00.120 --> 00:08:04.279
another buffer. This buffer matches the name of the

00:08:04.280 --> 00:08:07.959
session, but with surrounding asterisks. Evaluating a

00:08:07.960 --> 00:08:11.399
code block sends it into the REPL, and now I can work with my

00:08:11.400 --> 00:08:19.959
code blocks interactively. (That's not quite right.)

NOTE Has that block been eval'd?

00:08:19.960 --> 00:08:24.039
I primarily hack on Emacs Lisp, and textual changes to

00:08:24.040 --> 00:08:28.199
variables, functions, or macros--unless you habitually

00:08:28.200 --> 00:08:31.679
type C-c C-c--may not represent the state of your

00:08:31.680 --> 00:08:35.439
machine. A similar effect happens in any language that

00:08:35.440 --> 00:08:39.319
uses sessions. Sure, I can move the point to a block and

00:08:39.320 --> 00:08:42.799
evaluate, but I have three functions that allow me to

00:08:42.800 --> 00:08:44.734
evaluate all blocks in a buffer or all blocks in a subtree,

00:08:44.735 --> 00:08:50.199
or I can, without moving the point, evaluate any block I see.

00:08:50.200 --> 00:08:54.919
Now, this function here evaluates all blocks in a buffer.

00:08:54.920 --> 00:08:58.279
Someone mentioned calling this function when you first

00:08:58.280 --> 00:09:02.359
load a file. I'm not sure that's a good policy. I mean, have

00:09:02.360 --> 00:09:05.238
you not written a bug?

NOTE Evaluating code in a subtree

00:09:05.239 --> 00:09:08.559
Since this function right here

00:09:08.560 --> 00:09:12.039
evaluates only visible blocks, we can limit what Emacs

00:09:12.040 --> 00:09:18.799
evaluates to a single Org mode section. For instance, with

00:09:18.800 --> 00:09:23.759
the cursor in one section, I can evaluate just the blocks in

00:09:23.760 --> 00:09:26.871
that header section.

NOTE Evaluating code from a distance

00:09:26.872 --> 00:09:29.399
If I can see a block, why clumsily

00:09:29.400 --> 00:09:33.079
navigate to it when I can extend the avy project to just jump to

00:09:33.080 --> 00:09:40.479
it? For instance, let's pull this file up. I can jump to any of

00:09:40.480 --> 00:09:41.639
the four blocks.

00:09:41.640 --> 00:09:50.319
I think that's quite slick. Now why navigate to a code block

00:09:50.320 --> 00:09:55.799
solely to evaluate it? Yes, this is a terrible example, but

00:09:55.800 --> 00:09:59.679
these three blocks set a variable to different values. So

00:09:59.680 --> 00:10:02.599
without moving the point, I can evaluate any one of them.

00:10:02.600 --> 00:10:09.719
To be honest, the reason why I wrote this is because I often

00:10:09.720 --> 00:10:13.999
forget to evaluate a block after editing it. I've moved on,

00:10:14.000 --> 00:10:17.839
and I just don't want to jump back. Now, I can just evaluate

00:10:17.840 --> 00:10:22.359
from a distance. I apologize for the previous terrible

00:10:22.360 --> 00:10:26.019
examples, but I'm quite pleased with this feature.

NOTE Navigating by headers

00:10:26.020 --> 00:10:30.119
As I mentioned earlier, in a large code base, we organize code by

00:10:30.120 --> 00:10:33.839
library or module, and each file contains a class composed

00:10:33.840 --> 00:10:37.119
of methods, functions, variables, fields, et cetera.

00:10:37.120 --> 00:10:39.999
Literate programming in Org files allows me to add a

00:10:40.000 --> 00:10:43.159
semantic organization layer where I can group related

00:10:43.160 --> 00:10:46.919
concepts under headlines. Now, while this isn't specific

00:10:46.920 --> 00:10:50.799
to literate programming, I wrote a little user interface to

00:10:50.800 --> 00:10:54.296
allow me to jump to any heading in any Org file

00:10:54.297 --> 00:10:57.679
in a particular project.

00:10:57.680 --> 00:11:02.879
These are the headings in my Emacs configuration project.

00:11:02.880 --> 00:11:06.559
Notice the file name beforehand, before the colon

00:11:06.560 --> 00:11:09.759
character. The header name and its parent headers are

00:11:09.760 --> 00:11:14.799
after. Let me search for the LSP sections. Maybe I only want

00:11:14.800 --> 00:11:20.039
the one for Python. Now I use ripgrep to search the files and

00:11:20.040 --> 00:11:24.559
then some Lisp to parse the output. Unless someone has

00:11:24.560 --> 00:11:26.793
already done this, I should package this up on MELPA.

NOTE Navigating by function names

00:11:26.794 --> 00:11:32.199
What about jumping directly to the definition of a function,

00:11:32.200 --> 00:11:36.799
variable, or what have you? We can use Emacs's built-in xref

00:11:36.800 --> 00:11:39.879
library, but these functions don't understand that the

00:11:39.880 --> 00:11:45.319
source code is in Org files. When I started using Emacs

00:11:45.320 --> 00:11:49.479
30-something years ago, I would pre-index my source into

00:11:49.480 --> 00:11:53.799
tag files, but the dumb-jump project uses the newfangled and

00:11:53.800 --> 00:11:58.319
faster text search programs like ripgrep to find a symbol in

00:11:58.320 --> 00:12:02.319
real time. I followed this pattern and wrote an extension

00:12:02.320 --> 00:12:08.119
to the xref API. Now, I want to jump around my code from both

00:12:08.120 --> 00:12:14.519
code block or in the surrounding prose. I'm sure it

00:12:14.520 --> 00:12:18.199
comes as no surprise that my presentation is just an Org

00:12:18.200 --> 00:12:23.919
file. Let's suppose my cursor is on this symbol. I wrote this

00:12:23.920 --> 00:12:28.079
function for this demonstration. We can jump to the

00:12:28.080 --> 00:12:30.759
definition and I can jump back.

00:12:30.760 --> 00:12:37.639
Notice it jumped into an Org file and back out. References,

00:12:37.640 --> 00:12:42.279
unlike definitions, is where something is defined and

00:12:42.280 --> 00:12:46.919
where it's used. Well, you know how the xref system works.

00:12:46.920 --> 00:12:52.679
Here, I can jump to the definition or where it's

00:12:52.680 --> 00:12:59.519
used. Of course, and jump back. I think this is cool. This

00:12:59.520 --> 00:13:04.319
should be a nifty package on MELPA. But my code is specific to

00:13:04.320 --> 00:13:08.799
Lisp, and I'm not completely sure how to make it general. For

00:13:08.800 --> 00:13:13.399
instance, what is a symbol? If you know the language, this is

00:13:13.400 --> 00:13:17.679
obvious. But what should the language be when your cursor is

00:13:17.680 --> 00:13:22.639
in the prose of an Org file? Python only supports sequences

00:13:22.640 --> 00:13:25.559
of alphanumeric and underscores, but in Lisp, a symbol can

00:13:25.560 --> 00:13:30.399
be almost any character sequence. I've been stewing on how

00:13:30.400 --> 00:13:34.479
to do this. I have ideas like prompting during the first

00:13:34.480 --> 00:13:37.719
query or scanning the language based on the nearest code

00:13:37.720 --> 00:13:40.479
block. I think I'm babbling.

NOTE Why literate programming?

00:13:40.480 --> 00:13:47.199
In true geek fashion, I dived into the details before

00:13:47.200 --> 00:13:52.079
answering some better questions. In my original Literate

00:13:52.080 --> 00:13:55.479
DevOps talk, I explained the advantages of initially

00:13:55.480 --> 00:13:58.959
writing down your thoughts, your plans, goals... the

00:13:58.960 --> 00:14:02.879
user requirements. But what do you do with all that luscious

00:14:02.880 --> 00:14:06.359
prose afterwards? Well, you do the same thing you do to your

00:14:06.360 --> 00:14:09.279
initial code. You refactor that prose.

00:14:09.280 --> 00:14:14.759
Just because the tech surrounding your code is now a

00:14:14.760 --> 00:14:18.799
first-class citizen doesn't excuse bad code. You want

00:14:18.800 --> 00:14:23.165
something more from both your code and your prose.

NOTE LP prose isn't comments

00:14:23.166 --> 00:14:25.586
The prose of your literate program isn't

00:14:25.587 --> 00:14:28.667
just regurgitation of the code in the block.

00:14:28.668 --> 00:14:31.527
You want something more helpful.

00:14:31.528 --> 00:14:35.736
You're really writing a research paper to yourself.

00:14:35.737 --> 00:14:38.577
I know what you're thinking. You've seen my Git repos.

00:14:38.578 --> 00:14:41.858
I'm guilty and not always the best example.

00:14:41.859 --> 00:14:44.559
However, I do get great joy

00:14:44.560 --> 00:14:48.680
when I see someone ask about something in Emacs

00:14:48.681 --> 00:14:51.041
and my response is little more than a link

00:14:51.042 --> 00:14:55.799
to my online repo that I've rendered as a website.

NOTE Summary

00:14:55.800 --> 00:15:01.199
I'm out of time. I hope this has been interesting

00:15:01.200 --> 00:15:04.359
philosophically as well as practically, as I think

00:15:04.360 --> 00:15:08.559
literate programming is the cat's meow. I'm afraid this

00:15:08.560 --> 00:15:11.879
summary slide is about my home-baked solutions that fit my

00:15:11.880 --> 00:15:15.119
needs, but hopefully you can recognize your pain points and

00:15:15.120 --> 00:15:17.839
address them. If you don't need my Literate

00:15:17.840 --> 00:15:21.479
DevOps-specific techniques for connecting code blocks, I

00:15:21.480 --> 00:15:25.799
suggest using sessions by default. I highly recommend

00:15:25.800 --> 00:15:28.399
looking at your workflow and writing snippets to give you

00:15:28.400 --> 00:15:33.159
less typing for Org blocks. I now jump by headlines in my

00:15:33.160 --> 00:15:37.479
projects, but extending xref to support Org files made

00:15:37.480 --> 00:15:40.159
literate programming as easy as programming the

00:15:40.160 --> 00:15:44.319
old-fashioned way. I do need to make it more general to put up

00:15:44.320 --> 00:15:47.722
on MELPA, though. Thanks for watching.

00:15:47.723 --> 00:15:51.240
Happy hacking, my friends.