Can you believe it's been a decade since I started
pontificating on literate programming?
I am Howard Abrams. In 2015, I spoke at this EmacsConf
where I described my challenges I called Literate DevOps.
The conference wasn't completely virtual, even though I was.
My city of Portland was suffering a citywide electrical outage
and I was without power, so I gave the talk in a corner of my
friend's living room. People online asking questions and
wondering about literate programming... I also see comments
explaining why literate programming hasn't caught on in
corporate practice. I often don't engage. I mean, is the
online arguments and chatter over ignorance or
preference? Sure, we're wired differently. I mean, my
favorite programming languages put the parentheses
before the function name.
Literate programming has come a long way
since Knuth proposed it in the 19th century. I feel
it's come a long way just in the last 10 years. Obviously,
this interest is due to Org. I don't think I would bother if
all I had was Knuth's original preprocessor. But since I'm
talking to fellow nerds about an open source project
without corporate backing, let me change the title of my
talk and re-pitch Literate Programming in the 24th and a
Half Century!
People often ask if I still program that way.
I guess they want to know if there's any long-term benefits,
for many of our tools and our workflows, while initially
tantalizing, often don't last. But yes, when I sit down to
write a program, I create a file with an extension of .org.
I guess you can say I program literally.
Let me be transparent. Do I use literate programming during
my day job? Yes, but only for personal tools or for initial
investigation. At the end of the sprint, I tangle the file
and git commit that. My personal projects, on the other
hand, are Org files. Since I can't show you the code from
my day job, I'm afraid my example code will have a lot of
parentheses.
I'm sure you won't mind.
I like having my Emacs configuration in Org.
It's pretty bling. It has over 8,000
lines of code. I know, I can hear the screams and gasps over
the network. However, the surrounding prose in Org adds
10,000 lines, and those lines are non-wrapped paragraphs.
I mean, is that large? Sure, we've all worked on
larger, so I guess it's not huge. Come on, it's still
significant.
Advantages? Look who I'm talking to. I'm sure
you know the advantages, but indulge me. I feel that one
advantage of literate programming, especially with large
code bases, is how you can organize and manage the
complexity. Most programming languages tame large bases
by putting code in separate files. While Org can too, with
Org, we can group related functions together under
expandable headlines.
Here's one. You can see that
I've got different sections grouped together.
In my original talk, I mentioned how I would attempt to organize
my thoughts before coding. I appreciate how I can look back
at my notes. In my Emacs configuration, I review the prose to
help memorize key bindings.
My section on getting email working with Emacs using
notmuch means creating small collections of scripts and
configuration files. I can tangle them all from one Org
file. I like that I can explain each part separately.
You just can't beat having links back to Stack Overflow or
that GitHub repo where you stole, I mean, became inspired to
write your code.
Literate programming may push the boundaries of our
workflows and revealing some abrasion, but we aren't
solely working with Org. We have the flexibility of a Lisp
engine to file down those rough parts. You may have your
concerns. Perhaps you could reach out to me, and with
particular issues, maybe we can figure something out.
Here is my list of frictions, and the rest of my talk
demonstrates my answers and my hacks. The goal in literate
programming with Org is that it should not require more
effort than non-literate programming. For instance, I
shouldn't have to type much more than regular programming
to get my code literate. I also shouldn't have to worry about
the state between my Org file and the source code. I want
to be able to jump around my code just as easily.
Let me explain more. I've created some templates using
yasnippet. Since I was used to the old org-tempo feature,
my habit has all the snippets starting with a
< character. I'm not sure if I should demonstrate all of them
as you may be doing something similar. I like to build on top
of characters to remind me that if I just enter a <s, I
need to put in the language. But if I append a mnemonic, I can
get a full language. Why not do that with a full function
definition? In this case, I'm smooshing one yasnippet
inside another one in order to save myself some typing.
My point here is to pay attention to what slows you down or
hinders you from getting the advantages you want.
Do you ever forget to tangle your code? You can append this
code to the bottom of your Org file so that it gets tangled
every time you save. I've written a function so I can visit
that tangled file and then return. I've grouped all my
functions together. I've taken a cue from Charles Choi, you
know, kickingvegas, and his Casual feature set. But
instead of Transient, I've just made a hydra using
the major-mode-hydra package. Anyway, this allows me to use and
remember my micro-optimizations.
If you set the :comments property to link,
the tangled output is back-connected.
This allows us to edit the tangled code and have it update the
Org file. Personally, I don't like this. My source of truth
is the Org file, and I tangle as a one-way diode.
Often a block of code will reference a variable
or call a function to find in another block of code.
In my original literate DevOps talk,
I discussed how to use the output from one block into
another block by naming the first block and referencing it
with a :var for the second. However, if all the blocks use the
same language, you can use sessions, which create a
persistent REPL behind the scenes. Let's evaluate the
blocks of Python code in this file.
The evaluation created a Python REPL. It's available in
another buffer. This buffer matches the name of the
session, but with surrounding asterisks. Evaluating a
code block sends it into the REPL, and now I can work with my
code blocks interactively. (That's not quite right.)
I primarily hack on Emacs Lisp, and textual changes to
variables, functions, or macros--unless you habitually
type C-c C-c--may not represent the state of your
machine. A similar effect happens in any language that
uses sessions. Sure, I can move the point to a block and
evaluate, but I have three functions that allow me to
evaluate all blocks in a buffer or all blocks in a subtree,
or I can, without moving the point, evaluate any block I see.
Now, this function here evaluates all blocks in a buffer.
Someone mentioned calling this function when you first
load a file. I'm not sure that's a good policy. I mean, have
you not written a bug?
Since this function right here
evaluates only visible blocks, we can limit what Emacs
evaluates to a single Org mode section. For instance, with
the cursor in one section, I can evaluate just the blocks in
that header section.
If I can see a block, why clumsily
navigate to it when I can extend the avy project to just jump to
it? For instance, let's pull this file up. I can jump to any of
the four blocks.
I think that's quite slick. Now why navigate to a code block
solely to evaluate it? Yes, this is a terrible example, but
these three blocks set a variable to different values. So
without moving the point, I can evaluate any one of them.
To be honest, the reason why I wrote this is because I often
forget to evaluate a block after editing it. I've moved on,
and I just don't want to jump back. Now, I can just evaluate
from a distance. I apologize for the previous terrible
examples, but I'm quite pleased with this feature.
As I mentioned earlier, in a large code base, we organize code by
library or module, and each file contains a class composed
of methods, functions, variables, fields, et cetera.
Literate programming in Org files allows me to add a
semantic organization layer where I can group related
concepts under headlines. Now, while this isn't specific
to literate programming, I wrote a little user interface to
allow me to jump to any heading in any Org file
in a particular project.
These are the headings in my Emacs configuration project.
Notice the file name beforehand, before the colon
character. The header name and its parent headers are
after. Let me search for the LSP sections. Maybe I only want
the one for Python. Now I use ripgrep to search the files and
then some Lisp to parse the output. Unless someone has
already done this, I should package this up on MELPA.
What about jumping directly to the definition of a function,
variable, or what have you? We can use Emacs's built-in xref
library, but these functions don't understand that the
source code is in Org files. When I started using Emacs
30-something years ago, I would pre-index my source into
tag files, but the dumb-jump project uses the newfangled and
faster text search programs like ripgrep to find a symbol in
real time. I followed this pattern and wrote an extension
to the xref API. Now, I want to jump around my code from both
code block or in the surrounding prose. I'm sure it
comes as no surprise that my presentation is just an Org
file. Let's suppose my cursor is on this symbol. I wrote this
function for this demonstration. We can jump to the
definition and I can jump back.
Notice it jumped into an Org file and back out. References,
unlike definitions, is where something is defined and
where it's used. Well, you know how the xref system works.
Here, I can jump to the definition or where it's
used. Of course, and jump back. I think this is cool. This
should be a nifty package on MELPA. But my code is specific to
Lisp, and I'm not completely sure how to make it general. For
instance, what is a symbol? If you know the language, this is
obvious. But what should the language be when your cursor is
in the prose of an Org file? Python only supports sequences
of alphanumeric and underscores, but in Lisp, a symbol can
be almost any character sequence. I've been stewing on how
to do this. I have ideas like prompting during the first
query or scanning the language based on the nearest code
block. I think I'm babbling.
In true geek fashion, I dived into the details before
answering some better questions. In my original Literate
DevOps talk, I explained the advantages of initially
writing down your thoughts, your plans, goals... the
user requirements. But what do you do with all that luscious
prose afterwards? Well, you do the same thing you do to your
initial code. You refactor that prose.
Just because the tech surrounding your code is now a
first-class citizen doesn't excuse bad code. You want
something more from both your code and your prose.
The prose of your literate program isn't
just regurgitation of the code in the block.
You want something more helpful.
You're really writing a research paper to yourself.
I know what you're thinking. You've seen my Git repos.
I'm guilty and not always the best example.
However, I do get great joy
when I see someone ask about something in Emacs
and my response is little more than a link
to my online repo that I've rendered as a website.
I'm out of time. I hope this has been interesting
philosophically as well as practically, as I think
literate programming is the cat's meow. I'm afraid this
summary slide is about my home-baked solutions that fit my
needs, but hopefully you can recognize your pain points and
address them. If you don't need my Literate
DevOps-specific techniques for connecting code blocks, I
suggest using sessions by default. I highly recommend
looking at your workflow and writing snippets to give you
less typing for Org blocks. I now jump by headlines in my
projects, but extending xref to support Org files made
literate programming as easy as programming the
old-fashioned way. I do need to make it more general to put up
on MELPA, though. Thanks for watching.
Happy hacking, my friends.