[[!meta title="Literate Programming for the 21st Century"]]
[[!meta copyright="Copyright © 2024 Howard Abrams"]]
[[!inline pages="internal(2024/info/literate-nav)" raw="yes"]]
<!-- Initially generated with emacsconf-publish-talk-page and then left alone for manual editing -->
<!-- You can manually edit this file to update the abstract, add links, etc. --->
# Literate Programming for the 21st Century
Howard Abrams (he/him) - @howard@emacs.ch , <https://www.howardism.org>
[[!inline pages="internal(2024/info/literate-before)" raw="yes"]]
Donald Knuth’s idea of *literate programming* in the 80’s and 90’s was
interesting, but he didn’t realize what Emacs and Org can do in this
century. In this talk, I would like to go back your initial *dabblings*
with Org `src` blocks to show how you can program *literately* as quickly
as you can in any other mode.
Some of the tips and tricks include:
- Automatically keeping your lit code sync’d
- Easier code generation
- Jumping to Org headers to help organize code
- Jumping to code definitions with the xref interface
At the end of this talk, I hope to inspire you to try it again, as my
personal “go to” is *programming literately*.
I will be following the format and outline in my essay:
<https://howardism.org/Technical/Emacs/literate-writ-large.html>
About the speaker:
About ten years ago, I gave a talk I called *literate devops* and people
still ask me if I still use those techniques. For all my personal
projects, I do. Even projects that I share with others, I often *start*
programming with an Org file.
I will admit, programming within Org blocks has some burrs, but over
the years, I’ve filed them off with helper functions, snippets and
other features. Thought I would share these.
# Discussion
## Questions and answers
- Q: You touched on it briefly, but how do you handle things like
"C-h f" helpful info not being tied back to the defuns in src
block code when you "C-c C-c" them in the org buffers instead of
re-tangling it to the files, and other such things? Did you create
wrappers for jumping back and forth atop org's built-in mechanisms
to go back and forth between org/tangled files?
- A: (not yet answered)
- Q: Apropos large literate programs: what's the largest code base
you've ever tackled with the literate approach (esp. Emacs +
Org-mode)?
- A: The largest is the one I mentioned in the talk ... about
8000 lines of "code" and another "10000" lines of prose. I
think I came to 15,000 max (in code blocks only).
- Q: Have you ever used org-transclusion
([https://github.com/nobiot/org-transclusion](https://github.com/nobiot/org-transclusion))?
- A: Nope \... but I will
- I tried it out once, and had one hour of work deleted 🥲,
but it was from an issue already reported:
[https://github.com/nobiot/org-transclusion/issues/177](https://github.com/nobiot/org-transclusion/issues/177)
and
[https://github.com/nobiot/org-transclusion/issues/257](https://github.com/nobiot/org-transclusion/issues/257)
- Q: What is your usage of dynamic blocks in such workflows? Any
interesting use cases and custom ones?
- A: Not yet, will report back
- Q:Is the minibuffer being deliberately hidden in this video? (first
noticed this in the section previous to "Navigating by Function
Names")
- A: Not intentionally :) You may notice the minibuffer comes and
goes, sorry about that; not intentional (didn't quite "fix"
all of them) (Thanks for the answer, no worries.)
- Q:What's your take on Emacs+Org vs. Jupyter notebooks (for
interactive programming)?
- A: Not something I use right now. Tend to include things from
jupyter/python (e.g. numpy) that has been the biggest challenge
(not knowing that stuff all that well), things like matrix
multiplications are easy in jupyter not such much in org. May
make sense to stay where you are comfortable. Curious what the
community can do to make this transition easier
- You can't work with Jupyter in \> 1 language either (I think).
It's Py + SQL or R + SQL etc. Org allows 45+ languages in one
document (I often mix languages).
- Q: Do you think any programming language is more suited to literate
programming than another?
- A: R, C are my favorites (for literate programs). C (and C++)
have got great support. There are some great books implemented
in literate programming I think. The two that come to mind are
Physically Based Rendering: From Theory to Implementation and C
Interfaces and Implementations. The first is C++ and the
second C. Ty.
- \<gs-101\> Personally, when working with Org files, I have a
better time dealing with interpreted languages, such as Python,
because you can initialize a session and the code is all
conected. You can divide blocks however you seem fit.
- Q: related to above, do you use inline org function calls and org
babel library and such?
- A: Used to use more org-babel inline functions, found sound bugs
(maybe) 8yrs ago, right now my literate dev-ops is calling a lot
of backend programs so the org-babel has limited help in that
regard while in the emacs session things are "just available"
so that hasn't help much either
- Q: How do you handle the cases where org markup may sometimes
interfere with some of the code, in places where you can't use
"escapes" (\~ or = or \| \<- vertical bar), doubly so if you use
modes to not show these but the styled text instead, and so on?
- I think an example is in C when you assign to a pointer \*p = &i;
(In Org, you need to write (\*p) or ,\*p = &i; or it will be
mistaken for a headline \*
- A: (clarifying) when I'm making pros and I'm talking about a
function I've written somewhere else I'll use tildes and look
for those things so I can strip them off. Is that the spirit of
the question. (confirmed). Yes, I'll strip that off after
finding the function name, so I can still mark it correctly.
- Q: clarifying: when in code inside an org buffer, you don't get
to use \~ or = (verbatim/etc), and any font-locking interferes
with the proper display in the src blocks, that kind of
interference.
- Q: You said at the start that literate didn't catch on in corporate
DevOps - why not?
- A: I guess the big thing is not everyone is using Emacs and org
is needed to make it work really well.
- Q: I gotta ask: why not that full stack on Markdown, I'm sure it's
crossed your mind at least a few times how the same setup on
Markdown would be more interop-friendly with colleagues and such?
- A: It's a real good idea.
- Q: How does your management of "TODOs" (projects/tasks) interact
with this literate mindset, any insightful things you do on that
front?
- A: Yeah. Okay. So, uh, on to do's and that sort of thing, um. You know, a lot is the same as it was 10 years ago. At the beginning of every sprint, my company, we're using JIRA. All jokes are valid when I say that. But I just go to its web page. I copy all the tasks that I need to do that sprint. I wrote a function that takes that code and reformats it as a bunch of org tasks for me to do that I can just now have all my to-dos. And I start to work off of that from that point on. So it'll reference all the projects, and I do symlinks to every code base that I need to use. Most of the to-dos I put inside my code block is kind of just for me, just so I can remember it. Because the work that I have to do needs to be tracked a little more fine-grained, I guess. Yeah.
- Q: \<Donovan\> Do you LP also on larger projects? (More files &
nested directories)
- A: I haven't done nested directories, but I can now. Now that
i've realized I have the feature where I can just jump to any
projects and all the org-files and all the headings just show
up, that works in nested directories, that that's fun.
- Q: Have you used Cucumber/Gherkin/BDD and do you think it has a
strong overlap to what you talked about here?
- A: I tend to put the tests right next to the function, I like
tangle it out to different files; keeping things together is
nice. Many frameworks assume we'll have things seperated out
in a way that isn't useful to me. I like to go old-school on
that?
- Q: What granularity are you looking for re your org files and
contents, with respect to a codebase that it tangles to, or in
non-coding contexts?
- A: Great questions, really subjective. I change that all
time. I have an idea, I start to refine it. My goal at one
point was to have an emacs config that was really small and
simple and that just really doesn't happen, it's full of ideas
and things that are half-baked and i pull them out and polish
them up bit by bit so it ends up being like any code-base it
just keeps getting refined. Sub-trees, archiving are useful.
- I've found it useful to prune the init file back to minimal
every once in a while (actually, AI has been surprisingly
helpful - perhaps it helps that Emacs is ancient and hence there
is a lot of doc out there and much of it \... correct?)
## Notes
- My literate programming code extensions:
[https://www.howardabrams.com/git/howard/hamacs/src/branch/main/ha-org-literate.org](https://www.howardabrams.com/git/howard/hamacs/src/branch/main/ha-org-literate.org)
- My Emacs configuration written in a literate style:
[https://github.com/howardabrams/hamacs](https://github.com/howardabrams/hamacs)
- See it rendered here:
[https://howardabrams.com/hamacs/](https://howardabrams.com/hamacs/)
- My JOPS (Jump to Project Sections) code that "searches Org
headers":
[https://www.howardabrams.com/git/howard/jops](https://www.howardabrams.com/git/howard/jops)
\... temporary location?
- snippet on \<ssl: Cool one.
- 0_0 I need to do this.
- This (evaluating babel blocks) is also possible with the
Avy + Embark combo developed by karthik:
([https://karthinks.com/software/avy-can-do-anything/#avy-plus-embark-any-action-anywhere](https://karthinks.com/software/avy-can-do-anything/#avy-plus-embark-any-action-anywhere)),
just jump to a block and then "RET".
- Really good talk. I need to find out how to extend
xref to handle org files!
- same here, I asked a long-winded
question that was about that (before he touched a bit on it),
but feel there's more in terms of wrappers and such
- Denote has some pretty good use of dynamic blocks I
think
([https://protesilaos.com/emacs/denote#h:8b542c50-dcc9-4bca-8037-a36599b22779](https://protesilaos.com/emacs/denote#h:8b542c50-dcc9-4bca-8037-a36599b22779))
- There's also the dynamic blocks from org-nursery:
[https://github.com/chrisbarrett/nursery?tab=readme-ov-file#org-roam-dblocks-incubating](https://github.com/chrisbarrett/nursery?tab=readme-ov-file#org-roam-dblocks-incubating)
- there is an RFC in for 'cargo-script' which allows
building single-file crates - i think that will be quite useful in
ob-rust
- There's a ob-rust already and it uses
rust-script: [https://github.com/micanzhang/ob-rust](https://github.com/micanzhang/ob-rust),
but the developer wanted to use rustc instead.
- cargo-script RFC issue: [https://github.com/rust-lang/cargo/issues/12207](https://github.com/rust-lang/cargo/issues/12207)
- Not a Q, just a comment that we need more of your insightful
posts and videos! :) (sic)
- I am both hyped and scared by Howard's talk, some of his past insights into using org-mode (literate everything, many small spreadsheets, etc.) have changed the way I operate so thoroughly...
- 19th century? Isn't that in the 1800s?
- 21 1/4 century?
- you are not incorrect I believe - easy mistake to make, it's not intuitive!
- YouTube comment: 1:00 So Knuth invented literate programming in the 19th century? I knew he was old, but not that old!
- i think the only time i have worked with literate programming is Inform 7
- I find that the Julia support in org (babel) is good enough. Most of my programming is in Julia.
- re: Julia in Org: For ob-julia support I wrote an ob-julia that does a few more things than the one that ships with ESS. https://github.com/karthink/ob-julia
- (I'm here trying to milk Howard for all his fantastic insights in the Etherpad, hope that's not a faux-pas...!)
- I, at least, am enjoying the Q&A session so not a faux-pas as far as I'm concerned.
- YouTube comment: The best reason I've heard explaining why most programmers don't (and won't) use literate programming. It requires them to be literate in three languages: the programming language, the markup language, and most challenging of all, English.
- YouTube comment: Q: Literate programming is very appealing in theory, but it's difficult with languages like Go where your code is split across multiple files. Any suggestions?
Feedback:
- very cool
- Lovely talk, thank you!
- Awesome, thank you!
- That was wonderful, thank you! 😊
- Thank you Howard!
- Thanks for your continued work, Howard!
- A legend! ... loved the Ironsworn presentation from previous year.
- excellent presentation indeed
- Thank you for the marvelous talk!!
- Thanks for the presentation
- Your way of delivering is inspiring.
- god I wish I was that good a presenter
- I for one had been looking forward to this particular talk, so there's that :)
- YouTube comment: Legend!
- YouTube comment: Howard! Your videos have been such an amazing source of information. You voice is engrained in my brains haha
[[!inline pages="internal(2024/info/literate-after)" raw="yes"]]
[[!inline pages="internal(2024/info/literate-nav)" raw="yes"]]
