[[!meta title="Literate Programming for the 21st Century"]] [[!meta copyright="Copyright © 2024 Howard Abrams"]] [[!inline pages="internal(2024/info/literate-nav)" raw="yes"]] # Literate Programming for the 21st Century Howard Abrams (he/him) - @howard@emacs.ch , [[!inline pages="internal(2024/info/literate-before)" raw="yes"]] Donald Knuth’s idea of *literate programming* in the 80’s and 90’s was interesting, but he didn’t realize what Emacs and Org can do in this century. In this talk, I would like to go back your initial *dabblings* with Org `src` blocks to show how you can program *literately* as quickly as you can in any other mode. Some of the tips and tricks include: - Automatically keeping your lit code sync’d - Easier code generation - Jumping to Org headers to help organize code - Jumping to code definitions with the xref interface At the end of this talk, I hope to inspire you to try it again, as my personal “go to” is *programming literately*. I will be following the format and outline in my essay: About the speaker: About ten years ago, I gave a talk I called *literate devops* and people still ask me if I still use those techniques. For all my personal projects, I do. Even projects that I share with others, I often *start* programming with an Org file. I will admit, programming within Org blocks has some burrs, but over the years, I’ve filed them off with helper functions, snippets and other features. Thought I would share these. # Discussion ## Questions and answers - Q: Apropos large literate programs: what's the largest code base you've ever tackled with the literate approach (esp. Emacs + Org-mode)? - A: The largest is the one I mentioned in the talk \... about 8000 lines of "code" and another "10000" lines of prose. I think I came to 15,000 max (in code blocks only). - Q: You touched on it briefly, but how do you handle things like "C-h f" helpful info not being tied back to the defuns in src block code when you "C-c C-c" them in the org buffers instead of re-tangling it to the files, and other such things? Did you create wrappers for jumping  back and forth atop org's built-in mechanisms to go back and forth between org/tangled files? - A: - Q: Have you ever used org-transclusion ([https://github.com/nobiot/org-transclusion](https://github.com/nobiot/org-transclusion){rel="noreferrer noopener"})? - A: Nope \... but I will - I tried it out once, and had one hour of work deleted 🥲, but it was from an issue already reported: [https://github.com/nobiot/org-transclusion/issues/177](https://github.com/nobiot/org-transclusion/issues/177){rel="noreferrer noopener"} and [https://github.com/nobiot/org-transclusion/issues/257](https://github.com/nobiot/org-transclusion/issues/257){rel="noreferrer noopener"} - Q: What is your usage of dynamic blocks in such workflows? Any interesting use cases and custom ones? - A: Not yet, will report back - Q:Is the minibuffer being deliberately hidden in this video? (first noticed this in the section previous to "Navigating by Function Names") - A: Not intentionally :)  You may notice the minibuffer comes and goes, sorry about that; not intentional (didn't quite "fix" all of them) (Thanks for the answer, no worries.) - Q:What's your take on Emacs+Org vs. Jupyter notebooks (for interactive programming)? - A: Not something I use right now.  Tend to include things from jupyter/python (e.g. numpy) that has been the biggest challenge (not knowing that stuff all that well), things like matrix multiplications are easy in jupyter not such much in org. May make sense to stay where you are comfortable.  Curious what the community can do to make this transition easier - You can't work with Jupyter in \> 1 language either (I think). It's Py + SQL or R + SQL etc. Org allows 45+ languages in one document (I often mix languages). - Q: Do you think any programming language is more suited to literate programming than another? - A: R, C are my favorites (for literate programs). C (and C++) have got great support. There are some great books implemented in literate programming I think. The two that come to mind are Physically Based Rendering: From Theory to Implementation and C Interfaces and Implementations. The first is C++ and the second C. Ty. - A: \ Personally, when working with Org files, I have a better time dealing with interpreted languages, such as Python, because you can initialize a session and the code is all conected. You can divide blocks however you seem fit. - Q: related to above, do you use inline org function calls and org babel library and such? - A: usta-use more org-babel inline functions, found sound bugs (maybe) 8yrs ago, right now my literate dev-ops is calling a lot of backend programs so the org-babel has limited help in that regard while in the emacs session things are "just available" so that hasn't help much either - Q: How do you handle the cases where org markup may sometimes interfere with some of the code, in places where you can't use "escapes" (\~ or = or \| \<- vertical bar), doubly so if you use modes to not show these but the styled text instead, and so on? - I think an example is in C when you assign to a pointer  \*p = &i;   (In Org, you need to write (\*p) or ,\*p = &i; or it will be mistaken for a headline \* - A: (clarifying) when I'm making pros and I'm talking about a function I've written somewhere else I'll use tildes and look for those things so I can strip them off.  Is that the spirit of the question.  (confirmed).  Yes, I'll strip that off after finding the function name, so I can still mark it correctly. - Q: clarifying: when in code inside an org buffer, you don't get to use \~ or = (verbatim/etc), and any font-locking interferes with the proper display in the src blocks, that kind of interference. - Q: You said at the start that literate didn't catch on in corporate DevOps - why not? - A: I guess the big thing is not everyone is using Emacs and org is needed to make it work really well. - Q: I gotta ask: why not that full stack on Markdown, I'm sure it's crossed your mind at least a few times how the same setup on Markdown would be more interop-friendly with colleagues and such? - A:  It's a real good idea. - Q: How does your management of "TODOs" (projects/tasks) interact with this literate mindset, any insightful things you do on that front? - A:  - Q: \ Do you LP also on larger projects? (More files & nested directories)  - A:  I haven't done nested directories, but I can now. Now that i've realized I have the feature where I can just jump to any projects and all the org-files and all the headings just show up, that works in nested directories, that that's fun. - Q: Have you used Cucumber/Gherkin/BDD and do you think it has a strong overlap to what you talked about here? - A: I tend to put the tests right next to the function, I like tangle it out to different files; keeping things together is nice.  Many frameworks assume we'll have things seperated out in a way that isn't useful to me.  I like to go old-school on that? - Q: What granularity are you looking for re your org files and contents, with respect to a codebase that it tangles to, or in non-coding contexts? - A:  Great questions, really subjective.  I change that all time.  I have an idea, I start to refine it. My goal at one point was to have an emacs config that was really small and simple and that just really doesn't happen, it's full of ideas and things that are half-baked and i pull them out and polish them up bit by bit so it ends up being like any code-base it just keeps getting refined.  Sub-trees, archiving are useful. - I've found it useful to prune the init file back to minimal every once in a while (actually, AI has been surprisingly helpful - perhaps it helps that Emacs is ancient and hence there is a lot of doc out there and much of it \... correct?) ## Notes - My literate programming code extensions: [https://www.howardabrams.com/git/howard/hamacs/src/branch/main/ha-org-literate.org](https://www.howardabrams.com/git/howard/hamacs/src/branch/main/ha-org-literate.org){rel="noreferrer noopener"} - My Emacs configuration written in a literate style: [https://github.com/howardabrams/hamacs](https://github.com/howardabrams/hamacs){rel="noreferrer noopener"} - See it rendered here: [https://howardabrams.com/hamacs/](https://howardabrams.com/hamacs/){rel="noreferrer noopener"} - My JOPS (Jump to Project Sections) code that "searches Org headers": [https://www.howardabrams.com/git/howard/jops](https://www.howardabrams.com/git/howard/jops){rel="noreferrer noopener"} \... temporary location? - snippet on \ 0_0 I need to do this. - Your way of delivering is inspiring. - \ This (evaluating babel blocks) is also possible with the Avy + Embark combo developed by karthik: ([https://karthinks.com/software/avy-can-do-anything/#avy-plus-embark-any-action-anywhere](https://karthinks.com/software/avy-can-do-anything/#avy-plus-embark-any-action-anywhere){rel="noreferrer noopener"}), just jump to a block and then "RET". - \ god I wish I was that good a presenter - \ Really good talk.  I need to find out how to extend xref to handle org files! - \ ericsfraga: same here, I asked a long-winded question that was about that (before he touched a bit on it), but feel there's more in terms of wrappers and such - \ excellent presentation indeed - \ Thank you for the marvelous talk!! - \ A legend!  \... loved the Ironsworn presentation from previous year. - \ Denote has some pretty good use of dynamic blocks I think ([https://protesilaos.com/emacs/denote#h:8b542c50-dcc9-4bca-8037-a36599b22779](https://protesilaos.com/emacs/denote#h:8b542c50-dcc9-4bca-8037-a36599b22779){rel="noreferrer noopener"}) - There's also the dynamic blocks from org-nursery: [https://github.com/chrisbarrett/nursery?tab=readme-ov-file#org-roam-dblocks-incubating](https://github.com/chrisbarrett/nursery?tab=readme-ov-file#org-roam-dblocks-incubating){rel="noreferrer noopener"} - \ there is an RFC in for 'cargo-script' which allows building single-file crates - i think that will be quite useful in ob-rust - \ ellis: There's a ob-rust already and it uses rust-script: [https://github.com/micanzhang/ob-rust](https://github.com/micanzhang/ob-rust){rel="noreferrer noopener"}, but the developer wanted to use rustc instead. - cargo-script RFC issue: [https://github.com/rust-lang/cargo/issues/12207](https://github.com/rust-lang/cargo/issues/12207){rel="noreferrer noopener"} - Not a Q, just a comment that we need more of your insightful posts and videos! :) (sic) [[!inline pages="internal(2024/info/literate-after)" raw="yes"]] [[!inline pages="internal(2024/info/literate-nav)" raw="yes"]]