1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
|
# README-Driven Design
Adam Ard
[[!template id=vid src="https://mirror.csclub.uwaterloo.ca/emacsconf/2020/emacsconf-2020--14-readme-driven-design--adam-ard.webm" subtitles="/2020/subtitles/emacsconf-2020--14-readme-driven-design--adam-ard.vtt"]]
[Download compressed .webm video (21.4M)](https://mirror.csclub.uwaterloo.ca/emacsconf/2020/smaller/emacsconf-2020--14-readme-driven-design--adam-ard--vp9-q56-video-original-audio.webm)
[View transcript](#transcript)
Many source code projects these days begin with a README file. While
most people use markdown, if you use org-mode, you can use literate
programming to generate all of your source code directly from the
documentation. This strategy is a great way to keep your documentation
from getting outdated, and it allows you to use all the other
wonderful features of org-mode. Watch "README-Driven Design" to see
exactly how to make your README file a powerful literate document.
- [Companion Blog Post](http://adamard.com/literate_programming.html)
- [Extended Version of Demo File](https://github.com/adam-ard/literate-demo)
- [Literate Static Website](https://github.com/adam-ard/static-website-literate-demo)
<!-- from the pad --->
- Actual start and end time (EST): Start: 2020-11-28T14.15.00; End:
2020-11-28T14.34.46
# Questions
## If you put all your code in an Org file (in addition to prose), doesn't that make the file very large for medium/large projects? (Since all the code across all files is tangled from a single README.org)
You are right it would get pretty large. I haven't hit that point yet,
but plan to experiment with separate Org files that are imported into
a master file.
## If a collaborator edits the tangled file(s), is reverse-tangling in Org reliable? How do you integrate the reverse in a safe way?
So, I actually think this is the big unsolved problem right now. How
to do reverse tangling. As far as I know, Emacs doesn't do that. But
it would really cool. I think it is probably a hard problem.
- Actually it does! You have to enable comments that mark the
boundaries of the code blocks. (org-babel-detangle) ->
org-babel-detangle is pretty fragile right now.
- Oh wonderful! I will have to check that out. There is always more to
discover in Emacs. Thanks!
## Would this approach make it harder to collaborate with contributors who don't use Org? / How to rectify these difficulties? (Thank you!)
I have had some sucess at work by managing an Org file myself, then I
commit the tangled code and a README.md. I have to manually update my
Org file though when someone makes a change to the raw source
files. That process can be a pain. It would be awesome to find a way
to make this easier. So that non Emacs users can collaborate and be
unaware of the source Org file. To have an annotation free reverse
tangling process would be the holy grail of literate
programming. Would be a great thesis project for someone.
## Interesting. Did you ever use this approach on a large project? Could one incorporate also TDD into this workflow?
I have only really hit the medium size. But would love to try a larger
one. I have seen people write whole books in literate progamming
though. (Not sure if they were using Emacs) (one example:
<http://www.pbr-book.org/> ). Here is a pretty large one I found on
GitHub: <https://github.com/nakkaya/ferret>.
TDD is an interesting idea. I haven't tried doing that, but Org seems
flexible enough to build a workflow around that.
## Could you share the snippet for adding these source code blocks, it seems much better than the one I am using currently. Thanks!
Sure, it is documented in the literate programming demo here
(<https://github.com/adam-ard/literate-demo>).
## In Python, indentation is part of the syntax. How is this handled when <<foo>>-syntax is used for functions or even a few lines of codes that are get re-used in multiple functions? Does the user have to define different <<foo>> snippets for different indentations but otherwise identical code?
Not the speaker, but `:noweb` will add the prefix characters to all
lines, see
<https://orgmode.org/manual/Noweb-Reference-Syntax.html>. Python
identation is fine (and used as an example in the manual :))
- Exactly, I have done a lot of Python this way, it works great!
## Could this structure be used with a SQL query with the output being an Org table?
Yep, I have done that before too. Org will send the query to a
database and insert the results. It is super nice. You can add block
properties to set the hostname of the database too, so it isn't
limited to just databases running on your local machine.
## Why do you export to Markdown when GitHub and others are supporting rendering Org directly?
Good question. I do this because I usually work with people that don't
use Emacs :( so I usually take the source files and the Markdown and
commit them to Git. I keep the Org file to myself. If everyone used
Emacs, I wouldn't bother with that step.
## This file would be very useful to have for us for reference, could you also share it please?
Yep! See the links below for a couple template files. An extended one
from the talk is at: <https://github.com/adam-ard/literate-demo>
# Notes
- Adam Ard: I'll be answering questions here in the pad or in
#emacsconf (aard3).
- Companion Blog Post: <http://adamard.com/literate_programming.html>.
- Extended Version of Demo File:
<https://github.com/adam-ard/literate-demo>.
- Literate Static Website:
<https://github.com/adam-ard/static-website-literate-demo>.
- If you want to learn what GitHub is able to render in Org syntax:
<https://github.com/novoid/github-orgmode-tests>.
- <https://en.wikipedia.org/wiki/Literate_programming>.
- I am thinking about org-transclusion; similar ideas to deal with
notes instead of codes.
- FYI: <https://github.com/alphapapa/transclusion-in-emacs>.
<a name="transcript"></a>
# Transcript
00:00:03.600 --> 00:00:37.120
Adam: Hello! Welcome to Readme Driven
Design in Emacs by Adam Aard. If you're
a programmer, you're accustomed to
putting a README file at the root of
your project. It's usually a Markdown
file But if you use an Org Mode file
instead, you can take advantage of the
great features that Org Mode provides,
including literate programming, which
lets you generate your source code and
Markdown documentation dynamically.
I want to walk you through a little bit
of what this looks like.
00:00:37.120 --> 00:01:03.520
When you start a project, especially if
if you use something like Github you
begin with an automatically generated
README.md file. So just delete that
and instead create a README.org file.
Starting with an empty Org file,
like you see here, you can begin
by recording important information about
your project goals. You can add diagrams,
code snippets, to-do lists, time tracking
and much more.
00:01:03.520 --> 00:01:38.880
I'm going to drop in some documentation
that I've written about about my project
here, so you can see what this would
look like. As you can see, I have a
title, and a description, and then a
subsection as well as some code
snippets. You can see that Org Mode does
a great job of formatting lists, code
sections, diagrams, and so forth. It's
as good or better than Markdown, but
when you use it in Emacs you can do a
lot more.
00:01:38.880 --> 00:02:08.000
For example, you can dynamically create
diagrams using Graphviz from a text
description. If you go to this source
block here and hit C-c C-c, you'll see
that we generate a diagram dynamically
You can run these code snippets in place
and get the results to show up inside of
your file, which is a really powerful
paradigm.
00:02:08.000 --> 00:02:19.520
But most importantly, for my purposes
here, Org Mode provides you the ability
to do literate programming.
00:02:19.520 --> 00:02:34.720
So take a quick look at this diagram
that I generated here. It gives you a
quick overview of what I mean by
literate programming and how I'm using
it. You can see that we start with a
README.org file on top.
00:02:34.720 --> 00:03:17.120
At this point, we can do one of two
things: tangle or weave. Tangle is used
to describe the process of generating
source code, while weave is the process
of generating documentation. These are
terms that Donald Knuth used. He's the
one that came up with the idea of
literate programming in the early 1980s.
But this is really all that there is to
it. You just... You are simply using a
literate source file, in this case the
README.org, to generate the rest of the
project files, basically.
00:03:17.120 --> 00:03:59.479
So let's dig in to the details of how
this works. I hope you... Hopefully
you'll see how cool this is. So
returning to the file here. Let's assume
we have enough documentation now, that
we want to get started coding. So maybe
we'll just start with like a Hello World
app, just so we can make sure that our
environment is set up correctly. Let's
get started with a code block. So I
created a little snippet to help me add
a source block for literate programming
quickly. There's not much to it, but
there are some important annotations
here.
00:04:01.599 --> 00:04:55.360
Excuse me. There's a property called
:tangle and that takes a value of a file
name. Then there's also a :noweb
property called no-export. Basically,
the noexport--we'll explain that a
little bit more later It has to do with
how the tangling is done in the tangle
step versus the weave step. I'll explain
that a little bit more. But the tangle
field just simply tells Emacs where it
needs to generate the main.go file and
where it needs to put it on the file
system.
00:04:55.360 --> 00:05:21.520
You'll notice that we're going to use
Go. That's just the language that I've
been using the most lately, but this
programming strategy is
language-agnostic. You could use any
language or any mix of languages. You
could create some files in Python, some
files in Go, some files in Lisp, or
whatever you want.
00:05:24.720 --> 00:05:56.400
Let's create just a little Hello World.
Let's use another snippet here to
generate the basics of a Go program. I'm
just going to print Hello World. So
that's... And then let's make it a
section in our file. So now you can see,
we've got this snippet.
00:05:56.400 --> 00:06:42.319
When you have a source block in inside
of Org Mode, you can easily pop into a
language-specific buffer by typing C-c '
(single quote). So you can see, now I
have a buffer that's in go-mode and
gives you all the ability to edit like
you would normally. If you hit C-c '
(single quote) again, it goes back and
any changes you make will be updated
there. But you can do quite a bit just
inside of here too. There's quite a bit
of language-specific functionality just
in place, so you don't always have to go
over to a separate buffer. It's a nice
option sometimes.
00:06:42.319 --> 00:07:12.240
Now that you have the code in here,
you're going to want to run it. Right
now, it just lives here in this
documentation. You need to get a copy of
it into a separate file, and that's the
tangle process that you you need to
follow there. So I'm gonna drop in a
little bit more doc, a little bit more
documentation really quick here.
00:07:12.240 --> 00:07:44.879
Okay, all right. So just as a side note,
I like to follow this process. Whenever
I have an operation to perform, I'd like
to document it here with a snippet that
can be executed inline. Then I don't
have to leave Org Mode, and I don't have
to try to remember what I did later. So
instead of just trying to do an
operation, the first time I do
something, I take the time to figure out
what it is and document it, so then it's
recorded.
00:07:44.879 --> 00:08:14.400
So here we find that to do a tangle
operation, you run the command
org-babel-tangle, which is an Elisp
command. If you hit C-c C-c to run it in
place, you get the result of main.go,
which basically is telling us that we've
tangled one file called main.go. You can
see that that's true if you go to the
file system and you look.
00:08:14.400 --> 00:08:41.120
Now in our demo directory, we have a
README.org, we have that PNG that we
generated, but we also have a main.go.
If you visit that file, you'll see that
it's just the source code that was in
our documentation, which is exactly what
we expected and what we wanted. So
that's good.
So if we return to where we were at...
00:08:41.120 --> 00:09:43.012
Now we're at the point where we have a
file on the file system. Now we need to
build it and to run it. So let's follow
the same philosophy, where let's
document these operations that we're
going to perform. I'm dropping in a a
build instruction section and a run
instruction section. As you can see
here, we have a little a bash source
block, and another bash source block.
This one compiles. The go build command
is what compiles a file. Then the file
that gets generated should be called
demo. So we just run it here. If I type
C-c C-c, we get an empty results block.
When you compile things, no news is good
news. It means there's no errors.
00:09:43.012 --> 00:10:30.839
So presumably, we've created an
executable that's called demo. Let's
look again at the file system and
regenerate... Yep. What we have here is
a demo executable, which is exactly what
we wanted. Let's go back. Now we should
be able to run it. C-c C-c, and we get
Hello World as a result, which was
exactly what we were expecting. So
that's already pretty cool. You can do
that much.
00:10:33.040 --> 00:11:09.760
That's really just the tip of the
iceberg. To really use the more
impressive features of literate
programming, we need to do a little bit
more at least. Really, to get the full
benefit of it, we need to add some
sections that will cause Emacs to have
to tangle or assemble this file from
different pieces.
00:11:09.760 --> 00:11:36.240
Imagine that we wanted to take this file
and maybe kind of templatize it. So,
using literature programming syntax,
this angle bracket syntax, let's say
that we want to create an imports
section, a functions section, and then
maybe just a main section. We'll get rid
of this.
00:11:36.240 --> 00:11:56.639
So now you see, we've created something
that looks a little bit like a template
or a scaffolding or outline for what our
file is going to be. It looks a little
bit like pseudocode. What we're going to
have literate programming do is
dynamically insert those things into
those slots.
00:11:56.639 --> 00:12:36.639
So the first thing we need to do is...
So let's create a section called "Say
Hello." We want to add some
functionality that makes our program say
hello. So using a different snippet that
I have for creating something that I
call like a literate section, basically,
we create a another source block that's
almost the same as the one for the file.
It just has a few differences. Say we
want to drop code into the import
section and we want it to be in Go.
00:12:36.639 --> 00:13:14.399
Here we use the same :noweb no-export
syntax, but then we've added this
:noweb-ref imports, and this ties that
slot to this reference. It tells Emacs
that when you tangle, we want to stick
whatever's in here in that spot. You
skip the tangle file name section
because you're not actually creating a
file name. You're putting information
into an existing file. So here, we would
just add the "fmt" for the imports.
00:13:14.399 --> 00:14:10.320
Let's add another section for functions.
Let's just create a function called
sayHello that doesn't have any
arguments. No return types. All it does
is pretty much the same thing as we did
before: just print something. Let's just
say "Hello EmacsConf" this time. Now we
have a function, and now the function
won't do anything unless we invoke it.
Let's do one last literate section
called main. Make that Go source block.
Then let's invoke that function.
00:14:10.320 --> 00:14:39.839
Now you can see that we've got our
scaffolding outline, and then we have
the sections that we want to get tangled
or inserted. I've used this syntax. It's
kinda borrowed from literate programming
a little bit with a +=, so really it's
just saying that I want to append this
item into the import section It's really
just to make a little bit more clear
what's going on.
00:14:39.839 --> 00:14:57.760
When you generate documentation, you
won't see these particular property
annotations, and so you won't know
immediately that this section goes in
the imports area. So I usually put a
little bit of documentation on top
there, so that it's easy to see.
00:14:57.760 --> 00:15:21.120
You would, probably, if this was very
complicated, you'd put some
documentation above to explain what you
were doing, maybe right here. You could
picture yourself maybe explaining a
complicated algorithm or something up
here and having a nice way to document
it.
00:15:21.120 --> 00:15:28.045
So now that we've got that here in the
documentation, we need to figure out...
We need to make sure that it's going to
tangle properly.
00:15:28.045 --> 00:16:20.479
Your best friend at this point is a
keyboard shortcut that lets you preview
the tangled operation. If you say C-c
C-v C-v, it will create a new buffer
with the tangled contents and so you can
see here that the fmt import went to the
right place, that function went to the
right place, the function invocation
went to the right place. We're feeling
good. You can nest these things many
layers deep. If you came into the
sayHello function, you could add more
sections. It'll go through and it'll
keep track of all that and tangle it for
you so you really get a lot of freedom
and flexibility for how you want to
document things by doing this.
00:16:20.479 --> 00:16:57.645
So now that we've previewed it and we
feel good about it, we need to tangle so
we get the file on the file system. so
C-c C-c and get... just main.go comes
back again. C-c C-c and no errors come
back. Then if we did this right, when we
run this, we should get "Hello,
EmacsConf." So C-c C-c, Hello EmacsConf.
I think that's pretty, pretty cool,
actually.
00:16:57.645 --> 00:17:23.280
So we've got the breadcrumbs of the
process we've gone through to get to
this point, this initial document that
has some tangling in it. We have
documentation for how to tangle, how to
build, how to run. We've really built a
nice foundation for moving forward on
our project and a nice way of breaking
things out and documenting further.
00:17:23.280 --> 00:17:38.640
The last piece that we need to
take care of is the weave that
I showed you in the diagram above.
So one more time, we'll drop in
00:17:38.640 --> 00:18:35.520
some documentation, this time on how to
weave. It's really just an export
function. it's not... There's not a
separate weave command going on here.
we're just going to export what we've
got here into a Markdown format. We're
using org-gfm-export-to-markdown, which
is the Github style markdown. You can
use the other, more standard type as
well. Hit C-c C-c. Now you see we've got
a README file, and if you look in the
file system, we've got that right there.
If you go to something like ghostwriter
and open that file, now you can see that
it's generated some documentation.
00:18:35.520 --> 00:18:48.559
It puts a index at top at the top. I
usually turn that off. It's easy to do
that by putting a property at the top of
your Org file, but some people like to
have an index.
00:18:48.559 --> 00:19:22.802
Here you can see that it has generated
pretty nicely and formatted snippets
well, put the diagram in there, and then
it's preserved this literate programming
syntax, which is important because
that's how we want to view the
documentation. That's what the
no-exports property was trying to
maintain. no-exports means when you
export, do not try to tangle.
00:19:22.802 --> 00:19:43.600
Hopefully that makes more sense now. Now
you can see all the documentation. I
think it demonstrates a pretty useful
feature that's inside of Emacs.
Hopefully you'll have as much fun using
that as I have. So thanks!
|