path: root/2023/captions/emacsconf-2023-repl--repls-in-strange-places-lua-latex-lpeg-lpegrex-tikz--eduardo-ochs--main.vtt

WEBVTT captioned by sachac

NOTE Intro

00:00.000 --> 00:00:00.014
Hi, my name is Eduardo Ochs

00:00:00.015 --> 00:00:05.020
and the title of this talk is

00:05.020 --> 00:07.840
REPLs in Strange Places: Lua, LaTeX,

00:07.840 --> 00:10.440
LPeg, LPegRex, and TikZ.

00:10.440 --> 00:13.400
I'm the author of an Emacs package called eev,

00:13.400 --> 00:17.280
and this is a talk at the EmacsConf 2023,

00:17.280 --> 00:21.560
that is happening in December 2023 at the Internet.

NOTE Diagrams

00:21.560 --> 00:00:24.243
This is one of the examples of diagrams

00:00:24.244 --> 00:00:25.920
that we are going to see.

00:25.920 --> 00:28.400
Let me show how I generated it.

00:28.400 --> 00:33.600
One second, I have to use a smaller font here.

00:34.560 --> 00:39.080
This is a file called parse32.lua.

00:39.080 --> 00:42.600
Let me go back to this block of tests again.

00:42.600 --> 00:45.520
Now, if I run this,

00:46.720 --> 00:51.520
we get these outputs here at the right.

00:51.520 --> 00:53.800
Then in this line here,

00:53.800 --> 00:55.420
it generates a PDF,

00:55.420 --> 00:57.780
and if I type `F8` here,

00:57.780 --> 01:02.560
it shows the PDF in the lower right window.

NOTE eev

01:03.320 --> 01:08.920
Let me start by explaining briefly what is eev.

01:08.920 --> 00:01:10.889
First, it's something

00:01:10.890 --> 00:01:14.560
that appeared by accident in the mid 90s.

01:14.560 --> 01:20.640
I explained this story in my presentation at EmacsConf 2019.

01:20.640 --> 01:25.420
It's an Emacs package that is part of ELPA.

01:25.420 --> 01:28.720
It has at least 10 users.

01:28.720 --> 01:32.480
Those are the ones that I know by name.

01:32.480 --> 01:36.960
eev means "Emacs, execute verbosely".

01:36.960 --> 00:01:38.847
eev is something that treats eval-last-sexp

00:01:38.848 --> 00:01:43.400
as the central feature of Emacs.

01:43.400 --> 01:47.280
eev blurs the distinction between programmers and users,

01:47.280 --> 01:49.960
and it replaces the slogan,

01:49.960 --> 01:52.360
"users should not be forced to see Lisp"

01:52.360 --> 01:55.280
(that is something that Richard Stallman told me once),

01:55.280 --> 01:59.240
by "users should see Lisp instead of buttons,"

01:59.240 --> 02:03.400
and "new users should see Lisp in the first five minutes."

02:03.400 --> 02:08.160
I'm going to show some examples of that soon.

02:08.160 --> 00:02:11.565
eev uses code in the comments a lot

00:02:11.566 --> 00:02:14.120
and also tests in comments.

02:14.120 --> 00:02:16.781
I changed my way of presenting it,

00:02:16.782 --> 00:02:19.190
and it became very REPL-centric

00:02:19.191 --> 00:02:22.200
in the last few years, in the sense that

02:22.200 --> 00:02:26.460
I start by explaining its main features

00:02:26.461 --> 00:02:29.520
by its support for REPLs.

02:29.520 --> 00:02:32.078
eev supposes that we want to keep

00:02:32.079 --> 00:02:34.560
executable notes of everything.

02:34.560 --> 02:37.800
I'm also going to show examples of this in a second.

02:37.800 --> 02:42.120
eev has lots of videos for people who hate videos,

02:42.120 --> 00:02:44.270
and it tries to do everything

00:02:44.271 --> 00:02:47.640
with very little magic and without black boxes.

02:47.640 --> 02:51.360
I'm going to explain many of these things very soon.

NOTE Another figure

02:51.360 --> 02:58.200
This is a figure that I'm going to show in detail soon

02:58.200 --> 03:01.640
that is about something important about Lua.

03:01.640 --> 03:03.400
Oops, the font is very bad now,

03:03.400 --> 03:05.200
so let me change the font.

03:05.200 --> 03:07.760
The figure is this one.

03:09.880 --> 00:03:12.935
What most people do when they visit a file

00:03:12.936 --> 00:03:15.280
with something interesting in it,

03:15.280 --> 03:18.360
is that they just go there and they set a bookmark there,

03:18.360 --> 03:22.640
or they put the position in the register.

03:22.640 --> 00:03:29.781
But I prefer to keep links

00:03:29.782 --> 00:03:30.221
to everything that is interesting

00:03:30.222 --> 00:03:30.480
as an Elisp hyperlink.

03:30.480 --> 00:03:34.236
For example, this is an Elisp hyperlink to the file

00:03:34.237 --> 00:03:36.520
that goes to this anchor here,

03:36.520 --> 03:39.280
and to this string after this anchor.

03:39.280 --> 00:03:43.388
This is a variant that opens that file

00:03:43.389 --> 00:03:46.760
in the window at the right here,

03:46.760 --> 03:53.640
and this is a sexp that changes the font.

03:53.640 --> 03:57.880
I have a command with a very short name that does that,

03:57.880 --> 04:03.680
but I prefer to keep that as a one-liner.

04:03.680 --> 00:04:04.281
About the videos, we can see

00:04:04.282 --> 00:04:11.440
the list of first-class videos of eev by executing this,

04:11.440 --> 00:04:14.190
`M-x find-1stclassvideos`,

00:04:14.191 --> 00:04:18.480
or by running this alias here, `M-x 1c`,

04:18.480 --> 04:21.160
and then what we see is this.

04:21.160 --> 04:25.560
The first sexp here regenerates this buffer,

04:25.560 --> 04:29.120
so we can make a mess here and then run this,

04:29.120 --> 04:34.120
and the original buffer is regenerated again in a clean way.

04:34.120 --> 00:04:36.951
Each of these things here

00:04:36.952 --> 00:04:41.960
opens a buffer with information about a video.

04:41.960 --> 04:45.320
Let me take a specific example here.

04:45.320 --> 00:04:50.278
This video here is about one of the ancestors of this talk

00:04:50.279 --> 00:04:52.800
that is a library that I wrote

04:52.800 --> 00:04:56.906
for creating diagrams in LaTeX

00:04:56.907 --> 00:05:03.000
using a package called [pict2e-lua] using REPLs.

05:03.000 --> 00:05:10.028
Anyway, the thing is that if we run a sexp like this one

00:05:10.029 --> 00:05:16.314
and we don't have a local copy of the video,

00:05:16.315 --> 00:05:19.581
eev will try to load the local copy

00:05:19.582 --> 00:05:20.560
and instead of doing that by asking something

05:20.560 --> 00:05:25.381
like "do you want me to download the local copy"

00:05:25.382 --> 00:05:29.240
blah, blah, blah, it simply opens a buffer like this.

05:29.240 --> 00:05:35.514
And if we don't have a local copy yet,

00:05:35.515 --> 00:05:37.220
it will open a buffer like this one,

00:05:37.221 --> 00:05:40.030
in which these things here in comments

00:05:40.031 --> 00:05:42.678
are links to the documentation.

00:05:42.679 --> 00:05:45.780
This thing here explains the idea

00:05:45.781 --> 00:05:50.278
of local copies of files from the internet.

00:05:50.279 --> 00:05:54.120
There are more details here and here,

00:05:54.121 --> 00:05:59.580
and this is a script that we can execute line by line.

05:59.580 --> 00:06:04.276
So instead of this script being hidden behind the button

00:06:04.277 --> 00:06:07.160
that we just press after a question

06:07.160 --> 00:06:10.708
like "do you want me to do something," blah, blah, blah,

00:06:10.709 --> 00:06:13.228
"yes or no?", the script is visible here

00:06:13.229 --> 00:06:15.380
and we can execute it step by step.

06:15.380 --> 00:06:22.457
It creates a terminal with a shell here in the right window

00:06:22.458 --> 00:06:26.988
and when we type `F8` in these lines here,

00:06:26.989 --> 00:06:30.360
the lines are sent to this line.

06:30.360 --> 00:06:33.842
So this is going to download a copy of the video,

00:06:33.843 --> 00:06:38.134
the wget says that I already have a copy of the video

00:06:38.135 --> 00:06:40.360
and its subtitles and so on.

06:40.360 --> 00:06:45.739
And after getting a copy of the video

00:06:45.740 --> 00:06:52.960
we can run this exp here and it displays the video.

06:52.960 --> 00:06:55.486
I said that eev has lots of videos

00:06:55.487 --> 00:06:57.435
for people who hate videos.

00:06:57.436 --> 00:07:00.046
The idea is that very few people

00:07:00.047 --> 00:07:03.081
are going to watch the videos in real time.

00:07:03.082 --> 00:07:07.642
Most of the people that I know, or most of the people

00:07:07.643 --> 00:07:10.790
that are interested in eev in some way,

00:07:10.791 --> 00:07:14.320
they are going to watch just small sections of the video.

00:07:14.321 --> 00:07:17.574
Most of the time, they are just going to

00:07:17.575 --> 00:07:19.940
read the subtitles of the video.

07:19.940 --> 00:07:22.531
For each one of the videos

00:07:22.532 --> 00:07:25.295
we have a page about the video...

00:07:25.296 --> 00:07:29.673
Let me see if I have internet here...

00:07:29.674 --> 00:07:34.901
This is a page. Usually these pages

00:07:34.902 --> 00:07:38.427
have a link to another page,

00:07:38.428 --> 00:07:43.890
the page that has all the subtitles of the video,

00:07:43.891 --> 00:07:48.587
wherever. In this one, it's not so visible.

00:07:48.588 --> 00:07:52.001
Anyway, there are several ways of accessing

00:07:52.002 --> 00:07:53.963
the subtitles of the video.

00:07:53.964 --> 00:07:58.632
One of the ways is by running this sexp here

00:07:58.633 --> 00:08:01.064
that opens a file in Lua.

00:08:01.065 --> 00:08:05.440
That is what I use to generate the subtitles.

08:05.440 --> 00:08:10.368
By the way, each one of these things here

00:08:10.369 --> 00:08:14.680
is hyperlinked to a position of the video.

08:14.680 --> 00:08:21.280
If I type this, it goes to that position.

00:08:21.281 --> 00:08:24.580
Anyway, let me go back.

08:24.580 --> 00:08:28.557
So the tutorials of eev, the intros of eev

00:08:28.558 --> 00:08:32.350
that start with `find-` and end with `-intro`,

00:08:32.351 --> 00:08:35.332
they have lots of blocks that say

00:08:35.333 --> 00:08:37.612
video links like this one.

00:08:37.613 --> 00:08:41.972
These blocks have links to positions in videos.

00:08:41.973 --> 00:08:46.332
If we don't have a local copy of the video yet,

00:08:46.333 --> 00:08:49.015
the thing shows us a script

00:08:49.016 --> 00:08:52.560
that lets us download the local copy.

NOTE eev-wconfig, magic, and black boxes

08:52.560 --> 00:08:57.440
I said that I was going to explain

00:08:57.441 --> 00:09:01.855
what I mean by magic and black boxes,

00:09:01.856 --> 00:09:03.799
this is something that I've been trying

00:09:03.800 --> 00:09:05.111
to explain for a long time

00:09:05.112 --> 00:09:08.761
and I think that I got a very good explanation about that

00:09:08.762 --> 00:09:12.348
in a video that I made about something called eev-wconfig

00:09:12.349 --> 00:09:17.943
that is a tool for configuring eev on Windows without magic,

00:09:17.944 --> 00:09:20.878
without buttons that do things,

00:09:20.879 --> 00:09:24.380
without explaining what they are doing.

09:24.380 --> 00:09:28.078
This is a part of the subtitles of the video.

00:09:28.079 --> 00:09:29.445
Let me read that.

00:09:29.446 --> 00:09:32.531
eev-wconfig is an attempt to solve the problem

00:09:32.532 --> 00:09:35.349
of how to install these things on Windows

00:09:35.350 --> 00:09:40.040
both without magic and with very little magic.

09:40.040 --> 00:09:44.941
Remember this slogan: "Any sufficiently advanced technology

00:09:44.942 --> 00:09:47.600
is indistinguishable from magic."

09:47.600 --> 00:09:51.433
Here in this video I'm going to use the term magic

00:09:51.434 --> 00:09:55.262
as a shorthand for sufficiently advanced technology

00:09:55.263 --> 00:09:58.944
that is something that is complex and non obvious

00:09:58.945 --> 00:10:01.920
and that is indistinguishable from magic

00:10:01.921 --> 00:10:05.840
in the sense of being almost impossible to understand.

10:05.840 --> 00:10:09.272
And I'm also going to use the term black box

00:10:09.273 --> 00:10:11.408
as a near synonym for magic

00:10:11.409 --> 00:10:15.217
and sometimes the term black box is more convenient

00:10:15.218 --> 00:10:17.488
even though it's a bit longer,

00:10:17.489 --> 00:10:21.333
it has more letters because when I use the term black box

00:10:21.334 --> 00:10:23.521
it invites us to use expressions

00:10:23.522 --> 00:10:27.872
like opening the black box

00:10:27.873 --> 00:10:34.800
and I'm going to use that expression a lot.

10:34.800 --> 00:10:40.899
Now let me try to explain what is. Sorry,

00:10:40.900 --> 00:10:44.240
let me change the font.

NOTE Lua

10:44.240 --> 10:45.240
What is Lua?

10:45.240 --> 00:10:48.645
Lua is a minimalistic language

00:10:48.646 --> 00:10:53.040
in the sense of battery is not included.

10:53.040 --> 00:10:57.987
It uses associative tables for most of its data structures.

00:10:57.988 --> 00:11:00.874
It's so minimalistic

00:11:00.875 --> 00:11:05.147
that the default print function

00:11:05.148 --> 00:11:11.960
when we create an associative table and we ask it to print--

11:11.960 --> 00:11:14.914
we ask print to print an associative table--

00:11:14.915 --> 00:11:17.600
it just prints the address of the table.

11:17.600 --> 00:11:20.902
Here are some examples. Here is a table

00:11:20.903 --> 00:11:23.296
and when we ask to print it,

00:11:23.297 --> 00:11:28.200
it just says that it's the table at this address here.

11:28.200 --> 00:11:30.893
One of the things that most people do

00:11:30.894 --> 00:11:33.586
when they start using Lua is that either

00:11:33.587 --> 00:11:36.313
they download a package with a print to printing function

00:11:36.314 --> 00:11:39.520
or they write their own print to printing functions.

11:39.520 --> 00:11:41.975
My own print to printing function

00:11:41.976 --> 00:11:44.450
is called PP with upper case letters,

00:11:44.451 --> 00:11:46.757
and it works like this.

00:11:46.758 --> 00:11:52.072
It prints associative tables in a way like this.

00:11:52.073 --> 00:11:54.125
It says that for the key 1,

00:11:54.126 --> 00:11:56.602
the value associated to it is 2;

00:11:56.603 --> 00:11:59.078
for the key 2, the value is 3;

00:11:59.079 --> 00:12:09.240
and for the key 3, the value is 5.

12:09.240 --> 00:12:12.379
When I started using Lua, one of my favorite languages

00:12:12.380 --> 00:12:16.623
was also a language that used associative tables a lot.

00:12:16.624 --> 00:12:20.339
It was called Icon. I had to write my own

00:12:20.340 --> 00:12:23.394
print to printing functions for Icon,

00:12:23.395 --> 00:12:29.425
so I just had to port my print to printing functions to Lua.

00:12:29.426 --> 00:12:34.350
My first version looked something like this.

00:12:34.351 --> 00:12:40.080
it just had some global functions. Lots of them actually.

12:40.080 --> 00:12:41.933
After a while, I rewrote it.

00:12:41.934 --> 00:12:44.503
I rewrote it again and again and again.

00:12:44.504 --> 00:12:48.397
This is one of the versions

00:12:48.398 --> 00:12:53.751
that is not even the default at this point.

00:12:53.752 --> 00:12:57.702
`tos` is for to string. This is a demo.

00:12:57.703 --> 00:13:02.905
It's very modular, so it's easy to replace parts of it

00:13:02.906 --> 00:13:06.567
or to toggle flags. This is an example.

00:13:06.568 --> 00:13:11.797
If I try to print the table of methods for a certain class--

00:13:11.798 --> 00:13:16.245
I only need a smaller font--it prints the table like this

00:13:16.246 --> 00:13:18.510
with the names of the methods

00:13:18.511 --> 00:13:22.360
and then links to the source code of the functions.

13:22.360 --> 00:13:26.671
These links only make sense in Emacs and in eev.

00:13:26.672 --> 00:13:30.296
When we run a link like this one,

00:13:30.297 --> 00:13:34.013
it shows the source code and the window at the right.

00:13:34.014 --> 00:13:38.647
For some functions, the source code is three lines.

00:13:38.648 --> 00:13:42.320
For other ones, it's one line and whatever.

13:42.320 --> 00:13:44.149
Anyway, let me go back.

00:13:44.150 --> 00:13:47.490
Lua can be used in many different styles.

00:13:47.491 --> 00:13:52.240
Most people hate other people's styles.

13:52.240 --> 00:13:55.252
When I started using it in the year 2000,

00:13:55.253 --> 00:13:59.566
I learned most of the basic language in a single day.

00:13:59.567 --> 00:14:04.042
It was very similar to things that I was already using.

00:14:04.043 --> 00:14:08.388
Then I rewrote the mini language

00:14:08.389 --> 00:14:15.200
that I was using to generate the HTML for my pages in Lua.

14:15.200 --> 00:14:17.591
Actually, I had to rewrite it many times

00:14:17.592 --> 00:14:21.367
but the first version I certainly did

00:14:21.368 --> 00:14:25.740
in my first weeks or first months using Lua.

14:25.740 --> 00:14:29.653
In the beginning, I was just using it for writing programs

00:14:29.654 --> 00:14:32.876
that either didn't take any input at all

00:14:32.877 --> 00:14:36.727
because the input was already in the source file,

00:14:36.728 --> 00:14:39.400
or that worked as a Unix function:

14:39.400 --> 00:14:42.451
the Unix programs that would read files,

00:14:42.452 --> 00:14:49.560
process these files in some way, and output something.

14:49.560 --> 00:14:52.200
I mentioned the basic language here,

00:14:52.201 --> 00:14:55.625
I only learned how to use closures, meta tables,

00:14:55.626 --> 00:15:00.600
and coroutines many years later.

15:00.600 --> 00:15:02.588
In the beginning when I started using Lua

00:15:02.589 --> 00:15:04.199
it didn't have a package manager.

00:15:04.200 --> 00:15:08.299
It appeared later. It is called luarocks.

00:15:08.300 --> 00:15:13.180
It has had this package manager for several years.

15:13.180 --> 00:15:17.242
Most of the rocks for luarocks are poorly documented

00:15:17.243 --> 00:15:18.897
and hacker-unfriendly.

00:15:18.898 --> 00:15:22.079
You can't rely just on the documentation.

00:15:22.080 --> 00:15:25.188
You can't rely just on the source code,

00:15:25.189 --> 00:15:28.706
because--I mean, if you're a genius, of course you can,

00:15:28.707 --> 00:15:31.544
but for people who are either

00:15:31.545 --> 00:15:35.895
lazy or dumb or whatever like me, or unfocused,

00:15:35.896 --> 00:15:39.490
the source code is hard to understand

00:15:39.491 --> 00:15:41.760
and hard to tinker with.

15:41.760 --> 00:15:43.760
Some rocks are excellent.

00:15:43.761 --> 00:15:46.452
The best rocks are well documented,

00:15:46.453 --> 00:15:48.760
but they are hacker-unfriendly

15:48.760 --> 15:53.640
in the sense that I hope that I'll be able to explain soon.

15:53.640 --> 00:15:59.466
The best rocks use local variables and meta tables a lot,

00:15:59.467 --> 00:16:02.905
so if you are a beginner learning Lua

00:16:02.906 --> 00:16:08.245
you're not going to understand what their source codes do.

00:16:08.246 --> 00:16:10.960
They use lots of dirty tricks.

NOTE Object orientation in Lua

16:10.960 --> 16:14.360
Let me talk a bit about object orientation in Lua.

16:14.360 --> 00:16:18.275
It can be done in many ways. The main book about Lua

00:16:18.276 --> 00:16:20.140
called Programming in Lua

00:16:20.141 --> 00:16:22.865
by one of the authors of the language

00:16:22.866 --> 00:16:24.729
called Roberto Ierusalimschy

00:16:24.730 --> 00:16:28.680
presents several ways of doing object orientation in Lua.

16:28.680 --> 00:16:31.386
I hated all of these ways,

00:16:31.387 --> 00:16:36.056
and also the ways that I tried from the rocks.

00:16:36.057 --> 00:16:37.933
Then I wrote my own way

00:16:37.934 --> 00:16:40.347
of doing object orientation in Lua.

00:16:40.348 --> 00:16:45.653
It's very minimalistic. It's in this file here eoo.lua.

00:16:45.654 --> 00:16:49.898
The main code is just these five lines here.

00:16:49.899 --> 00:16:56.400
Here's an example of how it works.

16:56.400 --> 00:17:03.001
We define a class vector with some meta methods.

00:17:03.002 --> 00:17:06.294
This meta method here will tell Lua

00:17:06.295 --> 00:17:10.867
what to do when the user asks to add two vectors.

00:17:10.868 --> 00:17:13.986
This one here tells Lua what to do

00:17:13.987 --> 00:17:18.888
when the user asks Lua to convert a vector to a string

00:17:18.889 --> 00:17:22.007
and whatever. This one is something

00:17:22.008 --> 00:17:25.580
that I'm going to explain in a second.

17:25.580 --> 00:17:25.580
Here we create a vector

00:17:25.580 --> 00:17:25.580
with these coordinates 3 and 4.

00:17:25.581 --> 00:17:25.580
Here we create another vector.

17:31.600 --> 00:17:33.030
If we print here,

00:17:33.031 --> 00:17:37.153
then Lua uses the function here in the `__tostring`.

00:17:37.154 --> 00:17:38.907
If we add two vectors

00:17:38.908 --> 00:17:42.653
it uses the function here in the `__add` meta method.

00:17:42.654 --> 00:17:45.044
If we run the method `norm`,

00:17:45.045 --> 00:17:51.680
it is defined here in the table index.

17:51.680 --> 00:17:57.675
Anyway, even with this thing being so small,

00:17:57.676 --> 00:18:04.617
I used to forget how it works all the time.

00:18:04.618 --> 00:18:08.200
Actually, I always forget how things work,

00:18:08.201 --> 00:18:11.347
and I have to remember them somehow.

00:18:11.348 --> 00:18:14.920
I have to have tricks for remembering,

00:18:14.921 --> 00:18:19.769
and tricks for summarizing things and diagrams, and so on.

00:18:19.770 --> 00:18:23.309
Every time that I forgot how this thing worked,

00:18:23.310 --> 00:18:25.460
I went back to the source code

00:18:25.461 --> 00:18:28.148
and then I looked at the diagrams,

00:18:28.149 --> 00:18:30.994
or, of course, the first time,

00:18:30.995 --> 00:18:33.800
I had to draw the diagrams. I run the examples.

00:18:33.801 --> 00:18:35.539
Of course, in the beginning

00:18:35.540 --> 00:18:37.503
I thought that the code was clear.

00:18:37.504 --> 00:18:39.737
My examples were very brief.

00:18:39.738 --> 00:18:43.565
So I had to rewrite the examples many times

00:18:43.566 --> 00:18:50.880
until they became let's say perfect.

18:50.880 --> 00:18:54.090
I was saying that Lua can be used in many ways

00:18:54.091 --> 00:18:58.403
and in my way of using Lua, in my favorite way,

00:18:58.404 --> 00:19:03.075
everything can be inspected and modified from REPLs

00:19:03.076 --> 00:19:06.817
like we can do in Emacs and in SmallTalk, sort of.

00:19:06.818 --> 00:19:09.436
In my favorite way of using Lua

00:19:09.437 --> 00:19:11.531
there's no security at all.

00:19:11.532 --> 00:19:17.460
Everything can be changed at all times.

19:17.460 --> 00:19:19.822
Of course, most people hate that.

NOTE My init file

00:19:19.823 --> 00:19:22.185
My init file has lots of classes,

00:19:22.186 --> 00:19:24.701
and by the way, instead of

00:19:24.702 --> 00:19:28.636
keeping many small files with many things,

00:19:28.637 --> 00:19:33.000
I put lots of stuff in just one big init file.

19:33.000 --> 00:19:35.520
My init file has lots of classes,

00:19:35.521 --> 00:19:39.187
and lots of global functions and lots of cruft,

00:19:39.188 --> 00:19:43.700
so people hate that, of course.

19:43.700 --> 00:19:46.340
This is an example.

00:19:46.341 --> 00:19:52.415
This is the index at the top of my init file.

00:19:52.416 --> 00:19:57.541
The classes start here. Then we have some functions

00:19:57.542 --> 00:20:02.480
and then we have functions that load certain packages,

20:02.480 --> 20:06.580
and then we have cruft, whatever.

20:06.580 --> 00:20:08.923
Most people think that my style of using Lua

00:20:08.924 --> 00:20:10.122
is dirty and dangerous,

00:20:10.123 --> 00:20:14.287
and they wouldn't touch my Lua code with a 10 feet pole,

00:20:14.288 --> 00:20:15.968
but most of the things

00:20:15.969 --> 00:20:19.256
that I'm going to present here in this presentation

00:20:19.257 --> 00:20:23.050
are ideas that should be easy to port to other environments

00:20:23.051 --> 00:20:27.738
and other languages, especially the diagrams,

00:20:27.739 --> 00:20:31.000
so the code is not so important.

NOTE LaTeX and LuaLaTeX

20:31.000 --> 00:20:34.640
Now let me talk a bit about LaTeX, LuaLaTeX,

00:20:34.641 --> 00:20:39.249
that is LaTeX with the Lua interpreter embedded inside

00:20:39.250 --> 00:20:43.182
and two ways of generating pictures in LaTeX:

00:20:43.183 --> 00:20:45.115
TikZ that is very famous,

00:20:45.116 --> 00:20:48.311
and Pict2e that is not very famous

00:20:48.312 --> 00:20:50.644
and that is very low level,

00:20:50.645 --> 00:20:53.840
and I think that not many people use.

20:53.840 --> 00:20:57.048
I said before that when I learned Lua,

00:20:57.049 --> 00:21:00.354
I realized that it was very good

00:21:00.355 --> 00:21:03.360
for writing literal languages.

21:03.360 --> 00:21:05.986
I was doing my PhD at the time

00:21:05.987 --> 00:21:09.969
and typesetting the diagrams for my PhD thesis

00:21:09.970 --> 00:21:11.240
was very boring.

21:11.240 --> 00:21:13.864
One of the things that I did was that

00:21:13.865 --> 00:21:15.720
I created a literal language

00:21:15.721 --> 00:21:19.340
for typesetting the diagrams for me.

21:19.340 --> 00:21:22.821
It was called Dednat because initially

00:21:22.822 --> 00:21:26.800
it only generated diagrams for natural deduction

21:26.800 --> 21:30.440
and then it had several versions.

21:30.440 --> 00:21:34.379
These are the slides from my presentation about Dednat6

00:21:34.380 --> 00:21:38.990
and Dednat6 is an extensible semi preprocessor

00:21:38.991 --> 00:21:43.691
for LuaLaTeX that understands diagrams in ASCII art

00:21:43.692 --> 00:21:49.034
in the sense that when I have a TeX file that has this,

00:21:49.035 --> 00:21:52.143
and when Dednat6 is loaded,

00:21:52.144 --> 00:21:55.708
when I get the right comments,

00:21:55.709 --> 00:22:00.342
Dednat6 interprets this block here

00:22:00.343 --> 00:22:05.095
as something that defines this diagram.

00:22:05.096 --> 00:22:07.828
It interprets this diagram here,

00:22:07.829 --> 00:22:10.643
this diagram in the comments here,

00:22:10.644 --> 00:22:16.432
as something that defines a diagram called `foo`,

00:22:16.433 --> 00:22:23.064
a deduction called `foo`, and it generates this code here

00:22:23.065 --> 00:22:28.867
so that we can just invoke the definition of the deduction

00:22:28.868 --> 00:22:31.720
by typing `\ded{foo}`.

22:31.720 --> 00:22:36.148
Dednat6 also supports another language

00:22:36.149 --> 00:22:40.579
for typesetting bidimensional diagrams with arrows and stuff

00:22:40.580 --> 00:22:43.122
for category theory and blah blah.

00:22:43.123 --> 00:22:46.100
The specifications of these diagrams

22:46.100 --> 00:22:53.865
look like this. Here is a very good example.

00:22:53.866 --> 00:23:00.940
This is a huge diagram. Sorry, one second.

23:00.940 --> 00:23:04.757
So the source code that generates this diagram here

00:23:04.758 --> 00:23:08.628
is just this thing at the left. It's very visual.

00:23:08.629 --> 00:23:11.842
We can typeset the diagram in ASCII art here.

00:23:11.843 --> 00:23:14.434
Then in this part here,

00:23:14.435 --> 00:23:18.273
we tell how the nodes are to be joined,

00:23:18.274 --> 00:23:22.880
which arrows have to have annotations, and so on.

23:22.880 --> 00:23:30.950
This language is extensible in the sense that

00:23:30.951 --> 00:23:37.280
here, comments that start

23:37.280 --> 00:23:42.690
with `%:` are interpreted as definitions

00:23:42.691 --> 00:23:44.819
for tree diagrams,

00:23:44.820 --> 00:23:49.394
lines that start with `%D`

00:23:49.395 --> 00:23:53.761
define 2D diagrams with arrows and stuff,

00:23:53.762 --> 00:23:57.683
and lines that start with command `%L`

00:23:57.684 --> 00:24:01.197
contain blocks of Lua code that we can use

00:24:01.198 --> 00:24:05.527
to extend the interpreter on the flag.

00:24:05.528 --> 00:24:09.746
Anyway, here are some recent examples

00:24:09.747 --> 00:24:16.146
of diagrams that I used Dednat6 to typeset.

00:24:16.147 --> 00:24:24.494
This diagram here was generated by this specification here,

00:24:24.495 --> 00:24:30.619
and this diagram here with the curved arrows

00:24:30.620 --> 00:24:36.200
was generated by this specification here.

24:36.200 --> 00:24:39.174
Dednat6 was very easy to extend.

00:24:39.175 --> 00:24:43.043
At some point I started to use it to generate diagrams

00:24:43.044 --> 00:24:46.067
using Pict2e, mainly for the classes

00:24:46.068 --> 00:24:49.673
that I give at the university,

00:24:49.674 --> 00:24:53.861
I teach mathematics in a place.

00:24:53.862 --> 00:24:58.979
Let me show an animation. Here is a diagram

00:24:58.980 --> 00:25:02.325
that I generated with Dednat6.

00:25:02.326 --> 00:25:05.385
It is a flipbook animation,

00:25:05.386 --> 00:25:08.253
we type page up and page down,

00:25:08.254 --> 00:25:10.783
and we go to the next page of the book

00:25:10.784 --> 00:25:13.248
and to the previous page of the book.

00:25:13.249 --> 00:25:16.924
Here is the source code that generates that.

00:25:16.925 --> 00:25:19.700
This source code is not very visual,

00:25:19.701 --> 00:25:21.800
so it's quite clumsy to edit

25:21.800 --> 25:28.280
a diagram directly in the TeX file like that.

NOTE Manim

25:28.280 --> 00:25:33.437
These diagrams were inspired by something called Manim.

00:25:33.438 --> 00:25:36.713
I've forgotten the name of the guy, but it's a guy

00:25:36.714 --> 00:25:39.346
that makes many videos about mathematics,

00:25:39.347 --> 00:25:43.373
and he created this library called Manim

00:25:43.374 --> 00:25:46.417
for generating his animations.

00:25:46.418 --> 00:25:52.083
Other people adapted his library to make it more accessible.

00:25:52.084 --> 00:25:55.799
I tried to learn it, but each animation,

00:25:55.800 --> 00:25:59.191
even an animation with very few frames,

00:25:59.192 --> 00:26:03.770
each animation took ages to render, so it wasn't fun.

00:26:03.771 --> 00:26:08.345
Animations and PDFs can be rendered in seconds,

00:26:08.346 --> 00:26:11.160
so these things were fun for me

00:26:11.161 --> 00:26:17.440
because my laptop is very slow and Manim was not fun.

26:17.440 --> 00:26:24.263
Anyway, writing code like this inside a TeX file

00:26:24.264 --> 00:26:30.879
was not very fun because it was hard to debug.

NOTE Generating diagrams from REPLs

00:26:30.880 --> 00:26:36.510
In 2022, I started to play with ways

00:26:36.511 --> 00:26:42.000
of generating these diagrams from REPLs

26:42.000 --> 00:26:46.309
and I found a way for Pict2e and a way for Tikz.

00:26:46.310 --> 00:26:49.520
Each one of these ways became a video.

26:49.520 --> 00:26:53.220
If you go to the list of first class videos of eev,

00:26:53.221 --> 00:26:55.088
you're going to see

00:26:55.089 --> 00:26:59.010
that there's a video about Pict2e here

00:26:59.011 --> 00:27:01.158
and a video about Tikz.

00:27:01.159 --> 00:27:04.187
Here you have some information

00:27:04.188 --> 00:27:07.508
like length and explanation, etc.

00:27:07.509 --> 00:27:11.513
Here are the pages for these videos.

00:27:11.514 --> 00:27:15.991
My page about the video about Pict2e is like this.

00:27:15.992 --> 00:27:20.800
It has some diagrams. This one is much nicer.

27:20.800 --> 00:27:26.605
A lot of people watched that video. I mean,

00:27:26.606 --> 00:27:31.080
I thought that 250 people watched it.

27:31.080 --> 00:27:35.344
For me, that's a million of people.

00:27:35.345 --> 00:27:40.402
This video is about how to extract diagrams

00:27:40.403 --> 00:27:46.512
from the Tikz manual, and how to run those examples in REPL

00:27:46.513 --> 00:27:52.936
and modify them bit by bit. This is a screenshot.

00:27:52.937 --> 00:27:57.432
Let me go back. At that point,

00:27:57.433 --> 00:28:01.800
these things were just prototypes,

28:01.800 --> 00:28:04.803
the code was not very nice,

00:28:04.804 --> 00:28:07.592
and in this year I wrote...

00:28:07.593 --> 00:28:12.123
I was able to unify those two ways of generating PDFs,

00:28:12.124 --> 00:28:15.665
the one for Tikz and the one for Pict2e.

00:28:15.666 --> 00:28:19.196
I unified them with many other things

00:28:19.197 --> 00:28:21.298
that generated diagrams.

00:28:21.299 --> 00:28:28.430
The basis of these things is something called Show2.lua.

00:28:28.431 --> 00:28:33.935
I'm not going to just show its details now,

00:28:33.936 --> 00:28:38.775
but it's an extension that generates Tikz code.

00:28:38.776 --> 00:28:43.305
It's just this. We can specify a diagram

00:28:43.306 --> 00:28:50.538
with just a block like this, and then if we run `show00`,

00:28:50.539 --> 00:28:53.694
it returns a string

00:28:53.695 --> 00:28:59.480
that is just the inner body of the TeX file.

28:59.480 --> 00:29:02.306
If we run this, we see the whole TeX file.

00:29:02.307 --> 00:29:05.200
If we run this we save the TeX file and

29:05.200 --> 00:29:09.196
we compile the TeX file to generate a PDF.

00:29:09.197 --> 00:29:12.557
If we run this, we show the PDF

00:29:12.558 --> 00:29:15.078
in the lower right window.

00:29:15.079 --> 00:29:20.026
That's the same thing for all my recent programs

00:29:20.027 --> 00:29:23.810
that generate PDFs. They are all integrated.

00:29:23.811 --> 00:29:28.267
Here is the basis for all my modules

00:29:28.268 --> 00:29:31.423
that generate diagrams with Pict2e.

00:29:31.424 --> 00:29:34.337
Its demos are not very interesting.

00:29:34.338 --> 00:29:38.556
Let me show some demos of extensions

00:29:38.557 --> 00:29:43.723
that do interesting things. This is a diagram

00:29:43.724 --> 00:29:47.901
that I created by editing it in a REPL,

00:29:47.902 --> 00:29:51.871
I create several Pict objects here.

00:29:51.872 --> 00:29:57.818
If I execute this, it compiles the object and

00:29:57.819 --> 00:30:04.523
generates a PDF. If I tap this, here is the PDF.

00:30:04.524 --> 00:30:10.919
If I just ask Lua to display what is pux here,

00:30:10.920 --> 00:30:17.435
it shows the source code in Pict2e of the diagram.

00:30:17.436 --> 00:30:20.932
The nice thing is that it is indented,

00:30:20.933 --> 00:30:24.348
so it's easy to debug the Pict2e code.

00:30:24.349 --> 00:30:28.504
If anyone is interested, the module that does the tricks

00:30:28.505 --> 00:30:31.947
for indentation is very easy to understand.

00:30:31.948 --> 00:30:34.920
it has lots of tests and test blocks,

00:30:34.921 --> 00:30:47.438
and I think that its data structures are easy to understand.

00:30:47.439 --> 00:30:57.904
Anyway, here is another example, the show is here.

00:30:57.905 --> 00:31:03.240
It generates a 3D diagram.

NOTE Parsers

31:03.240 --> 00:31:05.897
Now let me talk about parsers

00:31:05.898 --> 00:31:08.997
and REPLs in a very strange place.

00:31:08.998 --> 00:31:15.148
I mean, using REPLs to build parsers step by step

00:31:15.149 --> 00:31:20.438
and replacing parts by more complex parts.

00:31:20.439 --> 00:31:24.492
I said that Lua is very minimalistic,

00:31:24.493 --> 00:31:26.964
and everybody knows that

00:31:26.965 --> 00:31:30.720
implementations of regular expressions

31:30.720 --> 00:31:33.973
are big and complex. So instead of coming

00:31:33.974 --> 00:31:36.375
with full regular expressions,

00:31:36.376 --> 00:31:41.320
Lua comes with something called patterns

00:31:41.321 --> 00:31:46.627
and a library function called string.match.

00:31:46.628 --> 00:31:50.820
Here is a copy of the part of the manual

00:31:50.821 --> 00:31:55.728
that explains a part of the syntax of patterns.

00:31:55.729 --> 00:32:00.090
Here's how string.match is described in the manual.

00:32:00.091 --> 00:32:04.536
It's just this: looks for the first match of pattern

00:32:04.537 --> 00:32:06.545
in the string s, blah blah blah.

00:32:06.546 --> 00:32:10.076
Then we have to go to the other section of the manual

00:32:10.077 --> 00:32:18.197
that explains patterns. Lua patterns are so simple,

00:32:18.198 --> 00:32:23.221
so limited, that they don't even have

00:32:23.222 --> 00:32:26.658
the alternation operator.

00:32:26.659 --> 00:32:33.135
Here is how it is described in the elisp manual,

00:32:33.136 --> 00:32:39.677
`\|` specifies an alternative, blah blah blah.

00:32:39.678 --> 00:32:45.880
When we want to build more complex regular expressions,

32:45.880 --> 00:32:48.365
patterns, grammars, etc.,

00:32:48.366 --> 00:32:52.668
we have to use an external library for that.

00:32:52.669 --> 00:32:56.207
No, sorry, a library that is external,

00:32:56.208 --> 00:33:01.471
but that was written by one of the authors of Lua itself.

00:33:01.472 --> 00:33:03.943
This library is called Lpeg.

00:33:03.944 --> 00:33:06.330
Its manual says Lpeg is

00:33:06.331 --> 00:33:09.653
a new pattern matching library for Lua

00:33:09.654 --> 00:33:14.772
based on parsing expression grammars (PEGs).

00:33:14.773 --> 00:33:17.780
The manual is very terse.

00:33:17.781 --> 00:33:21.945
I found it incredibly hard to read.

00:33:21.946 --> 00:33:25.415
it doesn't have any diagrams.

00:33:25.416 --> 00:33:28.769
It has some examples, though.

00:33:28.770 --> 00:33:34.016
The Lua wiki has a big page called Lpeg Tutorial

00:33:34.017 --> 00:33:39.560
with lots of examples, but it also doesn't have diagrams.

33:39.560 --> 00:33:43.131
I found some things incredibly hard to understand.

00:33:43.132 --> 00:33:45.080
For example, this is something

33:45.080 --> 00:33:48.061
that is in the manual of Lpeg that I saw,

00:33:48.062 --> 00:33:51.480
and I thought, wow, great, this makes all sense

33:51.480 --> 00:33:53.946
and this is going to be very useful.

00:33:53.947 --> 00:33:57.280
It's a way to build grammars that can be recursive

33:57.280 --> 00:34:01.300
and they sort of can encode BNF grammars,

00:34:01.301 --> 00:34:05.128
we just have to translate the BNF a bit

00:34:05.129 --> 00:34:07.592
to get rid of some recursions

00:34:07.593 --> 00:34:10.959
and to translate them to something else.

00:34:10.960 --> 00:34:14.337
The manual also has some things that I thought,

00:34:14.338 --> 00:34:17.845
oh no, I don't have any idea of what this thing does.

00:34:17.846 --> 00:34:20.768
In fact, I saw these things for the first time

00:34:20.769 --> 00:34:27.294
more than 10 years ago, and they only started to make sense

00:34:27.295 --> 00:34:32.272
one year ago. One example is group captures.

00:34:32.273 --> 00:34:37.834
Lpeg also comes with a module called the re module.

00:34:37.835 --> 00:34:43.182
Let me pronounce it in Portuguese, the re module.

00:34:43.183 --> 00:34:45.741
Its manual says the re module

00:34:45.742 --> 00:34:49.503
provided by the file re.lua in the distribution

00:34:49.504 --> 00:34:56.757
supports some odd conventional regular expression syntax

00:34:56.758 --> 00:35:04.520
for pattern usage within lpeg. This is a quick reference.

35:04.520 --> 00:35:08.464
This thing is very brief. It has some nice examples,

00:35:08.465 --> 00:35:11.120
but it's hard to understand anyway.

35:11.120 --> 00:35:14.180
Here are some comments about

00:35:14.181 --> 00:35:18.631
my attempts to learn re.lua. This is a class.

00:35:18.632 --> 00:35:22.361
In this case, it's a very small class.

00:35:22.362 --> 00:35:26.091
This file implements a `pm` method.

00:35:26.092 --> 00:35:31.400
I'm going to show examples of other `pm` methods very soon.

00:35:31.401 --> 00:35:34.696
So this is a `pm` method for re.lua

00:35:34.697 --> 00:35:40.301
that lets us compare the syntax of lua patterns, lpeg,

00:35:40.302 --> 00:35:46.631
and re. See this example here. If we run this,

00:35:46.632 --> 00:35:51.379
it loads my version of lpeg, no sorry,

00:35:51.380 --> 00:35:56.013
my version of lpegrex. It shows that

00:35:56.014 --> 00:36:01.552
when we apply the `pm` method to this Lua pattern,

00:36:01.553 --> 00:36:05.960
this lpeg pattern, and this re pattern,

36:05.960 --> 00:36:08.238
they all give the same results.

00:36:08.239 --> 00:36:11.512
So we can use this kind of thing here to show

00:36:11.513 --> 00:36:15.588
how to translate from Lua patterns that are familiar

00:36:15.589 --> 00:36:19.280
because they are similar to regular expressions,

36:19.280 --> 00:36:25.251
only weaker, to lpeg that is super weird,

00:36:25.252 --> 00:36:30.084
and to re that is not so weird.

00:36:30.085 --> 00:36:35.425
Anyway, the comment says that in 2012, I had a project

00:36:35.426 --> 00:36:38.590
that needed a precedence parser

00:36:38.591 --> 00:36:42.277
that could parse arithmetical expressions

00:36:42.278 --> 00:36:46.314
with the right precedences. At that point

00:36:46.315 --> 00:36:49.031
I was still struggling with pure lpeg

00:36:49.032 --> 00:36:51.318
and I couldn't do much with it,

00:36:51.319 --> 00:36:54.576
so I tried to learn re.lua instead,

00:36:54.577 --> 00:36:59.291
and I wrote this old class here that allowed me to use

00:36:59.292 --> 00:37:01.755
a preprocessor on patterns for Lua.

00:37:01.756 --> 00:37:04.834
The thing is that with this preprocessor

00:37:04.835 --> 00:37:10.920
I could specify precedence grammars using this thing here

00:37:10.921 --> 00:37:14.488
that worked but was super clumsy,

00:37:14.489 --> 00:37:18.160
and I gave up after a few attempts.

37:18.160 --> 00:37:25.632
And in 2022 I heard about something called lpegrex

00:37:25.633 --> 00:37:30.760
that was a kind of extension of re,

37:30.760 --> 00:37:33.045
and it was much more powerful than re.lua,

00:37:33.046 --> 00:37:35.784
but after a while I realized that

00:37:35.785 --> 00:37:38.765
it had the same defects as re.lua.

00:37:38.766 --> 00:37:40.698
Let me explain that,

00:37:40.699 --> 00:37:43.920
because it has all to do with the things

37:43.920 --> 00:37:52.026
about black boxes and magic that I told in the beginning.

00:37:52.027 --> 00:37:59.046
Neither re.lua or lpegrex had some features that I needed.

00:37:59.047 --> 00:38:05.374
They received a pattern that was specified as a string

00:38:05.375 --> 00:38:08.868
and it converted that into an lpeg pattern,

00:38:08.869 --> 00:38:12.520
but it didn't let us explore the lpeg patterns

38:12.520 --> 00:38:14.529
that it generated.

00:38:14.530 --> 00:38:20.662
Their code was written in a way that was REPL-unfriendly.

00:38:20.663 --> 00:38:26.281
I couldn't modify parts of the code bit by bit in a REPL

00:38:26.282 --> 00:38:28.943
and try to change the code

00:38:28.944 --> 00:38:32.035
without changing the original file, say.

00:38:32.036 --> 00:38:35.353
The code was very hard to explore, to hack,

00:38:35.354 --> 00:38:38.238
and to extend in my opinion.

00:38:38.239 --> 00:38:42.018
The documentation was not very clear,

00:38:42.019 --> 00:38:45.600
and I sent wonderful messages to the

38:45.600 --> 00:38:51.013
developer of lpegrex, and he was too busy to help me.

00:38:51.014 --> 00:38:57.210
He answered very briefly. To be honest, I felt rejected.

00:38:57.211 --> 00:39:03.200
I felt that I wasn't doing anything interesting, whatever.

NOTE ELpeg1.lua

39:03.200 --> 00:39:07.470
So in 2022, I was trying to learn lpegrex

00:39:07.471 --> 00:39:13.200
because I was thinking that it would solve my problems,

39:13.200 --> 00:39:16.666
but it didn't. It didn't have the features that I needed,

00:39:16.667 --> 00:39:18.400
and it was hard to extend and

39:18.400 --> 00:39:22.302
how to explore and hard to debug.

00:39:22.303 --> 00:39:28.960
I decided to rewrite it in a more hacker-friendly way,

39:28.960 --> 00:39:32.035
in the sense that it was modular

00:39:32.036 --> 00:39:37.160
and I could replace any part of the module from a REPL.

39:37.160 --> 00:39:42.640
My version of it was called ELpeg1.lua.

00:39:42.641 --> 00:39:48.120
I decided that in my version I wouldn't

39:48.120 --> 00:39:51.240
have the part that receives a grammar

00:39:51.241 --> 00:39:55.265
specified as a string and converts that to lpeg.

00:39:55.266 --> 00:39:58.984
I would just have the backend part

00:39:58.985 --> 00:40:02.171
that are the functions in lpeg

00:40:02.172 --> 00:40:06.320
that let us specify powerful grammars.

40:06.320 --> 00:40:15.161
Let me go back. Let me explain a bit about lpeg.

00:40:15.162 --> 00:40:20.780
Lua has coercions. The `+` expects to receive two numbers.

00:40:20.781 --> 00:40:25.930
If one of its arguments or both of them are strings,

00:40:25.931 --> 00:40:30.116
it converts the strings to numbers.

00:40:30.117 --> 00:40:37.091
So in this case here, `2 + "3"` returns the number 5.

00:40:37.092 --> 00:40:40.440
This is the concatenation operator.

00:40:40.441 --> 00:40:44.960
It expects to receive strings. In this case, it will

40:44.960 --> 00:40:47.541
convert the number 2 to the string "2"

00:40:47.542 --> 00:40:50.960
and the concatenation of these two things will be

40:50.960 --> 00:40:55.917
"23". Oops, sorry, "23" as a string.

00:40:55.918 --> 00:41:04.480
lpeg also has some coercions. I usually set these globals

41:04.480 --> 00:41:10.013
to let me write my grammars in a very compact way.

00:41:10.014 --> 00:41:13.811
So instead of lpeg.B, lpeg.C, etc.

00:41:13.812 --> 00:41:17.878
I use globals like B, C, and so on.

00:41:17.879 --> 00:41:21.720
And with these globals I can write

41:21.720 --> 00:41:36.047
things like this, `C(1) * "_"`. And lpeg knows that lpeg.C,

00:41:36.048 --> 00:41:41.147
sorry, it sort of expands these to lpeg.C,

00:41:41.148 --> 00:41:43.590
but lpeg.C expects to receive an lpeg pattern

00:41:43.591 --> 00:41:48.547
and 1 is not yet an lpeg pattern.

00:41:48.548 --> 00:41:53.347
So it is coerced into an lpeg pattern by calling lpeg.P.

00:41:53.348 --> 00:42:02.481
So this short thing here becomes equivalent to

00:42:02.482 --> 00:42:07.200
`lpeg.C(lpeg.P(1))`. And the multiplication when at least one of

42:07.200 --> 42:13.320
its arguments is an lpeg pattern, it expects to receive two lpeg patterns. And in this

42:13.320 --> 42:19.200
case the one at the right is just a string, so it is coerced to an lpeg pattern by using

42:19.200 --> 42:29.360
lpeg.P. With this idea we can sort of understand this comparison here. I mean, let me run it

42:29.360 --> 42:37.440
again. This first part is very similar to a regular expression here at the left. And

42:37.440 --> 42:51.440
when we apply this lpeg, sorry, this lua pattern to this subject here, the result is this thing

42:51.440 --> 42:56.040
here, this thing, this thing and this thing. I'm going to call each one of these results

42:56.040 --> 43:04.080
captures. So each of these things between parentheses captures a substring of the original

43:04.080 --> 43:11.520
string and these captured substrings are returned in a certain order. Here is how to express

43:11.520 --> 43:20.720
the same thing in lpeg. It's very cryptic, but it's a good way to understand some basic

43:20.720 --> 43:29.880
operators of lpeg. I mean, we can look at the menu and understand what C, S and R do

43:29.880 --> 43:41.360
and also exponentiation. And this strange thing here receives this string here, runs

43:41.360 --> 43:46.760
a function that I have defined that converts it to an object of a certain class and that

43:46.760 --> 43:55.240
class represents re patterns. So this thing is treated as a pattern for re.lua and it

43:55.240 --> 44:04.040
is matched again the string and it returns the same thing as the other one. Also this

44:04.040 --> 44:10.680
thing here also has a comparison with lpeg.rex, but these patterns are very, very trivial.

44:10.680 --> 44:17.340
They don't do anything very strange. So let's go back and see what kinds of very strange

44:17.340 --> 44:31.480
things there are. Here is the page of lpeg.rex at GitHub. Here is the documentation. It's

44:31.480 --> 44:39.120
relatively brief. It explains lpeg.rex as being an extension of re.lua. So

44:39.120 --> 44:44.880
it explains mainly the additional features. Here is a quick reference that explains only

44:44.880 --> 44:56.840
the additional features. Some of these things I was able to understand by struggling a lot,

44:56.840 --> 45:08.720
and some I wasn't able to even by spending several evenings trying to build examples.

45:08.720 --> 45:16.600
This is something very nice. lpeg.rex comes with some example parsers. Here is

45:16.600 --> 45:25.720
a parser that parses the lua grammar. I mean, this is the grammar for lua 5.4 at the end

45:25.720 --> 45:35.080
of the reference manual. It's just this. And this is a kind of BNF, and this is the BNF

45:35.080 --> 45:43.840
translated to the language of lpeg.rex. This thing uses many constructions that are

45:43.840 --> 45:51.840
in re.lua and some extra constructions that are described here. With these examples,

45:51.840 --> 46:00.280
I was able to understand some of these things here that are described here in the quick

46:00.280 --> 46:13.840
reference, but not all. So I wasn't able to use lpeg.rex by itself because some things

46:13.840 --> 46:21.560
didn't make much sense. I decided to re implement it in my own style because that

46:21.560 --> 46:29.120
would be a way to map... at the very least, map what I understood and what I didn't and

46:29.120 --> 46:38.160
learn one feature at a time, do comparisons and so on. Here I pointed to two features

46:38.160 --> 46:44.880
of lpeg. One I said, oh great, this thing can be used to define grammars, even recursive

46:44.880 --> 46:52.260
grammars and so on. And this is an "oh no" feature. One thing that didn't make any sense at all:

46:52.260 --> 47:01.760
group captures. One thing that I did to understand group captures was to represent them as diagrams.

47:01.760 --> 47:08.200
Of course in the beginning I was drawing these diagrams by hand, but then I realized that

47:08.200 --> 47:16.680
I could use the bits of lpeg that I already knew to build a grammar that would parse a

47:16.680 --> 47:24.400
little language and generate these diagrams in LaTeX. I was able to make this. In

47:24.400 --> 47:35.520
this diagram here, this thing above the arrow is a piece of Lua code that specifies an lpeg

47:35.520 --> 47:43.160
pattern. This thing here at the top is the string that is being matched and the things

47:43.160 --> 47:57.880
below the under braces are the captures that each thing captures. For example, this under

47:57.880 --> 48:03.520
brace here corresponds to this pattern here that parses a single character but doesn't

48:03.520 --> 48:10.680
return any captures. This thing here parses a single B and doesn't return any captures.

48:10.680 --> 48:17.800
This thing here parses a single character and captures it. And this thing here parses

48:17.800 --> 48:24.920
the character D and captures it. And this other thing here that transforms this pattern

48:24.920 --> 48:35.040
into another pattern returns first a capture with all the string that was parsed by this

48:35.040 --> 48:44.000
pattern here and then all the captures returned by this thing here before the column. So this

48:44.000 --> 48:51.840
was a way to build concrete examples for things that the lpeg manual was explaining in a very

48:51.840 --> 48:58.600
terse way and it worked for me. Some things that were very mysterious started to make

48:58.600 --> 49:10.200
sense and I started to have intelligent questions to ask in the mailing list. With that

49:10.200 --> 49:22.040
I was able to understand what are group captures that receive a name. Well, let me explain

49:22.040 --> 49:30.400
what this does. This thing here parses the empty string and returns this as a constant.

49:30.400 --> 49:38.520
So this is something that doesn't exist in regular expressions. It parses nothing and

49:38.520 --> 49:46.800
returns this as a capture. Then this thing here returns two constants here and parses

49:46.800 --> 49:54.440
the empty string. And this thing here, D, converts the results of this thing here into

49:54.440 --> 50:04.160
a group capture and stores it in the label D. And then here's another constant capture.

NOTE Building lists

50:04.160 --> 50:15.560
I realized that these things here were similar to how Lua specifies building lists.

50:15.560 --> 50:22.320
When we build a table and we say that the first element of the table is here, this element

50:22.320 --> 50:30.600
is put at the end of the table. After that we say D equals to say 42. We are putting

50:30.600 --> 50:39.880
the 42 in the slot whose key is D. This was happening with lpeg captures but there was

50:39.880 --> 50:49.320
something very strange. These group captures could hold more than one capture, more than

50:49.320 --> 50:57.400
one value. So there was something between lists and tables. I started to use this notation

50:57.400 --> 51:09.840
to explain in my notation what they were doing. Many things started to make sense, many mysterious

51:09.840 --> 51:20.480
sentences in the manual started to make sense but some didn't. But at least I was able to

51:20.480 --> 51:26.960
send some intelligent questions to the mailing list and the author of Lua and LPEG answered

51:26.960 --> 51:35.920
some of them. He was not very happy about my questions. He told me that those diagrams

51:35.920 --> 51:47.280
were a waste of time, the manual was perfectly clear and so on, whatever. So it was weird,

51:47.280 --> 51:56.440
but I was able to understand lots of things from his answers. So this is a copy of one

51:56.440 --> 52:02.040
of my messages, then there's another one, another one, some of the diagrams. Then he

52:02.040 --> 52:10.080
complained about these diagrams, he said that these things here that look like table constructions

52:10.080 --> 52:22.720
do not exist, whatever. Anyway, once I understood group captures, many features were very easy

52:22.720 --> 52:30.800
to understand and I started to be able to use lpeg to build some very interesting things.

52:30.800 --> 52:39.600
I was able to reproduce some of the features that I saw in lpegrex. Remember that this

52:39.600 --> 52:53.400
is a syntax of Lua. Here I was able to understand how these things here were translated to lpeg

52:53.400 --> 53:02.880
code, to lpeg patterns by using group captures in a certain way. I was able to implement

53:02.880 --> 53:13.080
them in ELpeg1.lua. after some time I was able to use ELpeg1.lua to build grammars that

53:13.080 --> 53:21.080
were able to parse arithmetical expressions with the right precedence. here's an example

53:21.080 --> 53:26.920
in which I built the grammar step by step and I test the current grammar and I replace

53:26.920 --> 53:34.400
a bit and then I test the new grammar and so on. You can see that the result is

53:34.400 --> 53:48.720
always a tree that is drawn in a nice two dimensional way. At this point, these powers

53:48.720 --> 53:57.760
here are returned as a list, as an operation power with several arguments here and then

53:57.760 --> 54:05.560
I apply a kind of parsing combinator here that transforms these trees into other trees

54:05.560 --> 54:14.200
and with these combinators here I can specify that the power is associative in a certain

54:14.200 --> 54:22.320
direction, the division is associative in another direction, the minus uses the same

54:22.320 --> 54:29.520
direction as a division and so on and they have the right precedences. So here are the

54:29.520 --> 54:42.360
tests. So here's my file ELpeg.lua, it has several classes. Each class has tests after

54:42.360 --> 54:52.040
it. I was able to implement something that lpeg.rex has that's called keywords, that's

54:52.040 --> 54:59.160
very useful for parsing problems in programming languages. I was able to implement something

54:59.160 --> 55:10.640
similar to the debugger, to the pack debugger that lpeg uses, but I was frustrated by some

55:10.640 --> 55:23.280
limitations of the debugger and I implemented my own that is much better and let me show

55:23.280 --> 55:33.800
something else. I was able to translate a good part of the Lua parser here to lpeg1.lua.

55:33.800 --> 55:45.880
I haven't finished yet but I have most of the translation here and after having all

55:45.880 --> 55:54.960
that I was able to build other grammars very quickly. Writing new parsers finally became

55:54.960 --> 56:08.560
fun and here's one example that I showed in the beginning. If I remember correctly, I

56:08.560 --> 56:15.680
took a figure from the Wikipedia, I don't have its link now, but I specify a grammar

56:15.680 --> 56:24.280
that parses exactly the example that appears in the Wikipedia. So with my grammar, considering

56:24.280 --> 56:36.680
that the top level entry is statement, when I parse this string here, the result is this

56:36.680 --> 56:44.840
tree and I can do some operations on that. I can define how this thing is to be converted

56:44.840 --> 56:53.400
into LaTeX. I can define other operations that convert trees into other trees and here's

56:53.400 --> 57:01.760
some tests of these operations. This is what I showed in the beginning. I'm not going to

57:01.760 --> 57:10.480
explain all the details of this thing now. This show converts this thing into LaTeX in

57:10.480 --> 57:39.320
the way specified by these instructions here. And here's the result, the LaTeX result.

57:39.320 --> 57:46.880
These diagrams here are generated by this file here that defines a simple grammar that

57:46.880 --> 57:56.720
parses this thing here and then LaTeX it in a certain way and also tests to check if this

57:56.720 --> 58:06.600
code here that is Lua code that generates an lpeg grammar, parses this subject here, and

58:06.600 --> 58:16.880
returns the expected result. So this is the code that I wanted to show. I wanted to show

58:16.880 --> 58:23.920
many more things, but I wasn't able to prepare them before the conference. And I hope that

58:23.920 --> 58:33.240
soon, for some value of soon, I'll be able to create REPL based tutorials for LPG, G and

58:33.240 --> 58:42.080
ELpeg1.Lua where Lpeg is something very famous. Here is a module of LPG. I could also do something

58:42.080 --> 58:55.600
like this for ELpeg, and ELpeg1.lua is the thing that I wrote, the one that has tests in comments

58:55.600 --> 59:04.920
and the tests usually generate trees and sometimes they generate tag code. So that's it, I wanted

59:04.920 --> 59:26.040
to present much more, but I wasn't able to prepare it. So sorry, thanks, bye!