Hi, my name is Eduard Duax, and the title of this talk is
Rapples in Strange Places, Lua, LaTeX, LPEG, LPEG-REX, and TIX.
I'm the author of an MX package called EEV, and this is a talk at the MXConf 2023
that is happening in December 2023 at the Internets.
And this is one of the examples of diagrams that we are going to see.
Let me show how I generate it.
One second. I have to use a smaller font here.
This is a file called parse32.lua. Let me go back to this block of tests again.
And now if I run this, we get these outputs here at the right.
And then in this line here, it generates a PDF.
And if I type f8 here, it shows the PDF in the lower right window.
Let me start by explaining briefly what is EEV.
First, it's something that appeared by accident in the mid-90s.
I explained this story in my presentation at the MXConf 2019.
It's a package. It's an MX package that is part of ELPA.
It has at least 10 users. Those are the ones that I know by name.
EEV means MX Execute Verbosely.
EEV is something that treats eval as the central feature of MX.
EEV blurs the distinction between programmers and users,
and it replaces the slogan,
users should not be forced to see Lisp, that is something that Richard Stallman told me once,
by users should see Lisp instead of buttons,
and new users should see Lisp in the first five minutes.
I'm going to show some examples of that soon.
EEV uses code in comments a lot, and also tests in comments.
I changed my way of presenting it, and it became very REPL-centric in the last few years,
in the sense that I start by explaining its main features by its support for REPLs.
EEV supposes that we want to keep executable notes of everything.
I'm also going to show examples of this in a second.
EEV has lots of videos for people who hate videos,
and it tries to do everything with very little magic and without black boxes.
I'm going to explain many of these things very soon.
This is a figure that I'm going to show in detail soon,
that is about something important about Lua.
Oops, the font is very bad now, so let me change the font.
The figure is this one, and
what most people do when they visit a file with something interesting on it
is that they just go there, and they set a bookmark there,
or they put the position in the register.
But I prefer to keep links to everything that is interesting as a list of hyperlinks.
So, for example, this is an at least hyperlink to the file
that goes to this anchor here, and to this string I've added.
And to this string after this anchor.
This is a variant that opens that file in the window at the right here.
And this is a sexpid that changes the font.
I have a command with a very short name that does that,
but I prefer to keep that as a one-liner.
About the videos, we can see the list of first-class videos of Eevee by executing this,
meta-x, find first-class videos, or by running this alias here,
meta-x1c, and then what we see is this.
The first sexpid here regenerates this buffer,
so we can make a mess here and then run this,
and the original buffer is regenerated again in a clean way.
Each of these things here opens a buffer with information about a video.
Let me take a specific example here.
This video here is about one of the ancestors of this talk,
that is a library that I wrote for
creating diagrams in LaTeX using a package called pic2e, using repls.
Anyway, the thing is that if we run a sexpid like this one,
and we don't have a local copy of the video,
Eevee will try to load the local copy,
and instead of doing that and by asking something like,
asking something like,
do you want me to download the local copy, blah, blah, blah, blah,
it simply opens a buffer like this.
I mean, if we don't have a local copy yet,
it will open a buffer like this one,
in which these things here in comments are links to the documentation.
I mean, this thing here explains the idea of local copies of files from the internet,
uh, there are more details here and here,
and this is a script that we can execute line by line.
So, instead of this script being hidden behind the button that we just
press after a question like,
do you want me to do something, blah, blah, blah, yes or no,
the script is visible here, and we can execute it step by step.
It creates a terminal with a shell here in the right window,
and when we type f8 in these lines here, the lines are sent to this line.
So, this is going to download the copy of the video,
the wget says that I already have a copy of the video in its subtitles, and so on.
And after getting a copy of the video, we can run this exp here, and it displays the video.
I said that Eevee has lots of videos for people who hate videos,
and the idea is that very few people are going to watch the videos in real time,
and most of the people that I know, or most of the people that are interested in Eevee in some way,
they are going to watch just small sections of the video,
and most of the time they're just going to read the subtitles of the video.
So, for each one of the videos, we have a page about the video.
Let me see if I have internet here.
Yes, this is a page,
and usually these pages have a link to another page,
the page that has all the subtitles of the video, wherever.
In this one, it's not so visible, but anyway,
there are several ways of accessing the subtitles of the video,
and one of the ways is by running this exp here,
is by running this exp here,
that opens a file in Lua that is what I use to generate the subtitles.
Anyway, by the way, each one of these things here is hyperlinked to a position of the video,
so if I type this in the right way, it goes to that position.
Anyway, let me go back.
Also, the tutorials of Eevee, the intros of Eevee,
let's start with find, and with intro,
they have lots of blocks that say video links, like this one,
and these blocks have links to the positions in videos,
and if we don't have a local copy of the video yet,
the thing shows us a script that lets us download the local copy.
Anyway, I said that I was going to explain what I mean by magic and black boxes.
This is something that I've been trying to explain for a long time,
and I think that I got a very good explanation about that
in a video that I made about something called Eevee wconfig,
that is a tool for configuring Eevee on Windows without magic,
on Windows without magic, without buttons that do things
without explaining what they are doing.
This is a part of the subtitles of the video.
Let me read that.
Eevee wconfig is an attempt to solve the problem
of how to install these things on Windows,
both without magic and with very little magic.
Remember this slogan,
any sufficiently advanced technology is indistinguishable from magic.
Here in this video, I'm going to use the term magic as a shorthand
for sufficiently advanced technology,
that is something that is complex and non-obvious,
and that is indistinguishable from magic
in the sense of being almost impossible to understand.
And I'm also going to use the term black box as a near synonym for magic,
and sometimes the term black box is more convenient,
even though it's a bit longer, it has more letters,
because when I use the term black box,
it invites us to use expressions like opening the black box,
and I'm going to use that expression a lot.
Now, let me try to explain what is...
Sorry, let me change the font.
What is Lua?
Lua is a minimalistic language in the sense of batteries not included.
It uses associative tables for most of its data structures,
and it's so minimalistic that it's the default print function.
When we create an associative table and we ask it to print...
We ask print to print an associative table,
it just prints the address of the table.
Here are some examples.
Here is a table, and when we ask print to print it,
it just says that it's the table at this address here.
So, one of the things that most people do when they start using Lua
is that either they download a package with a print to printing function,
or they write their own print to printing functions.
My own print to printing function is called pp,
with uppercase letters, and it works like this,
and it prints associative tables in a way like this.
It says that for the key 1,
the value associated to it is 2,
for the key 2, the value is 3,
and for the key 3, the value is 5.
When I started using Lua, one of my favorite languages
was also a language that used associative tables a lot.
It was called Icon,
and I had to write my own print to printing functions for Icon.
So, I just had to port my print to printing functions to Lua,
and my first version looked something like this.
It just had some global functions, lots of them actually,
and after a while I rewrote it,
and I rewrote it again, and again, and again,
and this is one of the versions that is not even the default at this point.
TOS is for toString,
and this is a demo.
It's very modular, so it's easy to replace parts of it or to toggle flags,
and this is an example.
If I try to print the table of methods for a certain class,
I'll need a smaller font,
it prints the table like this,
with the names of the methods,
and then links to the source code of the functions.
These links only make sense in Emacs and in Eevee,
and when we run a link like this one,
it shows the source code in the window at the right.
So, for some functions, the source code is three lines,
for other ones it's one line, and whatever.
Anyway, let me go back.
Lua can be used in many different styles.
Most people hate other people's styles.
When I started using it in the year 2000,
I learned most of the basic language in a single day.
It was very similar to things that I was already using,
and then I rewrote the mini language that I was using
to generate the HTML for my pages in Lua.
Actually, I had to rewrite it many times,
but the first version I certainly did in my first weeks
or first months using Lua.
In the beginning, I was just using it for writing programs
that either didn't take in any input at all,
because the input was already in the source file,
or that worked as a Unix function,
the Unix programs that would read files
and process these files in some way and output something.
I mentioned the basic language here.
I only learned how to use closures, metatables,
and coroutines many years later.
In the beginning, when I started using Lua,
it didn't have a package manager.
It appeared later.
It is called Lua-rocks.
It has had this package manager for several years.
Most of the rocks for Lua-rocks are poorly documented,
documented, and hacker-unfriendly,
so you can't rely just on the documentation,
and you can't rely just on the source code,
because, I mean, if you're a genius, of course you can,
but for people who are either lazy or dumb or whatever,
like me, or unfocused,
the source code is hard to understand
and hard to tinker with.
Some rocks are excellent.
The best rocks are well documented,
but they are hacker-unfriendly,
in the sense that I hope that I'll be able to explain soon.
The best rocks use local variables and metatables a lot,
so if you are a beginner learning Lua,
you're not going to understand what their source codes do.
They use lots of dirty tricks.
Let me talk a bit about object orientation in Lua.
It can be done in many ways.
The main book about Lua, called Programming in Lua,
by one of the authors of the language,
Roberto Ierzalimschi,
presents several ways of doing object orientation in Lua.
I hated all of these ways,
and also the ways that I tried from the rocks.
And then I wrote my own way,
wrote my own way of doing object orientation in Lua.
It's very minimalistic.
It's in this file here, eoo.lua.
The main code is just these five lines here,
and here's an example of how it works.
Here we define a class vector with some metamethods.
This metamethod here will tell Lua what to do
when the user asks to add two vectors.
This one here tells Lua what to do
when the user asks Lua to convert a vector to a string,
and whatever.
This one is something that I'm going to explain in a second.
So here we create a vector with these coordinates,
three and four.
Here we create another vector.
If we print here, then Lua uses the function here
in the toString.
If we add two vectors, it uses the function here
in the add metamethod.
And if we run the method norm,
it is defined here in the table index.
Anyway.
Even this thing being so small,
I used to forget how it's in odds worked all the time.
Actually, I always forget how things work,
and I have to remember them somehow.
And I have to have tricks for remembering
and tricks for summarizing things and diagrams and so on.
And every time that I forgot how to do it,
I just go back to the code.
And every time that I forgot how this thing worked,
I went back to the source code,
and then I looked at the diagrams.
Or, of course, in the first time,
I had to draw the diagrams.
And I run the examples.
And, of course, in the beginning,
I thought that the code was clear.
My examples were very brief.
And so I had to rewrite the examples many times
until they became, let's say, perfect.
And I was saying that Lua can be used in many ways.
And in my way of using Lua, my favorite way,
everything can be inspected and modified from REPLs,
like we can do in MX and in Smalltalk, or sort of.
So in my favorite way of using Lua,
there's no security at all.
Everything can be changed at all times.
Of course, most people hate that.
My init file has lots of classes.
And, by the way, instead of keeping many small files
with many things,
I put lots of stuff in just one big init file.
My init file has lots of classes
and lots of global functions and lots of crafts.
So people hate that, of course.
This is an example.
This is the index at the top of my init file.
The classes start here.
And then we have some functions.
And then we have functions that load certain packages.
And then we have craft.
Whatever.
Most people think that my style of using Lua
is dirty and dangerous.
And they wouldn't touch my Lua code.
With a 10-feet pole.
But most of the things that I'm going to present here
in this presentation are ideas that should be easy
to port to other environments and other languages.
Especially the diagrams.
So the code is not so important.
Now let me talk a bit about LaTeX.
Lua LaTeX.
That is LaTeX with the Lua interpreter embedded inside.
And two ways of generating pictures in LaTeX.
Text that is very famous.
And picture that is not very famous.
And that is very low level.
And I think that not many people use.
I said before that when I learned Lua,
I realized that it was very good for writing little languages.
I was doing my PhD at the time.
And typesetting the diagrams for my PhD thesis was very boring.
So one of the things that I did was that I created
a little language for typesetting the diagrams for me.
It was called DEDNOT.
Because initially it only generated diagrams
for natural deduction.
And then it had several versions.
These are the slides for my presentation about DEDNOT6.
And DEDNOT6 is an extensible semi-preprocessor for Lua LaTeX
that understands diagrams in ASCII art.
In the sense that when I have a LaTeX file that has this.
And when DEDNOT6 is loaded.
When I give the right commands, DEDNOT6 interprets this block here
as something that defines this diagram.
Oops, sorry.
It interprets this diagram here.
This diagram in the comments here
as something that defines a diagram called foo,
a deduction called foo.
And it generates this code here.
So that we can just invoke the definition of the deduction
by typing backslash DED foo.
And DEDNOT6 also supports another language
for typesetting bi-dimensional diagrams
with arrows and stuff for category theory and blah blah.
The specifications of these diagrams look like this.
Here is a very good example.
This is a huge diagram.
Sorry, one second.
So the source code that generates this diagram here
is just this thing at the left.
So it's very visual.
We can typeset the diagram in ASCII art here.
And then in this part here,
we tell how the nodes are to be joined,
which arrows have to have annotations and so on.
And this language is extensible in the sense that...
Where is that?
Here.
Comments that start with percent colon
are interpreted as definitions for three diagrams.
Lines that start with percent uppercase D
define 2D diagrams with arrows and stuff.
And lines that start with comment uppercase L
contain blocks of Lua code
that we can use to extend the interpreter on the flag.
Anyway, here are some recent examples of diagrams
that I used DEDNOT6 to typeset.
This diagram here
was generated by this specification here.
And this diagram here with the curved arrows
was generated by this specification here.
So DEDNOT6 was very easy to extend.
At that some point,
I started to use it to generate diagrams using Peaked Chewy,
mainly for the classes that I gave you.
For the classes that I give at the university.
I teach mathematics and whatever.
In a bad place.
Whatever.
Let me show an animation.
Here is a diagram that I generated with DEDNOT6.
And it is a flipbook animation.
Like we type page up and page down
and we go to the next page of the book
and to the previous page of the book.
And here is the source code that generates that.
This source code is not very visual.
So it's quite clumsy to edit a diagram
directly in the tag file like that.
These diagrams were inspired by something called Manim.
That's...
Oh, I've forgotten the name of the guy.
But it's a guy that makes many videos about mathematics.
And he created this library called Manim
for generating his animations.
Other people adapted his library to make it more accessible.
I tried to learn it.
But each animation,
even an animation with very few frames,
each animation took ages to render.
So it wasn't fun.
And animations and PDFs can be rendered in seconds.
So these things were fun for me
because my laptop is very slow and Manim was not fun.
Anyway, writing code like this inside the attack file
was not very fun because it was hard to debug.
So in 2022, I started to play with ways
of generating these diagrams from REPLs.
And I found a way for PIC2E and a way for TIX.
Each one of these ways became a video.
If you go to the list of first class videos of EEV,
you're going to see that there's a video about PIC2E here
and a video about TIX.
Here you have some information like length and explanation, etc.
And here are the pages for these videos.
My page about the video, about PIC2E is like this.
It has some diagrams, whatever.
And this one is much nicer.
And a lot of people watched that video.
I mean, I thought that 250 people watched it.
For me, that's a million of people.
And this video is about how to extract diagrams from the TIX manual
and how to run those examples in a REPL
and modify them bit by bit.
This is a screenshot.
But let me go back.
At that point, these things were just prototypes.
The code was not very nice.
And in this year, I was able to unify those two ways of generating PDFs,
the one for TIX and the one for PIC2E.
And I unified them with many other things that generated diagrams.
The basis of these things is something called showchew.lua.
I'm not going to show its details now.
But its extension that generates TIX code is just this.
So we can specify a diagram with just a block like this.
And then if we run show00 and it returns a string
that is just the inner body of the TIX file.
If we run this, we see the whole TIX file.
And if we run this, we save the TIX file
and we compile the TIX file to generate a PDF.
And if we run this, we show the PDF in the lower right window.
And that's the same thing for all my recent programs that generate PDFs.
They are all integrated.
Here is the one that the basis for all my modules that generate diagrams with PIC2E.
Its demos are not very interesting.
So let me show some demos of extensions that do interesting things.
So this is a diagram that I created by editing it in a REPL.
I create several PIC2E objects here.
And if I execute this, it compiles the object, generates a PDF.
And if I tap this, here is the PDF.
And if I just ask Lua to display what is books here,
it shows the source code in PIC2E of the diagram.
And the nice thing is that it is indented.
So it's easy to debug the PIC2E code.
If anyone is interested, the module that does the tricks for indentation
is very easy to understand.
It has lots of tests and test blocks.
And I think that its data structures are easy to understand.
Anyway, here is another example.
The show is here.
It generates a 3D diagram.
Now let me talk about parsers and REPLs in a very strange place.
I mean, using REPLs to build parsers step by step and replacing parts by more complex parts.
So I said that Lua is very minimalistic.
And everybody knows that implementations of regular expressions
are big and complex.
So instead of coming with full regular expressions, Lua comes with something called patterns
and a library function called string.match.
Here is a copy of the part of the manual that explains a part of the syntax of patterns.
Here's how string.match is described in the manual.
It's just this.
Looks for the first match of pattern in the string as blah, blah, blah.
And then we have to go to the other section of the manual that explains patterns.
Lua patterns are so simple, so limited, that they don't even have the alternation operator.
Here is how it is described in the ELLISP manual.
Backslash pipe specifies an alternative, blah, blah, blah.
When we want to build more complex regular expressions, patterns, grammars, etc.,
we have to use an external library for that.
No, sorry, a library that is external, but that was written by one of the authors of Lua itself.
This library is called LPEG, and its manual says,
LPEG is a new pattern matching library for Lua based on parsing expression grammars, PEGs.
The manual is very terse.
I found it incredibly hard to read.
It doesn't have any diagrams.
It has some examples, though.
And the Lua wiki has a big page called LPEG tutorial with lots of examples.
But it also doesn't have diagrams, and I found some things incredibly hard to understand.
For example, this is something that is in the manual of LPEG that I saw and I thought,
wow, great, this makes all sense, and this is going to be very useful.
It's a way to build grammars that can be recursive,
and they sort of can encode BNF grammars.
We just have to translate the BNF a bit to get rid of some recursions
and to translate them to something else.
And the manual also has some things that I thought,
oh, no, I don't have any idea of what this thing does.
And in fact, I saw these things for the first time more than 10 years ago,
and they only started to make sense one year ago.
One example is group captures.
LPEG also comes with a module called the RE module.
Let me pronounce it in Portuguese, the RE module.
And its manual says, the RE module provided by the file repo.luen
in the distribution supports a somewhat conventional
regular expression syntax for pattern usage within LPEG.
And this is a quick reference.
And this thing is very brief.
It has some nice examples, but it's hard to understand in a way.
And here are some comments about my attempts to learn HerdotLua.
This is a class.
In this case, it's a very small class.
And this file implements a PM method.
I'm going to show examples of other PM methods very soon.
So this is a PM method for HerdotLua that lets us compare the syntax
of Lua patterns, LPEG, and HerdotLua.
See this example here.
So if we run this, it loads my version of LPEG.
No, sorry, my version of LPEG-REX.
And it shows that when we apply the PM method to this Lua pattern,
this LPEG pattern, and this RE pattern, they all get the same results.
So we can use this thing, this kind of thing here to show how to translate from
Lua patterns that are familiar because they are similar to regular expressions, only weaker,
to LPEG that is super weird, and to RE that is not so weird.
Anyway, the comment says that in 2012, I had a project that needed a precedence parser that
could parse arithmetical expressions with the right precedences.
And at that point, I was still struggling with pure LPEG, and I couldn't do much with it.
So I tried to learn HerdotLua instead, and I wrote this old class here.
That allowed me to use preprocessor on patterns for Lua.
And the thing is that with this preprocessor, I could specify precedence grammars using this
thing here that worked, but it was super clumsy, and I gave up after a few attempts.
And in 2022, I heard about something called LPEG-REX that was an
a kind of extension of Re, and it was much more powerful than HerdotLua, but after a while,
I realized that it had the same defects as HerdotLua.
And let me explain that because it has all to do with the things about black boxes and magic
that I told in the beginning.
Both, I mean, sorry, neither HerdotLua or LPEG-REX had some features that I needed.
They didn't let us explain, sorry, they received a pattern that was specified as a string,
and it converted that into an LPEG pattern, but it didn't let us explore the LPEG patterns
that it was using.
So I had to use the LPEG-REX, and it didn't let me explore the LPEG patterns that it was
generated.
Their code was written in a way that was REPL unfriendly.
I couldn't modify parts of the code bit by bit in a REPL and try to change the code
without changing the original file, say.
The code was very hard to explore, to hack, and to extend, in my opinion.
The documentation was not very clear.
And I sent one or two messages to the developer of LPEG-REX, and he was too busy to help me.
He answered very briefly, and to be honest, I felt rejected.
I felt that I wasn't doing anything interesting.
Whatever, whatever.
So, in 2022, I was trying to learn LPEG-REX, because I was thinking that it would solve
my problems, but it didn't.
It didn't have the features that I needed, and it was hard to extend, and hard to explore,
and hard to debug.
I decided to rewrite it in a more hacker-friendly way, in the sense that it was modular, and
I could replace any part of the module from a REPL.
My version of it was called lpeg1.lua, and I decided that in my version, I wouldn't have
the part that receives a grammar specified as a string and converts that to LPEG.
I would just have the backend part that are the functions in LPEG that let us specify
powerful grammars.
So, let me go back.
Let me explain a bit about LPEG.
Lua has coercions.
The plus expects to receive two numbers.
If one of its arguments, or both of them, are strings, it converts the strings to numbers.
So, in this case here, 2 plus string 3 returns the number 5, and this is the concatenation
operator.
It expects to receive strings.
So, in this case, it will convert the number 2 to the string 2, and the concatenation of
these two things will be 23.
Sorry, 23 as a string.
LPEG also has some coercions.
I usually set these globals to let me write my grammars in a very compact way.
So, instead of lpeg.p, lpeg.c, etc., I use these globals like uppercase B, uppercase
C, and so on.
And with these globals, I can write things like this, c1 times string underscore.
And LPEG knows that lpeg.c, sorry, it sort of expands these to lpeg.c, but lpeg.c expects
to receive an LPEG pattern, and one is not yet an LPEG pattern, so it is coerced into
an LPEG pattern by calling lpeg.p.
So, this shorting here becomes equivalent to lpeg.c, lpeg.p1, and the multiplication,
when at least one of its arguments is an LPEG pattern, it expects to receive two LPEG patterns,
and in this case, the one at the right is just a string, so it is coerced to an LPEG
pattern by using lpeg.p.
With this idea, we can sort of understand this comparison here.
I mean, let me run it again.
This first part is very similar to a regular expression here at the left, and when we apply
this LPEG, sorry, this Lua pattern to this subject here, the result is that the Lua pattern
the result is this thing here, this thing, this thing, and this thing.
I'm going to call each one of these results captures, so each of these things between
parentheses captures a substring of the original string, and these captured substrings are
returned in a certain order.
Here is how to express the same thing in LPEG.
It's very cryptic, but it's a good way to understand some basic operators of LPEG.
I mean, we can look at the menu and understand what C, S, and R do, and also exponentiation.
And this strange thing here receives this string here, runs a function that I have defined
that converts it to an object of a certain class, and that class represents He patterns.
So this thing is treated as a pattern for He.Lua, and it is matched against the string,
and it returns the same thing as the other one.
Also, this thing here also has a comparison with LPEG-REGS, but these patterns are very
trivial.
They don't do anything very strange.
So let's go back and see what kinds of very strange things there are.
Here is the page of LPEG-REGS at GitHub.
Here's the documentation.
It's relatively brief.
It explains LPEG-REGS as being an extension of He.Lua.
So it explains mainly the additional features.
Here is a quick reference that explains only the additional features.
Some of these things I was able to understand by using the LPEG-REGS.
I was struggling a lot, and some I wasn't able to, even by spending several evenings
trying to build examples.
And this is something very nice.
LPEG-REGS comes with some example parsers, and here is a parser that parses the Lua grammar.
I mean, this is the grammar for Lua 5.4 at the end of the reference manual.
It's just this, and this is a kind of the BNF, and this is the BNF translated to
the language of LPEG-REGS.
So this thing uses many constructions that are in He.Lua and some extra constructions
that are described here.
And with these examples, I was able to understand some of these things here that are described
here in the quick reference, but not all.
So I wasn't able to use LPEG-REGS by itself because some things didn't make much sense,
and I decided to reimplement it in my own style.
Because that would be a way to map, at the very least, map what I understood and what
I didn't, and learn one feature at a time, do comparisons, and so on.
Here, I pointed to two features of LPEG.
One, I said, oh great, this thing can be used to define grammars, even recursive
grammars, and so on.
And this is an oh-no feature, one thing that didn't make any sense at all, group captures.
One thing that I did to understand group captures was to represent them as diagrams.
Of course, in the beginning, I was drawing these diagrams by hand, but then I realized
that I could use the bits of LPEG that I already knew to build a grammar that would
parse a little language and generate these diagrams in LaTeX.
And I was able to make this.
In this diagram here, this thing above the arrow is a piece of Lua code that specifies
an LPEG pattern.
This thing here at the top is the string that is being matched, and the things below the
under braces are the captures that each thing, sorry, each thing captures.
And for example, this under brace here corresponds to this pattern here that parses a single
character but doesn't return any captures.
This thing here parses a single B and doesn't return any captures.
This thing here parses a single character and captures it, and this thing here parses
the character D and captures it.
And this other thing here that transforms this pattern into another pattern returns
first a capture with all the string that was parsed by this pattern here, and then all
the captures returned by this thing here before the column.
So this was a way to build concrete examples for things that the LPEG manual was explaining
in a very terse way, and it worked for me.
Some things that were very mysterious started to make sense, and I started to have intelligent
questions to ask in the mailing list.
And with that, I was able to understand what are group captures and group captures that
receive a name.
Well, let me explain what this does.
This thing here captures, sorry, parses the empty string and returns this as a constant.
So this is something that doesn't exist in regular expressions.
It parses nothing and returns this as a capture.
Then this thing here returns these two constants here and parses the empty string, and this
thing here, d, converts the results of this thing here into a group capture and stores
it in the label d.
And then here's another constant capture.
And I realized that these things here were similar to how Lua specifies building lists.
When we build, sorry, a table, when we build a table and we say that the first element
of the table is here, this element is put at the end of the table.
When after that we say d equals to, say, 42, we are putting the 42 in the slot whose key
is d.
This was happening with LPEG captures, but there was something very strange.
These group captures could hold more than one capture, more than one value.
So there was something between lists and tables.
I started to use this notation to explain in my notation what they were doing.
Many things start, things stop.
Many things start, things started to make sense.
Many mysterious sentences in the manual started to make sense, but some didn't.
But at least I was able to send some intelligent questions to the mailing list, and the author
of Lua and LPEG answered some of them.
He was not very happy about my questions.
He told me that those diagrams were a waste of time, that the manual was perfectly clear,
and so on, whatever.
But I was able to, so it was weird, but I was able to understand lots of things from
his answers.
So this is a copy of one of my messages.
Then there's another one, another one.
Some were the diagrams, then he complained about these diagrams.
He said that these things here that look like table constructions do not exist, whatever.
Anyway, once I understood group captures, many features were very, very easy to understand,
and I started to be able to use LPEG to build some very interesting things.
I was able to reproduce some of the features that I saw in LPEG-REX.
Remember that this, where is that?
This is a syntax of Lua.
Here, I was able to understand how these things here were translated to LPEG code, to LPEG
patterns by using group captures in a certain way.
And I was able to implement them in lpeg1.lua.
And after some time, I was able to use lpeg1.lua to build grammars that were able to parse
arithmetical expressions with the right precedence.
And here's an example in which I built the grammar step by step, and I test the current
grammar, and I replace a bit, and then I test the new grammar, and so on.
And you can see that the result is always a tree that is drawn in a nice two-dimensional way.
At this point, these powers here are returned as a list, as an operation
pow with several arguments here.
And then I apply a kind of parsing combinator here that transforms these trees into other trees.
And with these combinators here, I can specify that the power is associative in a certain
direction, the division is associative in another direction, the minus is associative,
so it uses the same direction as a division, and so on.
And they have the right precedences.
So here are the tests.
So here's my file lpeg1.lua.
It has several classes.
Each class has tests after it.
I was able to implement something that lpeg-reqs has that's called keywords.
That's very useful for parsing programs in programming languages.
I was able to implement something similar to the debugger, to the PEG debugger that lpeg uses,
but I was frustrated by some limitations of that debugger,
and I implemented my own that is much better.
And let me show something else.
I was able to translate a good part of the Lua parser here to lpeg1.lua.
I haven't finished yet, but I have most of the translation here.
And after having all that, I was able to build other grammars very quickly.
Writing new parsers finally became fun, and here's one example that I showed in the beginning.
If I remember correctly, I took a figure from the Wikipedia.
I don't have its link now, but I specified a grammar that parses
exactly the example that appears in the Wikipedia.
So with my grammar, considering that the top-level entry is statement,
when I parse this string here, the result is this tree.
And I can do some operations on that.
I can define how this thing is to be converted into LaTeX.
I can define other operations that convert trees into other trees.
And here are some tests of these operations.
This is what I showed in the beginning.
I'm not going to explain all the details of this thing now.
This show converts this thing into LaTeX in the way specified by these instructions here.
Let's say that...
Well, whatever.
Really, whatever.
And here's the result, the LaTeX result.
And these diagrams here are generated by this file here that defines a simple grammar that parses
this thing here, and then LaTeX it in a certain way, and then also tests this thing.
To check if this code here, that is Lua code that generates an LPEG grammar,
parses this subject here and returns the expected result.
So this is the code that I wanted to show.
I wanted to show many more things, but I wasn't able to prepare them before the conference.
And I hope that soon, for some value of soon, I'll be able to create
REPL-based tutorials for LPEG here and lpeg1.lua, where LPEG is something very famous.
Here is a module of LPEG.
I could also do something like this for LPEG-REX, and lpeg1.lua is the thing that I wrote,
the one that has tests in comments, and the tests usually generate trees,
and sometimes they generate tag code.
Yeah, so that's it.
I wanted to present much more, but I wasn't able to prepare it.
So sorry.
Thanks.
Bye.