Hi, my name is Eduardo Ochs
and the title of this talk is
REPLs in Strange Places: Lua, LaTeX,
LPeg, LPegRex, and TikZ.
I'm the author of an Emacs package called eev,
and this is a talk at the EmacsConf 2023,
that is happening in December 2023 at the Internet.
This is one of the examples of diagrams
that we are going to see.
Let me show how I generated it.
One second, I have to use a smaller font here.
This is a file called parse32.lua.
Let me go back to this block of tests again.
Now, if I run this,
we get these outputs here at the right.
Then in this line here,
it generates a PDF,
and if I type `F8` here,
it shows the PDF in the lower right window.
Let me start by explaining briefly what is eev.
First, it's something
that appeared by accident in the mid 90s.
I explained this story in my presentation at EmacsConf 2019.
It's an Emacs package that is part of ELPA.
It has at least 10 users.
Those are the ones that I know by name.
eev means "Emacs, execute verbosely".
eev is something that treats eval-last-sexp
as the central feature of Emacs.
eev blurs the distinction between programmers and users,
and it replaces the slogan,
"users should not be forced to see Lisp"
(that is something that Richard Stallman told me once),
by "users should see Lisp instead of buttons,"
and "new users should see Lisp in the first five minutes."
I'm going to show some examples of that soon.
eev uses code in the comments a lot
and also tests in comments.
I changed my way of presenting it,
and it became very REPL-centric
in the last few years, in the sense that
I start by explaining its main features
by its support for REPLs.
eev supposes that we want to keep
executable notes of everything.
I'm also going to show examples of this in a second.
eev has lots of videos for people who hate videos,
and it tries to do everything
with very little magic and without black boxes.
I'm going to explain many of these things very soon.
This is a figure that I'm going to show in detail soon
that is about something important about Lua.
Oops, the font is very bad now,
so let me change the font.
The figure is this one.
What most people do when they visit a file
with something interesting in it,
is that they just go there and they set a bookmark there,
or they put the position in the register.
But I prefer to keep links
to everything that is interesting
as an Elisp hyperlink.
For example, this is an Elisp hyperlink to the file
that goes to this anchor here,
and to this string after this anchor.
This is a variant that opens that file
in the window at the right here,
and this is a sexp that changes the font.
I have a command with a very short name that does that,
but I prefer to keep that as a one-liner.
About the videos, we can see
the list of first-class videos of eev by executing this,
`M-x find-1stclassvideos`,
or by running this alias here, `M-x 1c`,
and then what we see is this.
The first sexp here regenerates this buffer,
so we can make a mess here and then run this,
and the original buffer is regenerated again in a clean way.
Each of these things here
opens a buffer with information about a video.
Let me take a specific example here.
This video here is about one of the ancestors of this talk
that is a library that I wrote
for creating diagrams in LaTeX
using a package called [pict2e-lua] using REPLs.
Anyway, the thing is that if we run a sexp like this one
and we don't have a local copy of the video,
eev will try to load the local copy
and instead of doing that by asking something
like "do you want me to download the local copy"
blah, blah, blah, it simply opens a buffer like this.
And if we don't have a local copy yet,
it will open a buffer like this one,
in which these things here in comments
are links to the documentation.
This thing here explains the idea
of local copies of files from the internet.
There are more details here and here,
and this is a script that we can execute line by line.
So instead of this script being hidden behind the button
that we just press after a question
like "do you want me to do something," blah, blah, blah,
"yes or no?", the script is visible here
and we can execute it step by step.
It creates a terminal with a shell here in the right window
and when we type `F8` in these lines here,
the lines are sent to this line.
So this is going to download a copy of the video,
the wget says that I already have a copy of the video
and its subtitles and so on.
And after getting a copy of the video
we can run this exp here and it displays the video.
I said that eev has lots of videos
for people who hate videos.
The idea is that very few people
are going to watch the videos in real time.
Most of the people that I know, or most of the people
that are interested in eev in some way,
they are going to watch just small sections of the video.
Most of the time, they are just going to
read the subtitles of the video.
For each one of the videos
we have a page about the video...
Let me see if I have internet here...
This is a page. Usually these pages
have a link to another page,
the page that has all the subtitles of the video,
wherever. In this one, it's not so visible.
Anyway, there are several ways of accessing
the subtitles of the video.
One of the ways is by running this sexp here
that opens a file in Lua.
That is what I use to generate the subtitles.
By the way, each one of these things here
is hyperlinked to a position of the video.
If I type this, it goes to that position.
Anyway, let me go back.
So the tutorials of eev, the intros of eev
that start with `find-` and end with `-intro`,
they have lots of blocks that say
video links like this one.
These blocks have links to positions in videos.
If we don't have a local copy of the video yet,
the thing shows us a script
that lets us download the local copy.
I said that I was going to explain
what I mean by magic and black boxes,
this is something that I've been trying
to explain for a long time
and I think that I got a very good explanation about that
in a video that I made about something called eev-wconfig
that is a tool for configuring eev on Windows without magic,
without buttons that do things,
without explaining what they are doing.
This is a part of the subtitles of the video.
Let me read that.
eev-wconfig is an attempt to solve the problem
of how to install these things on Windows
both without magic and with very little magic.
Remember this slogan: "Any sufficiently advanced technology
is indistinguishable from magic."
Here in this video I'm going to use the term magic
as a shorthand for sufficiently advanced technology
that is something that is complex and non obvious
and that is indistinguishable from magic
in the sense of being almost impossible to understand.
And I'm also going to use the term black box
as a near synonym for magic
and sometimes the term black box is more convenient
even though it's a bit longer,
it has more letters because when I use the term black box
it invites us to use expressions
like opening the black box
and I'm going to use that expression a lot.
Now let me try to explain what is. Sorry,
let me change the font.
What is Lua?
Lua is a minimalistic language
in the sense of battery is not included.
It uses associative tables for most of its data structures.
It's so minimalistic
that the default print function
when we create an associative table and we ask it to print--
we ask print to print an associative table--
it just prints the address of the table.
Here are some examples. Here is a table
and when we ask to print it,
it just says that it's the table at this address here.
One of the things that most people do
when they start using Lua is that either
they download a package with a print to printing function
or they write their own print to printing functions.
My own print to printing function
is called PP with upper case letters,
and it works like this.
It prints associative tables in a way like this.
It says that for the key 1,
the value associated to it is 2;
for the key 2, the value is 3;
and for the key 3, the value is 5.
When I started using Lua, one of my favorite languages
was also a language that used associative tables a lot.
It was called Icon. I had to write my own
print to printing functions for Icon,
so I just had to port my print to printing functions to Lua.
My first version looked something like this.
it just had some global functions. Lots of them actually.
After a while, I rewrote it.
I rewrote it again and again and again.
This is one of the versions
that is not even the default at this point.
`tos` is for to string. This is a demo.
It's very modular, so it's easy to replace parts of it
or to toggle flags. This is an example.
If I try to print the table of methods for a certain class--
I only need a smaller font--it prints the table like this
with the names of the methods
and then links to the source code of the functions.
These links only make sense in Emacs and in eev.
When we run a link like this one,
it shows the source code and the window at the right.
For some functions, the source code is three lines.
For other ones, it's one line and whatever.
Anyway, let me go back.
Lua can be used in many different styles.
Most people hate other people's styles.
When I started using it in the year 2000,
I learned most of the basic language in a single day.
It was very similar to things that I was already using.
Then I rewrote the mini language
that I was using to generate the HTML for my pages in Lua.
Actually, I had to rewrite it many times
but the first version I certainly did
in my first weeks or first months using Lua.
In the beginning, I was just using it for writing programs
that either didn't take any input at all
because the input was already in the source file,
or that worked as a Unix function:
the Unix programs that would read files,
process these files in some way, and output something.
I mentioned the basic language here,
I only learned how to use closures, meta tables,
and coroutines many years later.
In the beginning when I started using Lua
it didn't have a package manager.
It appeared later. It is called luarocks.
It has had this package manager for several years.
Most of the rocks for luarocks are poorly documented
and hacker-unfriendly.
You can't rely just on the documentation.
You can't rely just on the source code,
because--I mean, if you're a genius, of course you can,
but for people who are either
lazy or dumb or whatever like me, or unfocused,
the source code is hard to understand
and hard to tinker with.
Some rocks are excellent.
The best rocks are well documented,
but they are hacker-unfriendly
in the sense that I hope that I'll be able to explain soon.
The best rocks use local variables and meta tables a lot,
so if you are a beginner learning Lua
you're not going to understand what their source codes do.
They use lots of dirty tricks.
Let me talk a bit about object orientation in Lua.
It can be done in many ways. The main book about Lua
called Programming in Lua
by one of the authors of the language
called Roberto Ierusalimschy
presents several ways of doing object orientation in Lua.
I hated all of these ways,
and also the ways that I tried from the rocks.
Then I wrote my own way
of doing object orientation in Lua.
It's very minimalistic. It's in this file here eoo.lua.
The main code is just these five lines here.
Here's an example of how it works.
We define a class vector with some meta methods.
This meta method here will tell Lua
what to do when the user asks to add two vectors.
This one here tells Lua what to do
when the user asks Lua to convert a vector to a string
and whatever. This one is something
that I'm going to explain in a second.
Here we create a vector
with these coordinates 3 and 4.
Here we create another vector.
If we print here,
then Lua uses the function here in the `__tostring`.
If we add two vectors
it uses the function here in the `__add` meta method.
If we run the method `norm`,
it is defined here in the table index.
Anyway, even with this thing being so small,
I used to forget how it works all the time.
Actually, I always forget how things work,
and I have to remember them somehow.
I have to have tricks for remembering,
and tricks for summarizing things and diagrams, and so on.
Every time that I forgot how this thing worked,
I went back to the source code
and then I looked at the diagrams,
or, of course, the first time,
I had to draw the diagrams. I run the examples.
Of course, in the beginning
I thought that the code was clear.
My examples were very brief.
So I had to rewrite the examples many times
until they became let's say perfect.
I was saying that Lua can be used in many ways
and in my way of using Lua, in my favorite way,
everything can be inspected and modified from REPLs
like we can do in Emacs and in SmallTalk, sort of.
In my favorite way of using Lua
there's no security at all.
Everything can be changed at all times.
Of course, most people hate that.
My init file has lots of classes,
and by the way, instead of
keeping many small files with many things,
I put lots of stuff in just one big init file.
My init file has lots of classes,
and lots of global functions and lots of cruft,
so people hate that, of course.
This is an example.
This is the index at the top of my init file.
The classes start here. Then we have some functions
and then we have functions that load certain packages,
and then we have cruft, whatever.
Most people think that my style of using Lua
is dirty and dangerous,
and they wouldn't touch my Lua code with a 10 feet pole,
but most of the things
that I'm going to present here in this presentation
are ideas that should be easy to port to other environments
and other languages, especially the diagrams,
so the code is not so important.
Now let me talk a bit about LaTeX, LuaLaTeX,
that is LaTeX with the Lua interpreter embedded inside
and two ways of generating pictures in LaTeX:
TikZ that is very famous,
and Pict2e that is not very famous
and that is very low level,
and I think that not many people use.
I said before that when I learned Lua,
I realized that it was very good
for writing literal languages.
I was doing my PhD at the time
and typesetting the diagrams for my PhD thesis
was very boring.
One of the things that I did was that
I created a literal language
for typesetting the diagrams for me.
It was called Dednat because initially
it only generated diagrams for natural deduction
and then it had several versions.
These are the slides from my presentation about Dednat6
and Dednat6 is an extensible semi preprocessor
for LuaLaTeX that understands diagrams in ASCII art
in the sense that when I have a TeX file that has this,
and when Dednat6 is loaded,
when I get the right comments,
Dednat6 interprets this block here
as something that defines this diagram.
It interprets this diagram here,
this diagram in the comments here,
as something that defines a diagram called `foo`,
a deduction called `foo`, and it generates this code here
so that we can just invoke the definition of the deduction
by typing `\ded{foo}`.
Dednat6 also supports another language
for typesetting bidimensional diagrams with arrows and stuff
for category theory and blah blah.
The specifications of these diagrams
look like this. Here is a very good example.
This is a huge diagram. Sorry, one second.
So the source code that generates this diagram here
is just this thing at the left. It's very visual.
We can typeset the diagram in ASCII art here.
Then in this part here,
we tell how the nodes are to be joined,
which arrows have to have annotations, and so on.
This language is extensible in the sense that
here, comments that start
with `%:` are interpreted as definitions
for tree diagrams,
lines that start with `%D`
define 2D diagrams with arrows and stuff,
and lines that start with command `%L`
contain blocks of Lua code that we can use
to extend the interpreter on the flag.
Anyway, here are some recent examples
of diagrams that I used Dednat6 to typeset.
This diagram here was generated by this specification here,
and this diagram here with the curved arrows
was generated by this specification here.
Dednat6 was very easy to extend.
At some point I started to use it to generate diagrams
using Pict2e, mainly for the classes
that I give at the university,
I teach mathematics in a place.
Let me show an animation. Here is a diagram
that I generated with Dednat6.
It is a flipbook animation,
we type page up and page down,
and we go to the next page of the book
and to the previous page of the book.
Here is the source code that generates that.
This source code is not very visual,
so it's quite clumsy to edit
a diagram directly in the TeX file like that.
These diagrams were inspired by something called Manim.
I've forgotten the name of the guy, but it's a guy
that makes many videos about mathematics,
and he created this library called Manim
for generating his animations.
Other people adapted his library to make it more accessible.
I tried to learn it, but each animation,
even an animation with very few frames,
each animation took ages to render, so it wasn't fun.
Animations and PDFs can be rendered in seconds,
so these things were fun for me
because my laptop is very slow and Manim was not fun.
Anyway, writing code like this inside a TeX file
was not very fun because it was hard to debug.
In 2022, I started to play with ways
of generating these diagrams from REPLs
and I found a way for Pict2e and a way for Tikz.
Each one of these ways became a video.
If you go to the list of first class videos of eev,
you're going to see
that there's a video about Pict2e here
and a video about Tikz.
Here you have some information
like length and explanation, etc.
Here are the pages for these videos.
My page about the video about Pict2e is like this.
It has some diagrams. This one is much nicer.
A lot of people watched that video. I mean,
I thought that 250 people watched it.
For me, that's a million of people.
This video is about how to extract diagrams
from the Tikz manual, and how to run those examples in REPL
and modify them bit by bit. This is a screenshot.
Let me go back. At that point,
these things were just prototypes,
the code was not very nice,
and in this year I wrote...
I was able to unify those two ways of generating PDFs,
the one for Tikz and the one for Pict2e.
I unified them with many other things
that generated diagrams.
The basis of these things is something called Show2.lua.
I'm not going to just show its details now,
but it's an extension that generates Tikz code.
It's just this. We can specify a diagram
with just a block like this, and then if we run `show00`,
it returns a string
that is just the inner body of the TeX file.
If we run this, we see the whole TeX file.
If we run this we save the TeX file and
we compile the TeX file to generate a PDF.
If we run this, we show the PDF
in the lower right window.
That's the same thing for all my recent programs
that generate PDFs. They are all integrated.
Here is the basis for all my modules
that generate diagrams with Pict2e.
Its demos are not very interesting.
Let me show some demos of extensions
that do interesting things. This is a diagram
that I created by editing it in a REPL,
I create several Pict objects here.
If I execute this, it compiles the object and
generates a PDF. If I tap this, here is the PDF.
If I just ask Lua to display what is pux here,
it shows the source code in Pict2e of the diagram.
The nice thing is that it is indented,
so it's easy to debug the Pict2e code.
If anyone is interested, the module that does the tricks
for indentation is very easy to understand.
it has lots of tests and test blocks,
and I think that its data structures are easy to understand.
Anyway, here is another example, the show is here.
It generates a 3D diagram.
Now let me talk about parsers
and REPLs in a very strange place.
I mean, using REPLs to build parsers step by step
and replacing parts by more complex parts.
I said that Lua is very minimalistic,
and everybody knows that
implementations of regular expressions
are big and complex. So instead of coming
with full regular expressions,
Lua comes with something called patterns
and a library function called string.match.
Here is a copy of the part of the manual
that explains a part of the syntax of patterns.
Here's how string.match is described in the manual.
It's just this: looks for the first match of pattern
in the string s, blah blah blah.
Then we have to go to the other section of the manual
that explains patterns. Lua patterns are so simple,
so limited, that they don't even have
the alternation operator.
Here is how it is described in the elisp manual,
`\|` specifies an alternative, blah blah blah.
When we want to build more complex regular expressions,
patterns, grammars, etc.,
we have to use an external library for that.
No, sorry, a library that is external,
but that was written by one of the authors of Lua itself.
This library is called Lpeg.
Its manual says Lpeg is
a new pattern matching library for Lua
based on parsing expression grammars (PEGs).
The manual is very terse.
I found it incredibly hard to read.
it doesn't have any diagrams.
It has some examples, though.
The Lua wiki has a big page called Lpeg Tutorial
with lots of examples, but it also doesn't have diagrams.
I found some things incredibly hard to understand.
For example, this is something
that is in the manual of Lpeg that I saw,
and I thought, wow, great, this makes all sense
and this is going to be very useful.
It's a way to build grammars that can be recursive
and they sort of can encode BNF grammars,
we just have to translate the BNF a bit
to get rid of some recursions
and to translate them to something else.
The manual also has some things that I thought,
oh no, I don't have any idea of what this thing does.
In fact, I saw these things for the first time
more than 10 years ago, and they only started to make sense
one year ago. One example is group captures.
Lpeg also comes with a module called the re module.
Let me pronounce it in Portuguese, the re module.
Its manual says the re module
provided by the file re.lua in the distribution
supports some odd conventional regular expression syntax
for pattern usage within lpeg. This is a quick reference.
This thing is very brief. It has some nice examples,
but it's hard to understand anyway.
Here are some comments about
my attempts to learn re.lua. This is a class.
In this case, it's a very small class.
This file implements a `pm` method.
I'm going to show examples of other `pm` methods very soon.
So this is a `pm` method for re.lua
that lets us compare the syntax of lua patterns, lpeg,
and re. See this example here. If we run this,
it loads my version of lpeg, no sorry,
my version of lpegrex. It shows that
when we apply the `pm` method to this Lua pattern,
this lpeg pattern, and this re pattern,
they all give the same results.
So we can use this kind of thing here to show
how to translate from Lua patterns that are familiar
because they are similar to regular expressions,
only weaker, to lpeg that is super weird,
and to re that is not so weird.
Anyway, the comment says that in 2012, I had a project
that needed a precedence parser
that could parse arithmetical expressions
with the right precedences. At that point
I was still struggling with pure lpeg
and I couldn't do much with it,
so I tried to learn re.lua instead,
and I wrote this old class here that allowed me to use
a preprocessor on patterns for Lua.
The thing is that with this preprocessor
I could specify precedence grammars using this thing here
that worked but was super clumsy,
and I gave up after a few attempts.
And in 2022 I heard about something called lpegrex
that was a kind of extension of re,
and it was much more powerful than re.lua,
but after a while I realized that
it had the same defects as re.lua.
Let me explain that,
because it has all to do with the things
about black boxes and magic that I told in the beginning.
Neither re.lua or lpegrex had some features that I needed.
They received a pattern that was specified as a string
and it converted that into an lpeg pattern,
but it didn't let us explore the lpeg patterns
that it generated.
Their code was written in a way that was REPL-unfriendly.
I couldn't modify parts of the code bit by bit in a REPL
and try to change the code
without changing the original file, say.
The code was very hard to explore, to hack,
and to extend in my opinion.
The documentation was not very clear,
and I sent wonderful messages to the
developer of lpegrex, and he was too busy to help me.
He answered very briefly. To be honest, I felt rejected.
I felt that I wasn't doing anything interesting, whatever.
So in 2022, I was trying to learn lpegrex
because I was thinking that it would solve my problems,
but it didn't. It didn't have the features that I needed,
and it was hard to extend and
how to explore and hard to debug.
I decided to rewrite it in a more hacker-friendly way,
in the sense that it was modular
and I could replace any part of the module from a REPL.
My version of it was called ELpeg1.lua.
I decided that in my version I wouldn't
have the part that receives a grammar
specified as a string and converts that to lpeg.
I would just have the backend part
that are the functions in lpeg
that let us specify powerful grammars.
Let me go back. Let me explain a bit about lpeg.
Lua has coercions. The `+` expects to receive two numbers.
If one of its arguments or both of them are strings,
it converts the strings to numbers.
So in this case here, `2 + "3"` returns the number 5.
This is the concatenation operator.
It expects to receive strings. In this case, it will
convert the number 2 to the string "2"
and the concatenation of these two things will be
"23". Oops, sorry, "23" as a string.
lpeg also has some coercions. I usually set these globals
to let me write my grammars in a very compact way.
So instead of lpeg.B, lpeg.C, etc.
I use globals like B, C, and so on.
And with these globals I can write
things like this, `C(1) * "_"`. And lpeg knows that lpeg.C,
sorry, it sort of expands these to lpeg.C,
but lpeg.C expects to receive an lpeg pattern
and 1 is not yet an lpeg pattern.
So it is coerced into an lpeg pattern by calling lpeg.P.
So this short thing here becomes equivalent to
`lpeg.C(lpeg.P(1))`. And the multiplication when at least one of
its arguments is an lpeg pattern, it expects to receive two lpeg patterns. And in this
case the one at the right is just a string, so it is coerced to an lpeg pattern by using
lpeg.P. With this idea we can sort of understand this comparison here. I mean, let me run it
again. This first part is very similar to a regular expression here at the left. And
when we apply this lpeg, sorry, this lua pattern to this subject here, the result is this thing
here, this thing, this thing and this thing. I'm going to call each one of these results
captures. So each of these things between parentheses captures a substring of the original
string and these captured substrings are returned in a certain order. Here is how to express
the same thing in lpeg. It's very cryptic, but it's a good way to understand some basic
operators of lpeg. I mean, we can look at the menu and understand what C, S and R do
and also exponentiation. And this strange thing here receives this string here, runs
a function that I have defined that converts it to an object of a certain class and that
class represents re patterns. So this thing is treated as a pattern for re.lua and it
is matched again the string and it returns the same thing as the other one. Also this
thing here also has a comparison with lpeg.rex, but these patterns are very, very trivial.
They don't do anything very strange. So let's go back and see what kinds of very strange
things there are. Here is the page of lpeg.rex at GitHub. Here is the documentation. It's
relatively brief. It explains lpeg.rex as being an extension of re.lua. So
it explains mainly the additional features. Here is a quick reference that explains only
the additional features. Some of these things I was able to understand by struggling a lot,
and some I wasn't able to even by spending several evenings trying to build examples.
This is something very nice. lpeg.rex comes with some example parsers. Here is
a parser that parses the lua grammar. I mean, this is the grammar for lua 5.4 at the end
of the reference manual. It's just this. And this is a kind of BNF, and this is the BNF
translated to the language of lpeg.rex. This thing uses many constructions that are
in re.lua and some extra constructions that are described here. With these examples,
I was able to understand some of these things here that are described here in the quick
reference, but not all. So I wasn't able to use lpeg.rex by itself because some things
didn't make much sense. I decided to re implement it in my own style because that
would be a way to map... at the very least, map what I understood and what I didn't and
learn one feature at a time, do comparisons and so on. Here I pointed to two features
of lpeg. One I said, oh great, this thing can be used to define grammars, even recursive
grammars and so on. And this is an "oh no" feature. One thing that didn't make any sense at all:
group captures. One thing that I did to understand group captures was to represent them as diagrams.
Of course in the beginning I was drawing these diagrams by hand, but then I realized that
I could use the bits of lpeg that I already knew to build a grammar that would parse a
little language and generate these diagrams in LaTeX. I was able to make this. In
this diagram here, this thing above the arrow is a piece of Lua code that specifies an lpeg
pattern. This thing here at the top is the string that is being matched and the things
below the under braces are the captures that each thing captures. For example, this under
brace here corresponds to this pattern here that parses a single character but doesn't
return any captures. This thing here parses a single B and doesn't return any captures.
This thing here parses a single character and captures it. And this thing here parses
the character D and captures it. And this other thing here that transforms this pattern
into another pattern returns first a capture with all the string that was parsed by this
pattern here and then all the captures returned by this thing here before the column. So this
was a way to build concrete examples for things that the lpeg manual was explaining in a very
terse way and it worked for me. Some things that were very mysterious started to make
sense and I started to have intelligent questions to ask in the mailing list. With that
I was able to understand what are group captures that receive a name. Well, let me explain
what this does. This thing here parses the empty string and returns this as a constant.
So this is something that doesn't exist in regular expressions. It parses nothing and
returns this as a capture. Then this thing here returns two constants here and parses
the empty string. And this thing here, D, converts the results of this thing here into
a group capture and stores it in the label D. And then here's another constant capture.
I realized that these things here were similar to how Lua specifies building lists.
When we build a table and we say that the first element of the table is here, this element
is put at the end of the table. After that we say D equals to say 42. We are putting
the 42 in the slot whose key is D. This was happening with lpeg captures but there was
something very strange. These group captures could hold more than one capture, more than
one value. So there was something between lists and tables. I started to use this notation
to explain in my notation what they were doing. Many things started to make sense, many mysterious
sentences in the manual started to make sense but some didn't. But at least I was able to
send some intelligent questions to the mailing list and the author of Lua and LPEG answered
some of them. He was not very happy about my questions. He told me that those diagrams
were a waste of time, the manual was perfectly clear and so on, whatever. So it was weird,
but I was able to understand lots of things from his answers. So this is a copy of one
of my messages, then there's another one, another one, some of the diagrams. Then he
complained about these diagrams, he said that these things here that look like table constructions
do not exist, whatever. Anyway, once I understood group captures, many features were very easy
to understand and I started to be able to use lpeg to build some very interesting things.
I was able to reproduce some of the features that I saw in lpegrex. Remember that this
is a syntax of Lua. Here I was able to understand how these things here were translated to lpeg
code, to lpeg patterns by using group captures in a certain way. I was able to implement
them in ELpeg1.lua. after some time I was able to use ELpeg1.lua to build grammars that
were able to parse arithmetical expressions with the right precedence. here's an example
in which I built the grammar step by step and I test the current grammar and I replace
a bit and then I test the new grammar and so on. You can see that the result is
always a tree that is drawn in a nice two dimensional way. At this point, these powers
here are returned as a list, as an operation power with several arguments here and then
I apply a kind of parsing combinator here that transforms these trees into other trees
and with these combinators here I can specify that the power is associative in a certain
direction, the division is associative in another direction, the minus uses the same
direction as a division and so on and they have the right precedences. So here are the
tests. So here's my file ELpeg.lua, it has several classes. Each class has tests after
it. I was able to implement something that lpeg.rex has that's called keywords, that's
very useful for parsing problems in programming languages. I was able to implement something
similar to the debugger, to the pack debugger that lpeg uses, but I was frustrated by some
limitations of the debugger and I implemented my own that is much better and let me show
something else. I was able to translate a good part of the Lua parser here to lpeg1.lua.
I haven't finished yet but I have most of the translation here and after having all
that I was able to build other grammars very quickly. Writing new parsers finally became
fun and here's one example that I showed in the beginning. If I remember correctly, I
took a figure from the Wikipedia, I don't have its link now, but I specify a grammar
that parses exactly the example that appears in the Wikipedia. So with my grammar, considering
that the top level entry is statement, when I parse this string here, the result is this
tree and I can do some operations on that. I can define how this thing is to be converted
into LaTeX. I can define other operations that convert trees into other trees and here's
some tests of these operations. This is what I showed in the beginning. I'm not going to
explain all the details of this thing now. This show converts this thing into LaTeX in
the way specified by these instructions here. And here's the result, the LaTeX result.
These diagrams here are generated by this file here that defines a simple grammar that
parses this thing here and then LaTeX it in a certain way and also tests to check if this
code here that is Lua code that generates an lpeg grammar, parses this subject here, and
returns the expected result. So this is the code that I wanted to show. I wanted to show
many more things, but I wasn't able to prepare them before the conference. And I hope that
soon, for some value of soon, I'll be able to create REPL based tutorials for LPG, G and
ELpeg1.Lua where Lpeg is something very famous. Here is a module of LPG. I could also do something
like this for ELpeg, and ELpeg1.lua is the thing that I wrote, the one that has tests in comments
and the tests usually generate trees and sometimes they generate tag code. So that's it, I wanted
to present much more, but I wasn't able to prepare it. So sorry, thanks, bye!