Hi! My name is Eduardo Ochs
and the title of this talk is: REPLs
in strange places - Lua, LateX, LPeg, LPegRex,
and TikZ. I'm the author of an Emacs
package called eev, and this is a talk
at the EmacsConf 2023, that is happening in
December 2023, at the internets.
This is one of the
examples of diagrams that we are
going to see - let me show how I generate
it... one second,
I have to use a smaller font here...
this is a file called ParseTree2.lua...
let me go back to this block of tests again...
and now if I run
this...
we get these outputs here at the
right, and then in this line here it
generates a PDF, and if I type f8 here it
shows the PDF in the lower right window.
Let me start by explaining
briefly what is eev.
First: it is something that
appeared by accident in the mid-90s - I
explained this story in my
presentation at the EmacsConf 2019...
it's a package... it's an Emacs
package that is part of ELPA... it has at
least 10 users - those are the ones
that I know by name...
eev means `emacs-execute-verbosely'...
eev is something that treats eval-last-sexp
as the central feature of Emacs...
eev blurs the distinction between
programmers and users, and it replaces
the slogan "users should not be forced to
see Lisp", that is something that Richard
Stallman told me once, by "users should see
Lisp instead of buttons" and "new users
should see Lisp in the first 5 minutes"...
I'm going to show
some examples of that soon.
Eev uses code in comments a lot,
and also tests in comments...
I changed my way of presenting it
and it became very REPL-centric
in the last few years, in the
sense that I start by explaining its
main features by its support for REPLs...
eev supposes
that we want to keep
executable notes of everything - I'm also
going to show examples of this in a
second... eev has lots of "videos for
people who hate videos", and it tries to
do everything with very little magic and
without black boxes - I'm going to explain
many of these things very soon.
This is a figure that that I'm going
to show in details soon, that is
about something important about Lua...
the font is very bad now, so let me
change the font... the figure is this one...
and...
what most people do when they
visit a file with something
interesting on it is that they just go
there and they set a bookmark there, or
they put the position in a register...
but I prefer to keep
links to everything that is interesting
as elisp hyperlinks. So, for example, this is
an elisp hyperlink to a file, that goes
to this anchor here, and to this string
after this anchor... this is a variant
that opens that file in the window
at the right -
here... and this is
a sexp that changes the font. I
have a command with a very short name
that does that, but I
prefer to keep that as a one-liner.
About the videos... we can see
the list of first-class videos of eev
by executing this, M-x find-1stclassvideos,
or by running this alias here, M-x 1c...
and then what we see is this...
the first sexp here
regenerates this buffer - so we can make a
mess here and then run this and the
original buffer is regenerated again in
a clean way...
each of these things here
opens a buffer with information about
a video... let me take a specific
example here... this video here is about
one of the ancestors of this talk, that
is a library that I wrote
for creating diagrams in LaTeX using
a package called Pict2e using REPLs...
anyway...
the thing is that if we
run a sexp like this one and we don't
have a local copy of the video eev
will try to download to the local copy -
and instead of doing that by asking
something like "do you want me
to download the local copy? Blah
blah blah blah blah..." it simply opens a
buffer like this, I mean, if we don't
have a local copy yet it will open a
buffer like this one, in which these
things here in comments are links to the
documentation... I mean, this thing here
explains the idea of local copies
of files from the internet...
there are more details here, and here...
and this is a script that we
can execute line by line, so instead of
this script being hidden behind the
button that we just press after a
question like "Do you want me to do
something blah blah blah? Yes or no?"
the script is visible here and we can
execute it step by step... it creates a
terminal with a shell here in the
right window, and when we type f8 in
one of these lines here the lines are
sent... (...) so this is going
to download a copy of the video... the
wget says that I already have a copy of
the video and its subtitles... and so on.
And after getting a copy of the video
we can run this sexp here and it displays
the video.
I said that eev has lots of
"videos for people who hate videos", and
the idea is that very few
people are going to watch the videos in
real time... and most of the people that
I know - or: most of the people that
are interested in eev in some
way... they are going to watch just
small sections of the video, and most of
the time they're just going to read the
subtitles of the video. So, for each
one of the videos we have a page
about the video... let me see if I
have internet here... yes. This is a
page...
and usually these pages have a link
to another page that
has all the subtitles of the
video... uh, wherever... in this one
it's not so visible...
but anyway, there are several
ways of accessing the subtitles of the
video, and one of the ways is by running
this sexp here,
that opens a file in Lua that is
what I use to generate the
subtitles.
Anyway... by the way, these things... each
one of these things here is a hyperlink
to a position of the video, so if I type
this the right way it goes to that
position. Anyway, let me go back...
also, the tutorials of eev... the
"intros" of eev, that start with "find-" and
end with "-intro", they have lots of blocks
that say "[Video links:]", like this one, and
these blocks have links to positions
in videos, and if we don't have a local
copy of the video yet the thing shows
us a script that lets us download the
local copy.
Anyway, I said that I was going
to explain what I mean by "magic" and
"black boxes".
this is something that I've been
trying to explain for a long time, and I
think that I got a very good explanation
about that in a video that I made
about something called eev-wconfig, that
is a tool for configuring eev on
Windows without "magic" - without buttons
that do things without explaining what
they're doing.
This is a part of the subtitles
of the video, let me read that...
eev-wconfig is an attempt to solve the
problem of how to install these things
on Windows both without magic and with
very little
magic. Remember this slogan: "any
sufficiently advanced technology is
indistinguishable from
magic". Here in this video I'm going to
use the term magic as a shorthand
for sufficiently advanced technology,
that is something that is complex and
non-obvious and that is
indistinguishable from magic in the
sense of being almost impossible to
understand. And I'm also going to use a
the term "black box" as a near-synonym for
magic, and sometimes the term
"black box" is more convenient even though
it's a bit longer - it has more
letters - because when I use the term
black box it invites us to use
expressions like "opening the black box",
and I'm going to use that
expression a lot.
Now let me try to explain what is...
sorry, let me change the font...
what is Lua. Lua is a minimalistic
language, in the sense of
"batteries not included"... it uses
associative tables for most of its data
structures...
and it is so minimalistic
that its default print function, when
we tell... when we create an associative
table and we ask it to print...
when we ask "print" to print an
associative table it just prints the
address of the table. Here are some
examples... here is a table, and when we
ask "print" to print it it just says
that it's the table at this address here.
So, one of things that that most
people do when they start using Lua is
that either they download a package with
a pretty-printing function or they write
their own pretty-printing functions. My
own pretty-printing function is called
PP, with upper case letters, and it works
like this...
and it prints associative tables
in a way like this. It says that for
the key 1 the the value associated to
it is 2, for the key 2 the value is
3, and for the key 3 the value is 5.
When I started using Lua one of my
favorite languages was also a language
that used associative tables a lot -
it was called Icon...
and I had to write my own
pretty-printing functions for Icon, so
I just had to port my pretty-printing
functions to Lua... and my first
version looked at something like this... it
just had some some global functions... lots
of them, actually...
and after a while I rewrote it, and I
rewrote it again, and again, and again, and
this is one of the versions of that,
is not even the default at this
point...
"Tos" is for "to string"...
and this is a demo...
it's very modular, so it's easy to replace
parts of it, or to toggle flags... and this
is an example. If I try to print the
table of methods for a certain
class... I will need a smaller font...
it prints the table like this, with the
names of the methods and then links to
the source code of the functions...
these links only make sense in Emacs and
in eev...
and when we run a link like this one...
it shows the source code in the
window at the right. So, for some
functions the source code is three lines,
for other ones it's one line... and
whatever. Anyway, let me go
back... Lua can be used in many different
styles... most people hate other people's
styles... when I started using it in the
year 2000 I learned most of the basic
language in a single day - it was very
similar to things that I was already
using... and then I rewrote the the mini-
language that I was using to
generate the HTML for my pages
in Lua... actually I had to rewrite it
many times, but the first version I
certainly did in my first weeks or first
months using Lua...
In the beginning I was just using
it for writing programs that either
didn't take any input at all - because
the input was already in the source file -
or that worked as Unix programs,
that would read files
and process these files in some way
and output something.
I mentioned the "basic language" here...
I only learned how to use closures,
metatables, and coroutines many years later...
in the beginning, when I started using Lua,
it didn't have a package manager...
it appeared later, it is called
Luarocks... it has had this package
manager for several years, most
of the rocks for Luarocks are poorly
documented and hacker-unfriendly,
so you can't rely just on the
documentation and you can't rely just on the
source code, because, I mean... if you are
a genius of course you can, but for
people who are either lazy, or dumb, or
whatever, like me, or unfocused...
the source code is hard to
understand and hard to tinker with.
Some rocks are excellent. The
best rocks are well documented
but they are hacker-unfriendly
in a sense that I hope that
I'll be able to explain soon.
The best rocks use local
variables and metatables a lot -
so if you are beginner
learning Lua you're not going to
understand what their source code do...
they use lots of dirty tricks.
Let me talk a bit about object
orientation in Lua. It can be done in
many ways...
the main book about Lua, called
"Programming in Lua", by one of the authors
of the language, Roberto Ierusalimschy,
presents several ways of doing
object orientation in Lua... I hated all
of these ways - and also the ways that I
tried from the rocks.
And then I wrote my own way
of doing object orientation in Lua... it's
very minimalistic, it's in this file here,
eoo.lua... the main code is just this five
lines here...
and here's an example of how it works.
Here we define the class Vector,
with some metamethods...
this metamethod here will tell Lua
what to do when the
user asks to add two vectors, this one
here tells Lua what to do when the user
asks Lua to convert a vector to a string,
and... whatever, this one is
something that I'm going to explain in a
second. So, here we create a vector with
these coordinates, 3 and 4... here we create
another Vector... if we "print" here then Lua
uses this function here, in the __tostring...
if we add the two vectors it uses this
function here, in the __add metamethod, and
if we run the method :norm...
it is defined here, in the table __index.
Anyway...
Even this thing being so small I used
to forget how its innards worked all
the time. Actually I always forget how
things work and I have to remember them
somehow... and I have to have
tricks for remembering, and tricks for
summarizing things, and diagrams, and so
on. And every time that I forgot how this
thing worked I went back to the
source code, and then I looked at the
diagrams... or, of course, in the
first times I had to draw the diagrams...
and I run the examples, and of course in
in the beginning I thought that the code
was clear and my examples were very
brief, and so I had to rewrite the
examples many times until they became,
let's say...
perfect.
I was saying that Lua can be used in
many ways, and in my way of using Lua - in
my favorite way - everything can be
inspected and modified from REPLs,
like we can do in Emacs and in SmallTalk,
or sort of. So, in my
favorite way of using Lua there's no
security at all, everything can be
changed at all times.
Of course most people hate that...
My init file has lots of classes... by the
way, instead of keeping many small files
with many things I put lots of stuff
in just one big init file.
My init file has lots of classes,
and lots of global functions, and
lots of cruft - and people hate that,
of course. This is an example...
this is the index at the top
of my init file,
the classes start here, and then
we have some functions, and
then we have functions that load
certain packages, and then we have... cruft.
Whatever.
Most people think that my style
of using Lua is dirty, and dangerous...
and they wouldn't touch my Lua code
with a 10 feet pole... but most of the
things that I'm going to present here in
this presentation are ideas that should
be easy to port to other environments
and other languages, especially the
diagrams... so the code is not so important.
Now let me talk a bit about LuaLaTeX,
that is LaTeX with a Lua interpreter
embedded inside, and two ways
of generating pictures in LaTeX: TikZ,
that is very famous, and Pict2e, that is not
very famous and that is very low level...
and I think that not many people use it.
I said before that when I
learned Lua I realized that it was
very good for writing little
languages. I was doing my PhD at the
time and typesetting the diagrams for
my PhD thesis was very boring, so
one of the things that I did was that I
created a little language for typesetting
the diagrams for me. it was
called Dednat because initially
it only generated diagrams for
Natural Deduction, and then it had
several versions...
these are the slides for my
presentation about Dednat6... "Dednat6 is
an extensible semi-preprocessor for
LuaLaTeX that understands diagrams in
ASCII art"... in the sense that when I have
a .tex file that has this, and when
Dednat6 is loaded,
when I give the right commands
Dednat6 interprets this block here as
something that defines this
diagram... oops, sorry, it interprets this
diagram here, this diagram in
comments here, as something that defines
a diagram called foo... a deduction called
foo, and it generates this code here...
so that we can just invoke
the definition of the
deduction by typing \ded{foo}.
And Dednat6 also
supports another language for typesetting
bidimensional diagrams with
arrows and stuff for category Theory and
blah blah blah... the specifications of
these diagrams look like this...
here is a... sorry, here is a very good
example, this is a huge diagram...
sorry, one second...
so, the source code that generates
this diagram here is just this thing at
the left, so it's very visual... we can
typeset the diagram in ASCII art here and
then in this part here we tell how
the nodes are to be joined, which
arrows have to to have annotations, and
so on...
and this language is extensible in
the sense that... uh, where's that...
here: comments that start with "%:"
are interpreted as
definitions for tree diagrams,
lines that start with "%D"
define 2D diagrams with arrows and
stuff, and lines that start with "%L"
contain blocks of Lua code
that we can use to extend the interpreter
on-the-fly...
anyway, here are some recent
examples of diagrams that I used
Dednat6 to typeset... this diagram
here was generated by this
specification here...
and this diagram here with the
curved arrows was generated by this
specification here.
So, Dednat6 was very easy to extend,
and at some point I started to use it
to generate diagrams using Pict2e -
mainly for the classes that I give
at the University... I teach mathematics and
whatever... in a bad place. Whatever...
Let me show an animation... here is a
diagram that I generated with Dednat6,
and it is a flip book animation, like... we
type PgUp and PgDn and we go
to the next page of the book and to the
previous page of the book...
and here is the source code that generates
that. This source code is not very visual,
so it's quite clumsy to edit that
diagram directly in the .tex file like
that...
These diagrams were inspired
by something called my Manim, that...
I forgot the name of the guy, but
it's a guy that makes many videos about
Mathematics, and he created this library
called Manim for generating his
animations, and other people adapted
his library to make it more accessible...
I tried to learn it, but
each animation, even an animation
that has very few frames... each
animation took ages to render, so it
wasn't fun... and animations in PDFs can
be rendered in seconds. So these
things were fun for me, because my laptop
is very very slow, and my Manim was not fun.
Anyway, writing code like this
inside a .tex file was not very
fun because it was hard to
debug... so in 2022 I started to play
with ways of generating these
diagrams from REPLs, and I found a
way for Pict2e and a way for TikZ...
each one of these ways became a video...
if you go to the list of first-class
videos of eev you're going to see
that there's a video about Pict2e here
here and a video about TikZ...
here you have some some information
like length, an explanation, etc...
and here are the pages for these videos.
My page about the video about Pict2e
looks like this, it has some diagrams...
whatever... and this one is much
nicer, and a lot of people
watched that video... I mean, I think
that 250 people watched it - for me that's
a million of people...
and this video is about how to
extract diagrams from the manual... from
the TikZ manual and how to run those
examples in a REPL and modify
them bit by bit... this is a a
screenshot... but let me go back.
At that point these things were just
prototypes, the code was not very nice...
and in this year I wrote... I was able
to unify those two ways of generating PDFs,
the one for TikZ and the one for Pict2e,
and I unified them with many other
things that generated diagrams.
The basis of these things is
something called Show2.lua... I'm not going
to show its details now, but its
extension that generates TikZ code
is just this, so we can specify a
diagram with just a block like this,
and then uh if we
run :show00() it returns a string
that is just the body... the inner
body of the .tex file, if we run this we
see the whole .tex file, and if we run
this we save the .tex file and we
compile the .tex file to generate a PDF...
and if we run this we show the PDF in
the lower right window.
And that's the same thing for all
my recent programs that generate
PDFs - they are all
integrated... here is the one that...
the basis for all my modules that generate
diagrams with Pict2e...
its demos are not very interesting,
so let me show some demos of
extensions that do interesting things...
so, this is a diagram that I created
by editing it in a REPL...
I create several Pict objects here...
and if I execute this it
compiles an object, generates a PDF, and
if I tap this... here is the PDF.
And if I just ask Lua to
display what is "pux", here,
it shows the source code in Pict2e
of the diagram... and the
nice thing is that it is indented, so
it's easy to debug the Pict2e code.
If anyone is interested the
module that does the tricks for
indentation is very easy to understand...
it has lots of tests and test blocks,
and I think that its data
structures are easy to understand.
Anyway... here is another
example. The :show() is
here... it generates a 3D diagram.
Now let me talk about parsers and
REPLs in VERY strange places... I mean,
using REPLs to build parsers step by step
and" replacing parts by more complex
parts. So, I said that Lua is very
minimalistic, and everybody knows that
implementations of regular expressions
are big and complex..
so, instead of coming with
full regular expressions Lua comes with
something called "patterns" and a
library function called "string.match".
Here is
a copy of the part of the manual that
explains the syntax... a part of the
syntax of of patterns... here's how
string.match is described in the
manual - it's just this... "looks for
the first match of pattern in the string
as blah blah blah"... and then we have to
go to the other section of the menual
that explains patterns.
Lua patterns are so simple,
so limited, that they don't even
have the the alternation operator...
here is how it is described in the
elisp manual -
backslash-pipe specifies
an alternative, blah blah blah.
When we want to to build more
complex... regular expressions,
patterns, grammars, etc... we have to use
an external library for that... no,
sorry, a library that is external
but that was written by one of the
authors of Lua itself. This library
is called Lpeg, and its manual says...
"Lpeg is a new pattern matching library for
Lua based on Parsing Expression Grammars
(PEGs)". The manual is very terse, I
found it incredibly hard to read... it
doesn't have any diagrams - it has some
examples, though... and the Lua Wiki
has a big page called Lpeg Tutorial
with lots of examples...
but it it also doesn't have
diagrams and I found some things
incredibly hard to understand.
For example, this is something that is in
the the manual of Lpeg that I saw and I
thought: "Wow, great! This makes all sense
and is going to be very useful!"...
it's a way to to build
grammars that can be recursive,
and they sort of can encode BNF
grammars... we just have to translate the
BNF a bit to get rid of some
recursions and to translate them to
something else.
And the manual also has some things
that I thought: "Oh, no! I don't have any
idea of what this thing does"... and in fact
I saw these things for the first
time more than 10 years ago and they
only started to make sense one year ago.
One example is group captures.
Lpeg also comes with a
module called the Re module... let me
pronounce as it in Portuguese - the Re
module... its manual says: "The Re
module (provided by the file re.lua in the
distribution) supports a somewhat conventional
regular expression syntax for pattern usage
within lpeg"... and
this is a quick reference... this
thing is very brief, it has some nice
examples but it's hard to understand anyway...
and here are some comments about
my attempts to learn Re.lua. This is
a class... in this case it's a very small
class... this file implements a :pm()
method - I'm going to show examples of
other :pm() methods very soon - so, this is
a :pm() method for Re.lua that lets us
compare the syntax of Lua patterns, Lpeg,
and Re... let's see this example here... so,
if we run this it loads my version of
lpeg... no, sorry, my version of lpegrex...
and it shows that when we apply
the :pm() method to this Lua pattern, this
lpeg pattern, and this Re pattern
they all give the same results. So we can
use this thing... this kind of thing here
to show how to translate from Lua
patterns, that are familiar because
they're similar to regular expressions,
only weaker...
to lpeg, that is super weird
and to Re, that is not so weird.
Anyway, the comment says that in 2012
I had a project that needed a
precedence passer that could parse
arithmetical expressions with the right
precedences... and at that point I was
still struggling with pure lpeg, and I
couldn't do much with it, so I tried to
learn Re.lua instead, and I wrote this old
class here...
that allowed me to use a preprocessor
on patterns for Lua. And the thing is that
with this preprocessor I could
specify precedence grammars using this
thing here, that worked, but was super
clumsy... and I gave up after a few attempts.
and in 2022 I heard about something
called lpegrex,
that was a... a kind of extension or Re,
and it was much more powerful than re.lua,
but after a while I realized that it
had the same defects as re.lua...
and let me explain that, because
it has all to do with the things about
black boxes and magic that I told in the
beginning. Both... I mean, sorry, neither
re.lua or lpegrex had some features that
I needed... they didn't let us explore...
sorry, they received a pattern that was
specified as a string, and it converted
that into an lpeg pattern, but it didn't
let us explore the the lpeg patterns
that it generated...
their code was written in a way
that was REPL-unfriendly - I
couldn't modify parts of the code
bit by bit in a REPL and try to change
the code without changing the
original file... the code was very
hard to explore, to hack, and to extend -
in my opinion... the documentation was not
very clear... and I sent one or two messages
to the the developer of lpegrex and...
he was too busy to help me. He
answered it very briefly, and, uh, to be
honest I felt... rejected. I felt that I
wasn't doing anything interesting...
whatever, whatever...
So, in 2022 I was trying to learn lpegrex
because I was thinking that it would
solve my problems - but it didn't...
it didn't have the features that I needed,
it was hard to extend, hard to explore,
and hard to debug, and I
decided to rewrite it in a more
hacker-friendly way - in the sense that...
was modular, and I could replace any
part of the module from a REPL...
My version of it was called ELpeg1.lua...
and I decided that in my version I
wouldn't have the part that
receives a grammar specified as a string
and converts that to lpeg... I would
just have the backend part, that are the
functions in lpeg that let us specify
powerful grammars.
Let me go back. Let me explain a
bit about lpeg... Lua has
coercions: the + expects to receive
true numbers, and if one of its arguments,
or both of them, are strings, it converts
the string... the strings to numbers so in
this case here, 2+"3",
it returns the number 5,
and this is the concatenation
operator... it expects to receive
strings, so in this case it will
convert the number 2 to the string "2",
and the concatenation of thes two
things will be 23... oops, sorry, "23"
as a string.
Lpeg also has some coercions.
I usually set these
globals to let me write my grammars
in a very compact way, so instead
of lpeg.B, lpeg.C, etc I use these globals,
like uppercase B, uppercase C, and so on...
and with these globals I can write
things like this: C(1)*"_"...
and lpeg knows that lpeg.C...
it sort of expands this to lpeg.C,
but lpeg.C expects to receive
an lpeg pattern, and 1 is not yet an
lpeg pattern, so it is coerced into an
lpeg pattern by calling lpeg.P,
so this short thing here becomes
equivalent to lpeg.C(lpeg.P(1)), and the
multiplication, when at least one of its
arguments is an lpeg pattern... it expects
to receive two lpeg patterns, and in
this case the one at the right is
just a string, so it is coerced to an lpeg
pattern by using lpeg.P.
With this idea we can sort of
understand the comparison here. I mean,
let me run it again... this first part is
very similar to a regular expression
here at the left...
and when we apply this... Lua pattern
to this subject here the result
is this thing here, this thing, this
thing and this thing... I'm going to
call each one of these results
"captures", so each of these things
between parentheses "captures" a substring
of the original string and these
captured substrings are returned in a
certain order. Here is how to express the
same thing in lpeg...
it's very cryptic but it's a
good way to understand the some basic
operators of lpeg, I mean we can look at
the manual and understand and
what C, S and R do, and also
exponentiation... and this strange thing
here receives this string here, runs
a function that I have defined, that
converts it to an object of a certain
class, and that class
represents Re patterns, so this thing
is treated as a pattern for re.lua,
and it is matched against the string,
and it returns the same thing as the
other one.
Also, this thing here also has a
comparison with lpegrex, but these
patterns are very trivial, they
don't do anything very strange...
so let's go back and see what
kinds of very strange things there are.
Here is the page of lpegrex at github,
here's the documentation...
it's relatively brief,
it explains lpegrex as being an
extension of Re.lua, so it explains
mainly the additional features... here is a
quick reference that explains only the
additional features...
some of the these things
I was able to understand
by struggling a lot, and some I wasn't
able to even by spending several evenings
try to to build examples...
and this is something very nice. Lpegrex
comes with some example parsers... and
here is a parser that parses the Lua
grammar - I mean, this is the the grammar
for Lua 5.4 at the end of the
reference manual... it's just this... this
is in a kind of BNF, and this is the BNF
translated
to the language of lpegrex, so this
thing uses many constructions that are
in re.lua and some extra constructions that
are described here... and with these
examples I was able to to understand
some of the...
of these things here that are
described here in the quick
reference - but not all.
So, I wasn't able to use lpegrex
by itself, because some things didn't
make much sense, and I decided to
reimplement it in my own style,
because that would be a way to map...
to at the very least map what I had
understood and what I didn't, learn
one feature at a time, do comparisons, and
so on.
Here I pointed to two features of lpeg...
in one I said "Oh, great! This thing can
be used to to define grammars, even
recursive grammars", and so on...
and this is an "Oh, no!" feature - one
thing that didn't make any sense at all...
group captures. One thing that I did to
understand group captures was to
represent them as diagrams. Of course in
the beginning I was drawing these
diagrams by hand, but then I realized
that I could use the bits of lpeg
that I already knew to build a grammar
that would parse a little language and
generate these diagrams in LaTeX, and I was
able to make this.
In this diagram here
this thing above the arrow is Lua code...
a piece of Lua code that
specifies an lpeg pattern... this
thing here at the top is the string that
is being matched, and the things below
the underbraces are the captures that
each thing... sorry, that each thing
captures.
For example, this underbrace here
corresponds to this pattern here,
that parses a single character but
doesn't return any captures, this thing
here parses a single "b" and doesn't
return any captures, this thing here
parses a single character and captures
it, and this thing here parses the
character "d" and captures it... and this
other thing here transforms this
pattern into another pattern...
returns first a capture with all
the string that was parsed by this
pattern here, and then all the captures
returned by this thing here before
the ":".
So, this was a way to build
concrete examples for things that the
lpag manual was explaining in a very terse
way, and it worked for me - some things
that were very
mysterious started to make sense, and I
started to have intelligent questions
to ask in the mailing list.
And with that I was able to
understand what are group captures,
and group captures that receive a name...
Well, let me explain what this does.
This thing here captures... sorry, parses
the empty string and returns this as a
constant... so, this is something that
doesn't exist in regular expressions...
it parses nothing and
returns this as a capture... then this
thing here returns these two
constants here, and parses the empty
string, and this thing here converts
the results of this thing here into a
group capture, and stores it in the label
"d"... and then here's another constant
capture.
And I realized that these things
here were similar to how Lua
specifies building lists...
when we build... sorry, tables. When
we build a table, and we say that the
first element of the table is here, this
element is put at the end of the table...
when after the that would say d=42...
we are putting the 42
in the the slot whose key is "d".
This was happening with lpeg captures,
but there was something very strange...
these group captures could hold
more than one capture - more than one
value... so there was something between
lists and tables. I started to use this
notation to...
explain in my notation what they
were doing... many things started
to make sense, many mysterious
sentences in the manual started to
make sense... but some didn't...
but at least I was able to send
some intelligent questions to the
mailing lis,t and the author of Lua and
lpeg answered some of them...
he was not very happy about my
questions - he... told me that those
diagrams were a waste of time, the
manual was perfectly clear, and so on...
whatever - but I was able to...
so, it was weird, but I was able to
understand lots of things from his
answers. This is a copy of one of
my messages, then there's another one,
another one, some of them had diagrams...
then he complained about these diagrams,
he said that these things here, that look
like table constructors, "do not exist"...
whatever... anyway, once I understood
group captures many features
were very easy to understand
and I started to be able to use lpeg to
to build some very interesting things...
I was able to reproduce some
of the features that I saw in lpegrex -
remember that this... where is that?
this is the syntax of Lua... here -
I was able to understand
how these things here were translated to
lpeg code... to lpeg patterns
by using group captures in a certain
way... I was able to implement them
in ELpeg1.lua...
and after some time I was able to use
ELpeg1.lua to build grammars that
were able to parse
arithmetical expressions with the
right precedence... and here's an example
in which I built the grammar step by step...
and I test the current grammar, and I
replace a bit, and then I test the new
grammar and so on...
and you can see that the result is
always a tree that is drawn in a
nice two dimensional way...
At this point these powers here
are returned as a list,
as an operation "pow"
with several arguments, here... and then
I apply a kind of parsing combinator,
here... that transforms these trees into
other trees and with these combinators
here I can specify that the "^" is
associative in a certain direction...
that the "/" is associative in
another direction... the "-" uses
the same direction as a the "/",
and so on... and they have the
right precedences.
So, here are the tests...
here is my file ELpeg1.lua... it has
several classes, each class has tests
after it...
I was able to implement something
that lpegrex has, that is called
"keywords", that is very useful for parsing
programs in programming languages...
I was able to implement something
similar to the debugger... to the
lpeg debugger lpeg uses... I was
frustrated by some limitations of
the lpeg debugger, and I implemented
my own that is, uh... much better!...
Let me show something else... I was
able to translate a good part of the
Lua parser, here, to ELpeg1.lua... I haven't
finished yet, but I have most of the
the translation here...
and after having all that I was able to
build other grammars very quickly...
writing new parsers finally became fun.
And here's one example that I showed in the
beginning.
If I remember correctly...
I took a figure from the Wikipedia...
I don't have its link now...
but I specified a grammar that parses
exactly the example that appears
in the Wikipedia...
so, with my grammar, considering that
the top level entry is "Stmt", when I
parse this string here
the result is this tree...
and I can do some operations on that,
I can define how this thing is to be
converted into LaTeX,
I can define other operations
that convert trees into other trees, and
here are some tests of these operations...
This is what I showed in the beginning...
I'm not going to explain all the details
of this thing now...
this :show() converts this thing
into LaTeX in the way specified by these
instructions here, that says that...
well, whatever...
and here's the result - the LaTeXed result...
and these diagrams here are generated by
this file here, that defines a simple
grammar that parses this thing here,
and then LaTeXes it in a certain way, and
and also tests to check if this code here...
this Lua code that generates an lpeg grammar...
parses this subject here and
returns the expected result...
So: this is the code that I
wanted to show. I wanted to show many
more things but I wasn't able to prepare
them before the conference... and I hope
that soon - for some value of "soon" -
I'll be able to create REPL-based
tutorials for lpeg, Re, and ELpeg1.lua...
where lpeg is something very famous,
Re is a module of lpeg...
I could also do something like this
for lpegrex... and ELpeg1.lua is
the thing that I wrote, the one that
has test in comments, and the tests
usually generate trees, and sometimes
they generate TeX code.
Yeah, so that's it! I wanted to
present much more but I wasn't able to
prepare it... so: sorry, thanks, bye! =)