Hi, I'm Jeff Trull, and today I'm going to talk to you
about improving C++ compiler diagnostics
using overlays and other features from Emacs.
First an overview of my talk.
I'm going to cover what overlays are
and how you can use them in code,
then I'm going to talk about C++
and why its compiler errors can be so onerous.
Finally, we'll take that information
and build a new minor mode
using overlays and other Emacs features.
First of all, overlays.
What are they?
They are objects consisting of a buffer range
and a set of properties.
That means that they cover a region in a buffer.
The properties can be a certain set
of special property names,
in which case they can be used to cause
special effects in the buffer,
but they never change the underlying text.
You can use them for things like hiding things.
So, for example, overlays are working right now
in this window. `org-present`,
the technology I'm using for this presentation,
is hiding the asterisk before every headline,
as well as the things called emphasis markers;
that is, those things that make things look
monospaced for verbatim, or italic, or bold.
The special characters we use to mark off those sections
are also hidden by `org-present` using overlays.
But those things are still in the buffer
and they're still visible to code.
So if I run this little snippet of code down here,
it's going to go up to the headline "Overlays
and what they can do," and it's going to tell us
what's there in the buffer.
Let's go down and run this.
So according to this code, the contents of the buffer
to the left of the headline is a star in a space,
which means that even though we can't see that star,
it's still there, because it's hidden by an overlay.
And that's kind of the essence of what overlays are.
Let's do a simple overlay example.
We have some text on the right here,
which is a famous poem by William Carlos Williams,
which has been the subject of many memes.
Let's create an overlay that covers it.
I'll go down here and use this snippet of code here.
We'll go up to the top, and we'll mark everything
between `#+BEGIN_VERSE` and `#+END_VERSE`.
You can see we've created an overlay
from position 74 to 224.
Now we can take that overlay that we already created
and add a property, in this case a `face` property,
to change the appearance of the text.
This is a poem, and it's currently using
a face that is monospaced,
and so it looks like a computer program,
even though it's a poem.
I think it would be nicer to use something
with variable-width font, maybe with some serifs.
So let's give that a try.
Now you can see that the poem looks quite a bit different.
It looks more like what we'd see in a book.
We can also delete overlays.
So I've named this one.
So we can just go down and run `delete-overlay`
and get rid of it, and it'll go back to
the appearance it had before.
And there it is.
It's back to normal.
Now, if you're interested in changing all of the verses
inside an Org Mode file to a different face
or a different font family,
this isn't the way you'd really do it.
I'll just show you that real quick.
The right way is probably to change the `org-verse` face,
which is the face used for all of the verse blocks
inside your Org Mode file.
And so this is how you do it here:
`face-remap-add-relative`.
Let's give it a try.
It worked!
There are more advanced things that you can do
other than just changing fonts.
There's a whole long list of them in the manual,
but let's talk about the ones we're going to use today.
You can make text invisible, just like `org-present` did.
The simplest way is to set the `invisible` property to true,
so here's a code snippet that will do that.
What we're going to do is
go and find the word "plums" inside the poem,
and then we're going to make it invisible
by creating an overlay that covers it,
and then setting the invisible property to true.
Boom!
It's gone.
We've eaten the plums.
Visibility is a huge topic and very complicated.
There are powerful mechanisms for using it.
I suggest reading the manual
if you'd like to know more about that.
Another thing we can do with properties
is to add text either before or after an overlay.
Since we've made the word "plums" invisible,
or anything that you make invisible in the buffer,
if you add text then afterwards,
it looks like you've replaced the original words
with new words.
So let's add a property, a `before-string` property,
to the overlay that we used before
to make it seem as though we're eating cherries
instead of plums.
Boom!
There it is.
So that's how you can replace words using overlays.
You can also have custom properties
that you name and then use yourself.
For example, you can use it to mark regions in the buffer.
You can also use it to add information
to regions in the buffer for your own tracking
in a minor mode or something like that, which we will use.
Finally, two notes on properties.
We've been talking about overlay properties,
but there's also something called text properties.
Text properties are attached to text in a buffer.
When you copy that text, the properties come along with it.
If you modify the properties,
the buffer is considered modified.
Org Mode makes heavy use of text properties,
as we can see by running this little code snippet here,
which is going to tell us the properties
and the string attached
to the "Some poetry" headline on the right.
There's also some controversy regarding performance.
It may be that text properties
perform better than overlay properties,
so do some research
if you're going to make heavy use of them.
I prefer overlays because they're just easier to use.
C++ compiler output.
So my day job is C++ programmer,
and although I've been an Emacser for many years,
it can be a little bit of a chore dealing with errors.
The error messages that come out of the compiler
can be pretty hard to understand.
This has often been a barrier,
particularly for people who are new to C++.
So let's see what that's like.
I have an example
which is generously supplied by Ben Deane of Intel.
So let's see what it looks like
when you compile a C++ program
that has a difficult error in it.
Okay.
So you see we have a lot of fairly verbose messages.
The most verbose one I think is probably here.
This one here.
These are pretty bad.
I think there might be bigger ones.
Oh, yeah. Here we go.
Here's my favorite one.
You can see... Let's look for specialization... Basically,
this whole section of the buffer here,
that is specifying the specific types
that a function template was instantiated with.
And it's a lot there.
So if you're trying to figure out
what's wrong with your program
and you're looking at something like this,
it can be really, really hard to understand.
Okay.
Back to our presentation.
So it's often this way in C++
because we compose types from other types.
They can be long to begin with,
but then a couple of other factors come into play.
First of all, we can have default template arguments.
These are arguments you didn't write,
but that are implicitly there
and can sometimes refer
to the arguments that you did write,
which causes them to get a bit bigger,
such as these allocator arguments here and here.
Then there are type aliases.
For example, `std::string` here expands to
a type with three template arguments.
So you can imagine, when we combine
those two things together,
our simple vector of maps from strings to ints
becomes this humongous thing here, which...
Let's run the comparison.
Yeah.
So in summary, to properly understand an error
when you're a C++ programmer
requires knowing the exact types
that were supplied to your function.
And types are built recursively,
and therefore the types can--
the correct exact name for the type
can just be really huge
and have many levels and layers to it.
So when I was trying to understand
the things I'd done wrong,
especially when I was a newer C++ programmer,
but honestly still even recently,
if I was having a really intractable problem,
I would just copy the entire error message out,
stick it in the scratch buffer,
and then manually reformat it
so I could see what it was telling me
I'd actually called the function
or whatever it was with, the exact type.
I had to sit there
and go through the whole thing.
But there's a better way.
Now, anyway.
So what can Emacs do to help us with this problem?
First of all, if you think about a type name,
it's a lot like what we call S-expressions
or balanced expressions.
Lisp code itself is an S-expression.
It's basically things with parentheses
and little atoms or symbols in it,
or strings or numbers.
But parenthesized balanced expressions
are things that Emacs was actually built to deal with.
They were... I found an old manual from 1981,
and the two major modes that they recommended
or that they actually documented in the manual were
one, assembly language, and two, Lisp.
They mentioned that there were other modes,
but they didn't say anything about them.
So Lisp is something
with a really long history with Emacs.
Balanced expressions and manipulating them
and doing them efficiently
is just a thing that Emacs knows how to do,
and Emacs is good at it.
There's just a legacy
of algorithms and functions for doing it.
So we take types,
and we take the angle brackets in the types,
and we get the symbols right.
Then we can treat them
as though they were balanced expressions or S-expressions,
the same kind that Emacs is really good at handling.
Secondly, we can use overlays
to improve the readability of errors.
We can take long lines and break and indent them
using `before-string`s, so the same thing
I used to add "cherries" into the poem.
We can use that to insert new lines
followed by indentation
and produce a much nicer-looking listing of a type.
We can also use the `invisible` property
to hide unwanted detail.
Last of all, we can create a minor mode.
When we're compiling things in Emacs,
we often use `compilation-mode`.
`compilation-mode` allows you to install
compilation filters that run
when the compiler is producing output,
and at that time, then, we can add our overlays.
We can also add in minor-mode commands
that do whatever we want to the keymap.
In this case, we're going to show and hide
lower-level details interactively
so that we can see a simplified version
or a more detailed version of a type, depending on our needs.
First of all, parsing types as balanced expressions.
We need to be able to quickly locate
the boundaries and the contents
of parenthesized expressions,
or in this case, expressions in angle brackets.
We use a syntax table inside Emacs
to allow movement functions like `forward-list`
to jump between matching angle brackets.
By default, they're just parentheses.
First of all, let's look at our syntax table.
We're going to add here syntax entries
to handle angle brackets as though they were parentheses.
Then we have a lot of types
that have colons in them, and those are namespaces in C++.
By default, Emacs does not recognize them
as parts of symbols, so we're going to tell Emacs
that a colon is something called a symbol constituent,
that it can be part of a name.
Once we do that, then we can use our functions
like `forward-list`, `backward-word`,
all of the navigation and movement functions that we have
that do things, that do more complicated things
like S-expressions and so on,
can be used now with our angle brackets
and inside of our types.
The next thing we can do is
perform indent and fill with overlays.
We're going to use `before-string` properties
to break lines and create indentation
to make the output look a little better.
Today, we fill mostly text and we indent mostly code.
We fill text in order to prevent it
from running off the side of the right margin,
and we indent code to line up syntactic elements.
Back in the day, they had algorithms that could do both.
Those are what we're going to leverage.
We can use the `before-string` property
to insert a new line in the correct number of spaces
to emulate indentation.
As a simplified example, here's some code
that will indent 4 upon each open angle bracket.
Let's give it a try.
The next thing we're going to need to do is hide details.
So we have nested types, and the user is going to want to
be able to reveal lower-level or hide lower-level parts
of the nested type interactively
once we've already reformatted the error messages.
Let's see how we can do that using invisible properties.
The first thing we're going to do is
mark depths within the type.
When we're originally analyzing and formatting
and doing the indentation and the line breaks,
at the same time, we're going to go through
and mark the nested levels inside the type names,
just as this diagram shows.
So depth 1, for example, will be everything
inside the first level of angle brackets.
Depth 2 will be everything inside the second level,
and so on.
And then later on, when the users request it,
we can go and look at the depth that they've selected
and then mark those sections invisible.
Let's see how that might work.
First of all, let's delete the overlays
that we already have that created the indentation.
Now we're going to go and do that marking
with the custom depth properties here.
To prove that I didn't pull a fast one,
let's go and see what `describe-char` tells us
about the depths inside here.
Let's start here.
Okay, so inside this part here, `std::string`,
There are two overlays.
One of them is of depth 1, and the other is of depth 2,
which makes sense, because depth 1 is going to be
from about here to here,
and depth 2 is going to be from about here to this area.
So it's reasonable that there should be two,
and that's what we expect.
Now that we've marked the nested types with their depths,
let's experiment with hiding details.
This fragment of code takes a user-supplied depth,
in this case 2, and will hide,
based on those markings
that we've already made on the overlays,
the custom depth properties.
We'll take those and apply your requested level of detail.
So let's try it out.
Depth 2.
All right, that hid everything under the `std::map`,
so the deepest level.
If we make it 1, we should get a level higher than that.
So now level 1 and below are hidden.
Now if we put it back to 3, it should reveal everything.
So that's what we're going to use in our minor mode.
Let's have a demo.
We're going to revisit the initial example
with the minor mode installed.
Now we're going to have a compilation filter
that will run on every chunk of output
produced by the compiler.
It's going to add those overlays
with the line breaks and the indentation.
It's also going to add overlays
that mark up the nested types
with the depths for each region.
Let's add the hook for `tspew-mode`.
And now we can compile again.
All right, we can already see
that these things are formatted a little bit better
than they were before.
They're not all on one line.
Things are getting kind of lined up here.
Here's a good example.
And here's our big ugly one from before
with all the characters in it.
Let's try hiding some of this information.
We'll just slowly decrease the level of detail
and you can see how it works.
Over here, where there's these ellipses
next to string constant, the "..." there,
that's where we are starting to hide information
and go to the next level.
Hiding more, hiding more, hiding more.
Now we can go back and start adding it back.
You can see here now we just have about four layers,
which is a lot easier to understand.
And if we start understanding what it is
and we need more detail, we can just increase detail again.
And every time we increase or decrease detail,
it reformats so it still stays kind of consolidated
and nice looking.
Let's increase it a little bit more.
Okay, so you can see how that worked.
Let's go back to our presentation.
All right.
In conclusion, we saw how we could solve
a real problem for C++ programmers
by combining several Emacs features: overlays,
compilation mode extensions,
and balanced expression navigation using syntax tables.
Emacs is often compared unfavorably
to newer IDEs and editors with slicker user interfaces.
What Emacs has that they don't is powerful abstractions,
tons of libraries, and decades of work
by some of the luminaries in the field of software.
I think that this project would have been much harder to do
in a prettier but less powerful environment.
In short, there's plenty of hope for Emacs.
Thank you.