Thank you for joining me today. I'm Sameer Pradhan
from the Linguistic Data Consortium
at the University of Pennsylvania
and founder of cemantix.org .
Today we'll be addressing research
in computational linguistics,
also known as natural language processing
a sub area of artificial intelligence
with a focus on modeling and predicting
complex linguistic structures from various signals.
The work we present is limited to text and speech signals.
but it can be extended to other signals.
We propose an architecture,
and we call it GRAIL, which allows
the representation and aggregation
of such rich structures in a systematic fashion.
I'll demonstrate a proof of concept
for representing and manipulating data and annotations
for the specific purpose of building
machine learning models that simulate understanding.
These technologies have the potential for impact
in almost every conceivable field
that generates and uses data.
We process human language
when our brains receive and assimilate
various signals which are then manipulated
and interpreted within a syntactic structure.
it's a complex process that I have simplified here
for the purpose of comparison to machine learning.
Recent machine learning models tend to require
a large amount of raw, naturally occurring data
and a varying amount of manually enriched data,
commonly known as "annotations".
Owing to the complex and numerous nature
of linguistic phenomena, we have most often used
a divide and conquer approach.
The strength of this approach is that it allows us
to focus on a single, or perhaps a few related
linguistic phenomena.
The weaknesses are the universe of these phenomena
keep expanding, as language itself
evolves and changes over time,
and second, this approach requires an additional task
of aggregating the interpretations,
creating more opportunities for computer error.
Our challenge, then, is to find the sweet spot
that allows us to encode complex information
without the use of manual annotation,
or without the additional task of aggregation by computers.
So what do I mean by "annotation"?
In this talk the word annotation refers to
the manual assignment of certain attributes
to portions of a signal which is necessary
to perform the end task.
For example, in order for the algorithm
to accurately interpret a pronoun,
it needs to know that pronoun,
what that pronoun refers back to.
We may find this task trivial, however,
current algorithms repeatedly fail in this task.
So the complexities of understanding
in computational linguistics require annotation.
The world annotation itself is a useful example,
because it also reminds us
that words have multiple meetings
as annotation itself does—
just as I needed to define it in this context,
so that my message won't be misinterpreted.
So, too, must annotators do this for algorithms
through the manual intervention.
Learning from raw data
(commonly known as unsupervised learning)
poses limitations for machine learning.
As I described, modeling complex phenomena
need manual annotations.
The learning algorithm uses these annotations
as examples to build statistical models.
This is called supervised learning.
Without going into too much detail,
I'll simply note that the recent popularity
of the concept of deep learning
is that evolutionary step
where we have learned to train models
using trillions of parameters in ways that they can
learn richer hierarchical structures
from very large amounts of annotate, unannotated data.
These models can then be fine-tuned,
using varying amounts of annotated examples
depending on the complexity of the task
to generate better predictions.
As you might imagine, manually annotating
complex, linguistic phenomena
can be very specific, labor-intensive task.
For example, imagine if we were
to go back through this presentation
and connect all the pronouns
with the nouns to which they refer.
Even for a short 18 min presentation,
this would require hundreds of annotations.
The models we build are only as good
as the quality of the annotations we make.
We need guidelines
that ensure that the annotations are done
by at least two humans who have substantial agreement
with each other in their interpretations.
We know that if we try to trade a model using annotations
that are very subjective, or have more noise,
we will receive poor predictions.
Additionally, there is the concern of introducing
various unexpected biases into one's models.
So annotation is really both an art and a science.
In the remaining time,
we will turn to two fundamental questions.
First, how can we develop a unified representation
of data and annotations
that encompasses arbitrary levels of linguistic information?
There is a long history of attempting to answer
this first question.
This history is documented in our recent article,
and you can refer to that article.
It will be on the website.
It is as if we, as a community,
have been searching for our own Holy Grail.
The second question we will pose is
what role might Emacs, along with Org mode,
play in this process?
Well, the solution itself may not be tied to Emacs.
Emacs has built in capabilities
that could be useful for evaluating potential solutions.
It's also one of the most extensively documented
pieces of software and the most customizable
piece of software that I have ever come across,
and many would agree with that.
In order to approach this second question,
we turn to the complex structure of language itself.
At first glance, language appears to us
as a series of words.
Words form sentences, sentences form paragraphs,
and paragraphs form completed text.
If this was a sufficient description
of the complexity of language,
all of us would be able to speak and read
at least ten different languages.
We know it is much more complex than this.
There is a rich, underlying recursive tree structure--
in fact, many possible tree structures
which makes a particular sequence meaningful
and many others meaningless.
One of the better understood tree structures
is the syntactic structure.
While natural language
has rich ambiguities and complexities,
programming languages are designed to be parsed
and interpreted deterministically.
Emacs has been used for programming very effectively.
So there is a potential for using Emacs
as a tool for annotation.
This would significantly improve our current set of tools.
It is important to note that most of the annotation tools
that have been developed over the past few decades
have relied on graphical interfaces,
even those used for enriching textual information.
Most of the tools in current use
are designed for a end user to add very specific,
very restricted information.
We have not really made use of the potential
that an editor or a rich editing environment like Emacs
can add to the mix.
Emacs has long enabled the editing of, the manipulation of
complex embedded tree structures abundant in source code.
So it's not difficult to imagine that it would have
many capabilities that we we need
to represent actual language.
In fact, it already does that with features
that allow us to quickly navigate
through sentences and paragraphs,
and we don't need a few key strokes.
Or to add various text properties to text spans
to create overlays, to name but a few.
Emacs figured out this way to handle Unicode,
so you don't even have to worry about the complexity
of managing multiple languages.
It's built into Emacs. In fact, this is not the first time
Emacs has been used for linguistic analysis.
One of the breakthrough moments in language,
natural language processing was the creation
of manually created syntactic trees
for a 1 million word collection
of Wall Street Journal articles.
This was else around 1992
before Java or graphical interfaces were common.
The tool that was used to create that corpus was Emacs.
It was created at UPenn, and is famously known as
the Penn Treebank. '92 was about when
the Linguistic Data Consortium was also established,
and it's been about 30 years
that it has been creating various
language-related resources.
Org mode--in particular, the outlining mode,
or rather the enhanced form of outlining mode--
allows us to create rich outlines,
attaching properties to nodes,
and provides commands for easily customizing
sorting of various pieces of information
as per one's requirement.
This can also be a very useful tool.
This enhanced form of outline-mode adds more power to Emacs.
It provides commands for easily customizing
and filtering information,
while at the same time hiding unnecessary context.
It also allows structural editing.
This can be a very useful tool to enrich corpora
where we are focusing on limited amount of phenomena.
The two together allow us to create
a rich representation
that can simultaneously capture multiple possible sequences,
capture details necessary to recreate the original source,
allow the creation of hierarchical representation,
provide structural editing capabilities
that can take advantage of the concept of inheritance
within the tree structure.
Together they allow local manipulations of structures,
thereby minimizing data coupling.
The concept of tags in Org mode
complement the hierarchy part.
Hierarchies can be very rigid,
but to tags on hierarchies,
we can have a multifaceted representations.
As a matter of fact, Org mode has the ability for the tags
to have their own hierarchical structure
which further enhances the representational power.
All of this can be done as a sequence
of mostly functional data transformations,
because most of the capabilities
can be configured and customized.
It is not necessary to do everything at once.
Instead, it allows us to incrementally increase
the complexity of the representation.
Finally, all of this can be done
in plain-text representation
which comes with its own advantages.
Now let's take a simple example.
This is a a short video that I'll play.
The sentence is "I saw the moon with a telescope,"
and let's just make a copy of the sentence.
What we can do now is to see:
what does this sentence comprise?
It has a noun phrase "I,"
followed by a word "saw."
Then "the moon" is another noun phrase,
and "with the telescope" is a prepositional phrase.
Now one thing that you might remember,
from grammar school or syntax is that
there is a syntactic structure.
And if you in this particular case--
because we know that the moon is not typically
something that can hold the telescope,
that the seeing must be done by me or "I,"
and the telescope must be in my hand,
or "I" am viewing the moon with a telescope.
However, it is possible that in a different context
the moon could be referring to an animated character
in a animated series, and could actually hold the telescope.
And this is one of the most--
the oldest and one of the most--
and in that case the situation might be
that I'm actually seeing the moon holding a telescope...
I mean. The moon is holding the telescope,
and I'm just seeing the moon holding the telescope.
Complex linguistic ambiguity or linguistic
phenomena that requires world knowledge,
and it's called the PP attachment problem
where the propositional phrase attachment
can be ambiguous, and various different contextual cues
have to be used to resolve the ambiguity.
So in this case, as you saw,
both the readings are technically true,
depending on different contexts.
So one thing we could do is just
to cut the tree and duplicate it,
and then let's create another node
and call it an "OR" node.
And because we are saying,
this is one of the two interpretations.
Now let's call one interpretation "a",
and that interpretation essentially
is this child of that node "a"
and that says that the moon
is holding the telescope.
Now we can create another representation "b"
where we capture the other interpretation,
where this, the act, the moon or--I am actually
holding the telescope,
and watching the moon using it.
So now we have two separate interpretations
in the same structure,
and all we do--we're able to do is with this,
with very quick key strokes now...
While we are at it, let's add another interesting thing,
this node that represents "I":
"He." It can be "She".
It can be "the children," or it can be "The people".
Basically, any entity that has the capability to "see"
can be substituted in this particular node.
Let's see what we have here now.
We just are getting sort of a zoom view
of the entire structure, what we created,
and essentially you can see that
by just, you know, using a few keystrokes,
we were able to capture two different interpretations
of a a simple sentence,
and they are also able to add
these alternate pieces of information
that could help machine learning algorithms
generalize better.
All right.
Now, let's look at the next thing. So in a sense,
we can use this power of functional data structures
to represent various potentially conflicting
and structural readings of that piece of text.
In addition to that, we can also create more texts,
each with different structure,
and have them all in the same place.
This allows us to address the interpretation
of a static sentence that might be occurring in the world,
while simultaneously inserting information
that would add more value to it.
This makes the enrichment process also very efficient.
Additionally, we can envision
a power user of the future, or present,
who can not only annotate a span,
but also edit the information in situ
in a way that would help machine algorithms
generalize better by making more efficient use
of the annotations.
So together, Emacs and Org mode can speed up
the enrichment of the signals
in a way that allows us
to focus on certain aspects and ignore others.
Extremely complex landscape of rich structures
can be captured consistently,
in a fashion that allows computers
to understand language.
We can then build tools to enhance the tasks
that we do in our everyday life.
YAMR is acronym, or the file's type or specification
that we are creating to capture this new
rich representation.
We'll now look at an example of spontaneous speech
that occurs in spoken conversations.
Conversations frequently contain errors in speech:
interruptions, disfluencies,
verbal sounds such as cough or laugh,
and other noises.
In this sense, spontaneous speech is similar
to a functional data stream.
We cannot take back words that come out of our mouth,
but we tend to make mistakes, and we correct ourselves
as soon as we realize that we have made--
we have misspoken.
This process manifests through a combination
of a handful of mechanisms, including immediate correction
after an error, and we do this unconsciously.
Computers, on the other hand,
must be taught to understand these cases.
What we see here is a example document or outline,
or part of a document that illustrates
various different aspects of the representation.
We don't have a lot of time to go through
many of the details.
I would highly encourage you to play a...
I'm planning on making some videos, or ascii cinemas,
that I'll be posting, and you can,
if you're interested, you can go through those.
The idea here is to try to do
a slightly more complex use case.
But again, given the time constraint
and the amount of information
that needs to fit in the screen,
this may not be very informative,
but at least it will give you some idea
of what can be possible.
And in this particular case, what you're seeing is that
there is a sentence which is "What I'm I'm tr- telling now."
Essentially, there is a repetition of the word "I'm",
and then there is a partial word
that somebody tried to say "telling",
but started saying "tr-", and then corrected themselves
and said, "telling now."
So in this case, you see, we can capture words
or a sequence of words, or a sequence of tokens.
One thing to... An interesting thing to note is that in NLP,
sometimes we have to break typically
words that don't have spaces into two separate words,
especially contractions like "I'm",
so the syntactic parser needs needs two separate nodes.
But anyway, so I'll... You can see that here.
The other... This view. What this view shows is that
with each of the nodes in the sentence
or in the representation,
you can have a lot of different properties
that you can attach to them,
and these properties are typically hidden,
like you saw in the earlier slide.
But you can make use of all these properties
to do various kind of searches and filtering.
And on the right hand side here--
this is actually not a legitimate syntax--
but on the right are descriptions
of what each of these represent.
All the information is also available in the article.
You can see there... It shows how much rich context
you can capture.
This is just a closer snapshot
of the properties on the node,
and you can see we can have things like,
whether the word is a token or not,
or that it's incomplete, whether some words
might want to be filtered out for parsing,
and we can say this: PARSE_IGNORE,
or some words or restart markers...
We can mark, add a RESTART_MARKER, or sometimes,
some of these might have durations. Things like that.
The other fascinating thing of this representation
is that you can edit properties in the column view.
And suddenly, you have this tabular data structure
combined with the hierarchical data structure.
And as you can--you may not be able to see it here,
but what has also happened here is that
some of the tags have been inherited
from the earlier nodes.
And so you get a much fuller picture of things.
Essentially you, can filter out things
that you want to process,
process them, and then reintegrate it into the whole.
So, in conclusion, today we have proposed and demonstrated
the use of an architecture (GRAIL),
which allows the representation, manipulation,
and aggregation of rich linguistic structures
in a systematic fashion.
We have shown how GRAIL advances the tools
available for building machine learning models
that simulate understanding.
Thank you very much for your time and attention today.
My contact information is on this slide.
If you are interested in an additional example
that demonstrates the representation
of speech and written text together,
please continue watching.
Otherwise, you can stop here
and enjoy the rest of the conference.
Welcome to the bonus material.
I'm glad for those of you who are stuck around.
We are now going to examine an instance
of speech and text signals together
that produce multiple layers.
When we have--when we take a spoken conversation
and use the best language processing models available,
we suddenly hit a hard spot
because the tools are typically not trained
to filter out the unnecessary cruft
in order to automatically interpret
the part of what is being said
that is actually relevant.
Over time, language researchers
have created many interdependent layers of annotations,
yet the assumptions underlying them are seldom the same.
Piecing together such related but disjointed annotations
on their predictions poses a huge challenge.
This is another place where we can leverage
the data model underlying the Emacs editor,
along with the structural editing capabilities
of Org mode to improve current tools.
Let's take this very simple looking utterance.
"Um \{lipsmack\} and that's it. (\{laugh\})"
Looks like the person-- so this is--
what you are seeing here is a transcript of an audio signal
that has a lip smack and a laugh as part of it,
and there is also a "Um" like interjection.
So this has a few interesting noises
and specific things that would be illustrative
of what we are going to, how we are going to represent it.
Okay. So let's say you want to have
a syntactic analysis of this sentence or utterance.
One common technique people use
is just to remove the cruft, and, you know,
write some rules, clean up the utterance,
make it look like it's proper English,
and then, you know, tokenize it,
and basically just use standard tools to process it.
But in that process, they end up eliminating
valid pieces of signal that have meaning to others
studying different phenomena of language.
Here you have the rich transcript,
the input to the syntactic parser.
As you can see, there is a little tokenization happening
where you'll be inserting space
between "that" and the contracted is ('s),
and between the period and the "it,"
and the output of the syntactic parser is shown below.
which (surprise) is a S-expression.
Like I said, the parse trees, when they were created,
and still largely when they are used, are S-expressions,
and most of the viewers here
should not have much problem reading it.
You can see this tree structure
of this syntactic parser here.
Now let's say you want to integrate
phonetic information or phonetic layer
that's in the audio signal, and do some analysis.
Now, it would need you to do a few-- take a few steps.
First, you would need to align the transcript
with the audio. This process is called forced alignment,
where you already know what the transcript is,
and you have the audio, and you can get a good alignment
using both pieces of information.
And this is typically a technique that is used to
create training data for training
automatic speech recognizers.
One interesting thing is that in order to do
this forced alignment, you have to keep
the non-speech events in transcript,
because they consume some audio signal,
and if you don't have that signal,
the alignment process doesn't know exactly...
you know, it doesn't do a good job,
because it needs to align all parts of the signal
with something, either pause or silence or noise or words.
Interestingly, punctuations really don't factor in,
because we don't speak in punctuations.
So one of the things that you need to do
is remove most of the punctuations,
although you'll see there are some punctuations
that can be kept, or that are to be kept.
And the other thing is that the alignment has to be done
before tokenization, as it impacts pronunciation.
To show an example: Here you see "that's".
When it's one word,
it has a slightly different pronunciation
than when it is two words, which is "that is",
like you can see "is." And so,
if you split the tokens or split the words
in order for syntactic parser to process it,
you would end up getting the wrong phonetic analysis.
And if you have--if you process it
through the phonetic analysis,
and you don't know how to integrate it
with the tokenized syntax, you can, you know,
that can be pretty tricky. And a lot of time,
people write one-off pieces of code that handle these,
but the idea here is to try to have a general architecture
that seamlessly integrates all these pieces.
Then you do the syntactic parsing of the remaining tokens.
Then you align the data and the two annotations,
and then integrate the two layers.
Once that is done, then you can do all kinds of
interesting analysis, and test various hypotheses
and generate the statistics,
but without that you only are dealing
with one or the other part.
Let's just take a quick look at how each of the layers
that are involved look like.
So this is "Um \{lipsmack\}, and that's it. \{laugh\}"
This is the transcript, and on the right hand side,
you see the same thing as a transcript
listed in a vertical in a column.
You'll see why, in just a second.
And there are some place--
there are some rows that are empty,
some rows that are wider than the others, and we'll see why.
The next is the tokenized sentence
where you have space added,
you know space between these two tokens:
"that" and the apostrophe "s" ('s),
and the "it" and the "period".
And you see on the right hand side
that the tokens have attributes.
So there is a token index, and there are 1, 2,
you know 0, 1, 2, 3, 4, 5 tokens,
and each token has a start and end character,
and space (sp) also has a start and end character,
and space is represented by a "sp". And there are
these other things that we removed,
like the "\{LS\}" which is for "\{lipsmack\}"
and "\{LG\}" which is "\{laugh\}" are showing grayed out,
and you'll see why some of these things are grayed out
in a little bit.
This is what the forced alignment tool produces.
Basically, it takes the transcript,
and this is the transcript
that has slightly different symbols,
because different tools use different symbols
and their various configurational things.
But this is what is used to get an alignment
or time alignment with phones.
So this column shows the phones, and so each word...
So, for example, "and" has been aligned with these phones,
and these on the start and end
are essentially temporal or time stamps that it aligned--
that has been aligned to it.
Interestingly, sometimes we don't really have any pause
or any time duration between some words
and those are highlighted as gray here.
See, there's this space... Actually
it does not have any temporal content,
whereas this other space has some duration.
So the ones that have some duration are captured,
while the others are the ones that in the earlier diagram
we saw were left out.
And the aligner actually produces multiple files.
One of the files has a different, slightly different
variation on the same information,
and in this case, you can see
that the punctuation is missing,
and the punctuation is, you know, deliberately missing,
because there is no time associated with it,
and you see that it's not the tokenized sentence--
a tokenized word. This... Now it gives you a full table,
and you can't really look into it very carefully.
But we can focus on the part that seems legible,
or, you know, properly written sentence,
process it and reincorporate it back into the whole.
So if somebody wants to look at, for example,
how many pauses the person made while they were talking,
And they can actually measure the pause, the number,
the duration, and make connections between that
and the rich syntactic structure that is being produced.
And in order to do that, you have to get these layers
to align with each other,
and this table is just a tabular representation
of the information that we'll be storing in the YAMR file.
Congratulations! You have reached
the end of this demonstration.
Thank you for your time and attention.