WEBVTT captioned by sameer
NOTE Introduction
00:00:00.000 --> 00:00:05.839
Thank you for joining me today. I'm Sameer Pradhan
00:00:05.840 --> 00:00:07.799
from the Linguistic Data Consortium
00:00:07.800 --> 00:00:10.079
at the University of Pennsylvania
00:00:10.080 --> 00:00:14.519
and founder of cemantix.org .
00:00:14.520 --> 00:00:16.879
Today we'll be addressing research
00:00:16.880 --> 00:00:18.719
in computational linguistics,
00:00:18.720 --> 00:00:22.039
also known as natural language processing
00:00:22.040 --> 00:00:24.719
a sub area of artificial intelligence
00:00:24.720 --> 00:00:27.759
with a focus on modeling and predicting
00:00:27.760 --> 00:00:31.919
complex linguistic structures from various signals.
00:00:31.920 --> 00:00:35.799
The work we present is limited to text and speech signals.
00:00:35.800 --> 00:00:38.639
but it can be extended to other signals.
00:00:38.640 --> 00:00:40.799
We propose an architecture,
00:00:40.800 --> 00:00:42.959
and we call it GRAIL, which allows
00:00:42.960 --> 00:00:44.639
the representation and aggregation
00:00:44.640 --> 00:00:50.199
of such rich structures in a systematic fashion.
00:00:50.200 --> 00:00:52.679
I'll demonstrate a proof of concept
00:00:52.680 --> 00:00:56.559
for representing and manipulating data and annotations
00:00:56.560 --> 00:00:58.519
for the specific purpose of building
00:00:58.520 --> 00:01:02.879
machine learning models that simulate understanding.
00:01:02.880 --> 00:01:05.679
These technologies have the potential for impact
00:01:05.680 --> 00:01:09.119
in almost every conceivable field
00:01:09.120 --> 00:01:13.399
that generates and uses data.
NOTE Processing language
00:01:13.400 --> 00:01:15.039
We process human language
00:01:15.040 --> 00:01:16.719
when our brains receive and assimilate
00:01:16.720 --> 00:01:20.079
various signals which are then manipulated
00:01:20.080 --> 00:01:23.879
and interpreted within a syntactic structure.
00:01:23.880 --> 00:01:27.319
it's a complex process that I have simplified here
00:01:27.320 --> 00:01:30.759
for the purpose of comparison to machine learning.
00:01:30.760 --> 00:01:33.959
Recent machine learning models tend to require
00:01:33.960 --> 00:01:37.039
a large amount of raw, naturally occurring data
00:01:37.040 --> 00:01:40.199
and a varying amount of manually enriched data,
00:01:40.200 --> 00:01:43.199
commonly known as "annotations".
00:01:43.200 --> 00:01:45.959
Owing to the complex and numerous nature
00:01:45.960 --> 00:01:49.959
of linguistic phenomena, we have most often used
00:01:49.960 --> 00:01:52.999
a divide and conquer approach.
00:01:53.000 --> 00:01:55.399
The strength of this approach is that it allows us
00:01:55.400 --> 00:01:58.159
to focus on a single, or perhaps a few related
00:01:58.160 --> 00:02:00.439
linguistic phenomena.
00:02:00.440 --> 00:02:03.879
The weaknesses are the universe of these phenomena
00:02:03.880 --> 00:02:07.239
keep expanding, as language itself
00:02:07.240 --> 00:02:09.359
evolves and changes over time,
00:02:09.360 --> 00:02:13.119
and second, this approach requires an additional task
00:02:13.120 --> 00:02:14.839
of aggregating the interpretations,
00:02:14.840 --> 00:02:18.359
creating more opportunities for computer error.
00:02:18.360 --> 00:02:21.519
Our challenge, then, is to find the sweet spot
00:02:21.520 --> 00:02:25.239
that allows us to encode complex information
00:02:25.240 --> 00:02:27.719
without the use of manual annotation,
00:02:27.720 --> 00:02:34.559
or without the additional task of aggregation by computers.
NOTE Annotation
00:02:34.560 --> 00:02:37.119
So what do I mean by "annotation"?
00:02:37.120 --> 00:02:39.759
In this talk the word annotation refers to
00:02:39.760 --> 00:02:43.519
the manual assignment of certain attributes
00:02:43.520 --> 00:02:48.639
to portions of a signal which is necessary
00:02:48.640 --> 00:02:51.639
to perform the end task.
00:02:51.640 --> 00:02:54.439
For example, in order for the algorithm
00:02:54.440 --> 00:02:57.439
to accurately interpret a pronoun,
00:02:57.440 --> 00:03:00.279
it needs to know that pronoun,
00:03:00.280 --> 00:03:03.799
what that pronoun refers back to.
00:03:03.800 --> 00:03:06.719
We may find this task trivial, however,
00:03:06.720 --> 00:03:10.599
current algorithms repeatedly fail in this task.
00:03:10.600 --> 00:03:13.319
So the complexities of understanding
00:03:13.320 --> 00:03:16.639
in computational linguistics require annotation.
00:03:16.640 --> 00:03:20.799
The world annotation itself is a useful example,
00:03:20.800 --> 00:03:22.679
because it also reminds us
00:03:22.680 --> 00:03:25.119
that words have multiple meetings
00:03:25.120 --> 00:03:27.519
as annotation itself does—
00:03:27.520 --> 00:03:30.559
just as I needed to define it in this context,
00:03:30.560 --> 00:03:33.799
so that my message won't be misinterpreted.
00:03:33.800 --> 00:03:39.039
So, too, must annotators do this for algorithms
00:03:39.040 --> 00:03:43.239
through the manual intervention.
NOTE Learning from data
00:03:43.240 --> 00:03:44.759
Learning from raw data
00:03:44.760 --> 00:03:47.039
(commonly known as unsupervised learning)
00:03:47.040 --> 00:03:50.079
poses limitations for machine learning.
00:03:50.080 --> 00:03:53.039
As I described, modeling complex phenomena
00:03:53.040 --> 00:03:55.559
need manual annotations.
00:03:55.560 --> 00:03:58.559
The learning algorithm uses these annotations
00:03:58.560 --> 00:04:01.319
as examples to build statistical models.
00:04:01.320 --> 00:04:04.879
This is called supervised learning.
00:04:04.880 --> 00:04:06.319
Without going into too much detail,
00:04:06.320 --> 00:04:10.039
I'll simply note that the recent popularity
00:04:10.040 --> 00:04:12.519
of the concept of deep learning
00:04:12.520 --> 00:04:14.679
is that evolutionary step
00:04:14.680 --> 00:04:17.319
where we have learned to train models
00:04:17.320 --> 00:04:20.799
using trillions of parameters in ways that they can
00:04:20.800 --> 00:04:25.079
learn richer hierarchical structures
00:04:25.080 --> 00:04:29.399
from very large amounts of annotate, unannotated data.
00:04:29.400 --> 00:04:32.319
These models can then be fine-tuned,
00:04:32.320 --> 00:04:35.599
using varying amounts of annotated examples
00:04:35.600 --> 00:04:37.639
depending on the complexity of the task
00:04:37.640 --> 00:04:39.679
to generate better predictions.
NOTE Manual annotation
00:04:39.680 --> 00:04:44.919
As you might imagine, manually annotating
00:04:44.920 --> 00:04:47.359
complex, linguistic phenomena
00:04:47.360 --> 00:04:51.719
can be very specific, labor-intensive task.
00:04:51.720 --> 00:04:54.279
For example, imagine if we were
00:04:54.280 --> 00:04:56.399
to go back through this presentation
00:04:56.400 --> 00:04:58.399
and connect all the pronouns
00:04:58.400 --> 00:04:59.919
with the nouns to which they refer.
00:04:59.920 --> 00:05:03.239
Even for a short 18 min presentation,
00:05:03.240 --> 00:05:05.239
this would require hundreds of annotations.
00:05:05.240 --> 00:05:08.519
The models we build are only as good
00:05:08.520 --> 00:05:11.119
as the quality of the annotations we make.
00:05:11.120 --> 00:05:12.679
We need guidelines
00:05:12.680 --> 00:05:15.759
that ensure that the annotations are done
00:05:15.760 --> 00:05:19.719
by at least two humans who have substantial agreement
00:05:19.720 --> 00:05:22.119
with each other in their interpretations.
00:05:22.120 --> 00:05:25.599
We know that if we try to trade a model using annotations
00:05:25.600 --> 00:05:28.519
that are very subjective, or have more noise,
00:05:28.520 --> 00:05:30.919
we will receive poor predictions.
00:05:30.920 --> 00:05:33.679
Additionally, there is the concern of introducing
00:05:33.680 --> 00:05:37.079
various unexpected biases into one's models.
00:05:37.080 --> 00:05:44.399
So annotation is really both an art and a science.
NOTE How can we develop a unified representation?
00:05:44.400 --> 00:05:47.439
In the remaining time,
00:05:47.440 --> 00:05:49.999
we will turn to two fundamental questions.
00:05:50.000 --> 00:05:54.239
First, how can we develop a unified representation
00:05:54.240 --> 00:05:55.599
of data and annotations
00:05:55.600 --> 00:05:59.759
that encompasses arbitrary levels of linguistic information?
00:05:59.760 --> 00:06:03.839
There is a long history of attempting to answer
00:06:03.840 --> 00:06:04.839
this first question.
00:06:04.840 --> 00:06:08.839
This history is documented in our recent article,
00:06:08.840 --> 00:06:11.519
and you can refer to that article.
00:06:11.520 --> 00:06:16.719
It will be on the website.
00:06:16.720 --> 00:06:18.999
It is as if we, as a community,
00:06:19.000 --> 00:06:22.519
have been searching for our own Holy Grail.
NOTE What role might Emacs and Org mode play?
00:06:22.520 --> 00:06:26.519
The second question we will pose is
00:06:26.520 --> 00:06:30.159
what role might Emacs, along with Org mode,
00:06:30.160 --> 00:06:31.919
play in this process?
00:06:31.920 --> 00:06:35.359
Well, the solution itself may not be tied to Emacs.
00:06:35.360 --> 00:06:38.359
Emacs has built in capabilities
00:06:38.360 --> 00:06:42.599
that could be useful for evaluating potential solutions.
00:06:42.600 --> 00:06:45.759
It's also one of the most extensively documented
00:06:45.760 --> 00:06:48.519
pieces of software and the most customizable
00:06:48.520 --> 00:06:51.599
piece of software that I have ever come across,
00:06:51.600 --> 00:06:55.279
and many would agree with that.
NOTE The complex structure of language
00:06:55.280 --> 00:07:00.639
In order to approach this second question,
00:07:00.640 --> 00:07:03.919
we turn to the complex structure of language itself.
00:07:03.920 --> 00:07:07.679
At first glance, language appears to us
00:07:07.680 --> 00:07:09.879
as a series of words.
00:07:09.880 --> 00:07:13.439
Words form sentences, sentences form paragraphs,
00:07:13.440 --> 00:07:16.239
and paragraphs form completed text.
00:07:16.240 --> 00:07:19.039
If this was a sufficient description
00:07:19.040 --> 00:07:21.159
of the complexity of language,
00:07:21.160 --> 00:07:24.199
all of us would be able to speak and read
00:07:24.200 --> 00:07:26.559
at least ten different languages.
00:07:26.560 --> 00:07:29.279
We know it is much more complex than this.
00:07:29.280 --> 00:07:33.199
There is a rich, underlying recursive tree structure--
00:07:33.200 --> 00:07:36.439
in fact, many possible tree structures
00:07:36.440 --> 00:07:39.439
which makes a particular sequence meaningful
00:07:39.440 --> 00:07:42.079
and many others meaningless.
00:07:42.080 --> 00:07:45.239
One of the better understood tree structures
00:07:45.240 --> 00:07:47.119
is the syntactic structure.
00:07:47.120 --> 00:07:49.439
While natural language
00:07:49.440 --> 00:07:51.679
has rich ambiguities and complexities,
00:07:51.680 --> 00:07:55.119
programming languages are designed to be parsed
00:07:55.120 --> 00:07:56.999
and interpreted deterministically.
00:07:57.000 --> 00:08:02.159
Emacs has been used for programming very effectively.
00:08:02.160 --> 00:08:05.359
So there is a potential for using Emacs
00:08:05.360 --> 00:08:06.559
as a tool for annotation.
00:08:06.560 --> 00:08:10.799
This would significantly improve our current set of tools.
NOTE Annotation tools
00:08:10.800 --> 00:08:16.559
It is important to note that most of the annotation tools
00:08:16.560 --> 00:08:19.639
that have been developed over the past few decades
00:08:19.640 --> 00:08:22.879
have relied on graphical interfaces,
00:08:22.880 --> 00:08:26.919
even those used for enriching textual information.
00:08:26.920 --> 00:08:30.399
Most of the tools in current use
00:08:30.400 --> 00:08:36.159
are designed for a end user to add very specific,
00:08:36.160 --> 00:08:38.639
very restricted information.
00:08:38.640 --> 00:08:42.799
We have not really made use of the potential
00:08:42.800 --> 00:08:45.639
that an editor or a rich editing environment like Emacs
00:08:45.640 --> 00:08:47.239
can add to the mix.
00:08:47.240 --> 00:08:52.479
Emacs has long enabled the editing of, the manipulation of
00:08:52.480 --> 00:08:56.359
complex embedded tree structures abundant in source code.
00:08:56.360 --> 00:08:58.599
So it's not difficult to imagine that it would have
00:08:58.600 --> 00:09:00.359
many capabilities that we we need
00:09:00.360 --> 00:09:02.599
to represent actual language.
00:09:02.600 --> 00:09:04.759
In fact, it already does that with features
00:09:04.760 --> 00:09:06.399
that allow us to quickly navigate
00:09:06.400 --> 00:09:07.919
through sentences and paragraphs,
00:09:07.920 --> 00:09:09.799
and we don't need a few key strokes.
00:09:09.800 --> 00:09:13.599
Or to add various text properties to text spans
00:09:13.600 --> 00:09:17.039
to create overlays, to name but a few.
00:09:17.040 --> 00:09:22.719
Emacs figured out this way to handle Unicode,
00:09:22.720 --> 00:09:26.799
so you don't even have to worry about the complexity
00:09:26.800 --> 00:09:29.439
of managing multiple languages.
00:09:29.440 --> 00:09:34.039
It's built into Emacs. In fact, this is not the first time
00:09:34.040 --> 00:09:37.399
Emacs has been used for linguistic analysis.
00:09:37.400 --> 00:09:41.159
One of the breakthrough moments in language,
00:09:41.160 --> 00:09:44.439
natural language processing was the creation
00:09:44.440 --> 00:09:48.639
of manually created syntactic trees
00:09:48.640 --> 00:09:50.439
for a 1 million word collection
00:09:50.440 --> 00:09:52.399
of Wall Street Journal articles.
00:09:52.400 --> 00:09:54.879
This was else around 1992
00:09:54.880 --> 00:09:59.279
before Java or graphical interfaces were common.
00:09:59.280 --> 00:10:03.279
The tool that was used to create that corpus was Emacs.
00:10:03.280 --> 00:10:08.959
It was created at UPenn, and is famously known as
00:10:08.960 --> 00:10:12.719
the Penn Treebank. '92 was about when
00:10:12.720 --> 00:10:16.439
the Linguistic Data Consortium was also established,
00:10:16.440 --> 00:10:18.039
and it's been about 30 years
00:10:18.040 --> 00:10:20.719
that it has been creating various
00:10:20.720 --> 00:10:22.359
language-related resources.
NOTE Org mode
00:10:22.360 --> 00:10:28.519
Org mode--in particular, the outlining mode,
00:10:28.520 --> 00:10:32.399
or rather the enhanced form of outlining mode--
00:10:32.400 --> 00:10:35.599
allows us to create rich outlines,
00:10:35.600 --> 00:10:37.799
attaching properties to nodes,
00:10:37.800 --> 00:10:41.119
and provides commands for easily customizing
00:10:41.120 --> 00:10:43.879
sorting of various pieces of information
00:10:43.880 --> 00:10:45.639
as per one's requirement.
00:10:45.640 --> 00:10:50.239
This can also be a very useful tool.
00:10:50.240 --> 00:10:59.159
This enhanced form of outline-mode adds more power to Emacs.
00:10:59.160 --> 00:11:03.359
It provides commands for easily customizing
00:11:03.360 --> 00:11:05.159
and filtering information,
00:11:05.160 --> 00:11:08.999
while at the same time hiding unnecessary context.
00:11:09.000 --> 00:11:11.919
It also allows structural editing.
00:11:11.920 --> 00:11:16.039
This can be a very useful tool to enrich corpora
00:11:16.040 --> 00:11:20.919
where we are focusing on limited amount of phenomena.
00:11:20.920 --> 00:11:24.519
The two together allow us to create
00:11:24.520 --> 00:11:27.199
a rich representation
00:11:27.200 --> 00:11:32.999
that can simultaneously capture multiple possible sequences,
00:11:33.000 --> 00:11:38.759
capture details necessary to recreate the original source,
00:11:38.760 --> 00:11:42.079
allow the creation of hierarchical representation,
00:11:42.080 --> 00:11:44.679
provide structural editing capabilities
00:11:44.680 --> 00:11:47.439
that can take advantage of the concept of inheritance
00:11:47.440 --> 00:11:48.999
within the tree structure.
00:11:49.000 --> 00:11:54.279
Together they allow local manipulations of structures,
00:11:54.280 --> 00:11:56.199
thereby minimizing data coupling.
00:11:56.200 --> 00:11:59.119
The concept of tags in Org mode
00:11:59.120 --> 00:12:01.599
complement the hierarchy part.
00:12:01.600 --> 00:12:03.839
Hierarchies can be very rigid,
00:12:03.840 --> 00:12:06.039
but to tags on hierarchies,
00:12:06.040 --> 00:12:08.839
we can have a multifaceted representations.
00:12:08.840 --> 00:12:12.759
As a matter of fact, Org mode has the ability for the tags
00:12:12.760 --> 00:12:15.039
to have their own hierarchical structure
00:12:15.040 --> 00:12:18.639
which further enhances the representational power.
00:12:18.640 --> 00:12:22.639
All of this can be done as a sequence
00:12:22.640 --> 00:12:25.679
of mostly functional data transformations,
00:12:25.680 --> 00:12:27.439
because most of the capabilities
00:12:27.440 --> 00:12:29.759
can be configured and customized.
00:12:29.760 --> 00:12:32.799
It is not necessary to do everything at once.
00:12:32.800 --> 00:12:36.199
Instead, it allows us to incrementally increase
00:12:36.200 --> 00:12:37.919
the complexity of the representation.
00:12:37.920 --> 00:12:39.799
Finally, all of this can be done
00:12:39.800 --> 00:12:42.359
in plain-text representation
00:12:42.360 --> 00:12:45.479
which comes with its own advantages.
NOTE Example
00:12:45.480 --> 00:12:50.679
Now let's take a simple example.
00:12:50.680 --> 00:12:55.999
This is a a short video that I'll play.
00:12:56.000 --> 00:12:59.679
The sentence is "I saw the moon with a telescope,"
00:12:59.680 --> 00:13:03.999
and let's just make a copy of the sentence.
00:13:04.000 --> 00:13:09.199
What we can do now is to see:
00:13:09.200 --> 00:13:11.879
what does this sentence comprise?
00:13:11.880 --> 00:13:13.679
It has a noun phrase "I,"
00:13:13.680 --> 00:13:17.479
followed by a word "saw."
00:13:17.480 --> 00:13:21.359
Then "the moon" is another noun phrase,
00:13:21.360 --> 00:13:24.839
and "with the telescope" is a prepositional phrase.
00:13:24.840 --> 00:13:30.759
Now one thing that you might remember,
00:13:30.760 --> 00:13:36.119
from grammar school or syntax is that
00:13:36.120 --> 00:13:41.279
there is a syntactic structure.
00:13:41.280 --> 00:13:44.359
And if you in this particular case--
00:13:44.360 --> 00:13:47.919
because we know that the moon is not typically
00:13:47.920 --> 00:13:51.679
something that can hold the telescope,
00:13:51.680 --> 00:13:56.239
that the seeing must be done by me or "I,"
00:13:56.240 --> 00:14:01.039
and the telescope must be in my hand,
00:14:01.040 --> 00:14:04.479
or "I" am viewing the moon with a telescope.
00:14:04.480 --> 00:14:13.519
However, it is possible that in a different context
00:14:13.520 --> 00:14:17.159
the moon could be referring to an animated character
00:14:17.160 --> 00:14:22.319
in a animated series, and could actually hold the telescope.
00:14:22.320 --> 00:14:23.479
And this is one of the most--
00:14:23.480 --> 00:14:24.839
the oldest and one of the most--
00:14:24.840 --> 00:14:26.319
and in that case the situation might be
00:14:26.320 --> 00:14:30.959
that I'm actually seeing the moon holding a telescope...
00:14:30.960 --> 00:14:36.079
I mean. The moon is holding the telescope,
00:14:36.080 --> 00:14:40.959
and I'm just seeing the moon holding the telescope.
00:14:40.960 --> 00:14:47.999
Complex linguistic ambiguity or linguistic
00:14:48.000 --> 00:14:53.599
phenomena that requires world knowledge,
00:14:53.600 --> 00:14:55.719
and it's called the PP attachment problem
00:14:55.720 --> 00:14:59.239
where the propositional phrase attachment
00:14:59.240 --> 00:15:04.599
can be ambiguous, and various different contextual cues
00:15:04.600 --> 00:15:06.879
have to be used to resolve the ambiguity.
00:15:06.880 --> 00:15:09.079
So in this case, as you saw,
00:15:09.080 --> 00:15:11.199
both the readings are technically true,
00:15:11.200 --> 00:15:13.959
depending on different contexts.
00:15:13.960 --> 00:15:16.599
So one thing we could do is just
00:15:16.600 --> 00:15:19.919
to cut the tree and duplicate it,
00:15:19.920 --> 00:15:21.599
and then let's create another node
00:15:21.600 --> 00:15:24.479
and call it an "OR" node.
00:15:24.480 --> 00:15:26.119
And because we are saying,
00:15:26.120 --> 00:15:28.359
this is one of the two interpretations.
00:15:28.360 --> 00:15:32.159
Now let's call one interpretation "a",
00:15:32.160 --> 00:15:36.159
and that interpretation essentially
00:15:36.160 --> 00:15:39.319
is this child of that node "a"
00:15:39.320 --> 00:15:41.799
and that says that the moon
00:15:41.800 --> 00:15:43.999
is holding the telescope.
00:15:44.000 --> 00:15:46.359
Now we can create another representation "b"
00:15:46.360 --> 00:15:53.919
where we capture the other interpretation,
00:15:53.920 --> 00:15:59.959
where this, the act, the moon or--I am actually
00:15:59.960 --> 00:16:00.519
holding the telescope,
00:16:00.520 --> 00:16:06.799
and watching the moon using it.
00:16:06.800 --> 00:16:09.199
So now we have two separate interpretations
00:16:09.200 --> 00:16:11.679
in the same structure,
00:16:11.680 --> 00:16:15.519
and all we do--we're able to do is with this,
00:16:15.520 --> 00:16:18.159
with very quick key strokes now...
00:16:18.160 --> 00:16:22.439
While we are at it, let's add another interesting thing,
00:16:22.440 --> 00:16:25.159
this node that represents "I":
00:16:25.160 --> 00:16:28.919
"He." It can be "She".
00:16:28.920 --> 00:16:35.759
It can be "the children," or it can be "The people".
00:16:35.760 --> 00:16:45.039
Basically, any entity that has the capability to "see"
00:16:45.040 --> 00:16:53.359
can be substituted in this particular node.
00:16:53.360 --> 00:16:57.399
Let's see what we have here now.
00:16:57.400 --> 00:17:01.239
We just are getting sort of a zoom view
00:17:01.240 --> 00:17:04.599
of the entire structure, what we created,
00:17:04.600 --> 00:17:08.039
and essentially you can see that
00:17:08.040 --> 00:17:11.879
by just, you know, using a few keystrokes,
00:17:11.880 --> 00:17:17.839
we were able to capture two different interpretations
00:17:17.840 --> 00:17:20.879
of a a simple sentence,
00:17:20.880 --> 00:17:23.759
and they are also able to add
00:17:23.760 --> 00:17:27.799
these alternate pieces of information
00:17:27.800 --> 00:17:30.559
that could help machine learning algorithms
00:17:30.560 --> 00:17:32.439
generalize better.
00:17:32.440 --> 00:17:36.239
All right.
NOTE Different readings
00:17:36.240 --> 00:17:40.359
Now, let's look at the next thing. So in a sense,
00:17:40.360 --> 00:17:46.679
we can use this power of functional data structures
00:17:46.680 --> 00:17:50.239
to represent various potentially conflicting
00:17:50.240 --> 00:17:55.559
and structural readings of that piece of text.
00:17:55.560 --> 00:17:58.079
In addition to that, we can also create more texts,
00:17:58.080 --> 00:17:59.799
each with different structure,
00:17:59.800 --> 00:18:01.559
and have them all in the same place.
00:18:01.560 --> 00:18:04.239
This allows us to address the interpretation
00:18:04.240 --> 00:18:06.879
of a static sentence that might be occurring in the world,
00:18:06.880 --> 00:18:09.639
while simultaneously inserting information
00:18:09.640 --> 00:18:11.519
that would add more value to it.
00:18:11.520 --> 00:18:14.999
This makes the enrichment process also very efficient.
00:18:15.000 --> 00:18:19.519
Additionally, we can envision
00:18:19.520 --> 00:18:23.999
a power user of the future, or present,
00:18:24.000 --> 00:18:27.479
who can not only annotate a span,
00:18:27.480 --> 00:18:31.279
but also edit the information in situ
00:18:31.280 --> 00:18:34.639
in a way that would help machine algorithms
00:18:34.640 --> 00:18:36.879
generalize better by making more efficient use
00:18:36.880 --> 00:18:37.719
of the annotations.
00:18:37.720 --> 00:18:41.519
So together, Emacs and Org mode can speed up
00:18:41.520 --> 00:18:42.959
the enrichment of the signals
00:18:42.960 --> 00:18:44.519
in a way that allows us
00:18:44.520 --> 00:18:47.719
to focus on certain aspects and ignore others.
00:18:47.720 --> 00:18:50.839
Extremely complex landscape of rich structures
00:18:50.840 --> 00:18:53.039
can be captured consistently,
00:18:53.040 --> 00:18:55.639
in a fashion that allows computers
00:18:55.640 --> 00:18:56.759
to understand language.
00:18:56.760 --> 00:19:00.879
We can then build tools to enhance the tasks
00:19:00.880 --> 00:19:03.319
that we do in our everyday life.
00:19:03.320 --> 00:19:10.759
YAMR is acronym, or the file's type or specification
00:19:10.760 --> 00:19:15.239
that we are creating to capture this new
00:19:15.240 --> 00:19:17.679
rich representation.
NOTE Spontaneous speech
00:19:17.680 --> 00:19:21.959
We'll now look at an example of spontaneous speech
00:19:21.960 --> 00:19:24.799
that occurs in spoken conversations.
00:19:24.800 --> 00:19:28.599
Conversations frequently contain errors in speech:
00:19:28.600 --> 00:19:30.799
interruptions, disfluencies,
00:19:30.800 --> 00:19:33.959
verbal sounds such as cough or laugh,
00:19:33.960 --> 00:19:35.039
and other noises.
00:19:35.040 --> 00:19:38.199
In this sense, spontaneous speech is similar
00:19:38.200 --> 00:19:39.799
to a functional data stream.
00:19:39.800 --> 00:19:42.759
We cannot take back words that come out of our mouth,
00:19:42.760 --> 00:19:47.239
but we tend to make mistakes, and we correct ourselves
00:19:47.240 --> 00:19:49.039
as soon as we realize that we have made--
00:19:49.040 --> 00:19:50.679
we have misspoken.
00:19:50.680 --> 00:19:53.159
This process manifests through a combination
00:19:53.160 --> 00:19:56.279
of a handful of mechanisms, including immediate correction
00:19:56.280 --> 00:20:00.959
after an error, and we do this unconsciously.
00:20:00.960 --> 00:20:02.719
Computers, on the other hand,
00:20:02.720 --> 00:20:06.639
must be taught to understand these cases.
00:20:06.640 --> 00:20:12.799
What we see here is a example document or outline,
00:20:12.800 --> 00:20:18.119
or part of a document that illustrates
00:20:18.120 --> 00:20:22.919
various different aspects of the representation.
00:20:22.920 --> 00:20:25.919
We don't have a lot of time to go through
00:20:25.920 --> 00:20:28.239
many of the details.
00:20:28.240 --> 00:20:31.759
I would highly encourage you to play a...
00:20:31.760 --> 00:20:39.159
I'm planning on making some videos, or ascii cinemas,
00:20:39.160 --> 00:20:42.559
that I'll be posting, and you can,
00:20:42.560 --> 00:20:46.759
if you're interested, you can go through those.
00:20:46.760 --> 00:20:50.359
The idea here is to try to do
00:20:50.360 --> 00:20:54.599
a slightly more complex use case.
00:20:54.600 --> 00:20:57.639
But again, given the time constraint
00:20:57.640 --> 00:21:00.279
and the amount of information
00:21:00.280 --> 00:21:01.519
that needs to fit in the screen,
00:21:01.520 --> 00:21:05.559
this may not be very informative,
00:21:05.560 --> 00:21:08.399
but at least it will give you some idea
00:21:08.400 --> 00:21:10.439
of what can be possible.
00:21:10.440 --> 00:21:13.279
And in this particular case, what you're seeing is that
00:21:13.280 --> 00:21:18.319
there is a sentence which is "What I'm I'm tr- telling now."
00:21:18.320 --> 00:21:21.159
Essentially, there is a repetition of the word "I'm",
00:21:21.160 --> 00:21:23.279
and then there is a partial word
00:21:23.280 --> 00:21:25.159
that somebody tried to say "telling",
00:21:25.160 --> 00:21:29.599
but started saying "tr-", and then corrected themselves
00:21:29.600 --> 00:21:30.959
and said, "telling now."
00:21:30.960 --> 00:21:39.239
So in this case, you see, we can capture words
00:21:39.240 --> 00:21:44.919
or a sequence of words, or a sequence of tokens.
00:21:44.920 --> 00:21:52.279
One thing to... An interesting thing to note is that in NLP,
00:21:52.280 --> 00:21:55.319
sometimes we have to break typically
00:21:55.320 --> 00:22:01.199
words that don't have spaces into two separate words,
00:22:01.200 --> 00:22:04.119
especially contractions like "I'm",
00:22:04.120 --> 00:22:08.199
so the syntactic parser needs needs two separate nodes.
00:22:08.200 --> 00:22:11.199
But anyway, so I'll... You can see that here.
00:22:11.200 --> 00:22:15.759
The other... This view. What this view shows is that
00:22:15.760 --> 00:22:19.759
with each of the nodes in the sentence
00:22:19.760 --> 00:22:23.079
or in the representation,
00:22:23.080 --> 00:22:26.079
you can have a lot of different properties
00:22:26.080 --> 00:22:27.559
that you can attach to them,
00:22:27.560 --> 00:22:30.119
and these properties are typically hidden,
00:22:30.120 --> 00:22:32.719
like you saw in the earlier slide.
00:22:32.720 --> 00:22:35.599
But you can make use of all these properties
00:22:35.600 --> 00:22:39.439
to do various kind of searches and filtering.
00:22:39.440 --> 00:22:43.519
And on the right hand side here--
00:22:43.520 --> 00:22:48.799
this is actually not a legitimate syntax--
00:22:48.800 --> 00:22:51.279
but on the right are descriptions
00:22:51.280 --> 00:22:53.479
of what each of these represent.
00:22:53.480 --> 00:22:57.319
All the information is also available in the article.
00:22:57.320 --> 00:23:04.279
You can see there... It shows how much rich context
00:23:04.280 --> 00:23:05.879
you can capture.
00:23:05.880 --> 00:23:08.799
This is just a closer snapshot
00:23:08.800 --> 00:23:10.159
of the properties on the node,
00:23:10.160 --> 00:23:13.119
and you can see we can have things like,
00:23:13.120 --> 00:23:14.799
whether the word is a token or not,
00:23:14.800 --> 00:23:17.359
or that it's incomplete, whether some words
00:23:17.360 --> 00:23:19.959
might want to be filtered out for parsing,
00:23:19.960 --> 00:23:23.039
and we can say this: PARSE_IGNORE,
00:23:23.040 --> 00:23:25.519
or some words or restart markers...
00:23:25.520 --> 00:23:29.239
We can mark, add a RESTART_MARKER, or sometimes,
00:23:29.240 --> 00:23:31.999
some of these might have durations. Things like that.
NOTE Editing properties in column view
00:23:32.000 --> 00:23:38.799
The other fascinating thing of this representation
00:23:38.800 --> 00:23:42.599
is that you can edit properties in the column view.
00:23:42.600 --> 00:23:45.399
And suddenly, you have this tabular data structure
00:23:45.400 --> 00:23:48.879
combined with the hierarchical data structure.
00:23:48.880 --> 00:23:53.119
And as you can--you may not be able to see it here,
00:23:53.120 --> 00:23:56.879
but what has also happened here is that
00:23:56.880 --> 00:24:01.159
some of the tags have been inherited
00:24:01.160 --> 00:24:02.479
from the earlier nodes.
00:24:02.480 --> 00:24:07.919
And so you get a much fuller picture of things.
00:24:07.920 --> 00:24:13.919
Essentially you, can filter out things
00:24:13.920 --> 00:24:15.319
that you want to process,
00:24:15.320 --> 00:24:20.279
process them, and then reintegrate it into the whole.
NOTE Conclusion
00:24:20.280 --> 00:24:25.479
So, in conclusion, today we have proposed and demonstrated
00:24:25.480 --> 00:24:27.559
the use of an architecture (GRAIL),
00:24:27.560 --> 00:24:31.319
which allows the representation, manipulation,
00:24:31.320 --> 00:24:34.759
and aggregation of rich linguistic structures
00:24:34.760 --> 00:24:36.519
in a systematic fashion.
00:24:36.520 --> 00:24:41.359
We have shown how GRAIL advances the tools
00:24:41.360 --> 00:24:44.599
available for building machine learning models
00:24:44.600 --> 00:24:46.879
that simulate understanding.
00:24:46.880 --> 00:24:51.679
Thank you very much for your time and attention today.
00:24:51.680 --> 00:24:54.639
My contact information is on this slide.
00:24:54.640 --> 00:25:02.599
If you are interested in an additional example
00:25:02.600 --> 00:25:05.439
that demonstrates the representation
00:25:05.440 --> 00:25:08.039
of speech and written text together,
00:25:08.040 --> 00:25:10.719
please continue watching.
00:25:10.720 --> 00:25:12.199
Otherwise, you can stop here
00:25:12.200 --> 00:25:15.279
and enjoy the rest of the conference.
NOTE Bonus material
00:25:15.280 --> 00:25:39.079
Welcome to the bonus material.
00:25:39.080 --> 00:25:43.959
I'm glad for those of you who are stuck around.
00:25:43.960 --> 00:25:46.559
We are now going to examine an instance
00:25:46.560 --> 00:25:49.159
of speech and text signals together
00:25:49.160 --> 00:25:51.479
that produce multiple layers.
00:25:51.480 --> 00:25:54.839
When we have--when we take a spoken conversation
00:25:54.840 --> 00:25:58.719
and use the best language processing models available,
00:25:58.720 --> 00:26:00.679
we suddenly hit a hard spot
00:26:00.680 --> 00:26:03.239
because the tools are typically not trained
00:26:03.240 --> 00:26:05.359
to filter out the unnecessary cruft
00:26:05.360 --> 00:26:07.559
in order to automatically interpret
00:26:07.560 --> 00:26:09.559
the part of what is being said
00:26:09.560 --> 00:26:11.799
that is actually relevant.
00:26:11.800 --> 00:26:14.639
Over time, language researchers
00:26:14.640 --> 00:26:17.719
have created many interdependent layers of annotations,
00:26:17.720 --> 00:26:21.039
yet the assumptions underlying them are seldom the same.
00:26:21.040 --> 00:26:25.039
Piecing together such related but disjointed annotations
00:26:25.040 --> 00:26:28.039
on their predictions poses a huge challenge.
00:26:28.040 --> 00:26:30.719
This is another place where we can leverage
00:26:30.720 --> 00:26:33.119
the data model underlying the Emacs editor,
00:26:33.120 --> 00:26:35.359
along with the structural editing capabilities
00:26:35.360 --> 00:26:38.519
of Org mode to improve current tools.
00:26:38.520 --> 00:26:42.839
Let's take this very simple looking utterance.
00:26:42.840 --> 00:26:48.039
"Um \{lipsmack\} and that's it. (\{laugh\})"
00:26:48.040 --> 00:26:50.319
Looks like the person-- so this is--
00:26:50.320 --> 00:26:54.519
what you are seeing here is a transcript of an audio signal
00:26:54.520 --> 00:27:00.759
that has a lip smack and a laugh as part of it,
00:27:00.760 --> 00:27:04.199
and there is also a "Um" like interjection.
00:27:04.200 --> 00:27:08.199
So this has a few interesting noises
00:27:08.200 --> 00:27:13.999
and specific things that would be illustrative
00:27:14.000 --> 00:27:20.479
of what we are going to, how we are going to represent it.
NOTE Syntactic analysis
00:27:20.480 --> 00:27:25.839
Okay. So let's say you want to have
00:27:25.840 --> 00:27:28.879
a syntactic analysis of this sentence or utterance.
00:27:28.880 --> 00:27:30.959
One common technique people use
00:27:30.960 --> 00:27:32.879
is just to remove the cruft, and, you know,
00:27:32.880 --> 00:27:35.079
write some rules, clean up the utterance,
00:27:35.080 --> 00:27:36.719
make it look like it's proper English,
00:27:36.720 --> 00:27:40.239
and then, you know, tokenize it,
00:27:40.240 --> 00:27:43.079
and basically just use standard tools to process it.
00:27:43.080 --> 00:27:47.279
But in that process, they end up eliminating
00:27:47.280 --> 00:27:51.119
valid pieces of signal that have meaning to others
00:27:51.120 --> 00:27:52.799
studying different phenomena of language.
00:27:52.800 --> 00:27:56.479
Here you have the rich transcript,
00:27:56.480 --> 00:28:00.119
the input to the syntactic parser.
00:28:00.120 --> 00:28:05.919
As you can see, there is a little tokenization happening
00:28:05.920 --> 00:28:07.199
where you'll be inserting space
00:28:07.200 --> 00:28:12.119
between "that" and the contracted is ('s),
00:28:12.120 --> 00:28:15.599
and between the period and the "it,"
00:28:15.600 --> 00:28:18.199
and the output of the syntactic parser is shown below.
00:28:18.200 --> 00:28:21.639
which (surprise) is a S-expression.
00:28:21.640 --> 00:28:24.919
Like I said, the parse trees, when they were created,
00:28:24.920 --> 00:28:29.799
and still largely when they are used, are S-expressions,
00:28:29.800 --> 00:28:32.999
and most of the viewers here
00:28:33.000 --> 00:28:35.119
should not have much problem reading it.
00:28:35.120 --> 00:28:37.279
You can see this tree structure
00:28:37.280 --> 00:28:39.279
of this syntactic parser here.
NOTE Forced alignment
00:28:39.280 --> 00:28:40.919
Now let's say you want to integrate
00:28:40.920 --> 00:28:44.479
phonetic information or phonetic layer
00:28:44.480 --> 00:28:49.119
that's in the audio signal, and do some analysis.
00:28:49.120 --> 00:28:57.519
Now, it would need you to do a few-- take a few steps.
00:28:57.520 --> 00:29:01.679
First, you would need to align the transcript
00:29:01.680 --> 00:29:06.479
with the audio. This process is called forced alignment,
00:29:06.480 --> 00:29:10.399
where you already know what the transcript is,
00:29:10.400 --> 00:29:14.599
and you have the audio, and you can get a good alignment
00:29:14.600 --> 00:29:17.599
using both pieces of information.
00:29:17.600 --> 00:29:20.119
And this is typically a technique that is used to
00:29:20.120 --> 00:29:23.079
create training data for training
00:29:23.080 --> 00:29:25.839
automatic speech recognizers.
00:29:25.840 --> 00:29:29.639
One interesting thing is that in order to do
00:29:29.640 --> 00:29:32.879
this forced alignment, you have to keep
00:29:32.880 --> 00:29:35.799
the non-speech events in transcript,
00:29:35.800 --> 00:29:39.079
because they consume some audio signal,
00:29:39.080 --> 00:29:41.399
and if you don't have that signal,
00:29:41.400 --> 00:29:44.399
the alignment process doesn't know exactly...
00:29:44.400 --> 00:29:45.759
you know, it doesn't do a good job,
00:29:45.760 --> 00:29:50.039
because it needs to align all parts of the signal
00:29:50.040 --> 00:29:54.999
with something, either pause or silence or noise or words.
00:29:55.000 --> 00:29:59.719
Interestingly, punctuations really don't factor in,
00:29:59.720 --> 00:30:01.559
because we don't speak in punctuations.
00:30:01.560 --> 00:30:04.239
So one of the things that you need to do
00:30:04.240 --> 00:30:05.679
is remove most of the punctuations,
00:30:05.680 --> 00:30:08.039
although you'll see there are some punctuations
00:30:08.040 --> 00:30:12.599
that can be kept, or that are to be kept.
NOTE Alignment before tokenization
00:30:12.600 --> 00:30:15.319
And the other thing is that the alignment has to be done
00:30:15.320 --> 00:30:20.159
before tokenization, as it impacts pronunciation.
00:30:20.160 --> 00:30:24.399
To show an example: Here you see "that's".
00:30:24.400 --> 00:30:26.919
When it's one word,
00:30:26.920 --> 00:30:31.959
it has a slightly different pronunciation
00:30:31.960 --> 00:30:35.679
than when it is two words, which is "that is",
00:30:35.680 --> 00:30:38.399
like you can see "is." And so,
00:30:38.400 --> 00:30:44.279
if you split the tokens or split the words
00:30:44.280 --> 00:30:48.119
in order for syntactic parser to process it,
00:30:48.120 --> 00:30:51.599
you would end up getting the wrong phonetic analysis.
00:30:51.600 --> 00:30:54.239
And if you have--if you process it
00:30:54.240 --> 00:30:55.319
through the phonetic analysis,
00:30:55.320 --> 00:30:59.159
and you don't know how to integrate it
00:30:59.160 --> 00:31:02.719
with the tokenized syntax, you can, you know,
00:31:02.720 --> 00:31:07.519
that can be pretty tricky. And a lot of time,
00:31:07.520 --> 00:31:10.759
people write one-off pieces of code that handle these,
00:31:10.760 --> 00:31:14.279
but the idea here is to try to have a general architecture
00:31:14.280 --> 00:31:17.239
that seamlessly integrates all these pieces.
00:31:17.240 --> 00:31:21.319
Then you do the syntactic parsing of the remaining tokens.
00:31:21.320 --> 00:31:24.799
Then you align the data and the two annotations,
00:31:24.800 --> 00:31:27.959
and then integrate the two layers.
00:31:27.960 --> 00:31:31.359
Once that is done, then you can do all kinds of
00:31:31.360 --> 00:31:33.919
interesting analysis, and test various hypotheses
00:31:33.920 --> 00:31:35.279
and generate the statistics,
00:31:35.280 --> 00:31:39.359
but without that you only are dealing
00:31:39.360 --> 00:31:42.879
with one or the other part.
NOTE Layers
00:31:42.880 --> 00:31:48.319
Let's just take a quick look at how each of the layers
00:31:48.320 --> 00:31:51.159
that are involved look like.
00:31:51.160 --> 00:31:56.719
So this is "Um \{lipsmack\}, and that's it. \{laugh\}"
00:31:56.720 --> 00:32:00.159
This is the transcript, and on the right hand side,
00:32:00.160 --> 00:32:04.199
you see the same thing as a transcript
00:32:04.200 --> 00:32:06.239
listed in a vertical in a column.
00:32:06.240 --> 00:32:08.199
You'll see why, in just a second.
00:32:08.200 --> 00:32:09.879
And there are some place--
00:32:09.880 --> 00:32:11.279
there are some rows that are empty,
00:32:11.280 --> 00:32:15.079
some rows that are wider than the others, and we'll see why.
00:32:15.080 --> 00:32:19.319
The next is the tokenized sentence
00:32:19.320 --> 00:32:20.959
where you have space added,
00:32:20.960 --> 00:32:23.599
you know space between these two tokens:
00:32:23.600 --> 00:32:26.599
"that" and the apostrophe "s" ('s),
00:32:26.600 --> 00:32:28.079
and the "it" and the "period".
00:32:28.080 --> 00:32:30.679
And you see on the right hand side
00:32:30.680 --> 00:32:33.559
that the tokens have attributes.
00:32:33.560 --> 00:32:36.439
So there is a token index, and there are 1, 2,
00:32:36.440 --> 00:32:38.839
you know 0, 1, 2, 3, 4, 5 tokens,
00:32:38.840 --> 00:32:41.479
and each token has a start and end character,
00:32:41.480 --> 00:32:45.799
and space (sp) also has a start and end character,
00:32:45.800 --> 00:32:50.399
and space is represented by a "sp". And there are
00:32:50.400 --> 00:32:54.319
these other things that we removed,
00:32:54.320 --> 00:32:56.239
like the "\{LS\}" which is for "\{lipsmack\}"
00:32:56.240 --> 00:32:59.399
and "\{LG\}" which is "\{laugh\}" are showing grayed out,
00:32:59.400 --> 00:33:02.439
and you'll see why some of these things are grayed out
00:33:02.440 --> 00:33:03.399
in a little bit.
00:33:03.400 --> 00:33:11.919
This is what the forced alignment tool produces.
00:33:11.920 --> 00:33:17.159
Basically, it takes the transcript,
00:33:17.160 --> 00:33:19.159
and this is the transcript
00:33:19.160 --> 00:33:24.119
that has slightly different symbols,
00:33:24.120 --> 00:33:26.239
because different tools use different symbols
00:33:26.240 --> 00:33:28.159
and their various configurational things.
00:33:28.160 --> 00:33:33.679
But this is what is used to get an alignment
00:33:33.680 --> 00:33:36.039
or time alignment with phones.
00:33:36.040 --> 00:33:40.079
So this column shows the phones, and so each word...
00:33:40.080 --> 00:33:43.879
So, for example, "and" has been aligned with these phones,
00:33:43.880 --> 00:33:46.879
and these on the start and end
00:33:46.880 --> 00:33:52.959
are essentially temporal or time stamps that it aligned--
00:33:52.960 --> 00:33:54.279
that has been aligned to it.
00:33:54.280 --> 00:34:00.759
Interestingly, sometimes we don't really have any pause
00:34:00.760 --> 00:34:05.159
or any time duration between some words
00:34:05.160 --> 00:34:08.199
and those are highlighted as gray here.
00:34:08.200 --> 00:34:12.759
See, there's this space... Actually
00:34:12.760 --> 00:34:17.799
it does not have any temporal content,
00:34:17.800 --> 00:34:21.319
whereas this other space has some duration.
00:34:21.320 --> 00:34:24.839
So the ones that have some duration are captured,
00:34:24.840 --> 00:34:29.519
while the others are the ones that in the earlier diagram
00:34:29.520 --> 00:34:31.319
we saw were left out.
NOTE Variations
00:34:31.320 --> 00:34:37.639
And the aligner actually produces multiple files.
00:34:37.640 --> 00:34:44.399
One of the files has a different, slightly different
00:34:44.400 --> 00:34:46.679
variation on the same information,
00:34:46.680 --> 00:34:49.999
and in this case, you can see
00:34:50.000 --> 00:34:52.399
that the punctuation is missing,
00:34:52.400 --> 00:34:57.599
and the punctuation is, you know, deliberately missing,
00:34:57.600 --> 00:35:02.279
because there is no time associated with it,
00:35:02.280 --> 00:35:06.439
and you see that it's not the tokenized sentence--
00:35:06.440 --> 00:35:17.119
a tokenized word. This... Now it gives you a full table,
00:35:17.120 --> 00:35:21.239
and you can't really look into it very carefully.
00:35:21.240 --> 00:35:25.879
But we can focus on the part that seems legible,
00:35:25.880 --> 00:35:28.559
or, you know, properly written sentence,
00:35:28.560 --> 00:35:32.879
process it and reincorporate it back into the whole.
00:35:32.880 --> 00:35:35.879
So if somebody wants to look at, for example,
00:35:35.880 --> 00:35:39.679
how many pauses the person made while they were talking,
00:35:39.680 --> 00:35:42.919
And they can actually measure the pause, the number,
00:35:42.920 --> 00:35:46.279
the duration, and make connections between that
00:35:46.280 --> 00:35:49.639
and the rich syntactic structure that is being produced.
00:35:49.640 --> 00:35:57.279
And in order to do that, you have to get these layers
00:35:57.280 --> 00:35:59.039
to align with each other,
00:35:59.040 --> 00:36:04.359
and this table is just a tabular representation
00:36:04.360 --> 00:36:08.679
of the information that we'll be storing in the YAMR file.
00:36:08.680 --> 00:36:11.719
Congratulations! You have reached
00:36:11.720 --> 00:36:13.479
the end of this demonstration.
00:36:13.480 --> 00:36:17.000
Thank you for your time and attention.
