you
You
Wait, so there's a little something though. How are we gonna get the audio?
Give me just a second
Ah, so can we just get a confirmation? Can you start playing the talk now Samir, please?
So for everyone on the stream bear with us just a little bit we're trying to get it working right now
I'm not getting any audio from you Samir
Samir you might want to unmute yourself onto BBB. So if you could pause the video go to BBB and unmute yourself
Okay, Samir, can you hear me now? Yeah, okay, so
Oh, let me start. Where is it? Okay. There we go
That sounds great, okay, we'll give you just a second to get squared away here and thanks everybody on the stream for bearing with us
Okay, sure Samir. Can you now start playing your talk? Yeah
I'm Samir Pradhan from the Linguistic Data Consortium at the University of Pennsylvania. Can you pause the talk for a second?
What happened?
Oh, you don't have audio. The thing was no audio is
Oh
Okay, Samir, sorry, we were just doing some last-minute checks. So yes do exactly the same thing as you did will be fine and we'll manage on our end
Sorry everyone on the stream. We're just trying to do some last-minute shuffling
And you are muted on BBB right now, so you will probably need to pause the talk for a second
And you are muted on BBB right now, so you will probably need to unmute yourself on BBB and then start the talk
So Samir, right now, sorry, could you, no, it's not working. You need to unmute yourself on BBB
So right now you need to click the button, the microphone
Yes, you toggled it off again. Toggle it on again, please
What am I doing wrong?
So do not unmute yourself now. Leave your microphone on and press, go back to the beginning of your video and press play
Yes, from various signals
The work we present is limited to a limited number of people
So do not unmute yourself now. Leave your microphone on and press, go back to the beginning of your video and press play
Yes, from various signals
The work we present is limited to text and speech
Good approaching
But it can be extended
Thank you for joining me today. I am Samir Pradhan from the Linguistic Data Consortium at the University of Pennsylvania
And founder of osmantics.org
We research in computational linguistics, also known as natural language processing, a sub-area of artificial intelligence
With a focus on modeling and predicting complex linguistic structures from various signals
The work we present is limited to text and speech, but it can be extended to other signals
We propose an architecture, and we call it GRAIL, which allows the representation and aggregation of such rich structures in a systematic fashion
I'll demonstrate a proof of concept for representing and manipulating data and annotations for the specific purpose of building machine learning models that simulate understanding
These technologies have the potential for impact in almost any conceivable field that generates and uses data
We process human language when our brains receive and assimilate various signals, which are then manipulated and interpreted within a syntactic structure
It's a complex process that I have simplified here for the purpose of comparison to machine learning
Recent machine learning models tend to require a large amount of raw, naturally occurring data, and a varying amount of manually enriched data, commonly known as annotations
Owing to the complex and numerous nature of linguistic phenomena, we have most often used a divide-and-conquer approach
The strength of this approach is that it allows us to focus on a single or perhaps a few related linguistic phenomena
The weaknesses are the universe of these phenomena keep expanding as language itself evolves and changes over time
And second, this approach requires an additional task of aggregating annotations, creating more opportunities for computer error
Our challenge then is to find the sweet spot that allows us to encode complex information without the use of manual annotation or without the additional task of aggregation by computers
So what do I mean by annotation?
In this talk, the word annotation refers to the manual assignment of certain attributes to portions of a signal which is necessary to perform the end task
For example, in order for the algorithm to accurately interpret a pronoun, it needs to know what that pronoun refers back to
We may find this task trivial, however, current algorithms repeatedly fail in this task
So the complexities of understanding in computational linguistics require annotation
The word annotation itself is a useful example because it also reminds us that words have multiple meanings, as annotation itself does
Just as I needed to define it in this context so that my message won't be misinterpreted, so too must annotators do at least for algorithms through manual intervention
Learning from raw data, commonly known as unsupervised learning, poses limitations for machine learning
As I described, modeling complex phenomena need manual annotations
The learning algorithm uses these annotations as examples to build statistical models
This is called supervised learning
Without going into too much detail, I'll simply note that the recent popularity of the concept of deep learning is an evolutionary step where we have learned to train models using trillions of parameters in ways that they can learn richer hierarchical structures from very large amounts of annotated data
These models can then be fine-tuned using varying amounts of annotated examples, depending on the complexity of the task, to generate better predictions
As you might imagine, manually annotating complex linguistic phenomena can be a very specific, labor-intensive task
For example, imagine if we were to go back through this presentation and connect all the pronouns with the nouns to which they refer
Even for a short, 18-minute presentation, this would require hundreds of annotations
The models we build are only as good as the quality of the annotations we make
We need guidelines that ensure that the annotations are done by at least two humans who have substantial agreement with each other in their interpretations
We know that if we try to train a model using annotations that are very subjective or have more noise, we will receive poor predictions
Additionally, there is the concern of introducing various unexpected biases into one's models
So, annotation is really both an art and a science
In the remaining time, we will turn to two fundamental questions
First, how can we develop a unified representation of data and annotations that encompasses arbitrary levels of linguistic information?
There is a long history of attempting to answer this first question
This history is documented in our recent article
It is as if we as a community have been searching for our own holy grail
The second question we will pose on is what role might Emacs, along with Org Mode, play in this process?
While the solution itself may not be tied to Emacs, Emacs has built-in capabilities that could be useful for evaluating potential solutions
It is also one of the most extensively documented pieces of software and the most customizable piece of software that I have ever come across
Many would agree with that
In order to approach this second question, we turn to the complex structure of language itself
At first glance, language appears to us as a series of words
Words form sentences, sentences form paragraphs, and paragraphs form completed texts
If this was a sufficient description of the complexity of language, all of us would be able to read at least 10 different languages
We know it is much more complex than this
There is a rich, underlying, recursive tree structure
In fact, many possible tree structures, which makes a particular sequence, and many others
One of the better understood tree structures is the syntactic structure
While a natural language has rich ambiguities and complexities, programming languages are designed to be parsed and imprinted deterministically
Emacs has been used for programming very effectively, so there is a potential for using Emacs as a tool for annotation
This would significantly improve our current set of tools
It is important to note that most of the annotation tools that have been developed over the past few decades have relied on graphical indices
Even those used for enumerated textual indices
Most of the tools in use are designed for a user to add very specific, very restricted information
It has not really made use of the potential that an editor, rich editing environment like Emacs, can add to the mix
Emacs has long been able to edit and manipulate complex, embedded tree structures dependent in source code
It is difficult to imagine the capabilities that we represent naturally
In fact, it always does that, with features that allow us to quickly navigate sentences and graphs with a few keystrokes
Add various text studies, create overlays to name, etc.
Emacs has built up way too many control units, so we don't have to worry about the complexity of managing more languages
In fact, this is not the first time Emacs has used control linguistic sequences
One of the true moments in language natural language processing was the creation of a newly created syntactic tree for a million word collection of Wall Street articles
This was about 1990, before Java or Oracle interfaces were common
The tool that was used to create that corpus was Emacs, and it was created by Penn, known as the Penn Treebank
And in 1992, about when the Linguistic Consortium was established, it's been about 30 years that it has been creating various language related resources
The first outlining mode, in particular the outlining mode, rather enhanced the outlining mode
Allows us to create red outlines, attaching properties to nodes, and why this command is really customizing some of the various pieces of information as per one's requirement
This is also a very useful tool
This enhanced outlining mode provides more power to Emacs
It provides command for easily customizing, entering information, and at the same time hiding unnecessary context
It allows control editing, this could be a very useful tool when we are focused on a limited amount of data
The tool together allows us to create a rich representation that can simultaneously capture multiple possible sequences
Capture details necessary to read the original sources
Allows us to create hierarchical representation, wide structural capabilities that can take advantage of the concept of editance within the tree structure
Together allow local manipulance structure, thereby minimizing data coupling
The concept of tag outlining mode complements the hierarchy pattern
Hierarchies can be very rigid, but through tags on hierarchies we can have multi-faceted representations
As a matter of fact, outlining mode has the ability for tags to do their own hierarchical structure
Further enhances the representational power
All of this can be done as sequence, mostly for functional transformation
Because most capabilities can be configured and customized, it is not necessary to do everything at once
It allows us to intervene in the complexity of the representation
Finally, all of this can be done in plain tag representation
It has its own advantages
Now let's look at a simple example
The sentence is the thought of the moon with a telescope
Let's just make a view of the sentence
What is interesting is that it has a noun phrase i followed by an arrow to star
Then the moon is another phrase and the telescope is a positional phrase
Now, one thing you might remember from grammar school syntax is that there is a syntactical structure
And in this particular case
Because we know that the moon is not something that can hold the telescope
That seeing must be by me, or by eye, and the telescope must be in my hand, or I am viewing the moon with a telescope
However, it is possible that in a different context, the moon could be referred to an animated picture
And could hold the telescope, in that case, the situation might be that I am actually seeing a moon holding a telescope
And this is one of the most complex linguistic phenomena that requires world knowledge
And it is called the P-attachment problem, where the positional phrases can be ambiguous and various different cues have to be used to reduce the ambiguity
So in this case, as you saw, both the readings are technically true depending on the context
So one thing we could do is cut the tree and duplicate it, and then create another node and call it another node
And because this is one of the two interpreters, let's call one division A
And that division essentially is a tile of zone A, and it says that the moon is holding the telescope
Now we create another representation, where we capture the other interpretation, where the moon, or I am holding the telescope
Sorry everyone to interrupt the audio here. Sameer, can you move your mouse a little bit? Just move it to the corner of your screen, thank you
Now we have two separate interpreters in the same structure, and all we have to do is very quickly add a few keystrokes
Now let's add another interesting thing. This is two different interpreters. It can be A, it can be B, it can be C, it can be D, or it can be D
Basically, any entity that has the ability to see can be substituted in this particular node
And let's see what we have here. Now we are just getting a zoom view of the entire structure we have created
Essentially, you can see that by just using a few keystrokes, we are able to capture two different interpretations of a simple sentence
And we are also able to add various alternate pieces of information that could help machine algorithms generalize better
Now let's go to the next thing. In a sense, we can use the power of functional constructors to represent very potentially conflicting and structured readings
In addition to this, we can also create a text with different structure and have them in the same place. This allows us to address the interpretation of certain sentences that may be occurring in the world
While simultaneously giving information that can be more valuable. This makes the enrichment process all very efficient. Additionally, we can enrich the power of users of the feature or button who can not only expand, but also add information into it
In a way, that could help machine algorithms generalize better by making efficient use of their functions. Together, UX and Ardmo can speed the enrichment of nodes in a way that allows us to focus on certain aspects and ignore others
Extremely complex landscape structures can be captured consistently in a function that allows computers to understand the language
We can then use tools to enhance the tests that we do in our everyday life
However, this is the acronym or the type of specification that we are creating to capture this new and present virtual adaptation
We will now look at an example of spontaneous speech that occurs in spoken conversations. Conversations consistently contain interest in speech, interrupts, disfluency, verbal nouns such as talk or laugh, and other noises
Since spontaneous speech is simply a functional stream, we cannot take back words that come out of our mouths. We tend to make mistakes and correct ourselves as soon as we realize that we have spoken
This process manifests through a combination of a handful of mechanisms, including immediate action after an error, and we do this unconsciously
What we've taught here is an example of a language that has various aspects of the representation
We don't have time to go through many of the details. I would highly encourage you to play. I'm making some videos for ASCII cinemas that I'll be posting and if you're interested you can go through those
The idea here is to try a slightly more complex use case, but given the time consumption and the amount of information that can fit in the screen, this should be very informative
But at least you'll see some idea of what can be followed. In this particular case, you're saying that there's a sense which is what I am telling now
Essentially, there is a repetition of the I am, then there is a proper word, nobody can try to say the same thing but start by saying true, and then correct themselves by telling now
So in this case, we can capture a sequence of words
The interesting thing is that in NLB, sometimes we have to typically use words that have this interpretation of the context of I am
You can see that here, this view shows that with each of the words in the sentence or in the representation, you can have a lot of different properties that can attach to them
And these properties are typical then, like in the earlier slide, but you can use the cues of all these properties to various kind of searches and filtering
And the slide here is actually not a legitimate text, on the right are descriptions of what each of these present. This information is also available in the article and you can see there
But it shows how rich a context you can capture, it's just a closer snapshot of the properties on the word
And you can see we can have like whether the word is broken or not, it's incomplete, whether some words want to be filtered for parsing, and say this is ignored
Or some words are restart marks, we can add a restart marker, sometimes some of these migrations
The other fascinating thing about this presentation is that you can edit properties in the content view
So you have this pillar data structure and combining hierarchical data structure, as you can see, you may not be able to see here
What has also happened here is that some of the tags have been inherited from earlier groups, and so you get a much better picture of things, and essentially you can filter out things that you want to access, access them, and then integrate it into the model
So in conclusion today we have posed and implemented the use of the architecture layout, which allows representation, manipulation, and recognition of rich linguistic structure in systematic fashion
We've shown how Google advances tools available for building machine learning models to simulate understanding
Thank you Verj for your attention and contact information on this slide
If you are interested in an additional sample to demonstrate the representation of speech and retext together, continue, otherwise we'll stop here
Is it okay to stop?
Yes Amir, it's okay to stop now. Thank you so much for your talk. Are you able to see the pad on your end?
In the etherpad?
Yes, in the etherpad, do you have the link?
I'm there, nothing has happened so far
I'm going to put a link to the pad, give me just a second right now. Colwyn, feel free to interrupt me whenever you're here
I'm actually looking at the pad, I don't think anything is added in
There don't seem to be questions yet, yes. It's probably because of the audio problem people might have a little bit of trouble hearing you talk
Do you have anything else you'd like to add on your talk maybe? Because I think it was an excruciating process to get it out to us
You had to get a lot of darlings in the process didn't you?
Yeah, in the process of preparing this talk you had to select a lot of stuff that you wanted to include in your talk
Can I ask you to put on your webcam or something? Are you able to do this?
I'm starting to see a few questions come in. Just let us know when you're ready
Colwyn, I'll let you take over. Can you hear me?
Yeah, I hear you, the audio is just a little bit choppy, but we'll just talk slowly and hopefully that will work fine
Well thanks for the great talk, that was just kind of mind blowing actually, I'm looking forward to re-watching it probably two or three times
Who is this?
This is Colwyn again
Okay, so we do have a few questions coming in
I'm going to answer them
Okay, well I can read them to you and then we'll transcribe your answers if you'd like to answer them live
Oh, I see, let me do that. The identity you've come up as the pantry, that has been to depth and people are putting out perfect scores on that
But that's not quite the point, I mean sometimes
Oh, I should also speak slowly
Sometimes the research community goes too far and reuses the evaluations and doesn't really transfer to domains
But our richer and newer data that are available is always, we're in the process, I am currently and a couple of my colleagues, we're getting new data so that we can actually make sure the learning model is better
Oh shoot, and then I failed to unmute myself on the stream here
And I think you're answering in text right now one of these, so I'll just let you drive
So one thing I'll add is, please read the question that you're answering when you read out your answers
Oh, I see, yes
And we're showing the pad on the stream so people are seeing the text and that's probably a good approach considering we're having a little shakiness with the audio
In fact, I think my audio may be pretty stable, so I'll just start reading out both the questions and the answers
But Samir, if you want to, you're welcome to interrupt me if you want to expand on your remarks at all
So the first question was, has the 92U pin corpus of articles feat been reproduced over and over again using these tools
The answer was not quite, that was sort of a first wave, the particular corpus was the first one that started a revolution, kind of
But there are more corpus being made available, in fact I spent about 8 years, a decade ago, building a much larger corpus with more layers of information
And it is called the Onto Notes, it covers Chinese and Arabic, DARPA funded, this is freely available for research to anyone, anywhere
That was quite a feature, quite a feat
The next question, is this only for natural languages like English or more general, would this be used for programming languages
Samir said, I am using English as a use case, but the idea is to have it completely multilingual
I cannot think why you would want to use it for programming languages, in fact the AST in programming languages is sort of what we are trying to build upon
So that one can capture the abstract representation and help the models learn better
These days the models are trained on a boatload of data, and so they tend to be overfitted to the data
So if you have a smaller data set, which is not quite the same as the one that you had the training data for, then the models really do poorly
It is sometimes compared to learning the sine function, using the points on the sine wave, as opposed to deriving the function itself
You can get close, but then you cannot really do a lot better with that model
This is sort of what is happening with the deep learning hype
It is not to say that there hasn't been a significant advancement in the tech, in the technologies
But to say that the models can learn is an extreme overstatement
Awesome answer. I'm going to scroll my copy of the pad down just a little bit, and we'll just take a moment to start looking at the next question
So I'll read that out. Reminds me of the advantages of pre-computer copy and paste, cut up paper and rearrange, but having more stuff with your pieces
Right. Kind of like that, but more intelligent than copy-paste, because you could have various local constraints that would ensure the information is consistent with the whole
I am also envisioning this as a use case of hooks
And if you can have rich local dependencies, then you can be sure, as much as you can, that the information signal is not too corrupted
Have you used it on real life situations? No. I am probably the only person who is doing this crazy thing
It would be nice, or rather, I have a feeling that something like this, if worked upon for a while, by many people, by many, might lead to a really really potent tool for the masses
I feel strongly about using, sorry, I feel strongly about giving such power to the users
And be able to edit and share the data openly, so that they are not stuck in some corporate vault somewhere
Amen. One thing at a time. Plus one for that as well.
Alright, and I will read out the next question. Do you see this as a format for this type of annotation specifically, or something more general that can be used for interlinear glosses, lexicons, etc?
Absolutely. In fact, the project I mentioned, One Notes, has multiple layers of annotation, one of them being the propositional structure, which it uses for a large lexicon that covers about 15k verbs, so 15,000 verbs, and nouns
and all their argument structures that we have been seeing so far in the corpora
This is about a million propositions that have been released recently
We just recently celebrated a 20th birthday of the Corpus. It is called the Prop Bank.
There is an interesting history of the banks. It started with Tree Bank, and then there was Prop Bank, with a capital B
But then, when we were developing Onto Notes, which contains syntax, named entities, conference resolution, propositions, word sense, all in the same hole
Sorry for the interruption. We have about 5 minutes and 15 seconds.
That sounds good. If you want to just read it out, then. I think that would be the most important thing, that people can hear your answers, and I and the other volunteers will be going through and trying to transcribe this.
So go for it.
So, Samuel, just to make sure, did you have something to say right now?
Oh, okay. I think these are all good questions, and there is a lot of it, and clearly the amount of time is not enough.
But I am trying to figure out how to have a community that can help such a person.
One of the things that I am thinking that this could make possible is to take all the disparate resources that have inconsistent or not quite compatible additions on them,
and which are right now just iso of data, small island of data floating in the sea. But representation could really bring them all together, and then they could be much richer, full, and consistent.
Like you said, one of you was asking about the islands and the subcorporas that have sentiment and information.
I am, yeah, there's a lot of various. Common people, the way it could be used for common people is to potentially make them available that currently doesn't recognize the current models on dual land,
so that more people can use the data and not be biased towards one or the other.
And there are some things, when people train these models using huge amounts of data, no matter how big the data is, it is a small cross-section of the universe of data, and depending on what drop select will be your model, those will be the seconds for those.
And some people will be interested in using them on purpose X, but somebody else might want to use them on purpose Y, and if the data is not in, then it's harder to do that.
Okay, so I think we've got just about 100 seconds left, so if you have any closing remarks you want to share, and then we'll start transitioning.
Thank you so much, I really appreciate, this was a great experience, frankly.
I've never had a complete pre-related level of talk before, I guess, in a way it was for a different audience. It was extremely helpful, and I learned that planning sort of tried to create a community.
Thank you so much.
I'll take it over, we are going to move to the next talk. Thank you so much, Samir, and sorry for the technical difficulty.
As Corbin said, we will try to manage as much of the information that was shared during this Q&A, we will file everything away where we can use it, and make captions and all this, so don't worry about the difficulty.
Thank you.