diff options
Diffstat (limited to '2022/captions')
| -rw-r--r-- | 2022/captions/emacsconf-2022-grail--graila-generalized-representation-and-aggregation-of-information-layers--sameer-pradhan--main.vtt | 1945 |
1 files changed, 1945 insertions, 0 deletions
diff --git a/2022/captions/emacsconf-2022-grail--graila-generalized-representation-and-aggregation-of-information-layers--sameer-pradhan--main.vtt b/2022/captions/emacsconf-2022-grail--graila-generalized-representation-and-aggregation-of-information-layers--sameer-pradhan--main.vtt new file mode 100644 index 00000000..a642f94a --- /dev/null +++ b/2022/captions/emacsconf-2022-grail--graila-generalized-representation-and-aggregation-of-information-layers--sameer-pradhan--main.vtt @@ -0,0 +1,1945 @@ +WEBVTT captioned by sameer + +NOTE Introduction + +00:00:00.000 --> 00:00:05.839 +Thank you for joining me today. I'm Sameer Pradhan + +00:00:05.840 --> 00:00:07.799 +from the Linguistic Data Consortium + +00:00:07.800 --> 00:00:10.079 +at the University of Pennsylvania + +00:00:10.080 --> 00:00:14.519 +and founder of cemantix.org . + +00:00:14.520 --> 00:00:16.879 +Today we'll be addressing research + +00:00:16.880 --> 00:00:18.719 +in computational linguistics, + +00:00:18.720 --> 00:00:22.039 +also known as natural language processing + +00:00:22.040 --> 00:00:24.719 +a sub area of artificial intelligence + +00:00:24.720 --> 00:00:27.759 +with a focus on modeling and predicting + +00:00:27.760 --> 00:00:31.919 +complex linguistic structures from various signals. + +00:00:31.920 --> 00:00:35.799 +The work we present is limited to text and speech signals. + +00:00:35.800 --> 00:00:38.639 +but it can be extended to other signals. + +00:00:38.640 --> 00:00:40.799 +We propose an architecture, + +00:00:40.800 --> 00:00:42.959 +and we call it GRAIL, which allows + +00:00:42.960 --> 00:00:44.639 +the representation and aggregation + +00:00:44.640 --> 00:00:50.199 +of such rich structures in a systematic fashion. + +00:00:50.200 --> 00:00:52.679 +I'll demonstrate a proof of concept + +00:00:52.680 --> 00:00:56.559 +for representing and manipulating data and annotations + +00:00:56.560 --> 00:00:58.519 +for the specific purpose of building + +00:00:58.520 --> 00:01:02.879 +machine learning models that simulate understanding. + +00:01:02.880 --> 00:01:05.679 +These technologies have the potential for impact + +00:01:05.680 --> 00:01:09.119 +in almost every conceivable field + +00:01:09.120 --> 00:01:13.399 +that generates and uses data. + +NOTE Processing language + +00:01:13.400 --> 00:01:15.039 +We process human language + +00:01:15.040 --> 00:01:16.719 +when our brains receive and assimilate + +00:01:16.720 --> 00:01:20.079 +various signals which are then manipulated + +00:01:20.080 --> 00:01:23.879 +and interpreted within a syntactic structure. + +00:01:23.880 --> 00:01:27.319 +it's a complex process that I have simplified here + +00:01:27.320 --> 00:01:30.759 +for the purpose of comparison to machine learning. + +00:01:30.760 --> 00:01:33.959 +Recent machine learning models tend to require + +00:01:33.960 --> 00:01:37.039 +a large amount of raw, naturally occurring data + +00:01:37.040 --> 00:01:40.199 +and a varying amount of manually enriched data, + +00:01:40.200 --> 00:01:43.199 +commonly known as "annotations". + +00:01:43.200 --> 00:01:45.959 +Owing to the complex and numerous nature + +00:01:45.960 --> 00:01:49.959 +of linguistic phenomena, we have most often used + +00:01:49.960 --> 00:01:52.999 +a divide and conquer approach. + +00:01:53.000 --> 00:01:55.399 +The strength of this approach is that it allows us + +00:01:55.400 --> 00:01:58.159 +to focus on a single, or perhaps a few related + +00:01:58.160 --> 00:02:00.439 +linguistic phenomena. + +00:02:00.440 --> 00:02:03.879 +The weaknesses are the universe of these phenomena + +00:02:03.880 --> 00:02:07.239 +keep expanding, as language itself + +00:02:07.240 --> 00:02:09.359 +evolves and changes over time, + +00:02:09.360 --> 00:02:13.119 +and second, this approach requires an additional task + +00:02:13.120 --> 00:02:14.839 +of aggregating the interpretations, + +00:02:14.840 --> 00:02:18.359 +creating more opportunities for computer error. + +00:02:18.360 --> 00:02:21.519 +Our challenge, then, is to find the sweet spot + +00:02:21.520 --> 00:02:25.239 +that allows us to encode complex information + +00:02:25.240 --> 00:02:27.719 +without the use of manual annotation, + +00:02:27.720 --> 00:02:34.559 +or without the additional task of aggregation by computers. + +NOTE Annotation + +00:02:34.560 --> 00:02:37.119 +So what do I mean by "annotation"? + +00:02:37.120 --> 00:02:39.759 +In this talk the word annotation refers to + +00:02:39.760 --> 00:02:43.519 +the manual assignment of certain attributes + +00:02:43.520 --> 00:02:48.639 +to portions of a signal which is necessary + +00:02:48.640 --> 00:02:51.639 +to perform the end task. + +00:02:51.640 --> 00:02:54.439 +For example, in order for the algorithm + +00:02:54.440 --> 00:02:57.439 +to accurately interpret a pronoun, + +00:02:57.440 --> 00:03:00.279 +it needs to know that pronoun, + +00:03:00.280 --> 00:03:03.799 +what that pronoun refers back to. + +00:03:03.800 --> 00:03:06.719 +We may find this task trivial, however, + +00:03:06.720 --> 00:03:10.599 +current algorithms repeatedly fail in this task. + +00:03:10.600 --> 00:03:13.319 +So the complexities of understanding + +00:03:13.320 --> 00:03:16.639 +in computational linguistics require annotation. + +00:03:16.640 --> 00:03:20.799 +The world annotation itself is a useful example, + +00:03:20.800 --> 00:03:22.679 +because it also reminds us + +00:03:22.680 --> 00:03:25.119 +that words have multiple meetings + +00:03:25.120 --> 00:03:27.519 +as annotation itself does— + +00:03:27.520 --> 00:03:30.559 +just as I needed to define it in this context, + +00:03:30.560 --> 00:03:33.799 +so that my message won't be misinterpreted. + +00:03:33.800 --> 00:03:39.039 +So, too, must annotators do this for algorithms + +00:03:39.040 --> 00:03:43.239 +through the manual intervention. + +NOTE Learning from data + +00:03:43.240 --> 00:03:44.759 +Learning from raw data + +00:03:44.760 --> 00:03:47.039 +(commonly known as unsupervised learning) + +00:03:47.040 --> 00:03:50.079 +poses limitations for machine learning. + +00:03:50.080 --> 00:03:53.039 +As I described, modeling complex phenomena + +00:03:53.040 --> 00:03:55.559 +need manual annotations. + +00:03:55.560 --> 00:03:58.559 +The learning algorithm uses these annotations + +00:03:58.560 --> 00:04:01.319 +as examples to build statistical models. + +00:04:01.320 --> 00:04:04.879 +This is called supervised learning. + +00:04:04.880 --> 00:04:06.319 +Without going into too much detail, + +00:04:06.320 --> 00:04:10.039 +I'll simply note that the recent popularity + +00:04:10.040 --> 00:04:12.519 +of the concept of deep learning + +00:04:12.520 --> 00:04:14.679 +is that evolutionary step + +00:04:14.680 --> 00:04:17.319 +where we have learned to train models + +00:04:17.320 --> 00:04:20.799 +using trillions of parameters in ways that they can + +00:04:20.800 --> 00:04:25.079 +learn richer hierarchical structures + +00:04:25.080 --> 00:04:29.399 +from very large amounts of annotate, unannotated data. + +00:04:29.400 --> 00:04:32.319 +These models can then be fine-tuned, + +00:04:32.320 --> 00:04:35.599 +using varying amounts of annotated examples + +00:04:35.600 --> 00:04:37.639 +depending on the complexity of the task + +00:04:37.640 --> 00:04:39.679 +to generate better predictions. + +NOTE Manual annotation + +00:04:39.680 --> 00:04:44.919 +As you might imagine, manually annotating + +00:04:44.920 --> 00:04:47.359 +complex, linguistic phenomena + +00:04:47.360 --> 00:04:51.719 +can be very specific, labor-intensive task. + +00:04:51.720 --> 00:04:54.279 +For example, imagine if we were + +00:04:54.280 --> 00:04:56.399 +to go back through this presentation + +00:04:56.400 --> 00:04:58.399 +and connect all the pronouns + +00:04:58.400 --> 00:04:59.919 +with the nouns to which they refer. + +00:04:59.920 --> 00:05:03.239 +Even for a short 18 min presentation, + +00:05:03.240 --> 00:05:05.239 +this would require hundreds of annotations. + +00:05:05.240 --> 00:05:08.519 +The models we build are only as good + +00:05:08.520 --> 00:05:11.119 +as the quality of the annotations we make. + +00:05:11.120 --> 00:05:12.679 +We need guidelines + +00:05:12.680 --> 00:05:15.759 +that ensure that the annotations are done + +00:05:15.760 --> 00:05:19.719 +by at least two humans who have substantial agreement + +00:05:19.720 --> 00:05:22.119 +with each other in their interpretations. + +00:05:22.120 --> 00:05:25.599 +We know that if we try to trade a model using annotations + +00:05:25.600 --> 00:05:28.519 +that are very subjective, or have more noise, + +00:05:28.520 --> 00:05:30.919 +we will receive poor predictions. + +00:05:30.920 --> 00:05:33.679 +Additionally, there is the concern of introducing + +00:05:33.680 --> 00:05:37.079 +various unexpected biases into one's models. + +00:05:37.080 --> 00:05:44.399 +So annotation is really both an art and a science. + +NOTE How can we develop a unified representation? + +00:05:44.400 --> 00:05:47.439 +In the remaining time, + +00:05:47.440 --> 00:05:49.999 +we will turn to two fundamental questions. + +00:05:50.000 --> 00:05:54.239 +First, how can we develop a unified representation + +00:05:54.240 --> 00:05:55.599 +of data and annotations + +00:05:55.600 --> 00:05:59.759 +that encompasses arbitrary levels of linguistic information? + +00:05:59.760 --> 00:06:03.839 +There is a long history of attempting to answer + +00:06:03.840 --> 00:06:04.839 +this first question. + +00:06:04.840 --> 00:06:08.839 +This history is documented in our recent article, + +00:06:08.840 --> 00:06:11.519 +and you can refer to that article. + +00:06:11.520 --> 00:06:16.719 +It will be on the website. + +00:06:16.720 --> 00:06:18.999 +It is as if we, as a community, + +00:06:19.000 --> 00:06:22.519 +have been searching for our own Holy Grail. + +NOTE What role might Emacs and Org mode play? + +00:06:22.520 --> 00:06:26.519 +The second question we will pose is + +00:06:26.520 --> 00:06:30.159 +what role might Emacs, along with Org mode, + +00:06:30.160 --> 00:06:31.919 +play in this process? + +00:06:31.920 --> 00:06:35.359 +Well, the solution itself may not be tied to Emacs. + +00:06:35.360 --> 00:06:38.359 +Emacs has built in capabilities + +00:06:38.360 --> 00:06:42.599 +that could be useful for evaluating potential solutions. + +00:06:42.600 --> 00:06:45.759 +It's also one of the most extensively documented + +00:06:45.760 --> 00:06:48.519 +pieces of software and the most customizable + +00:06:48.520 --> 00:06:51.599 +piece of software that I have ever come across, + +00:06:51.600 --> 00:06:55.279 +and many would agree with that. + +NOTE The complex structure of language + +00:06:55.280 --> 00:07:00.639 +In order to approach this second question, + +00:07:00.640 --> 00:07:03.919 +we turn to the complex structure of language itself. + +00:07:03.920 --> 00:07:07.679 +At first glance, language appears to us + +00:07:07.680 --> 00:07:09.879 +as a series of words. + +00:07:09.880 --> 00:07:13.439 +Words form sentences, sentences form paragraphs, + +00:07:13.440 --> 00:07:16.239 +and paragraphs form completed text. + +00:07:16.240 --> 00:07:19.039 +If this was a sufficient description + +00:07:19.040 --> 00:07:21.159 +of the complexity of language, + +00:07:21.160 --> 00:07:24.199 +all of us would be able to speak and read + +00:07:24.200 --> 00:07:26.559 +at least ten different languages. + +00:07:26.560 --> 00:07:29.279 +We know it is much more complex than this. + +00:07:29.280 --> 00:07:33.199 +There is a rich, underlying recursive tree structure-- + +00:07:33.200 --> 00:07:36.439 +in fact, many possible tree structures + +00:07:36.440 --> 00:07:39.439 +which makes a particular sequence meaningful + +00:07:39.440 --> 00:07:42.079 +and many others meaningless. + +00:07:42.080 --> 00:07:45.239 +One of the better understood tree structures + +00:07:45.240 --> 00:07:47.119 +is the syntactic structure. + +00:07:47.120 --> 00:07:49.439 +While natural language + +00:07:49.440 --> 00:07:51.679 +has rich ambiguities and complexities, + +00:07:51.680 --> 00:07:55.119 +programming languages are designed to be parsed + +00:07:55.120 --> 00:07:56.999 +and interpreted deterministically. + +00:07:57.000 --> 00:08:02.159 +Emacs has been used for programming very effectively. + +00:08:02.160 --> 00:08:05.359 +So there is a potential for using Emacs + +00:08:05.360 --> 00:08:06.559 +as a tool for annotation. + +00:08:06.560 --> 00:08:10.799 +This would significantly improve our current set of tools. + +NOTE Annotation tools + +00:08:10.800 --> 00:08:16.559 +It is important to note that most of the annotation tools + +00:08:16.560 --> 00:08:19.639 +that have been developed over the past few decades + +00:08:19.640 --> 00:08:22.879 +have relied on graphical interfaces, + +00:08:22.880 --> 00:08:26.919 +even those used for enriching textual information. + +00:08:26.920 --> 00:08:30.399 +Most of the tools in current use + +00:08:30.400 --> 00:08:36.159 +are designed for a end user to add very specific, + +00:08:36.160 --> 00:08:38.639 +very restricted information. + +00:08:38.640 --> 00:08:42.799 +We have not really made use of the potential + +00:08:42.800 --> 00:08:45.639 +that an editor or a rich editing environment like Emacs + +00:08:45.640 --> 00:08:47.239 +can add to the mix. + +00:08:47.240 --> 00:08:52.479 +Emacs has long enabled the editing of, the manipulation of + +00:08:52.480 --> 00:08:56.359 +complex embedded tree structures abundant in source code. + +00:08:56.360 --> 00:08:58.599 +So it's not difficult to imagine that it would have + +00:08:58.600 --> 00:09:00.359 +many capabilities that we we need + +00:09:00.360 --> 00:09:02.599 +to represent actual language. + +00:09:02.600 --> 00:09:04.759 +In fact, it already does that with features + +00:09:04.760 --> 00:09:06.399 +that allow us to quickly navigate + +00:09:06.400 --> 00:09:07.919 +through sentences and paragraphs, + +00:09:07.920 --> 00:09:09.799 +and we don't need a few key strokes. + +00:09:09.800 --> 00:09:13.599 +Or to add various text properties to text spans + +00:09:13.600 --> 00:09:17.039 +to create overlays, to name but a few. + +00:09:17.040 --> 00:09:22.719 +Emacs figured out this way to handle Unicode, + +00:09:22.720 --> 00:09:26.799 +so you don't even have to worry about the complexity + +00:09:26.800 --> 00:09:29.439 +of managing multiple languages. + +00:09:29.440 --> 00:09:34.039 +It's built into Emacs. In fact, this is not the first time + +00:09:34.040 --> 00:09:37.399 +Emacs has been used for linguistic analysis. + +00:09:37.400 --> 00:09:41.159 +One of the breakthrough moments in language, + +00:09:41.160 --> 00:09:44.439 +natural language processing was the creation + +00:09:44.440 --> 00:09:48.639 +of manually created syntactic trees + +00:09:48.640 --> 00:09:50.439 +for a 1 million word collection + +00:09:50.440 --> 00:09:52.399 +of Wall Street Journal articles. + +00:09:52.400 --> 00:09:54.879 +This was else around 1992 + +00:09:54.880 --> 00:09:59.279 +before Java or graphical interfaces were common. + +00:09:59.280 --> 00:10:03.279 +The tool that was used to create that corpus was Emacs. + +00:10:03.280 --> 00:10:08.959 +It was created at UPenn, and is famously known as + +00:10:08.960 --> 00:10:12.719 +the Penn Treebank. '92 was about when + +00:10:12.720 --> 00:10:16.439 +the Linguistic Data Consortium was also established, + +00:10:16.440 --> 00:10:18.039 +and it's been about 30 years + +00:10:18.040 --> 00:10:20.719 +that it has been creating various + +00:10:20.720 --> 00:10:22.359 +language-related resources. + +NOTE Org mode + +00:10:22.360 --> 00:10:28.519 +Org mode--in particular, the outlining mode, + +00:10:28.520 --> 00:10:32.399 +or rather the enhanced form of outlining mode-- + +00:10:32.400 --> 00:10:35.599 +allows us to create rich outlines, + +00:10:35.600 --> 00:10:37.799 +attaching properties to nodes, + +00:10:37.800 --> 00:10:41.119 +and provides commands for easily customizing + +00:10:41.120 --> 00:10:43.879 +sorting of various pieces of information + +00:10:43.880 --> 00:10:45.639 +as per one's requirement. + +00:10:45.640 --> 00:10:50.239 +This can also be a very useful tool. + +00:10:50.240 --> 00:10:59.159 +This enhanced form of outline-mode adds more power to Emacs. + +00:10:59.160 --> 00:11:03.359 +It provides commands for easily customizing + +00:11:03.360 --> 00:11:05.159 +and filtering information, + +00:11:05.160 --> 00:11:08.999 +while at the same time hiding unnecessary context. + +00:11:09.000 --> 00:11:11.919 +It also allows structural editing. + +00:11:11.920 --> 00:11:16.039 +This can be a very useful tool to enrich corpora + +00:11:16.040 --> 00:11:20.919 +where we are focusing on limited amount of phenomena. + +00:11:20.920 --> 00:11:24.519 +The two together allow us to create + +00:11:24.520 --> 00:11:27.199 +a rich representation + +00:11:27.200 --> 00:11:32.999 +that can simultaneously capture multiple possible sequences, + +00:11:33.000 --> 00:11:38.759 +capture details necessary to recreate the original source, + +00:11:38.760 --> 00:11:42.079 +allow the creation of hierarchical representation, + +00:11:42.080 --> 00:11:44.679 +provide structural editing capabilities + +00:11:44.680 --> 00:11:47.439 +that can take advantage of the concept of inheritance + +00:11:47.440 --> 00:11:48.999 +within the tree structure. + +00:11:49.000 --> 00:11:54.279 +Together they allow local manipulations of structures, + +00:11:54.280 --> 00:11:56.199 +thereby minimizing data coupling. + +00:11:56.200 --> 00:11:59.119 +The concept of tags in Org mode + +00:11:59.120 --> 00:12:01.599 +complement the hierarchy part. + +00:12:01.600 --> 00:12:03.839 +Hierarchies can be very rigid, + +00:12:03.840 --> 00:12:06.039 +but to tags on hierarchies, + +00:12:06.040 --> 00:12:08.839 +we can have a multifaceted representations. + +00:12:08.840 --> 00:12:12.759 +As a matter of fact, Org mode has the ability for the tags + +00:12:12.760 --> 00:12:15.039 +to have their own hierarchical structure + +00:12:15.040 --> 00:12:18.639 +which further enhances the representational power. + +00:12:18.640 --> 00:12:22.639 +All of this can be done as a sequence + +00:12:22.640 --> 00:12:25.679 +of mostly functional data transformations, + +00:12:25.680 --> 00:12:27.439 +because most of the capabilities + +00:12:27.440 --> 00:12:29.759 +can be configured and customized. + +00:12:29.760 --> 00:12:32.799 +It is not necessary to do everything at once. + +00:12:32.800 --> 00:12:36.199 +Instead, it allows us to incrementally increase + +00:12:36.200 --> 00:12:37.919 +the complexity of the representation. + +00:12:37.920 --> 00:12:39.799 +Finally, all of this can be done + +00:12:39.800 --> 00:12:42.359 +in plain-text representation + +00:12:42.360 --> 00:12:45.479 +which comes with its own advantages. + +NOTE Example + +00:12:45.480 --> 00:12:50.679 +Now let's take a simple example. + +00:12:50.680 --> 00:12:55.999 +This is a a short video that I'll play. + +00:12:56.000 --> 00:12:59.679 +The sentence is "I saw the moon with a telescope," + +00:12:59.680 --> 00:13:03.999 +and let's just make a copy of the sentence. + +00:13:04.000 --> 00:13:09.199 +What we can do now is to see: + +00:13:09.200 --> 00:13:11.879 +what does this sentence comprise? + +00:13:11.880 --> 00:13:13.679 +It has a noun phrase "I," + +00:13:13.680 --> 00:13:17.479 +followed by a word "saw." + +00:13:17.480 --> 00:13:21.359 +Then "the moon" is another noun phrase, + +00:13:21.360 --> 00:13:24.839 +and "with the telescope" is a prepositional phrase. + +00:13:24.840 --> 00:13:30.759 +Now one thing that you might remember, + +00:13:30.760 --> 00:13:36.119 +from grammar school or syntax is that + +00:13:36.120 --> 00:13:41.279 +there is a syntactic structure. + +00:13:41.280 --> 00:13:44.359 +And if you in this particular case-- + +00:13:44.360 --> 00:13:47.919 +because we know that the moon is not typically + +00:13:47.920 --> 00:13:51.679 +something that can hold the telescope, + +00:13:51.680 --> 00:13:56.239 +that the seeing must be done by me or "I," + +00:13:56.240 --> 00:14:01.039 +and the telescope must be in my hand, + +00:14:01.040 --> 00:14:04.479 +or "I" am viewing the moon with a telescope. + +00:14:04.480 --> 00:14:13.519 +However, it is possible that in a different context + +00:14:13.520 --> 00:14:17.159 +the moon could be referring to an animated character + +00:14:17.160 --> 00:14:22.319 +in a animated series, and could actually hold the telescope. + +00:14:22.320 --> 00:14:23.479 +And this is one of the most-- + +00:14:23.480 --> 00:14:24.839 +the oldest and one of the most-- + +00:14:24.840 --> 00:14:26.319 +and in that case the situation might be + +00:14:26.320 --> 00:14:30.959 +that I'm actually seeing the moon holding a telescope... + +00:14:30.960 --> 00:14:36.079 +I mean. The moon is holding the telescope, + +00:14:36.080 --> 00:14:40.959 +and I'm just seeing the moon holding the telescope. + +00:14:40.960 --> 00:14:47.999 +Complex linguistic ambiguity or linguistic + +00:14:48.000 --> 00:14:53.599 +phenomena that requires world knowledge, + +00:14:53.600 --> 00:14:55.719 +and it's called the PP attachment problem + +00:14:55.720 --> 00:14:59.239 +where the propositional phrase attachment + +00:14:59.240 --> 00:15:04.599 +can be ambiguous, and various different contextual cues + +00:15:04.600 --> 00:15:06.879 +have to be used to resolve the ambiguity. + +00:15:06.880 --> 00:15:09.079 +So in this case, as you saw, + +00:15:09.080 --> 00:15:11.199 +both the readings are technically true, + +00:15:11.200 --> 00:15:13.959 +depending on different contexts. + +00:15:13.960 --> 00:15:16.599 +So one thing we could do is just + +00:15:16.600 --> 00:15:19.919 +to cut the tree and duplicate it, + +00:15:19.920 --> 00:15:21.599 +and then let's create another node + +00:15:21.600 --> 00:15:24.479 +and call it an "OR" node. + +00:15:24.480 --> 00:15:26.119 +And because we are saying, + +00:15:26.120 --> 00:15:28.359 +this is one of the two interpretations. + +00:15:28.360 --> 00:15:32.159 +Now let's call one interpretation "a", + +00:15:32.160 --> 00:15:36.159 +and that interpretation essentially + +00:15:36.160 --> 00:15:39.319 +is this child of that node "a" + +00:15:39.320 --> 00:15:41.799 +and that says that the moon + +00:15:41.800 --> 00:15:43.999 +is holding the telescope. + +00:15:44.000 --> 00:15:46.359 +Now we can create another representation "b" + +00:15:46.360 --> 00:15:53.919 +where we capture the other interpretation, + +00:15:53.920 --> 00:15:59.959 +where this, the act, the moon or--I am actually + +00:15:59.960 --> 00:16:00.519 +holding the telescope, + +00:16:00.520 --> 00:16:06.799 +and watching the moon using it. + +00:16:06.800 --> 00:16:09.199 +So now we have two separate interpretations + +00:16:09.200 --> 00:16:11.679 +in the same structure, + +00:16:11.680 --> 00:16:15.519 +and all we do--we're able to do is with this, + +00:16:15.520 --> 00:16:18.159 +with very quick key strokes now... + +00:16:18.160 --> 00:16:22.439 +While we are at it, let's add another interesting thing, + +00:16:22.440 --> 00:16:25.159 +this node that represents "I": + +00:16:25.160 --> 00:16:28.919 +"He." It can be "She". + +00:16:28.920 --> 00:16:35.759 +It can be "the children," or it can be "The people". + +00:16:35.760 --> 00:16:45.039 +Basically, any entity that has the capability to "see" + +00:16:45.040 --> 00:16:53.359 +can be substituted in this particular node. + +00:16:53.360 --> 00:16:57.399 +Let's see what we have here now. + +00:16:57.400 --> 00:17:01.239 +We just are getting sort of a zoom view + +00:17:01.240 --> 00:17:04.599 +of the entire structure, what we created, + +00:17:04.600 --> 00:17:08.039 +and essentially you can see that + +00:17:08.040 --> 00:17:11.879 +by just, you know, using a few keystrokes, + +00:17:11.880 --> 00:17:17.839 +we were able to capture two different interpretations + +00:17:17.840 --> 00:17:20.879 +of a a simple sentence, + +00:17:20.880 --> 00:17:23.759 +and they are also able to add + +00:17:23.760 --> 00:17:27.799 +these alternate pieces of information + +00:17:27.800 --> 00:17:30.559 +that could help machine learning algorithms + +00:17:30.560 --> 00:17:32.439 +generalize better. + +00:17:32.440 --> 00:17:36.239 +All right. + +NOTE Different readings + +00:17:36.240 --> 00:17:40.359 +Now, let's look at the next thing. So in a sense, + +00:17:40.360 --> 00:17:46.679 +we can use this power of functional data structures + +00:17:46.680 --> 00:17:50.239 +to represent various potentially conflicting + +00:17:50.240 --> 00:17:55.559 +and structural readings of that piece of text. + +00:17:55.560 --> 00:17:58.079 +In addition to that, we can also create more texts, + +00:17:58.080 --> 00:17:59.799 +each with different structure, + +00:17:59.800 --> 00:18:01.559 +and have them all in the same place. + +00:18:01.560 --> 00:18:04.239 +This allows us to address the interpretation + +00:18:04.240 --> 00:18:06.879 +of a static sentence that might be occurring in the world, + +00:18:06.880 --> 00:18:09.639 +while simultaneously inserting information + +00:18:09.640 --> 00:18:11.519 +that would add more value to it. + +00:18:11.520 --> 00:18:14.999 +This makes the enrichment process also very efficient. + +00:18:15.000 --> 00:18:19.519 +Additionally, we can envision + +00:18:19.520 --> 00:18:23.999 +a power user of the future, or present, + +00:18:24.000 --> 00:18:27.479 +who can not only annotate a span, + +00:18:27.480 --> 00:18:31.279 +but also edit the information in situ + +00:18:31.280 --> 00:18:34.639 +in a way that would help machine algorithms + +00:18:34.640 --> 00:18:36.879 +generalize better by making more efficient use + +00:18:36.880 --> 00:18:37.719 +of the annotations. + +00:18:37.720 --> 00:18:41.519 +So together, Emacs and Org mode can speed up + +00:18:41.520 --> 00:18:42.959 +the enrichment of the signals + +00:18:42.960 --> 00:18:44.519 +in a way that allows us + +00:18:44.520 --> 00:18:47.719 +to focus on certain aspects and ignore others. + +00:18:47.720 --> 00:18:50.839 +Extremely complex landscape of rich structures + +00:18:50.840 --> 00:18:53.039 +can be captured consistently, + +00:18:53.040 --> 00:18:55.639 +in a fashion that allows computers + +00:18:55.640 --> 00:18:56.759 +to understand language. + +00:18:56.760 --> 00:19:00.879 +We can then build tools to enhance the tasks + +00:19:00.880 --> 00:19:03.319 +that we do in our everyday life. + +00:19:03.320 --> 00:19:10.759 +YAMR is acronym, or the file's type or specification + +00:19:10.760 --> 00:19:15.239 +that we are creating to capture this new + +00:19:15.240 --> 00:19:17.679 +rich representation. + +NOTE Spontaneous speech + +00:19:17.680 --> 00:19:21.959 +We'll now look at an example of spontaneous speech + +00:19:21.960 --> 00:19:24.799 +that occurs in spoken conversations. + +00:19:24.800 --> 00:19:28.599 +Conversations frequently contain errors in speech: + +00:19:28.600 --> 00:19:30.799 +interruptions, disfluencies, + +00:19:30.800 --> 00:19:33.959 +verbal sounds such as cough or laugh, + +00:19:33.960 --> 00:19:35.039 +and other noises. + +00:19:35.040 --> 00:19:38.199 +In this sense, spontaneous speech is similar + +00:19:38.200 --> 00:19:39.799 +to a functional data stream. + +00:19:39.800 --> 00:19:42.759 +We cannot take back words that come out of our mouth, + +00:19:42.760 --> 00:19:47.239 +but we tend to make mistakes, and we correct ourselves + +00:19:47.240 --> 00:19:49.039 +as soon as we realize that we have made-- + +00:19:49.040 --> 00:19:50.679 +we have misspoken. + +00:19:50.680 --> 00:19:53.159 +This process manifests through a combination + +00:19:53.160 --> 00:19:56.279 +of a handful of mechanisms, including immediate correction + +00:19:56.280 --> 00:20:00.959 +after an error, and we do this unconsciously. + +00:20:00.960 --> 00:20:02.719 +Computers, on the other hand, + +00:20:02.720 --> 00:20:06.639 +must be taught to understand these cases. + +00:20:06.640 --> 00:20:12.799 +What we see here is a example document or outline, + +00:20:12.800 --> 00:20:18.119 +or part of a document that illustrates + +00:20:18.120 --> 00:20:22.919 +various different aspects of the representation. + +00:20:22.920 --> 00:20:25.919 +We don't have a lot of time to go through + +00:20:25.920 --> 00:20:28.239 +many of the details. + +00:20:28.240 --> 00:20:31.759 +I would highly encourage you to play a... + +00:20:31.760 --> 00:20:39.159 +I'm planning on making some videos, or ascii cinemas, + +00:20:39.160 --> 00:20:42.559 +that I'll be posting, and you can, + +00:20:42.560 --> 00:20:46.759 +if you're interested, you can go through those. + +00:20:46.760 --> 00:20:50.359 +The idea here is to try to do + +00:20:50.360 --> 00:20:54.599 +a slightly more complex use case. + +00:20:54.600 --> 00:20:57.639 +But again, given the time constraint + +00:20:57.640 --> 00:21:00.279 +and the amount of information + +00:21:00.280 --> 00:21:01.519 +that needs to fit in the screen, + +00:21:01.520 --> 00:21:05.559 +this may not be very informative, + +00:21:05.560 --> 00:21:08.399 +but at least it will give you some idea + +00:21:08.400 --> 00:21:10.439 +of what can be possible. + +00:21:10.440 --> 00:21:13.279 +And in this particular case, what you're seeing is that + +00:21:13.280 --> 00:21:18.319 +there is a sentence which is "What I'm I'm tr- telling now." + +00:21:18.320 --> 00:21:21.159 +Essentially, there is a repetition of the word "I'm", + +00:21:21.160 --> 00:21:23.279 +and then there is a partial word + +00:21:23.280 --> 00:21:25.159 +that somebody tried to say "telling", + +00:21:25.160 --> 00:21:29.599 +but started saying "tr-", and then corrected themselves + +00:21:29.600 --> 00:21:30.959 +and said, "telling now." + +00:21:30.960 --> 00:21:39.239 +So in this case, you see, we can capture words + +00:21:39.240 --> 00:21:44.919 +or a sequence of words, or a sequence of tokens. + +00:21:44.920 --> 00:21:52.279 +One thing to... An interesting thing to note is that in NLP, + +00:21:52.280 --> 00:21:55.319 +sometimes we have to break typically + +00:21:55.320 --> 00:22:01.199 +words that don't have spaces into two separate words, + +00:22:01.200 --> 00:22:04.119 +especially contractions like "I'm", + +00:22:04.120 --> 00:22:08.199 +so the syntactic parser needs needs two separate nodes. + +00:22:08.200 --> 00:22:11.199 +But anyway, so I'll... You can see that here. + +00:22:11.200 --> 00:22:15.759 +The other... This view. What this view shows is that + +00:22:15.760 --> 00:22:19.759 +with each of the nodes in the sentence + +00:22:19.760 --> 00:22:23.079 +or in the representation, + +00:22:23.080 --> 00:22:26.079 +you can have a lot of different properties + +00:22:26.080 --> 00:22:27.559 +that you can attach to them, + +00:22:27.560 --> 00:22:30.119 +and these properties are typically hidden, + +00:22:30.120 --> 00:22:32.719 +like you saw in the earlier slide. + +00:22:32.720 --> 00:22:35.599 +But you can make use of all these properties + +00:22:35.600 --> 00:22:39.439 +to do various kind of searches and filtering. + +00:22:39.440 --> 00:22:43.519 +And on the right hand side here-- + +00:22:43.520 --> 00:22:48.799 +this is actually not a legitimate syntax-- + +00:22:48.800 --> 00:22:51.279 +but on the right are descriptions + +00:22:51.280 --> 00:22:53.479 +of what each of these represent. + +00:22:53.480 --> 00:22:57.319 +All the information is also available in the article. + +00:22:57.320 --> 00:23:04.279 +You can see there... It shows how much rich context + +00:23:04.280 --> 00:23:05.879 +you can capture. + +00:23:05.880 --> 00:23:08.799 +This is just a closer snapshot + +00:23:08.800 --> 00:23:10.159 +of the properties on the node, + +00:23:10.160 --> 00:23:13.119 +and you can see we can have things like, + +00:23:13.120 --> 00:23:14.799 +whether the word is a token or not, + +00:23:14.800 --> 00:23:17.359 +or that it's incomplete, whether some words + +00:23:17.360 --> 00:23:19.959 +might want to be filtered out for parsing, + +00:23:19.960 --> 00:23:23.039 +and we can say this: PARSE_IGNORE, + +00:23:23.040 --> 00:23:25.519 +or some words or restart markers... + +00:23:25.520 --> 00:23:29.239 +We can mark, add a RESTART_MARKER, or sometimes, + +00:23:29.240 --> 00:23:31.999 +some of these might have durations. Things like that. + +NOTE Editing properties in column view + +00:23:32.000 --> 00:23:38.799 +The other fascinating thing of this representation + +00:23:38.800 --> 00:23:42.599 +is that you can edit properties in the column view. + +00:23:42.600 --> 00:23:45.399 +And suddenly, you have this tabular data structure + +00:23:45.400 --> 00:23:48.879 +combined with the hierarchical data structure. + +00:23:48.880 --> 00:23:53.119 +And as you can--you may not be able to see it here, + +00:23:53.120 --> 00:23:56.879 +but what has also happened here is that + +00:23:56.880 --> 00:24:01.159 +some of the tags have been inherited + +00:24:01.160 --> 00:24:02.479 +from the earlier nodes. + +00:24:02.480 --> 00:24:07.919 +And so you get a much fuller picture of things. + +00:24:07.920 --> 00:24:13.919 +Essentially you, can filter out things + +00:24:13.920 --> 00:24:15.319 +that you want to process, + +00:24:15.320 --> 00:24:20.279 +process them, and then reintegrate it into the whole. + +NOTE Conclusion + +00:24:20.280 --> 00:24:25.479 +So, in conclusion, today we have proposed and demonstrated + +00:24:25.480 --> 00:24:27.559 +the use of an architecture (GRAIL), + +00:24:27.560 --> 00:24:31.319 +which allows the representation, manipulation, + +00:24:31.320 --> 00:24:34.759 +and aggregation of rich linguistic structures + +00:24:34.760 --> 00:24:36.519 +in a systematic fashion. + +00:24:36.520 --> 00:24:41.359 +We have shown how GRAIL advances the tools + +00:24:41.360 --> 00:24:44.599 +available for building machine learning models + +00:24:44.600 --> 00:24:46.879 +that simulate understanding. + +00:24:46.880 --> 00:24:51.679 +Thank you very much for your time and attention today. + +00:24:51.680 --> 00:24:54.639 +My contact information is on this slide. + +00:24:54.640 --> 00:25:02.599 +If you are interested in an additional example + +00:25:02.600 --> 00:25:05.439 +that demonstrates the representation + +00:25:05.440 --> 00:25:08.039 +of speech and written text together, + +00:25:08.040 --> 00:25:10.719 +please continue watching. + +00:25:10.720 --> 00:25:12.199 +Otherwise, you can stop here + +00:25:12.200 --> 00:25:15.279 +and enjoy the rest of the conference. + +NOTE Bonus material + +00:25:15.280 --> 00:25:39.079 +Welcome to the bonus material. + +00:25:39.080 --> 00:25:43.959 +I'm glad for those of you who are stuck around. + +00:25:43.960 --> 00:25:46.559 +We are now going to examine an instance + +00:25:46.560 --> 00:25:49.159 +of speech and text signals together + +00:25:49.160 --> 00:25:51.479 +that produce multiple layers. + +00:25:51.480 --> 00:25:54.839 +When we have--when we take a spoken conversation + +00:25:54.840 --> 00:25:58.719 +and use the best language processing models available, + +00:25:58.720 --> 00:26:00.679 +we suddenly hit a hard spot + +00:26:00.680 --> 00:26:03.239 +because the tools are typically not trained + +00:26:03.240 --> 00:26:05.359 +to filter out the unnecessary cruft + +00:26:05.360 --> 00:26:07.559 +in order to automatically interpret + +00:26:07.560 --> 00:26:09.559 +the part of what is being said + +00:26:09.560 --> 00:26:11.799 +that is actually relevant. + +00:26:11.800 --> 00:26:14.639 +Over time, language researchers + +00:26:14.640 --> 00:26:17.719 +have created many interdependent layers of annotations, + +00:26:17.720 --> 00:26:21.039 +yet the assumptions underlying them are seldom the same. + +00:26:21.040 --> 00:26:25.039 +Piecing together such related but disjointed annotations + +00:26:25.040 --> 00:26:28.039 +on their predictions poses a huge challenge. + +00:26:28.040 --> 00:26:30.719 +This is another place where we can leverage + +00:26:30.720 --> 00:26:33.119 +the data model underlying the Emacs editor, + +00:26:33.120 --> 00:26:35.359 +along with the structural editing capabilities + +00:26:35.360 --> 00:26:38.519 +of Org mode to improve current tools. + +00:26:38.520 --> 00:26:42.839 +Let's take this very simple looking utterance. + +00:26:42.840 --> 00:26:48.039 +"Um \{lipsmack\} and that's it. (\{laugh\})" + +00:26:48.040 --> 00:26:50.319 +Looks like the person-- so this is-- + +00:26:50.320 --> 00:26:54.519 +what you are seeing here is a transcript of an audio signal + +00:26:54.520 --> 00:27:00.759 +that has a lip smack and a laugh as part of it, + +00:27:00.760 --> 00:27:04.199 +and there is also a "Um" like interjection. + +00:27:04.200 --> 00:27:08.199 +So this has a few interesting noises + +00:27:08.200 --> 00:27:13.999 +and specific things that would be illustrative + +00:27:14.000 --> 00:27:20.479 +of what we are going to, how we are going to represent it. + +NOTE Syntactic analysis + +00:27:20.480 --> 00:27:25.839 +Okay. So let's say you want to have + +00:27:25.840 --> 00:27:28.879 +a syntactic analysis of this sentence or utterance. + +00:27:28.880 --> 00:27:30.959 +One common technique people use + +00:27:30.960 --> 00:27:32.879 +is just to remove the cruft, and, you know, + +00:27:32.880 --> 00:27:35.079 +write some rules, clean up the utterance, + +00:27:35.080 --> 00:27:36.719 +make it look like it's proper English, + +00:27:36.720 --> 00:27:40.239 +and then, you know, tokenize it, + +00:27:40.240 --> 00:27:43.079 +and basically just use standard tools to process it. + +00:27:43.080 --> 00:27:47.279 +But in that process, they end up eliminating + +00:27:47.280 --> 00:27:51.119 +valid pieces of signal that have meaning to others + +00:27:51.120 --> 00:27:52.799 +studying different phenomena of language. + +00:27:52.800 --> 00:27:56.479 +Here you have the rich transcript, + +00:27:56.480 --> 00:28:00.119 +the input to the syntactic parser. + +00:28:00.120 --> 00:28:05.919 +As you can see, there is a little tokenization happening + +00:28:05.920 --> 00:28:07.199 +where you'll be inserting space + +00:28:07.200 --> 00:28:12.119 +between "that" and the contracted is ('s), + +00:28:12.120 --> 00:28:15.599 +and between the period and the "it," + +00:28:15.600 --> 00:28:18.199 +and the output of the syntactic parser is shown below. + +00:28:18.200 --> 00:28:21.639 +which (surprise) is a S-expression. + +00:28:21.640 --> 00:28:24.919 +Like I said, the parse trees, when they were created, + +00:28:24.920 --> 00:28:29.799 +and still largely when they are used, are S-expressions, + +00:28:29.800 --> 00:28:32.999 +and most of the viewers here + +00:28:33.000 --> 00:28:35.119 +should not have much problem reading it. + +00:28:35.120 --> 00:28:37.279 +You can see this tree structure + +00:28:37.280 --> 00:28:39.279 +of this syntactic parser here. + +NOTE Forced alignment + +00:28:39.280 --> 00:28:40.919 +Now let's say you want to integrate + +00:28:40.920 --> 00:28:44.479 +phonetic information or phonetic layer + +00:28:44.480 --> 00:28:49.119 +that's in the audio signal, and do some analysis. + +00:28:49.120 --> 00:28:57.519 +Now, it would need you to do a few-- take a few steps. + +00:28:57.520 --> 00:29:01.679 +First, you would need to align the transcript + +00:29:01.680 --> 00:29:06.479 +with the audio. This process is called forced alignment, + +00:29:06.480 --> 00:29:10.399 +where you already know what the transcript is, + +00:29:10.400 --> 00:29:14.599 +and you have the audio, and you can get a good alignment + +00:29:14.600 --> 00:29:17.599 +using both pieces of information. + +00:29:17.600 --> 00:29:20.119 +And this is typically a technique that is used to + +00:29:20.120 --> 00:29:23.079 +create training data for training + +00:29:23.080 --> 00:29:25.839 +automatic speech recognizers. + +00:29:25.840 --> 00:29:29.639 +One interesting thing is that in order to do + +00:29:29.640 --> 00:29:32.879 +this forced alignment, you have to keep + +00:29:32.880 --> 00:29:35.799 +the non-speech events in transcript, + +00:29:35.800 --> 00:29:39.079 +because they consume some audio signal, + +00:29:39.080 --> 00:29:41.399 +and if you don't have that signal, + +00:29:41.400 --> 00:29:44.399 +the alignment process doesn't know exactly... + +00:29:44.400 --> 00:29:45.759 +you know, it doesn't do a good job, + +00:29:45.760 --> 00:29:50.039 +because it needs to align all parts of the signal + +00:29:50.040 --> 00:29:54.999 +with something, either pause or silence or noise or words. + +00:29:55.000 --> 00:29:59.719 +Interestingly, punctuations really don't factor in, + +00:29:59.720 --> 00:30:01.559 +because we don't speak in punctuations. + +00:30:01.560 --> 00:30:04.239 +So one of the things that you need to do + +00:30:04.240 --> 00:30:05.679 +is remove most of the punctuations, + +00:30:05.680 --> 00:30:08.039 +although you'll see there are some punctuations + +00:30:08.040 --> 00:30:12.599 +that can be kept, or that are to be kept. + +NOTE Alignment before tokenization + +00:30:12.600 --> 00:30:15.319 +And the other thing is that the alignment has to be done + +00:30:15.320 --> 00:30:20.159 +before tokenization, as it impacts pronunciation. + +00:30:20.160 --> 00:30:24.399 +To show an example: Here you see "that's". + +00:30:24.400 --> 00:30:26.919 +When it's one word, + +00:30:26.920 --> 00:30:31.959 +it has a slightly different pronunciation + +00:30:31.960 --> 00:30:35.679 +than when it is two words, which is "that is", + +00:30:35.680 --> 00:30:38.399 +like you can see "is." And so, + +00:30:38.400 --> 00:30:44.279 +if you split the tokens or split the words + +00:30:44.280 --> 00:30:48.119 +in order for syntactic parser to process it, + +00:30:48.120 --> 00:30:51.599 +you would end up getting the wrong phonetic analysis. + +00:30:51.600 --> 00:30:54.239 +And if you have--if you process it + +00:30:54.240 --> 00:30:55.319 +through the phonetic analysis, + +00:30:55.320 --> 00:30:59.159 +and you don't know how to integrate it + +00:30:59.160 --> 00:31:02.719 +with the tokenized syntax, you can, you know, + +00:31:02.720 --> 00:31:07.519 +that can be pretty tricky. And a lot of time, + +00:31:07.520 --> 00:31:10.759 +people write one-off pieces of code that handle these, + +00:31:10.760 --> 00:31:14.279 +but the idea here is to try to have a general architecture + +00:31:14.280 --> 00:31:17.239 +that seamlessly integrates all these pieces. + +00:31:17.240 --> 00:31:21.319 +Then you do the syntactic parsing of the remaining tokens. + +00:31:21.320 --> 00:31:24.799 +Then you align the data and the two annotations, + +00:31:24.800 --> 00:31:27.959 +and then integrate the two layers. + +00:31:27.960 --> 00:31:31.359 +Once that is done, then you can do all kinds of + +00:31:31.360 --> 00:31:33.919 +interesting analysis, and test various hypotheses + +00:31:33.920 --> 00:31:35.279 +and generate the statistics, + +00:31:35.280 --> 00:31:39.359 +but without that you only are dealing + +00:31:39.360 --> 00:31:42.879 +with one or the other part. + +NOTE Layers + +00:31:42.880 --> 00:31:48.319 +Let's just take a quick look at how each of the layers + +00:31:48.320 --> 00:31:51.159 +that are involved look like. + +00:31:51.160 --> 00:31:56.719 +So this is "Um \{lipsmack\}, and that's it. \{laugh\}" + +00:31:56.720 --> 00:32:00.159 +This is the transcript, and on the right hand side, + +00:32:00.160 --> 00:32:04.199 +you see the same thing as a transcript + +00:32:04.200 --> 00:32:06.239 +listed in a vertical in a column. + +00:32:06.240 --> 00:32:08.199 +You'll see why, in just a second. + +00:32:08.200 --> 00:32:09.879 +And there are some place-- + +00:32:09.880 --> 00:32:11.279 +there are some rows that are empty, + +00:32:11.280 --> 00:32:15.079 +some rows that are wider than the others, and we'll see why. + +00:32:15.080 --> 00:32:19.319 +The next is the tokenized sentence + +00:32:19.320 --> 00:32:20.959 +where you have space added, + +00:32:20.960 --> 00:32:23.599 +you know space between these two tokens: + +00:32:23.600 --> 00:32:26.599 +"that" and the apostrophe "s" ('s), + +00:32:26.600 --> 00:32:28.079 +and the "it" and the "period". + +00:32:28.080 --> 00:32:30.679 +And you see on the right hand side + +00:32:30.680 --> 00:32:33.559 +that the tokens have attributes. + +00:32:33.560 --> 00:32:36.439 +So there is a token index, and there are 1, 2, + +00:32:36.440 --> 00:32:38.839 +you know 0, 1, 2, 3, 4, 5 tokens, + +00:32:38.840 --> 00:32:41.479 +and each token has a start and end character, + +00:32:41.480 --> 00:32:45.799 +and space (sp) also has a start and end character, + +00:32:45.800 --> 00:32:50.399 +and space is represented by a "sp". And there are + +00:32:50.400 --> 00:32:54.319 +these other things that we removed, + +00:32:54.320 --> 00:32:56.239 +like the "\{LS\}" which is for "\{lipsmack\}" + +00:32:56.240 --> 00:32:59.399 +and "\{LG\}" which is "\{laugh\}" are showing grayed out, + +00:32:59.400 --> 00:33:02.439 +and you'll see why some of these things are grayed out + +00:33:02.440 --> 00:33:03.399 +in a little bit. + +00:33:03.400 --> 00:33:11.919 +This is what the forced alignment tool produces. + +00:33:11.920 --> 00:33:17.159 +Basically, it takes the transcript, + +00:33:17.160 --> 00:33:19.159 +and this is the transcript + +00:33:19.160 --> 00:33:24.119 +that has slightly different symbols, + +00:33:24.120 --> 00:33:26.239 +because different tools use different symbols + +00:33:26.240 --> 00:33:28.159 +and their various configurational things. + +00:33:28.160 --> 00:33:33.679 +But this is what is used to get an alignment + +00:33:33.680 --> 00:33:36.039 +or time alignment with phones. + +00:33:36.040 --> 00:33:40.079 +So this column shows the phones, and so each word... + +00:33:40.080 --> 00:33:43.879 +So, for example, "and" has been aligned with these phones, + +00:33:43.880 --> 00:33:46.879 +and these on the start and end + +00:33:46.880 --> 00:33:52.959 +are essentially temporal or time stamps that it aligned-- + +00:33:52.960 --> 00:33:54.279 +that has been aligned to it. + +00:33:54.280 --> 00:34:00.759 +Interestingly, sometimes we don't really have any pause + +00:34:00.760 --> 00:34:05.159 +or any time duration between some words + +00:34:05.160 --> 00:34:08.199 +and those are highlighted as gray here. + +00:34:08.200 --> 00:34:12.759 +See, there's this space... Actually + +00:34:12.760 --> 00:34:17.799 +it does not have any temporal content, + +00:34:17.800 --> 00:34:21.319 +whereas this other space has some duration. + +00:34:21.320 --> 00:34:24.839 +So the ones that have some duration are captured, + +00:34:24.840 --> 00:34:29.519 +while the others are the ones that in the earlier diagram + +00:34:29.520 --> 00:34:31.319 +we saw were left out. + +NOTE Variations + +00:34:31.320 --> 00:34:37.639 +And the aligner actually produces multiple files. + +00:34:37.640 --> 00:34:44.399 +One of the files has a different, slightly different + +00:34:44.400 --> 00:34:46.679 +variation on the same information, + +00:34:46.680 --> 00:34:49.999 +and in this case, you can see + +00:34:50.000 --> 00:34:52.399 +that the punctuation is missing, + +00:34:52.400 --> 00:34:57.599 +and the punctuation is, you know, deliberately missing, + +00:34:57.600 --> 00:35:02.279 +because there is no time associated with it, + +00:35:02.280 --> 00:35:06.439 +and you see that it's not the tokenized sentence-- + +00:35:06.440 --> 00:35:17.119 +a tokenized word. This... Now it gives you a full table, + +00:35:17.120 --> 00:35:21.239 +and you can't really look into it very carefully. + +00:35:21.240 --> 00:35:25.879 +But we can focus on the part that seems legible, + +00:35:25.880 --> 00:35:28.559 +or, you know, properly written sentence, + +00:35:28.560 --> 00:35:32.879 +process it and reincorporate it back into the whole. + +00:35:32.880 --> 00:35:35.879 +So if somebody wants to look at, for example, + +00:35:35.880 --> 00:35:39.679 +how many pauses the person made while they were talking, + +00:35:39.680 --> 00:35:42.919 +And they can actually measure the pause, the number, + +00:35:42.920 --> 00:35:46.279 +the duration, and make connections between that + +00:35:46.280 --> 00:35:49.639 +and the rich syntactic structure that is being produced. + +00:35:49.640 --> 00:35:57.279 +And in order to do that, you have to get these layers + +00:35:57.280 --> 00:35:59.039 +to align with each other, + +00:35:59.040 --> 00:36:04.359 +and this table is just a tabular representation + +00:36:04.360 --> 00:36:08.679 +of the information that we'll be storing in the YAMR file. + +00:36:08.680 --> 00:36:11.719 +Congratulations! You have reached + +00:36:11.720 --> 00:36:13.479 +the end of this demonstration. + +00:36:13.480 --> 00:36:17.000 +Thank you for your time and attention. |