WEBVTT captioned by sachac
00:00:00.000 --> 00:00:04.359
Hi, I'm Blaine Mooers. I'm an associate professor
00:00:04.360 --> 00:00:06.519
of biochemistry at the University of Oklahoma
00:00:06.520 --> 00:00:09.319
Health Sciences Center in Oklahoma City.
00:00:09.320 --> 00:00:12.959
My lab studies the role of RNA structure in RNA editing.
00:00:12.960 --> 00:00:17.199
We use X-ray crystallography to study the structures
00:00:17.200 --> 00:00:19.919
of these RNAs. We spend a lot of time in the lab
00:00:19.920 --> 00:00:22.719
preparing our samples for structural studies,
00:00:22.720 --> 00:00:26.719
and then we also spend a lot of time at the computer
00:00:26.720 --> 00:00:29.719
analyzing the resulting data.
00:00:29.720 --> 00:00:33.039
I was seeking ways of using voice computing
00:00:33.040 --> 00:00:37.399
to try to enhance my productivity.
00:00:37.400 --> 00:00:41.319
I divide voice computing into three activities,
00:00:41.320 --> 00:00:44.959
speech-to-text or dictation, speech-to-commands,
00:00:44.960 --> 00:00:47.639
and speech-to-code. I'll be talking about
00:00:47.640 --> 00:00:50.159
speech-to-text and speech-to-commands today
00:00:50.160 --> 00:00:55.079
because these are two activities
00:00:55.080 --> 00:00:57.319
that are probably most broadly applicable
00:00:57.320 --> 00:01:02.559
to the workflows of people attending this conference.
00:01:02.560 --> 00:01:06.799
This talk will not be about Emacspeak.
00:01:06.800 --> 00:01:11.359
This is a verbal program for converting text to speech.
00:01:11.360 --> 00:01:13.319
We're talking about the flow of information
00:01:13.320 --> 00:01:16.519
opposite direction, speech-to-text.
00:01:16.520 --> 00:01:20.599
We need an Emacs Listens. We don't have one,
00:01:20.600 --> 00:01:25.479
so I had to seek help from outside the Emacs world
00:01:25.480 --> 00:01:30.639
via the Voice In Plus. This runs in
00:01:30.640 --> 00:01:33.639
the Google Chrome web browser,
00:01:33.640 --> 00:01:36.719
and it's very good for speech-to-text
00:01:36.720 --> 00:01:39.519
and very easy to learn how to use.
00:01:39.520 --> 00:01:41.999
It also has some speech-to-commands.
00:01:42.000 --> 00:01:44.799
However, Talon Voice is much better
00:01:44.800 --> 00:01:47.559
with the speech-to-commands,
00:01:47.560 --> 00:01:53.519
and it's also great at speech-to-code.
NOTE Motivations
00:01:53.520 --> 00:01:57.239
So, the motivations are, obviously, as I mentioned already,
00:01:57.240 --> 00:01:59.159
for improved productivity.
00:01:59.160 --> 00:02:00.399
So, if you're a fast typist
00:02:00.400 --> 00:02:05.199
who types faster than they can speak,
00:02:05.200 --> 00:02:07.079
then nonetheless you might still benefit
00:02:07.080 --> 00:02:09.279
from voice computing when you grow tired of
00:02:09.280 --> 00:02:12.199
using the keyboard. On the other hand,
00:02:12.200 --> 00:02:15.199
you might be a slow typist who talks faster
00:02:15.200 --> 00:02:17.519
than they can type.
00:02:17.520 --> 00:02:19.759
In this case, you're definitely going to
00:02:19.760 --> 00:02:22.859
benefit from dictation because you'll be able to
00:02:22.860 --> 00:02:29.359
encode more words in text documents in a given day.
00:02:29.360 --> 00:02:33.639
If you're a coder, then you may get a kick out of
00:02:33.640 --> 00:02:36.999
opening programs and websites and coding projects
00:02:37.000 --> 00:02:39.279
by using your voice.
00:02:39.280 --> 00:02:41.719
Then there are health-related reasons.
00:02:41.720 --> 00:02:44.599
You may have impaired use of your hands, eyes, or both
00:02:44.600 --> 00:02:49.199
due to accident or disease, or you may suffer from
00:02:49.200 --> 00:02:53.519
a repetitive stress injury. Many of us have this
00:02:53.520 --> 00:02:55.759
in a mild but chronic form of it.
00:02:55.760 --> 00:02:59.039
We can't take a three-month sabbatical from the keyboard
00:02:59.040 --> 00:03:05.519
without losing our jobs, so these injuries tend to persist.
00:03:05.520 --> 00:03:06.679
And then you may have learned
00:03:06.680 --> 00:03:09.959
that it's not good for your health to sit
00:03:09.960 --> 00:03:11.919
for prolonged periods of time
00:03:11.920 --> 00:03:14.919
with your staring at a computer screen.
00:03:14.920 --> 00:03:21.799
You can actually dictate to your computer from 20 feet away
00:03:21.800 --> 00:03:24.999
while looking out the window,
00:03:25.000 --> 00:03:27.779
thereby giving your lower body a break
00:03:27.780 --> 00:03:33.239
and your eyes a break.
NOTE Data
00:03:33.240 --> 00:03:35.639
I'm not God, so I have to bring data.
00:03:35.640 --> 00:03:38.039
I have two data points here,
00:03:38.040 --> 00:03:42.399
the number of words that I wrote in June and July this year
00:03:42.400 --> 00:03:45.159
and in September and October.
00:03:45.160 --> 00:03:49.519
I adopted the use of voice computing
00:03:49.520 --> 00:03:53.919
in the middle of August. As you can see,
00:03:53.920 --> 00:03:58.679
I got a over three-fold increase in my output.
NOTE Voice In in the Chrome Store
00:03:58.680 --> 00:04:07.119
So this is the Chrome store website for voice-in.
00:04:07.120 --> 00:04:11.119
So it's only available for Google Chrome.
00:04:11.120 --> 00:04:13.239
You just hit the install button to install it.
00:04:13.240 --> 00:04:16.639
To configure it, you need to select a language.
00:04:16.640 --> 00:04:19.559
It has support for 40 languages
00:04:19.560 --> 00:04:23.119
and it supports about a dozen different dialects of English,
00:04:23.120 --> 00:04:29.959
including Australian. It works on web pages with text areas,
00:04:29.960 --> 00:04:33.319
so it works. I use it regularly
00:04:33.320 --> 00:04:37.879
on Overleaf and 750words.com,
00:04:37.880 --> 00:04:42.279
a distraction-free environment for writing.
00:04:42.280 --> 00:04:46.239
It also works in webmails. It works in Google.
00:04:46.780 --> 00:04:51.319
It works in Jupyter Lab, of course,
00:04:51.320 --> 00:04:52.879
because that runs in the browser.
00:04:52.880 --> 00:04:57.999
It also works in Jupyter Notebook and Colab Notebook.
00:04:58.000 --> 00:05:01.319
It should work in Cloudmacs.
00:05:01.320 --> 00:05:04.159
I've mapped option-L to opening Voice In
00:05:04.160 --> 00:05:09.119
when the cursor is on a web page that has a text area.
00:05:09.120 --> 00:05:16.879
So that's the main limiting factor.
NOTE Built-in commands in Voice In Plus
00:05:16.880 --> 00:05:19.159
So it has a number of built-in commands.
00:05:19.160 --> 00:05:24.879
You can turn it off by saying stop dictation.
00:05:24.880 --> 00:05:26.119
It doesn't distinguish between
00:05:26.120 --> 00:05:28.799
a command mode and a dictation mode.
00:05:28.800 --> 00:05:33.599
It has undo command. When you use a command,
00:05:33.600 --> 00:05:36.919
copy that to a copy of selection.
00:05:36.920 --> 00:05:40.079
And the `press` commands are used in the browser,
00:05:40.080 --> 00:05:44.839
so you press Enter to issue a command or a text
00:05:44.840 --> 00:05:50.319
that has been written in a web form,
00:05:50.320 --> 00:05:55.279
and then "press tab" will open up the next tab
00:05:55.280 --> 00:05:58.599
in a web browser. The scroll up and down
00:05:58.600 --> 00:06:02.379
will allow you to navigate a web page.
00:06:02.380 --> 00:06:05.819
I've put together a quiz about these commands
00:06:05.820 --> 00:06:09.559
so that you can go through this quiz several times
00:06:09.560 --> 00:06:14.699
until you get at least 90 percent of them correct,
00:06:14.700 --> 00:06:16.679
90 percent of the questions correct.
00:06:16.680 --> 00:06:20.599
In order to boost your recall of the commands,
00:06:20.600 --> 00:06:23.799
I have a Python script that you can probably
00:06:23.800 --> 00:06:26.559
pound through the quiz with
00:06:26.560 --> 00:06:32.159
in less than a minute, once you know the commands.
00:06:32.160 --> 00:06:35.599
I also provide an Elisp version of this quiz,
00:06:35.600 --> 00:06:41.739
but it's a little slower to operate.
NOTE Common errors
00:06:41.740 --> 00:06:43.399
These are some common errors
00:06:43.400 --> 00:06:45.399
that I've run into with Voice In.
00:06:45.400 --> 00:06:50.319
It likes to contract statements like "I will" into "I'll".
00:06:50.320 --> 00:06:55.599
Contractions are not used in formal writing,
00:06:55.600 --> 00:07:00.359
and most of my writing is formal writing, so this annoys me.
00:07:00.360 --> 00:07:04.759
I will show you how I corrected for that problem.
00:07:04.760 --> 00:07:10.039
It also drops the first word in sentences quite often.
00:07:10.040 --> 00:07:13.359
This might be some speech issue that I have.
00:07:13.360 --> 00:07:17.599
It inserts the wrong word because it's not in the dictionary
00:07:17.600 --> 00:07:22.619
that was used to train it. So, for example,
00:07:22.620 --> 00:07:26.919
the word PyMOL is the name of a lexicographic program
00:07:26.920 --> 00:07:31.639
that we use in our field. It doesn't recognize PyMOL.
00:07:31.640 --> 00:07:34.239
Instead, it substitutes in the word "primal".
00:07:34.240 --> 00:07:38.399
Since I don't use "primal" very often,
00:07:38.400 --> 00:07:42.299
I've mapped the word "primal" to "PyMOL"
00:07:42.300 --> 00:07:45.659
in some custom commands I'll talk about in a minute.
00:07:45.660 --> 00:07:50.439
Then there's a problem that the commands that exist
00:07:50.440 --> 00:07:54.439
might get executed when you speak them when, in fact,
00:07:54.440 --> 00:07:58.839
you wanted to use the words in those commands
00:07:58.840 --> 00:08:01.439
during your dictation.
00:08:01.440 --> 00:08:07.119
So this is a problem, a pitfall of Voice In,
00:08:07.120 --> 00:08:08.919
in that it doesn't have a command mode
00:08:08.920 --> 00:08:14.759
that's separate from a dictation mode.
NOTE Custom speech-to-text commands
00:08:14.760 --> 00:08:20.319
So you can set up through a very easy-to-use GUI
00:08:20.320 --> 00:08:26.959
custom voice commands mapped to what you want inserted.
00:08:26.960 --> 00:08:32.399
So this is how misinterpreted words can be corrected.
00:08:32.400 --> 00:08:35.759
You just map the misinterpreted word to the intended word.
00:08:35.760 --> 00:08:42.839
You can also map the contractions to their expansions.
00:08:42.840 --> 00:08:46.959
I did this for 94 English contractions,
00:08:46.960 --> 00:08:50.139
and you can find this on GitHub.
00:08:50.140 --> 00:08:56.079
You can also insert acronyms and expand those acronyms.
00:08:56.080 --> 00:09:00.239
I apply the same approach to the first names of colleagues.
00:09:00.240 --> 00:09:03.759
I say "expand Fred", for example,
00:09:03.760 --> 00:09:06.999
to get Fred's first and last name with the spelling
00:09:07.000 --> 00:09:12.599
of his very long German name.
00:09:12.600 --> 00:09:19.399
You can also insert other trivia like favorite URLs.
00:09:19.400 --> 00:09:24.559
You can insert a lot of text snippets,
00:09:24.560 --> 00:09:34.799
and so it handles correctly multi-line snippets.
00:09:34.800 --> 00:09:39.419
You just have to enclose them in double quotes.
00:09:39.420 --> 00:09:45.039
You can even insert BibTeX cite keys for references
00:09:45.040 --> 00:09:46.879
that you use frequently. All fields
00:09:46.880 --> 00:09:59.419
have certain key references for certain methods or topics.
00:09:59.420 --> 00:10:05.079
Then it has a set of commands that you can customize
00:10:05.080 --> 00:10:08.199
for the purpose of speech to commands
00:10:08.200 --> 00:10:09.679
to get the computer to do something
00:10:09.680 --> 00:10:15.399
like open up a specific website or save the current writing.
00:10:15.400 --> 00:10:19.919
In this case, we have "press" is a mapping of
00:10:19.920 --> 00:10:27.759
is applied to the command `s` for saving current writing.
00:10:27.760 --> 00:10:28.099
You can change the language,
00:10:28.100 --> 00:10:37.539
and you can change the case of the text.
NOTE Introducing Talon Voice
00:10:37.540 --> 00:10:41.039
But the speech to command repertoire is quite limited
00:10:41.040 --> 00:10:49.759
in Voice In, so it's now time to pick up on Talon Voice.
00:10:49.760 --> 00:10:54.119
This is an open source project. It's free.
00:10:54.120 --> 00:10:57.399
It is highly configurable via TalonScript,
00:10:57.400 --> 00:10:58.959
which is a subset of Python.
00:10:58.960 --> 00:11:03.039
You can use either TalonScript or Python to configure it,
00:11:03.040 --> 00:11:06.279
but it's easier to code up your configuration
00:11:06.280 --> 00:11:08.399
in TalonScript.
00:11:08.400 --> 00:11:10.759
It has a Python interpreter embedded in it,
00:11:10.760 --> 00:11:12.999
so you don't have to mess around with installing
00:11:13.000 --> 00:11:14.559
yet another Python interpreter.
00:11:14.560 --> 00:11:21.519
It runs on all platforms, and it has a dictation mode
00:11:21.520 --> 00:11:24.599
that's separate from a command mode.
00:11:24.600 --> 00:11:25.599
You can activate it,
00:11:25.600 --> 00:11:31.359
and it'll be in a listening state asleep.
00:11:31.360 --> 00:11:36.279
You just bark out Talon Wake to start to wake it up,
00:11:36.280 --> 00:11:43.799
and Talon Sleep to have it go into a listening state.
00:11:43.800 --> 00:11:47.919
It has a very welcoming community
00:11:47.920 --> 00:11:50.919
in the Talon Slack channel.
00:11:50.920 --> 00:11:56.399
Then I need to point out that there's several packages
00:11:56.400 --> 00:11:59.199
that others have developed that run on top of Talon,
00:11:59.200 --> 00:12:03.079
but one of particular note is by Pokey Rule.
00:12:03.080 --> 00:12:08.119
He has on his website some really well-done videos
00:12:08.120 --> 00:12:11.479
that demonstrate how he uses Cursorless
00:12:11.480 --> 00:12:17.239
to move the cursor around using voice commands.
00:12:17.240 --> 00:12:20.559
This, however, runs on VS Code.
00:12:20.560 --> 00:12:23.359
At least that's the text editor
00:12:23.360 --> 00:12:28.399
for which he's primarily developing Cursorless.
NOTE Talon GUI
00:12:28.400 --> 00:12:35.519
So, I followed the protocol outlined by Tara Roys.
00:12:35.520 --> 00:12:38.759
She has a collection of tutorials
00:12:38.760 --> 00:12:44.599
on YouTube as well as on GitHub that are quite helpful.
00:12:44.600 --> 00:12:49.479
I followed her tutorial for installing
00:12:49.480 --> 00:12:51.359
Talend on macOS without any issues,
00:12:51.360 --> 00:12:55.319
but allow for half an hour to an hour
00:12:55.320 --> 00:12:57.719
to go through the process. When you're done,
00:12:57.720 --> 00:13:02.199
you'll have this Talon icon appear in the toolbar
00:13:02.200 --> 00:13:06.119
on the Mac. When it has this diagonal line across it,
00:13:06.120 --> 00:13:09.539
that means it's in the sleep state.
00:13:09.540 --> 00:13:13.519
So, this leads to cascading pull-down menus.
00:13:13.520 --> 00:13:19.639
This is it for the GUI interface.
00:13:19.640 --> 00:13:26.519
One of your first tasks is to select a large language model
00:13:26.520 --> 00:13:30.439
or language model that will be used to interpret
00:13:30.440 --> 00:13:35.179
the sounds that you generate as words.
00:13:35.180 --> 00:13:38.959
And the other kind of key feature is that there's a,
00:13:38.960 --> 00:13:43.399
under scripting, there's a view log pull-down
00:13:43.400 --> 00:13:48.399
that opens up a window displaying the log file.
00:13:48.400 --> 00:13:52.879
Whenever you make a change in a Talon configuration file,
00:13:52.880 --> 00:13:55.079
that change is implemented immediately.
00:13:55.080 --> 00:13:57.599
You do not have to restart Talon
00:13:57.600 --> 00:14:02.539
to get the change to take effect.
00:14:02.540 --> 00:14:04.759
So, this is an example of a Talon file.
00:14:04.760 --> 00:14:10.499
It has two components. It has a header above the dash that describes
00:14:10.500 --> 00:14:14.919
the scope of the commands contained below the dash.
00:14:14.920 --> 00:14:19.739
Each command is separated by a blank line.
00:14:19.740 --> 00:14:24.239
If a voice command is mapped to multiple actions,
00:14:24.240 --> 00:14:30.999
these are listed separately on indented lines
00:14:31.000 --> 00:14:33.599
below the first line.
00:14:33.600 --> 00:14:39.419
The words that are in square brackets are optional.
00:14:39.420 --> 00:14:44.319
So, I have mapped the word toggle voice in,
00:14:44.320 --> 00:14:46.319
or the phrase toggle voice in,
00:14:46.320 --> 00:14:51.279
to the keyboard shortcut Alt L
00:14:51.280 --> 00:14:54.999
in order to toggle on or off voice in.
00:14:55.000 --> 00:14:57.879
If I toggle voice in on,
00:14:57.880 --> 00:15:01.759
I need to immediately toggle off Talon,
00:15:01.760 --> 00:15:09.079
and this is done through this key command for Control T,
00:15:09.080 --> 00:15:11.079
which is mapped to speech toggle.
00:15:11.080 --> 00:15:20.399
Speech toggle. Then there are,
00:15:20.400 --> 00:15:24.079
there's a couple other examples.
00:15:24.080 --> 00:15:26.439
So, if there's no header present,
00:15:26.440 --> 00:15:29.599
it's an optional feature of Talon files,
00:15:29.600 --> 00:15:32.639
then the commands in the file will apply in all situations,
00:15:32.640 --> 00:15:36.959
in all modes. Here we have two restrictions.
00:15:36.960 --> 00:15:38.959
This is only, these commands will only work
00:15:38.960 --> 00:15:42.959
when using the iTerm2 terminal emulator for the Mac,
00:15:42.960 --> 00:15:48.239
and then only when the title of the window in iTerm2
00:15:48.240 --> 00:15:52.439
has this particular address, which corresponds to,
00:15:52.440 --> 00:15:55.559
which is what appears when I've logged into
00:15:55.560 --> 00:16:00.059
the supercomputer at the University of Oklahoma.
00:16:00.060 --> 00:16:03.479
So, one of the commands in this file is checkjobs.
00:16:03.480 --> 00:16:05.539
It's mapped to an alias,
00:16:05.540 --> 00:16:10.919
a bash alias called cj for "check jobs",
00:16:10.920 --> 00:16:17.079
which in turn is mapped to a script called checkjobs.sh
00:16:17.080 --> 00:16:20.399
that, when it's run, returns a listing
00:16:20.400 --> 00:16:23.219
of the pending and running jobs on the supercomputer
00:16:23.220 --> 00:16:26.080
in a format that I find pleasing.
00:16:26.081 --> 00:16:34.559
So, this backslash n after cj, new line character,
00:16:34.560 --> 00:16:39.839
enters the command. So, I don't have to do that
00:16:39.840 --> 00:16:43.799
as an additional step. And then, likewise,
00:16:43.800 --> 00:16:46.799
here's a similar setup for interacting with
00:16:46.800 --> 00:16:52.499
a Ubuntu virtual machine.
NOTE Recommendations
00:16:52.500 --> 00:16:55.919
So, in terms of picking up voice computing,
00:16:55.920 --> 00:16:57.479
these are my recommendations.
00:16:57.480 --> 00:16:59.759
You're going to run into more errors
00:16:59.760 --> 00:17:01.479
than you may like initially,
00:17:01.480 --> 00:17:07.839
and so you need some patience in dealing with those.
00:17:07.840 --> 00:17:09.919
And also, it'll take you a while
00:17:09.920 --> 00:17:16.799
to get your head wrapped around Talon and how it works.
00:17:16.800 --> 00:17:19.439
You'll definitely want to use these custom commands
00:17:19.440 --> 00:17:21.479
to correct the errors or shortcomings
00:17:21.480 --> 00:17:26.919
of the language models. And you've seen how,
00:17:26.920 --> 00:17:29.879
by opening up projects by voice commands,
00:17:29.880 --> 00:17:31.359
you can reduce friction
00:17:31.360 --> 00:17:36.659
in terms of restarting work on a project.
00:17:36.660 --> 00:17:40.399
You've seen how Voice In is preferred
00:17:40.400 --> 00:17:44.879
for more accurate dictation.
00:17:44.880 --> 00:17:48.079
I think my error rate is about 1 to 2 percent.
00:17:48.080 --> 00:17:53.879
That is, 1 to 2 out of 100 words are incorrect
00:17:53.880 --> 00:17:56.319
versus Talon Voice where I think
00:17:56.320 --> 00:17:59.879
the error rate is closer to 5 percent.
00:18:00.840 --> 00:18:04.759
I have put together contractions also for Talon,
00:18:04.760 --> 00:18:07.479
and they can be found here on GitHub.
00:18:07.480 --> 00:18:12.959
And I also have a quiz of 600 questions
00:18:12.960 --> 00:18:17.719
about some basic Talon commands.
00:18:17.720 --> 00:18:20.999
So, I'd like to thank the people who've helped me out
00:18:21.000 --> 00:18:22.159
on the Talon Slack channel
00:18:22.160 --> 00:18:25.799
and members of the Oklahoma Data Science Workshop
00:18:25.800 --> 00:18:29.879
where I gave an hour-long talk on this topic
00:18:29.880 --> 00:18:30.959
several weeks ago.
00:18:30.960 --> 00:18:34.159
I'd like to thank my friends
00:18:34.160 --> 00:18:37.399
at the Berlin and Austin Emacs Meetup
00:18:37.400 --> 00:18:42.659
and at the M-x Research Slack channel.
00:18:42.660 --> 00:18:45.119
And I thank these grant funding agencies
00:18:45.120 --> 00:18:48.880
for supporting my work. I'll be happy to take any questions.
