path: root/2020/subtitles/emacsconf-2020--23-incremental-parsing-with-emacs-tree-sitter--tuan-anh-nguyen-autogen.vtt

WEBVTT

00:00:01.520 --> 00:00:04.400
hello everyone my name is toniang

00:00:04.400 --> 00:00:07.200
I've been using amax for about 10 years

00:00:07.200 --> 00:00:09.280
today I'm going to talk about 360

00:00:09.280 --> 00:00:11.519
a new imax package that allows ems to

00:00:11.519 --> 00:00:13.759
pass multiple programming languages

00:00:13.759 --> 00:00:17.840
in real time

00:00:17.840 --> 00:00:21.840
so what is the problem statement

00:00:21.840 --> 00:00:23.359
in order to support programming

00:00:23.359 --> 00:00:24.960
functionalities for a particular

00:00:24.960 --> 00:00:25.760
language

00:00:25.760 --> 00:00:27.680
a text editor needs to have some degree

00:00:27.680 --> 00:00:29.679
of language understanding

00:00:29.679 --> 00:00:31.840
traditionally text editors have relied

00:00:31.840 --> 00:00:33.840
very heavily on regular expressions for

00:00:33.840 --> 00:00:34.960
this

00:00:34.960 --> 00:00:38.320
e-max is no different most language

00:00:38.320 --> 00:00:39.280
major modes use

00:00:39.280 --> 00:00:40.879
regular expressions for syntax

00:00:40.879 --> 00:00:42.960
highlighting code navigation

00:00:42.960 --> 00:00:46.239
folding indexing and so on regular

00:00:46.239 --> 00:00:47.440
expressions are

00:00:47.440 --> 00:00:50.559
problematic for a couple of reasons

00:00:50.559 --> 00:00:53.600
they're slow and inaccurate they also

00:00:53.600 --> 00:00:54.000
make

00:00:54.000 --> 00:00:56.800
the code hard to read and write

00:00:56.800 --> 00:00:57.440
sometimes

00:00:57.440 --> 00:00:59.199
it's because the regular expressions

00:00:59.199 --> 00:01:01.199
themselves are very hairy

00:01:01.199 --> 00:01:04.000
and sometimes because they are just not

00:01:04.000 --> 00:01:05.199
powerful enough

00:01:05.199 --> 00:01:07.840
some helper code is usually needed to

00:01:07.840 --> 00:01:11.200
pass more intricate language features

00:01:11.200 --> 00:01:13.280
that also illustrates the core problem

00:01:13.280 --> 00:01:16.159
with regular expressions

00:01:16.159 --> 00:01:18.400
in that they are not powerful enough to

00:01:18.400 --> 00:01:21.119
pass programming languages

00:01:21.119 --> 00:01:22.640
an example feature that regular

00:01:22.640 --> 00:01:25.040
expressions cannot handle very well

00:01:25.040 --> 00:01:27.520
is string interpolation which is a very

00:01:27.520 --> 00:01:28.320
common feature

00:01:28.320 --> 00:01:31.680
in many modern programming languages

00:01:31.680 --> 00:01:34.079
it would be much nicer if image somehow

00:01:34.079 --> 00:01:35.840
had structural understanding of source

00:01:35.840 --> 00:01:36.479
code

00:01:36.479 --> 00:01:39.520
like ides do

00:01:39.520 --> 00:01:41.119
there have been multiple efforts to

00:01:41.119 --> 00:01:42.960
bring this kind of programming language

00:01:42.960 --> 00:01:45.280
understanding into Emacs

00:01:45.280 --> 00:01:47.119
there are language specific persons

00:01:47.119 --> 00:01:48.640
written in elise

00:01:48.640 --> 00:01:50.240
they can be thought of as the next

00:01:50.240 --> 00:01:52.320
logical step of the glue code on top

00:01:52.320 --> 00:01:54.960
of tribal expressions moving from

00:01:54.960 --> 00:01:56.000
partial local

00:01:56.000 --> 00:01:58.079
pattern recognition into a full-fledged

00:01:58.079 --> 00:01:59.840
parser

00:01:59.840 --> 00:02:01.439
the most prominent example of this

00:02:01.439 --> 00:02:03.040
approach is probably the famous

00:02:03.040 --> 00:02:06.479
js2 mode

00:02:06.479 --> 00:02:10.080
however this approach has several issues

00:02:10.080 --> 00:02:12.959
parsing is computationally expensive and

00:02:12.959 --> 00:02:13.680
imagine

00:02:13.680 --> 00:02:16.800
is not good at that kind of stuff

00:02:16.800 --> 00:02:18.400
furthermore maintenance is very

00:02:18.400 --> 00:02:20.840
troublesome in order to work on these

00:02:20.840 --> 00:02:22.160
process

00:02:22.160 --> 00:02:23.599
first you have to know at least well

00:02:23.599 --> 00:02:25.599
enough and then you have to be

00:02:25.599 --> 00:02:27.760
comfortable with writing a

00:02:27.760 --> 00:02:30.319
recursive ascendant parser while

00:02:30.319 --> 00:02:32.080
constantly keeping up with changes to

00:02:32.080 --> 00:02:34.000
the language itself

00:02:34.000 --> 00:02:36.879
which can be evolving very quickly like

00:02:36.879 --> 00:02:39.360
javascript for example

00:02:39.360 --> 00:02:41.599
together these constraints significantly

00:02:41.599 --> 00:02:45.680
reduce the pull of potential maintenance

00:02:45.680 --> 00:02:47.760
the biggest issue though in my opinion

00:02:47.760 --> 00:02:49.680
is lack of the set of generic

00:02:49.680 --> 00:02:52.879
and reusable apis this makes them very

00:02:52.879 --> 00:02:54.319
hard to use

00:02:54.319 --> 00:02:55.920
for minor modes that want to deal with

00:02:55.920 --> 00:02:57.920
cross-cutting concerns across multiple

00:02:57.920 --> 00:02:59.920
languages

00:02:59.920 --> 00:03:01.760
the other approach which has been

00:03:01.760 --> 00:03:03.599
gaining a lot of momentum in recent

00:03:03.599 --> 00:03:04.319
years

00:03:04.319 --> 00:03:06.560
is externalizing language understanding

00:03:06.560 --> 00:03:08.159
to another process

00:03:08.159 --> 00:03:12.239
also known as language server protocol

00:03:12.239 --> 00:03:14.480
this second approach is actually a very

00:03:14.480 --> 00:03:16.560
interesting one

00:03:16.560 --> 00:03:18.400
my decoupling language understanding

00:03:18.400 --> 00:03:21.280
from the editing facility itself

00:03:21.280 --> 00:03:23.760
the usb servers can attract a lot more

00:03:23.760 --> 00:03:25.120
contributors

00:03:25.120 --> 00:03:28.959
which makes maintenance easier however

00:03:28.959 --> 00:03:32.400
they also have several issues available

00:03:32.400 --> 00:03:34.720
being a separate process they are

00:03:34.720 --> 00:03:36.000
usually more resource

00:03:36.000 --> 00:03:39.920
intensive and depending on the language

00:03:39.920 --> 00:03:42.159
the usb server itself can bring with it

00:03:42.159 --> 00:03:44.640
a host of additional dependencies

00:03:44.640 --> 00:03:47.680
external to Emacs which may message to

00:03:47.680 --> 00:03:50.640
install and manage

00:03:50.640 --> 00:03:53.760
furthermore json over rpc has pretty

00:03:53.760 --> 00:03:55.120
high latency

00:03:55.120 --> 00:03:57.840
for one-off tasks like jumping to source

00:03:57.840 --> 00:04:00.879
or on-demand completion is great

00:04:00.879 --> 00:04:03.040
but for things like code highlighting

00:04:03.040 --> 00:04:06.000
the latency is just too much

00:04:06.000 --> 00:04:08.319
I was using rust and I was following the

00:04:08.319 --> 00:04:10.480
community effort to improve its id

00:04:10.480 --> 00:04:11.760
support

00:04:11.760 --> 00:04:13.680
hoping to integrate some of that into

00:04:13.680 --> 00:04:15.760
Emacs itself

00:04:15.760 --> 00:04:17.600
then I heard someone from community

00:04:17.600 --> 00:04:19.759
mention tree sitter

00:04:19.759 --> 00:04:23.360
and I decided to check it out

00:04:23.360 --> 00:04:25.520
basically trisita is an incremental

00:04:25.520 --> 00:04:28.720
parsing library and a parser generator

00:04:28.720 --> 00:04:31.000
it was introduced by the item editor in

00:04:31.000 --> 00:04:33.040
2018

00:04:33.040 --> 00:04:35.680
besides item is also being integrated

00:04:35.680 --> 00:04:36.960
into the neo-vim

00:04:36.960 --> 00:04:41.040
editor and github is using it to power

00:04:41.040 --> 00:04:42.479
their source code analysis and

00:04:42.479 --> 00:04:45.840
navigation features

00:04:45.840 --> 00:04:48.639
it is written in c and can be compiled

00:04:48.639 --> 00:04:49.199
for all

00:04:49.199 --> 00:04:53.120
major platforms it can even be compiled

00:04:53.120 --> 00:04:56.080
to web assembly to run on the web that's

00:04:56.080 --> 00:04:57.600
how github is using it

00:04:57.600 --> 00:05:00.800
on their website

00:05:00.800 --> 00:05:02.960
so why is trisita an interesting

00:05:02.960 --> 00:05:05.840
solution to this problem

00:05:05.840 --> 00:05:07.360
there are multiple features that make it

00:05:07.360 --> 00:05:10.000
an attractive option

00:05:10.000 --> 00:05:12.400
it is designed to be fast by being

00:05:12.400 --> 00:05:13.680
incremental

00:05:13.680 --> 00:05:15.680
the initial parts of a typical big fight

00:05:15.680 --> 00:05:18.160
can take tens of milliseconds

00:05:18.160 --> 00:05:20.240
while subsequent incremental processes

00:05:20.240 --> 00:05:22.560
are sub milliseconds

00:05:22.560 --> 00:05:24.720
it achieves this by using structural

00:05:24.720 --> 00:05:26.240
sharing

00:05:26.240 --> 00:05:29.360
meaning replacing only affected nodes

00:05:29.360 --> 00:05:32.960
in the old tree when it needs to

00:05:32.960 --> 00:05:36.000
also unlike lsp being in the same

00:05:36.000 --> 00:05:37.120
process

00:05:37.120 --> 00:05:40.639
it has much lower latency

00:05:40.639 --> 00:05:42.880
secondly it provides a uniform

00:05:42.880 --> 00:05:44.960
programming interface

00:05:44.960 --> 00:05:47.039
the same data structures and functions

00:05:47.039 --> 00:05:48.720
work on parse trees of different

00:05:48.720 --> 00:05:50.400
languages

00:05:50.400 --> 00:05:52.160
syntax knows of different languages

00:05:52.160 --> 00:05:54.160
differ only by their types

00:05:54.160 --> 00:05:57.360
and their possible child nodes this

00:05:57.360 --> 00:05:58.960
is a big advantage over language

00:05:58.960 --> 00:06:02.240
specific parcels

00:06:02.240 --> 00:06:04.880
thirdly it's written in self-contained

00:06:04.880 --> 00:06:06.880
embeddable c

00:06:06.880 --> 00:06:09.680
as I mentioned previously it can even be

00:06:09.680 --> 00:06:10.400
compiled

00:06:10.400 --> 00:06:13.759
to webassembly this makes integrating it

00:06:13.759 --> 00:06:15.199
into various editors

00:06:15.199 --> 00:06:18.240
quite easy without having to install

00:06:18.240 --> 00:06:22.880
any external dependencies

00:06:22.880 --> 00:06:24.639
one thing that is not mentioned here is

00:06:24.639 --> 00:06:28.000
that being a parcel generator

00:06:28.000 --> 00:06:31.039
scrummers are declarative

00:06:31.039 --> 00:06:34.880
together with being editor independent

00:06:34.880 --> 00:06:36.720
this makes the pool of potential

00:06:36.720 --> 00:06:38.160
contributors

00:06:38.160 --> 00:06:42.400
much larger so I was convinced

00:06:42.400 --> 00:06:45.520
that trisito is a good fit for Emacs

00:06:45.520 --> 00:06:48.000
last year I started writing the bindings

00:06:48.000 --> 00:06:48.720
using

00:06:48.720 --> 00:06:50.960
dynamic model support introduced in imax

00:06:50.960 --> 00:06:53.280
25.

00:06:53.280 --> 00:06:55.360
dynamic module means there is platform

00:06:55.360 --> 00:06:58.479
specific native code involved

00:06:58.479 --> 00:07:00.560
but since they are pre-compiled binaries

00:07:00.560 --> 00:07:02.880
for the three major platforms

00:07:02.880 --> 00:07:06.319
it should work in most places currently

00:07:06.319 --> 00:07:08.319
the core functionalities are in a pretty

00:07:08.319 --> 00:07:09.440
good shape

00:07:09.440 --> 00:07:12.560
syntax highlighting is working nicely

00:07:12.560 --> 00:07:14.840
the whole thing is split into three

00:07:14.840 --> 00:07:16.080
packages

00:07:16.080 --> 00:07:17.759
tree sitter is the main package that

00:07:17.759 --> 00:07:20.319
other packages should depend on

00:07:20.319 --> 00:07:22.800
tree system lens is the language bundle

00:07:22.800 --> 00:07:24.000
that includes support

00:07:24.000 --> 00:07:27.199
for most common languages

00:07:27.199 --> 00:07:30.080
and finally the core apis are in the

00:07:30.080 --> 00:07:32.160
package tsc

00:07:32.160 --> 00:07:36.160
which stands for trees the core

00:07:36.160 --> 00:07:38.800
it is the implicit dependency of the

00:07:38.800 --> 00:07:43.520
three-seater package

00:07:43.520 --> 00:07:46.000
the main package includes the miner mode

00:07:46.000 --> 00:07:47.520
3-seater mode

00:07:47.520 --> 00:07:49.840
this provides the base for other major

00:07:49.840 --> 00:07:52.560
or minor modes to build on

00:07:52.560 --> 00:07:55.280
using image change tracking hooks it

00:07:55.280 --> 00:07:55.840
enables

00:07:55.840 --> 00:07:58.080
incremental parsing and provides a

00:07:58.080 --> 00:08:00.800
syntax tree that is always up to date

00:08:00.800 --> 00:08:04.080
after any edits in a buffer

00:08:04.080 --> 00:08:06.560
there is also a basic debug mode that

00:08:06.560 --> 00:08:10.080
shows the parse tree in another buffer

00:08:10.080 --> 00:08:13.360
here is a quick demo

00:08:13.360 --> 00:08:15.759
here I mean an empty python buffer with

00:08:15.759 --> 00:08:17.520
three seater enabled

00:08:17.520 --> 00:08:19.440
I'm going to turn on the debug mode to

00:08:19.440 --> 00:08:26.560
see the parse tree

00:08:26.560 --> 00:08:28.720
since the buffer is empty there is only

00:08:28.720 --> 00:08:30.639
one node in the syntax tree the top

00:08:30.639 --> 00:08:33.279
level module node

00:08:33.279 --> 00:09:11.040
let's try typing some code

00:09:11.040 --> 00:09:13.600
as you can see as I type into the python

00:09:13.600 --> 00:09:14.640
buffer

00:09:14.640 --> 00:09:19.120
the syntax tree updates in real time

00:09:19.120 --> 00:09:21.120
the other minor mode included in the

00:09:21.120 --> 00:09:23.279
main package is 3-seater

00:09:23.279 --> 00:09:26.640
hl mode it overrides font-lock mode and

00:09:26.640 --> 00:09:28.480
provides its own set of phases

00:09:28.480 --> 00:09:31.839
and customization options it is query

00:09:31.839 --> 00:09:32.800
driven

00:09:32.800 --> 00:09:35.200
that means instead of regular

00:09:35.200 --> 00:09:36.240
expressions

00:09:36.240 --> 00:09:38.720
it uses a list like query language to

00:09:38.720 --> 00:09:40.320
map syntax notes

00:09:40.320 --> 00:09:43.760
to highlighting phrases I'm going to

00:09:43.760 --> 00:09:45.760
open a python file with small snippets

00:09:45.760 --> 00:09:54.320
that showcase syntax highlighting

00:09:54.320 --> 00:09:55.920
so this is the default highlighting

00:09:55.920 --> 00:10:00.880
provided by python mode

00:10:00.880 --> 00:10:02.839
this is the highlighting enabled by tree

00:10:02.839 --> 00:10:04.640
sitter

00:10:04.640 --> 00:10:07.680
as you can see string interpolation

00:10:07.680 --> 00:10:11.680
and decorators are highlighted correctly

00:10:11.680 --> 00:10:17.440
function calls are also highlighted

00:10:17.440 --> 00:10:20.240
you can also note that property

00:10:20.240 --> 00:10:21.839
assessors

00:10:21.839 --> 00:10:24.640
and property assignments are highlighted

00:10:24.640 --> 00:10:27.440
differently

00:10:27.440 --> 00:10:29.360
what I like the most about this is that

00:10:29.360 --> 00:10:30.880
new bindings are consistently

00:10:30.880 --> 00:10:32.640
highlighted

00:10:32.640 --> 00:10:36.320
this included local variable

00:10:36.320 --> 00:10:39.760
function parameters and property

00:10:39.760 --> 00:10:45.760
mutations

00:10:45.760 --> 00:10:48.000
before going through the three queries

00:10:48.000 --> 00:10:49.279
and the syntax highlighting

00:10:49.279 --> 00:10:51.680
customization options

00:10:51.680 --> 00:10:53.760
let's take a brief look at the core data

00:10:53.760 --> 00:10:55.040
structures and functions

00:10:55.040 --> 00:10:58.079
that tree sitter provides

00:10:58.079 --> 00:10:59.839
so parsing is done with the help of a

00:10:59.839 --> 00:11:02.240
generic parser object

00:11:02.240 --> 00:11:04.160
a single parser object can be used to

00:11:04.160 --> 00:11:06.000
pass different languages

00:11:06.000 --> 00:11:08.320
by sending different language objects to

00:11:08.320 --> 00:11:09.279
it

00:11:09.279 --> 00:11:10.880
the language objects themselves are

00:11:10.880 --> 00:11:14.079
loaded from shared libraries

00:11:14.079 --> 00:11:16.079
since three seater mode already handles

00:11:16.079 --> 00:11:17.360
the parsing part

00:11:17.360 --> 00:11:19.440
we will instead focus on the functions

00:11:19.440 --> 00:11:20.800
that inspect nodes

00:11:20.800 --> 00:11:25.279
and in the resulting path tree

00:11:25.279 --> 00:11:27.200
we can ask tree sitter what is the

00:11:27.200 --> 00:11:44.240
syntax node at point

00:11:44.240 --> 00:11:47.200
uh is it an opaque object so this is not

00:11:47.200 --> 00:11:48.480
very useful

00:11:48.480 --> 00:12:03.760
we can instead ask what is its type

00:12:03.760 --> 00:12:06.560
so his type is the symbol comparison

00:12:06.560 --> 00:12:08.959
operator

00:12:08.959 --> 00:12:11.600
trees there are two kinds of nodes

00:12:11.600 --> 00:12:13.680
anonymous nodes and named nodes

00:12:13.680 --> 00:12:15.519
anonymous nodes correspond to simple

00:12:15.519 --> 00:12:17.040
grammar elements

00:12:17.040 --> 00:12:19.839
like keywords operators punctuations and

00:12:19.839 --> 00:12:21.279
so on

00:12:21.279 --> 00:12:24.160
name nodes on the other hand grammar

00:12:24.160 --> 00:12:25.920
elements that are interesting enough for

00:12:25.920 --> 00:12:26.639
their own

00:12:26.639 --> 00:12:30.320
to have a name like an identifier an

00:12:30.320 --> 00:12:31.839
expression

00:12:31.839 --> 00:12:35.440
or a function definition

00:12:35.440 --> 00:12:37.760
name node types are symbols while

00:12:37.760 --> 00:12:42.639
anonymous node types are strings

00:12:42.639 --> 00:12:46.320
for example if we are on this

00:12:46.320 --> 00:12:49.760
comparison operator

00:12:49.760 --> 00:12:55.920
the node type should be a string

00:12:55.920 --> 00:12:57.920
we can also get other information about

00:12:57.920 --> 00:12:58.959
the node

00:12:58.959 --> 00:13:09.680
for example what is this text

00:13:09.680 --> 00:13:20.800
or where it is in the buffer

00:13:20.800 --> 00:13:43.199
or what is its parent

00:13:43.199 --> 00:13:46.160
there are many other apis to query or

00:13:46.160 --> 00:13:46.839
not

00:13:46.839 --> 00:13:52.639
properties

00:13:52.639 --> 00:13:54.399
tree sitter allows searching for

00:13:54.399 --> 00:13:58.240
structural patterns within a parse tree

00:13:58.240 --> 00:14:01.440
it does so through a list like language

00:14:01.440 --> 00:14:03.519
this language supports by the matching

00:14:03.519 --> 00:14:04.639
by node types

00:14:04.639 --> 00:14:07.760
field names and predicates

00:14:07.760 --> 00:14:10.079
it also allows capturing nodes for

00:14:10.079 --> 00:14:12.639
further processing

00:14:12.639 --> 00:14:37.680
let's try to see some examples

00:14:37.680 --> 00:14:41.040
so in this very simple query we just

00:14:41.040 --> 00:14:43.839
try to highlight all the identifiers in

00:14:43.839 --> 00:14:49.040
the buffer

00:14:49.040 --> 00:14:51.920
this s side tells trisito to capture a

00:14:51.920 --> 00:14:53.120
node

00:14:53.120 --> 00:14:55.839
in the context of the query builder it's

00:14:55.839 --> 00:14:57.360
not very important

00:14:57.360 --> 00:15:00.320
but in normal highlighting query this

00:15:00.320 --> 00:15:01.760
will determine

00:15:01.760 --> 00:15:06.639
the face used to highlight the note

00:15:06.639 --> 00:15:08.800
suppose we want to capture all the

00:15:08.800 --> 00:15:10.320
function names

00:15:10.320 --> 00:15:13.519
instead of just any identifier

00:15:13.519 --> 00:15:29.440
you can improve the query like this

00:15:29.440 --> 00:15:31.600
uh this will highlight the whole

00:15:31.600 --> 00:15:32.639
definition

00:15:32.639 --> 00:15:35.519
but we only want to capture the function

00:15:35.519 --> 00:15:36.399
name

00:15:36.399 --> 00:15:39.600
which means the identifier

00:15:39.600 --> 00:15:42.800
here so we

00:15:42.800 --> 00:15:46.320
move the capture to after the identifier

00:15:46.320 --> 00:15:49.600
node

00:15:49.600 --> 00:15:51.759
if we want to capture the class names as

00:15:51.759 --> 00:15:52.959
well

00:15:52.959 --> 00:16:10.079
we just add another pattern

00:16:10.079 --> 00:16:20.320
let's look at a more practical example

00:16:20.320 --> 00:16:22.959
here we can see that single quotes

00:16:22.959 --> 00:16:23.759
strings and

00:16:23.759 --> 00:16:25.600
double quotes screens are highlighted

00:16:25.600 --> 00:16:27.279
the same

00:16:27.279 --> 00:16:30.399
but in some places

00:16:30.399 --> 00:16:33.440
because of some coding conventions

00:16:33.440 --> 00:16:35.440
it may be desirable to highlight them

00:16:35.440 --> 00:16:37.279
differently for example if

00:16:37.279 --> 00:16:39.680
the string is single quoted we may want

00:16:39.680 --> 00:16:40.880
to highlight it

00:16:40.880 --> 00:16:44.399
as a constant

00:16:44.399 --> 00:16:46.160
let's try to see whether we can

00:16:46.160 --> 00:16:47.600
distinguish these

00:16:47.600 --> 00:16:56.240
two cases

00:16:56.240 --> 00:17:00.639
so here we get all the strings

00:17:00.639 --> 00:17:04.079
if we want to see if it's single quotes

00:17:04.079 --> 00:17:04.559
or

00:17:04.559 --> 00:17:08.799
double quote strings

00:17:08.799 --> 00:17:11.039
we can try looking at the first

00:17:11.039 --> 00:17:12.480
character

00:17:12.480 --> 00:17:15.280
of the string I mean the first character

00:17:15.280 --> 00:17:16.720
of the note

00:17:16.720 --> 00:17:19.360
to check whether it's a single quote or

00:17:19.360 --> 00:17:33.600
a double quote

00:17:33.600 --> 00:17:36.080
yeah so for that we use the three

00:17:36.080 --> 00:17:36.799
setters

00:17:36.799 --> 00:17:40.160
support for predicate in this case

00:17:40.160 --> 00:17:43.360
we use a match predicate

00:17:43.360 --> 00:17:46.080
to check whether the string where the

00:17:46.080 --> 00:17:46.799
note

00:17:46.799 --> 00:17:50.320
starts with a single quote and with this

00:17:50.320 --> 00:17:51.280
pattern

00:17:51.280 --> 00:17:58.840
we only capture the single quotes

00:17:58.840 --> 00:18:00.400
strings

00:18:00.400 --> 00:18:03.760
let's try to give it a different face

00:18:03.760 --> 00:18:13.039
so we copy the pattern

00:18:13.039 --> 00:18:18.640
and we add this pattern

00:18:18.640 --> 00:18:25.120
pop item only

00:18:25.120 --> 00:18:28.400
but we also want to give the

00:18:28.400 --> 00:18:31.440
capture a different name

00:18:31.440 --> 00:18:40.840
let's say we want to highlight it as a

00:18:40.840 --> 00:18:46.559
keyword

00:18:46.559 --> 00:19:06.320
and now if we refresh the buffer

00:19:06.320 --> 00:19:08.799
we see that single quote strings are

00:19:08.799 --> 00:19:10.320
highlighted as

00:19:10.320 --> 00:19:14.400
keywords

00:19:14.400 --> 00:19:16.400
the highlighting patterns can also be

00:19:16.400 --> 00:19:19.200
set for a single project

00:19:19.200 --> 00:19:23.440
using directory local variable

00:19:23.440 --> 00:19:26.880
for example let's take a look at

00:19:26.880 --> 00:19:35.760
ems source code

00:19:35.760 --> 00:19:40.400
so in image c source there are a lot of

00:19:40.400 --> 00:19:43.760
uses of these different macros

00:19:43.760 --> 00:19:47.679
to define functions

00:19:47.679 --> 00:19:51.200
and you can see

00:19:51.200 --> 00:19:53.520
this is actually the function name but

00:19:53.520 --> 00:19:55.760
it's highlighted as the

00:19:55.760 --> 00:19:59.120
string so what we want

00:19:59.120 --> 00:20:03.679
is to somehow recognize this pattern

00:20:03.679 --> 00:20:07.600
and highlight it

00:20:07.600 --> 00:20:11.280
as highlight this part

00:20:11.280 --> 00:20:14.559
with the function phase instead

00:20:14.559 --> 00:20:17.679
in order to do that

00:20:17.679 --> 00:20:20.240
we put a pattern in this project

00:20:20.240 --> 00:20:21.760
directory local

00:20:21.760 --> 00:20:31.760
settings file

00:20:31.760 --> 00:20:34.799
so we can put this button in the c

00:20:34.799 --> 00:20:40.159
mode section

00:20:40.159 --> 00:20:48.000
and now if we enable tree sitter

00:20:48.000 --> 00:20:50.480
you can see that this is the highlighted

00:20:50.480 --> 00:20:53.200
uh

00:20:53.200 --> 00:20:55.520
as a normal function definition so this

00:20:55.520 --> 00:20:56.559
is the function

00:20:56.559 --> 00:21:01.200
face like we wanted

00:21:01.200 --> 00:21:03.760
the pattern for this is actually pretty

00:21:03.760 --> 00:21:07.200
simple

00:21:07.200 --> 00:21:10.720
it's only

00:21:10.720 --> 00:21:14.720
only this part so

00:21:14.720 --> 00:21:17.440
if it's a function call where the name

00:21:17.440 --> 00:21:19.679
of the function is different

00:21:19.679 --> 00:21:21.600
then we highlight the different as a

00:21:21.600 --> 00:21:24.240
keyword

00:21:24.240 --> 00:21:27.360
and then the first string element we

00:21:27.360 --> 00:21:28.159
highlighted

00:21:28.159 --> 00:21:35.360
as a function name

00:21:35.360 --> 00:21:37.679
since the language objects are actually

00:21:37.679 --> 00:21:39.280
native code

00:21:39.280 --> 00:21:40.799
they have to be compiled for each

00:21:40.799 --> 00:21:43.440
platform that we want to support

00:21:43.440 --> 00:21:45.600
this will become a big obstacle for

00:21:45.600 --> 00:21:48.159
3-seater adoption

00:21:48.159 --> 00:21:50.240
therefore I've created a language window

00:21:50.240 --> 00:21:52.960
package 3-seater length

00:21:52.960 --> 00:21:54.960
that takes care of pre-compiling the

00:21:54.960 --> 00:21:56.320
grammars the

00:21:56.320 --> 00:21:59.679
most common grammars for all three major

00:21:59.679 --> 00:22:01.600
platforms

00:22:01.600 --> 00:22:04.080
it also takes care of distributing these

00:22:04.080 --> 00:22:05.360
binaries

00:22:05.360 --> 00:22:08.080
and provides some highlighting queries

00:22:08.080 --> 00:22:11.440
for some of the languages

00:22:11.440 --> 00:22:13.760
it should be noted that this package

00:22:13.760 --> 00:22:15.919
should be treated as a temporary

00:22:15.919 --> 00:22:19.919
distribution mechanism only

00:22:19.919 --> 00:22:22.240
to help with bootstrapping three-seaters

00:22:22.240 --> 00:22:24.720
adoption

00:22:24.720 --> 00:22:27.760
the plan is that eventually these files

00:22:27.760 --> 00:22:29.760
should be provided by the language major

00:22:29.760 --> 00:22:32.480
modes themselves

00:22:32.480 --> 00:22:35.120
but in order to do that we need better

00:22:35.120 --> 00:22:36.320
tooling

00:22:36.320 --> 00:22:40.240
so we're not there yet

00:22:40.240 --> 00:22:42.559
since the call already works reasonably

00:22:42.559 --> 00:22:43.280
well

00:22:43.280 --> 00:22:44.640
there are several areas that would

00:22:44.640 --> 00:22:46.320
benefit from the community's

00:22:46.320 --> 00:22:49.120
contribution

00:22:49.120 --> 00:22:51.520
so three seaters upstream language

00:22:51.520 --> 00:22:52.640
prepositories

00:22:52.640 --> 00:22:54.400
already contain highlighting queries on

00:22:54.400 --> 00:22:55.679
their own

00:22:55.679 --> 00:22:58.480
however they are pretty basic and they

00:22:58.480 --> 00:23:00.480
may not fit well with existing emax

00:23:00.480 --> 00:23:02.559
conventions

00:23:02.559 --> 00:23:04.320
therefore the language bundle has its

00:23:04.320 --> 00:23:07.120
own set of highlighting queries

00:23:07.120 --> 00:23:10.559
this requires maintenance until language

00:23:10.559 --> 00:23:11.600
measurements adopt

00:23:11.600 --> 00:23:13.760
three sitter and maintain the queries on

00:23:13.760 --> 00:23:16.640
their own

00:23:16.640 --> 00:23:18.480
the queries are actually quite easy to

00:23:18.480 --> 00:23:22.000
write as you've already seen

00:23:22.000 --> 00:23:24.240
you just need to be familiar with the

00:23:24.240 --> 00:23:25.360
language

00:23:25.360 --> 00:23:30.000
familiar enough to come up with sensible

00:23:30.000 --> 00:23:35.200
highlighting patterns

00:23:35.200 --> 00:23:37.600
and if you are a maintainer of a

00:23:37.600 --> 00:23:39.679
language major mode

00:23:39.679 --> 00:23:42.320
you may want to consider integrating

00:23:42.320 --> 00:23:43.360
tree sitter into

00:23:43.360 --> 00:23:46.960
your mode initially maybe as an

00:23:46.960 --> 00:23:50.080
optional feature the integration is

00:23:50.080 --> 00:23:53.279
actually pretty straightforward

00:23:53.279 --> 00:23:56.640
especially for syntax highlighting

00:23:56.640 --> 00:24:01.520
or alternatively

00:24:01.520 --> 00:24:03.760
you can also try writing a new major

00:24:03.760 --> 00:24:04.640
mode

00:24:04.640 --> 00:24:08.000
from scratch that relies on tree sitter

00:24:08.000 --> 00:24:12.559
from the very beginning

00:24:12.559 --> 00:24:16.320
the code for such a major mode is

00:24:16.320 --> 00:24:19.679
quite simple for example

00:24:19.679 --> 00:24:23.200
this is the proposed

00:24:23.200 --> 00:24:26.240
what mode for web assembly

00:24:26.240 --> 00:24:31.039
the code is just

00:24:31.039 --> 00:24:34.559
like one page of code not

00:24:34.559 --> 00:24:39.520
not a lot

00:24:39.520 --> 00:24:42.720
you can also try writing new minor modes

00:24:42.720 --> 00:24:46.559
or writing integration packages

00:24:46.559 --> 00:24:50.080
for example a lot of package a lot of

00:24:50.080 --> 00:24:50.880
packages

00:24:50.880 --> 00:24:54.559
may benefit from tree sitter integration

00:24:54.559 --> 00:24:58.840
but no one has written the integration

00:24:58.840 --> 00:25:02.960
yet

00:25:02.960 --> 00:25:05.039
if you are interested in 3-seater you

00:25:05.039 --> 00:25:06.720
can use these links to

00:25:06.720 --> 00:25:10.320
learn more about it I think that's it

00:25:10.320 --> 00:25:11.440
for me today

00:25:11.440 --> 00:25:18.159
I'm happy to answer any questions