diff options
Diffstat (limited to '')
| -rw-r--r-- | 2020/info/23.md | 374 |
1 files changed, 372 insertions, 2 deletions
diff --git a/2020/info/23.md b/2020/info/23.md index f87ed299..7f20b2fd 100644 --- a/2020/info/23.md +++ b/2020/info/23.md @@ -1,8 +1,9 @@ # Incremental Parsing with emacs-tree-sitter Tuấn-Anh Nguyễn -[[!template id=vid src="https://mirror.csclub.uwaterloo.ca/emacsconf/2020/emacsconf-2020--23-incremental-parsing-with-emacs-tree-sitter--tuan-anh-nguyen.webm"]] -[Download compressed .webm video (21.8M)](https://mirror.csclub.uwaterloo.ca/emacsconf/2020/smaller/emacsconf-2020--23-incremental-parsing-with-emacs-tree-sitter--tuan-anh-nguyen--vp9-q56-video-original-audio.webm) +[[!template vidid="mainVideo" id=vid src="https://mirror.csclub.uwaterloo.ca/emacsconf/2020/emacsconf-2020--23-incremental-parsing-with-emacs-tree-sitter--tuan-anh-nguyen.webm" subtitles="/2020/subtitles/emacsconf-2020--23-incremental-parsing-with-emacs-tree-sitter--tuan-anh-nguyen.vtt"]] +[Download compressed .webm video (21.8M)](https://mirror.csclub.uwaterloo.ca/emacsconf/2020/smaller/emacsconf-2020--23-incremental-parsing-with-emacs-tree-sitter--tuan-anh-nguyen--vp9-q56-video-original-audio.webm) +[View transcript](#transcript) [[!template id=vid src="https://mirror.csclub.uwaterloo.ca/emacsconf/2020/emacsconf-2020--23-incremental-parsing-with-emacs-tree-sitter--questions--tuan-anh-nguyen.webm" download="Download Q&A video"]] [Download compressed Q&A .webm video (16.4M)](https://mirror.csclub.uwaterloo.ca/emacsconf/2020/smaller/emacsconf-2020--23-incremental-parsing-with-emacs-tree-sitter--questions--tuan-anh-nguyen--vp9-q56-video-original-audio.webm) @@ -175,3 +176,372 @@ Yes, it is just matter of paperwork. - LSP has high latency and is resource intensive, oft. - An updated video version was uploaded after the event, with the missing introduction to Tree-sitter added. + +<a name="transcript"></a> +# Transcript + +[[!template text="Hello, everyone! My name is Tuấn-Anh." start="00:00:01.520" video="mainVideo" id=subtitle]] +[[!template text="I've been using Emacs for about 10 years." start="00:00:04.400" video="mainVideo" id=subtitle]] +[[!template text="Today, I'm going to talk about tree-sitter," start="00:00:07.200" video="mainVideo" id=subtitle]] +[[!template text="a new Emacs package that allows Emacs" start="00:00:09.280" video="mainVideo" id=subtitle]] +[[!template text="to parse multiple programming languages in real-time." start="00:00:11.351" video="mainVideo" id=subtitle]] +[[!template new="1" text="So what is the problem statement?" start="00:00:17.840" video="mainVideo" id=subtitle]] +[[!template text="In order to support programming functionalities" start="00:00:21.840" video="mainVideo" id=subtitle]] +[[!template text="for a particular language," start="00:00:24.131" video="mainVideo" id=subtitle]] +[[!template text="a text editor needs to have some degree" start="00:00:25.760" video="mainVideo" id=subtitle]] +[[!template text="of language understanding." start="00:00:27.680" video="mainVideo" id=subtitle]] +[[!template text="Traditionally, text editors have relied" start="00:00:29.679" video="mainVideo" id=subtitle]] +[[!template text="very heavily on regular expressions for this." start="00:00:31.840" video="mainVideo" id=subtitle]] +[[!template text="Emacs is no different." start="00:00:34.960" video="mainVideo" id=subtitle]] +[[!template text="Most language major modes use regular expressions" start="00:00:37.013" video="mainVideo" id=subtitle]] +[[!template text="for syntax-highlighting, code navigation," start="00:00:40.170" video="mainVideo" id=subtitle]] +[[!template text="folding, indexing, and so on." start="00:00:42.960" video="mainVideo" id=subtitle]] +[[!template text="Regular expressions are problematic for a couple of reasons." start="00:00:46.618" video="mainVideo" id=subtitle]] +[[!template text="They're slow and inaccurate." start="00:00:50.559" video="mainVideo" id=subtitle]] +[[!template text="They also make the code hard to read and write." start="00:00:53.778" video="mainVideo" id=subtitle]] +[[!template text="Sometimes it's because the regular expressions themselves are very hairy," start="00:00:56.800" video="mainVideo" id=subtitle]] +[[!template text="and sometimes because they are just not powerful enough." start="00:01:01.199" video="mainVideo" id=subtitle]] +[[!template text="Some helper code is usually needed" start="00:01:05.199" video="mainVideo" id=subtitle]] +[[!template text="to parse more intricate language features." start="00:01:08.625" video="mainVideo" id=subtitle]] +[[!template text="That also illustrates the core problem with regular expressions," start="00:01:11.200" video="mainVideo" id=subtitle]] +[[!template text="in that they are not powerful enough to parse programming languages." start="00:01:16.159" video="mainVideo" id=subtitle]] +[[!template text="An example feature that regular expressions cannot handle very well" start="00:01:21.119" video="mainVideo" id=subtitle]] +[[!template text="is string interpolation, which is a very common feature" start="00:01:25.040" video="mainVideo" id=subtitle]] +[[!template text="in many modern programming languages." start="00:01:28.320" video="mainVideo" id=subtitle]] +[[!template new="1" text="It would be much nicer if Emacs somehow" start="00:01:31.680" video="mainVideo" id=subtitle]] +[[!template text="had structural understanding of source code, like IDEs do." start="00:01:34.079" video="mainVideo" id=subtitle]] +[[!template text="There have been multiple efforts" start="00:01:39.520" video="mainVideo" id=subtitle]] +[[!template text="to bring this kind of programming language understanding into Emacs." start="00:01:41.981" video="mainVideo" id=subtitle]] +[[!template text="There are language-specific parsers" start="00:01:45.280" video="mainVideo" id=subtitle]] +[[!template text="written in Elisp" start="00:01:47.119" video="mainVideo" id=subtitle]] +[[!template text="that can be thought of" start="00:01:48.640" video="mainVideo" id=subtitle]] +[[!template text="as the next logical step of the glue code" start="00:01:50.675" video="mainVideo" id=subtitle]] +[[!template text="on top of regular expressions," start="00:01:51.989" video="mainVideo" id=subtitle]] +[[!template text="moving from partial local pattern recognition" start="00:01:53.856" video="mainVideo" id=subtitle]] +[[!template text="into a full-fledged parser." start="00:01:57.356" video="mainVideo" id=subtitle]] +[[!template text="The most prominent example of this approach" start="00:01:59.840" video="mainVideo" id=subtitle]] +[[!template text="is probably the famous js2-mode." start="00:02:02.023" video="mainVideo" id=subtitle]] +[[!template new="1" text="However, this approach has several issues." start="00:02:06.479" video="mainVideo" id=subtitle]] +[[!template text="Parsing is computationally expensive," start="00:02:10.080" video="mainVideo" id=subtitle]] +[[!template text="and Emacs Lisp is not good at that kind of stuff." start="00:02:12.606" video="mainVideo" id=subtitle]] +[[!template new="1" text="Furthermore, maintenance is very troublesome." start="00:02:16.800" video="mainVideo" id=subtitle]] +[[!template text="In order to work on these parsers," start="00:02:19.156" video="mainVideo" id=subtitle]] +[[!template text="first, you have to know Elisp well enough," start="00:02:22.160" video="mainVideo" id=subtitle]] +[[!template text="and then you have to be comfortable with" start="00:02:24.239" video="mainVideo" id=subtitle]] +[[!template text="writing a recursive descending parser," start="00:02:26.606" video="mainVideo" id=subtitle]] +[[!template text="while constantly keeping up with changes to the language itself," start="00:02:29.739" video="mainVideo" id=subtitle]] +[[!template text="which can be evolving very quickly," start="00:02:34.000" video="mainVideo" id=subtitle]] +[[!template text="like Javascript, for example." start="00:02:36.356" video="mainVideo" id=subtitle]] +[[!template new="1" text="Together, these constraints significantly reduce" start="00:02:39.360" video="mainVideo" id=subtitle]] +[[!template text="the pool of potential maintainers." start="00:02:42.373" video="mainVideo" id=subtitle]] +[[!template text="The biggest issue, though, in my opinion," start="00:02:45.680" video="mainVideo" id=subtitle]] +[[!template text="is lack of the set of generic and reusable APIs." start="00:02:47.760" video="mainVideo" id=subtitle]] +[[!template text="This makes them very hard to use" start="00:02:52.139" video="mainVideo" id=subtitle]] +[[!template text="for minor modes that want to deal with" start="00:02:54.319" video="mainVideo" id=subtitle]] +[[!template text="cross-cutting concerns across multiple languages." start="00:02:55.920" video="mainVideo" id=subtitle]] +[[!template new="1" text="The other approach which has been" start="00:02:59.920" video="mainVideo" id=subtitle]] +[[!template text="gaining a lot of momentum in recent years" start="00:03:01.760" video="mainVideo" id=subtitle]] +[[!template text="is externalizing language understanding" start="00:03:04.319" video="mainVideo" id=subtitle]] +[[!template text="to another process," start="00:03:06.560" video="mainVideo" id=subtitle]] +[[!template text="also known as language server protocol." start="00:03:08.159" video="mainVideo" id=subtitle]] +[[!template new="1" text="This second approach is actually a very interesting one." start="00:03:12.239" video="mainVideo" id=subtitle]] +[[!template text="By decoupling language understanding" start="00:03:16.560" video="mainVideo" id=subtitle]] +[[!template text="from the editing facility itself," start="00:03:18.400" video="mainVideo" id=subtitle]] +[[!template text="the LSP servers can attract a lot more contributors," start="00:03:21.280" video="mainVideo" id=subtitle]] +[[!template text="which makes maintenance easier." start="00:03:25.120" video="mainVideo" id=subtitle]] +[[!template new="1" text="However, they also have several issues of their own." start="00:03:27.189" video="mainVideo" id=subtitle]] +[[!template text="Being a separate process," start="00:03:32.400" video="mainVideo" id=subtitle]] +[[!template text="they are usually more resource-intensive," start="00:03:34.089" video="mainVideo" id=subtitle]] +[[!template text="and depending on the language," start="00:03:37.073" video="mainVideo" id=subtitle]] +[[!template text="the LSP server itself can bring with it" start="00:03:39.920" video="mainVideo" id=subtitle]] +[[!template text="a host of additional dependencies" start="00:03:42.159" video="mainVideo" id=subtitle]] +[[!template text="external to Emacs, which may be messy to install and manage." start="00:03:44.640" video="mainVideo" id=subtitle]] +[[!template new="1" text="Furthermore, JSON over RPC has pretty high latency." start="00:03:50.640" video="mainVideo" id=subtitle]] +[[!template text="For one-off tasks like jumping to source" start="00:03:55.120" video="mainVideo" id=subtitle]] +[[!template text="or on-demand completion, it's great." start="00:03:57.840" video="mainVideo" id=subtitle]] +[[!template text="But for things like code highlighting," start="00:04:00.879" video="mainVideo" id=subtitle]] +[[!template text="the latency is just too much." start="00:04:03.040" video="mainVideo" id=subtitle]] +[[!template new="1" text="I was using Rust and I was following the" start="00:04:06.000" video="mainVideo" id=subtitle]] +[[!template text="community effort to improve its IDE support," start="00:04:08.319" video="mainVideo" id=subtitle]] +[[!template text="hoping to integrate some of that into Emacs itself." start="00:04:11.760" video="mainVideo" id=subtitle]] +[[!template text="Then I heard someone from the community mention tree-sitter," start="00:04:15.760" video="mainVideo" id=subtitle]] +[[!template text="and I decided to check it out." start="00:04:19.759" video="mainVideo" id=subtitle]] +[[!template text="Basically, tree-sitter is an incremental parsing library and a parser generator." start="00:04:23.360" video="mainVideo" id=subtitle]] +[[!template text="It was introduced by the Atom editor in 2018." start="00:04:28.720" video="mainVideo" id=subtitle]] +[[!template text="Besides Atom, it is also being integrated" start="00:04:33.040" video="mainVideo" id=subtitle]] +[[!template text="into the NeoVim editor," start="00:04:35.923" video="mainVideo" id=subtitle]] +[[!template text="and Github is using it to power" start="00:04:37.623" video="mainVideo" id=subtitle]] +[[!template text="their source code analysis" start="00:04:41.040" video="mainVideo" id=subtitle]] +[[!template text="and navigation features." start="00:04:42.423" video="mainVideo" id=subtitle]] +[[!template text="It is written in C and can be compiled" start="00:04:45.840" video="mainVideo" id=subtitle]] +[[!template text="for all major platforms." start="00:04:48.639" video="mainVideo" id=subtitle]] +[[!template text="It can even be compiled" start="00:04:50.623" video="mainVideo" id=subtitle]] +[[!template text="to web assembly to run on the web." start="00:04:53.120" video="mainVideo" id=subtitle]] +[[!template text="That's how Github is using it on their website." start="00:04:55.323" video="mainVideo" id=subtitle]] +[[!template new="1" text="So why is tree-sitter an interesting solution to this problem?" start="00:05:00.800" video="mainVideo" id=subtitle]] +[[!template text="There are multiple features that make it an attractive option." start="00:05:05.840" video="mainVideo" id=subtitle]] +[[!template text="It is designed to be fast." start="00:05:10.000" video="mainVideo" id=subtitle]] +[[!template text="By being incremental," start="00:05:11.839" video="mainVideo" id=subtitle]] +[[!template text="the initial parse of a typical big file" start="00:05:13.680" video="mainVideo" id=subtitle]] +[[!template text="can take tens of milliseconds," start="00:05:15.680" video="mainVideo" id=subtitle]] +[[!template text="while subsequent incremental processes" start="00:05:18.160" video="mainVideo" id=subtitle]] +[[!template text="are sub-millisecond." start="00:05:20.240" video="mainVideo" id=subtitle]] +[[!template text="It achieves this by using structural sharing," start="00:05:22.560" video="mainVideo" id=subtitle]] +[[!template text="meaning replacing only affected nodes" start="00:05:26.240" video="mainVideo" id=subtitle]] +[[!template text="in the old tree when it needs to." start="00:05:29.360" video="mainVideo" id=subtitle]] +[[!template text="Also, unlike LSP, being in the same process," start="00:05:32.960" video="mainVideo" id=subtitle]] +[[!template text="it has much lower latency." start="00:05:37.120" video="mainVideo" id=subtitle]] +[[!template new="1" text="Secondly, it provides a uniform programming interface." start="00:05:40.639" video="mainVideo" id=subtitle]] +[[!template text="The same data structures and functions" start="00:05:44.960" video="mainVideo" id=subtitle]] +[[!template text="work on parse trees of different languages." start="00:05:47.039" video="mainVideo" id=subtitle]] +[[!template text="Syntax nodes of different languages" start="00:05:50.400" video="mainVideo" id=subtitle]] +[[!template text="differ only by their types" start="00:05:52.160" video="mainVideo" id=subtitle]] +[[!template text="and their possible child nodes." start="00:05:54.160" video="mainVideo" id=subtitle]] +[[!template text="This is a big advantage over language-specific parsers." start="00:05:55.723" video="mainVideo" id=subtitle]] +[[!template text="Thirdly, it's written in self-contained embeddable C." start="00:06:02.240" video="mainVideo" id=subtitle]] +[[!template text="As I mentioned previously, it can even be compiled to webassembly." start="00:06:06.880" video="mainVideo" id=subtitle]] +[[!template text="This makes integrating it into various editors quite easy" start="00:06:11.723" video="mainVideo" id=subtitle]] +[[!template text="without having to install any external dependencies." start="00:06:16.106" video="mainVideo" id=subtitle]] +[[!template new="1" text="One thing that is not mentioned here" start="00:06:22.880" video="mainVideo" id=subtitle]] +[[!template text="is that being a parser generator," start="00:06:25.503" video="mainVideo" id=subtitle]] +[[!template text="its grammars are declarative." start="00:06:28.000" video="mainVideo" id=subtitle]] +[[!template text="Together with being editor-independent," start="00:06:31.039" video="mainVideo" id=subtitle]] +[[!template text="this makes the pool of potential contributors much larger." start="00:06:34.880" video="mainVideo" id=subtitle]] +[[!template new="1" text="So I was convinced that tree-sitter is a good fit for Emacs." start="00:06:39.139" video="mainVideo" id=subtitle]] +[[!template text="Last year, I started writing the bindings" start="00:06:45.520" video="mainVideo" id=subtitle]] +[[!template text="using dynamic module support introduced in Emacs 25." start="00:06:48.000" video="mainVideo" id=subtitle]] +[[!template text="Dynamic module means there is platform-specific native code involved," start="00:06:53.280" video="mainVideo" id=subtitle]] +[[!template text="but since there are pre-compiled binaries" start="00:06:58.479" video="mainVideo" id=subtitle]] +[[!template text="for the three major platforms," start="00:07:00.560" video="mainVideo" id=subtitle]] +[[!template text="it should work in most places." start="00:07:02.880" video="mainVideo" id=subtitle]] +[[!template text="Currently, the core functionalities are in a pretty good shape." start="00:07:04.706" video="mainVideo" id=subtitle]] +[[!template text="Syntax highlighting is working nicely." start="00:07:09.440" video="mainVideo" id=subtitle]] +[[!template new="1" text="The whole thing is split into three packages." start="00:07:12.560" video="mainVideo" id=subtitle]] +[[!template text="tree-sitter is the main package that other packages should depend on." start="00:07:16.080" video="mainVideo" id=subtitle]] +[[!template text="tree-sitter-langs is the language bundle" start="00:07:20.319" video="mainVideo" id=subtitle]] +[[!template text="that includes support" start="00:07:22.800" video="mainVideo" id=subtitle]] +[[!template text="for most common languages." start="00:07:24.000" video="mainVideo" id=subtitle]] +[[!template text="And finally, the core APIs are in the package tsc," start="00:07:27.199" video="mainVideo" id=subtitle]] +[[!template text="which stands for tree-sitter-core." start="00:07:32.160" video="mainVideo" id=subtitle]] +[[!template text="It is the implicit dependency of the" start="00:07:36.160" video="mainVideo" id=subtitle]] +[[!template text="tree-sitter package." start="00:07:38.800" video="mainVideo" id=subtitle]] +[[!template text="The main package includes the minor mode tree-sitter-mode." start="00:07:43.520" video="mainVideo" id=subtitle]] +[[!template text="This provides the base for other major or minor modes to build on." start="00:07:47.520" video="mainVideo" id=subtitle]] +[[!template text="Using Emacs's change tracking hooks," start="00:07:52.560" video="mainVideo" id=subtitle]] +[[!template text="it enables incremental parsing" start="00:07:54.839" video="mainVideo" id=subtitle]] +[[!template text="and provides a syntax tree that is always up to date" start="00:07:57.073" video="mainVideo" id=subtitle]] +[[!template text="after any edits in a buffer." start="00:08:00.800" video="mainVideo" id=subtitle]] +[[!template text="There is also a basic debug mode" start="00:08:04.080" video="mainVideo" id=subtitle]] +[[!template text="that shows the parse tree in another buffer." start="00:08:06.223" video="mainVideo" id=subtitle]] +[[!template new="1" text="Here is a quick demo." start="00:08:10.080" video="mainVideo" id=subtitle]] +[[!template text="Here I'm in an empty Python buffer" start="00:08:13.360" video="mainVideo" id=subtitle]] +[[!template text="with tree-sitter enabled." start="00:08:15.673" video="mainVideo" id=subtitle]] +[[!template text="I'm going to turn on the debug mode to" start="00:08:17.520" video="mainVideo" id=subtitle]] +[[!template text="see the parse tree." start="00:08:19.440" video="mainVideo" id=subtitle]] +[[!template text="Since the buffer is empty," start="00:08:26.560" video="mainVideo" id=subtitle]] +[[!template text="there is only one node in the syntax tree:" start="00:08:28.106" video="mainVideo" id=subtitle]] +[[!template text="the top-level module node." start="00:08:30.423" video="mainVideo" id=subtitle]] +[[!template text="Let's try typing some code." start="00:08:33.279" video="mainVideo" id=subtitle]] +[[!template text="As you can see, as I type into the Python buffer," start="00:09:11.040" video="mainVideo" id=subtitle]] +[[!template text="the syntax tree updates in real time." start="00:09:14.640" video="mainVideo" id=subtitle]] +[[!template new="1" text="The other minor mode included in the main package" start="00:09:19.120" video="mainVideo" id=subtitle]] +[[!template text="is tree-sitter-hl-mode." start="00:09:22.039" video="mainVideo" id=subtitle]] +[[!template text="It overrides font-lock mode" start="00:09:24.389" video="mainVideo" id=subtitle]] +[[!template text="and provides its own set of phases" start="00:09:26.349" video="mainVideo" id=subtitle]] +[[!template text="and customization options" start="00:09:28.480" video="mainVideo" id=subtitle]] +[[!template text="It is query-driven." start="00:09:30.139" video="mainVideo" id=subtitle]] +[[!template text="That means instead of regular expressions," start="00:09:32.800" video="mainVideo" id=subtitle]] +[[!template text="it uses a Lisp-like query language" start="00:09:36.240" video="mainVideo" id=subtitle]] +[[!template text="to map syntax nodes" start="00:09:39.518" video="mainVideo" id=subtitle]] +[[!template text="to highlighting phrases." start="00:09:40.320" video="mainVideo" id=subtitle]] +[[!template text="I'm going to open a python file with small snippets" start="00:09:41.923" video="mainVideo" id=subtitle]] +[[!template text="that showcase syntax highlighting." start="00:09:45.760" video="mainVideo" id=subtitle]] +[[!template text="So this is the default highlighting" start="00:09:54.320" video="mainVideo" id=subtitle]] +[[!template text="provided by python-mode." start="00:09:55.920" video="mainVideo" id=subtitle]] +[[!template text="This is the highlighting enabled by tree-sitter." start="00:10:00.880" video="mainVideo" id=subtitle]] +[[!template text="As you can see, string interpolation" start="00:10:04.640" video="mainVideo" id=subtitle]] +[[!template text="and decorators are highlighted correctly." start="00:10:07.680" video="mainVideo" id=subtitle]] +[[!template text="Function calls are also highlighted." start="00:10:11.680" video="mainVideo" id=subtitle]] +[[!template text="You can also note that property accessors" start="00:10:17.440" video="mainVideo" id=subtitle]] +[[!template text="and property assignments are highlighted differently." start="00:10:21.839" video="mainVideo" id=subtitle]] +[[!template text="What I like the most about this is that" start="00:10:27.440" video="mainVideo" id=subtitle]] +[[!template text="new bindings are consistently highlighted." start="00:10:29.360" video="mainVideo" id=subtitle]] +[[!template text="This included local variables," start="00:10:32.640" video="mainVideo" id=subtitle]] +[[!template text="function parameters, and property mutations." start="00:10:36.320" video="mainVideo" id=subtitle]] +[[!template new="1" text="Before going through the tree queries" start="00:10:45.760" video="mainVideo" id=subtitle]] +[[!template text="and the syntax highlighting" start="00:10:48.000" video="mainVideo" id=subtitle]] +[[!template text="customization options," start="00:10:49.279" video="mainVideo" id=subtitle]] +[[!template text="let's take a brief look at" start="00:10:51.680" video="mainVideo" id=subtitle]] +[[!template text="the core data structures and functions" start="00:10:53.339" video="mainVideo" id=subtitle]] +[[!template text="that tree-sitter provides." start="00:10:55.040" video="mainVideo" id=subtitle]] +[[!template text="So parsing is done with the help of" start="00:10:58.079" video="mainVideo" id=subtitle]] +[[!template text="a generic parser object." start="00:11:00.743" video="mainVideo" id=subtitle]] +[[!template text="A single parser object can be used to" start="00:11:02.240" video="mainVideo" id=subtitle]] +[[!template text="parse different languages" start="00:11:04.160" video="mainVideo" id=subtitle]] +[[!template text="by sending different language objects to it." start="00:11:06.000" video="mainVideo" id=subtitle]] +[[!template text="The language objects themselves are" start="00:11:09.279" video="mainVideo" id=subtitle]] +[[!template text="loaded from shared libraries." start="00:11:10.880" video="mainVideo" id=subtitle]] +[[!template text="Since tree-sitter-mmode already handles" start="00:11:14.079" video="mainVideo" id=subtitle]] +[[!template text="the parsing part," start="00:11:16.079" video="mainVideo" id=subtitle]] +[[!template text="we will instead focus on the functions" start="00:11:17.360" video="mainVideo" id=subtitle]] +[[!template text="that inspect nodes," start="00:11:19.440" video="mainVideo" id=subtitle]] +[[!template text="and in the resulting path tree," start="00:11:20.800" video="mainVideo" id=subtitle]] +[[!template text="we can ask tree-sitter what is" start="00:11:25.279" video="mainVideo" id=subtitle]] +[[!template text="the syntax node at point." start="00:11:27.030" video="mainVideo" id=subtitle]] +[[!template text="This is an opaque object, so this is not very useful." start="00:11:44.240" video="mainVideo" id=subtitle]] +[[!template text="We can instead ask what is its type." start="00:11:48.480" video="mainVideo" id=subtitle]] +[[!template text="So its type is the symbol comparison operator." start="00:12:03.760" video="mainVideo" id=subtitle]] +[[!template new="1" text="In tree-sitter, there are two kinds of nodes," start="00:12:08.959" video="mainVideo" id=subtitle]] +[[!template text="anonymous nodes and named nodes." start="00:12:11.600" video="mainVideo" id=subtitle]] +[[!template text="Anonymous nodes correspond to simple grammar elements" start="00:12:13.680" video="mainVideo" id=subtitle]] +[[!template text="like keywords, operators, punctuations, and so on." start="00:12:17.040" video="mainVideo" id=subtitle]] +[[!template text="Name nodes, on the other hand, are grammar elements" start="00:12:21.279" video="mainVideo" id=subtitle]] +[[!template text="that are interesting enough on their own" start="00:12:24.656" video="mainVideo" id=subtitle]] +[[!template text="to have a name, like an identifier," start="00:12:26.639" video="mainVideo" id=subtitle]] +[[!template text="an expression, or a function definition." start="00:12:30.029" video="mainVideo" id=subtitle]] +[[!template text="Name node types are symbols," start="00:12:35.440" video="mainVideo" id=subtitle]] +[[!template text="while anonymous node types are strings." start="00:12:37.323" video="mainVideo" id=subtitle]] +[[!template text="For example, if we are on this comparison operator," start="00:12:42.639" video="mainVideo" id=subtitle]] +[[!template text="the node type should be a string." start="00:12:49.760" video="mainVideo" id=subtitle]] +[[!template text="We can also get other information about the node." start="00:12:55.920" video="mainVideo" id=subtitle]] +[[!template text="For example: what is this text," start="00:12:58.959" video="mainVideo" id=subtitle]] +[[!template text="or where it is in the buffer," start="00:13:09.680" video="mainVideo" id=subtitle]] +[[!template text="or what is its parent." start="00:13:20.800" video="mainVideo" id=subtitle]] +[[!template new="1" text="There are many other APIs to query" start="00:13:43.199" video="mainVideo" id=subtitle]] +[[!template text="our node's properties." start="00:13:46.106" video="mainVideo" id=subtitle]] +[[!template text="tree-sitter allows searching" start="00:13:52.639" video="mainVideo" id=subtitle]] +[[!template text="for structural patterns within a parse tree." start="00:13:54.234" video="mainVideo" id=subtitle]] +[[!template text="It does so through a Lisp-like language." start="00:13:58.240" video="mainVideo" id=subtitle]] +[[!template text="This language supports matching by node types," start="00:14:01.440" video="mainVideo" id=subtitle]] +[[!template text="field names, and predicates." start="00:14:04.639" video="mainVideo" id=subtitle]] +[[!template text="It also allows capturing nodes for further processing." start="00:14:07.760" video="mainVideo" id=subtitle]] +[[!template text="Let's try to see some examples." start="00:14:12.639" video="mainVideo" id=subtitle]] +[[!template text="So in this very simple query," start="00:14:37.680" video="mainVideo" id=subtitle]] +[[!template text="we just try to highlight all the identifiers in the buffer." start="00:14:40.206" video="mainVideo" id=subtitle]] +[[!template text="This s side tells tree-sitter to capture a node." start="00:14:49.040" video="mainVideo" id=subtitle]] +[[!template text="In the context of the query builder," start="00:14:53.120" video="mainVideo" id=subtitle]] +[[!template text="it's not very important," start="00:14:55.507" video="mainVideo" id=subtitle]] +[[!template text="but in normal highlighting query," start="00:14:57.360" video="mainVideo" id=subtitle]] +[[!template text="this will determine" start="00:14:59.706" video="mainVideo" id=subtitle]] +[[!template text="the face used to highlight the note." start="00:15:01.760" video="mainVideo" id=subtitle]] +[[!template text="Suppose we want to capture" start="00:15:06.639" video="mainVideo" id=subtitle]] +[[!template text="all the function names," start="00:15:08.256" video="mainVideo" id=subtitle]] +[[!template text="instead of just any identifier." start="00:15:10.320" video="mainVideo" id=subtitle]] +[[!template text="You can improve the query like this." start="00:15:13.519" video="mainVideo" id=subtitle]] +[[!template text="This will highlight the whole definition." start="00:15:29.440" video="mainVideo" id=subtitle]] +[[!template text="But we only want to capture the function name," start="00:15:32.639" video="mainVideo" id=subtitle]] +[[!template text="which means the identifier here." start="00:15:36.399" video="mainVideo" id=subtitle]] +[[!template text="So we move the capture to after the identifier node." start="00:15:41.054" video="mainVideo" id=subtitle]] +[[!template text="If we want to capture the class names as well," start="00:15:49.600" video="mainVideo" id=subtitle]] +[[!template text="we just add another pattern." start="00:15:52.959" video="mainVideo" id=subtitle]] +[[!template new="1" text="Let's look at a more practical example." start="00:16:10.079" video="mainVideo" id=subtitle]] +[[!template text="Here we can see that single-quoted strings" start="00:16:20.320" video="mainVideo" id=subtitle]] +[[!template text="and double-quoted strings are highlighted the same." start="00:16:23.468" video="mainVideo" id=subtitle]] +[[!template text="But in some places," start="00:16:27.279" video="mainVideo" id=subtitle]] +[[!template text="because of some coding conventions," start="00:16:30.399" video="mainVideo" id=subtitle]] +[[!template text="it may be desirable to highlight them differently." start="00:16:33.440" video="mainVideo" id=subtitle]] +[[!template text="For example, if the string is single-quoted," start="00:16:36.373" video="mainVideo" id=subtitle]] +[[!template text="we may want to highlight it as a constant." start="00:16:39.073" video="mainVideo" id=subtitle]] +[[!template text="Let's try to see whether we can" start="00:16:44.399" video="mainVideo" id=subtitle]] +[[!template text="distinguish these two cases." start="00:16:46.160" video="mainVideo" id=subtitle]] +[[!template text="So here we get all the strings." start="00:16:56.240" video="mainVideo" id=subtitle]] +[[!template text="If we want to see if it's single quotes" start="00:17:00.639" video="mainVideo" id=subtitle]] +[[!template text="or double quote strings," start="00:17:04.079" video="mainVideo" id=subtitle]] +[[!template text="we can try looking at the first character of the string--" start="00:17:08.799" video="mainVideo" id=subtitle]] +[[!template text="I mean the first character of the node--" start="00:17:13.436" video="mainVideo" id=subtitle]] +[[!template text="to check whether it's a single quote or a double quote." start="00:17:16.720" video="mainVideo" id=subtitle]] +[[!template text="So for that, we use tree-sitter's support for predicates." start="00:17:33.600" video="mainVideo" id=subtitle]] +[[!template text="In this case, we use a match predicate" start="00:17:38.920" video="mainVideo" id=subtitle]] +[[!template text="to check whether the string-- whether the node starts" start="00:17:43.360" video="mainVideo" id=subtitle]] +[[!template text="with a single quote." start="00:17:47.339" video="mainVideo" id=subtitle]] +[[!template text="And with this pattern," start="00:17:49.556" video="mainVideo" id=subtitle]] +[[!template text="we only capture the single-quotes strings." start="00:17:51.280" video="mainVideo" id=subtitle]] +[[!template text="Let's try to give it a different face." start="00:18:00.400" video="mainVideo" id=subtitle]] +[[!template text="So we copy the pattern," start="00:18:03.760" video="mainVideo" id=subtitle]] +[[!template text="and we add this pattern for Python only." start="00:18:13.039" video="mainVideo" id=subtitle]] +[[!template text="But we also want to give the capture a different name." start="00:18:25.120" video="mainVideo" id=subtitle]] +[[!template text="Let's say we want to highlight it as a keyword." start="00:18:31.440" video="mainVideo" id=subtitle]] +[[!template text="And now, if we refresh the buffer," start="00:18:46.559" video="mainVideo" id=subtitle]] +[[!template text="we see that single quote strings" start="00:19:06.320" video="mainVideo" id=subtitle]] +[[!template text="are highlighted as keywords." start="00:19:08.523" video="mainVideo" id=subtitle]] +[[!template new="1" text="The highlighting patterns" start="00:19:14.400" video="mainVideo" id=subtitle]] +[[!template text="can also be set for a single project" start="00:19:15.751" video="mainVideo" id=subtitle]] +[[!template text="using directory-local variables." start="00:19:19.200" video="mainVideo" id=subtitle]] +[[!template text="For example, let's take a look at Emacs's source code." start="00:19:23.440" video="mainVideo" id=subtitle]] +[[!template text="So in Emacs's C source, there are a lot of uses" start="00:19:35.760" video="mainVideo" id=subtitle]] +[[!template text="of these different macros" start="00:19:41.123" video="mainVideo" id=subtitle]] +[[!template text="to define functions," start="00:19:43.760" video="mainVideo" id=subtitle]] +[[!template text="and you can see this is actually the function name," start="00:19:47.679" video="mainVideo" id=subtitle]] +[[!template text="but it's highlighted as the string." start="00:19:53.256" video="mainVideo" id=subtitle]] +[[!template text="So what we want is to somehow recognize this pattern" start="00:19:56.373" video="mainVideo" id=subtitle]] +[[!template text="and highlight it." start="00:20:03.679" video="mainVideo" id=subtitle]] +[[!template text="Highlight this part" start="00:20:07.600" video="mainVideo" id=subtitle]] +[[!template text="with the function face instead." start="00:20:11.280" video="mainVideo" id=subtitle]] +[[!template text="In order to do that," start="00:20:14.559" video="mainVideo" id=subtitle]] +[[!template text="we put a pattern in this project's directory-local settings file." start="00:20:17.679" video="mainVideo" id=subtitle]] +[[!template text="So we can put this button in the C mode section." start="00:20:31.760" video="mainVideo" id=subtitle]] +[[!template text="And now, if we enable tree-sitter," start="00:20:40.159" video="mainVideo" id=subtitle]] +[[!template text="you can see that this is highlighted" start="00:20:48.000" video="mainVideo" id=subtitle]] +[[!template text="as a normal function definition." start="00:20:53.200" video="mainVideo" id=subtitle]] +[[!template text="So this is the function face like we wanted." start="00:20:55.056" video="mainVideo" id=subtitle]] +[[!template text="The pattern for this is actually pretty simple." start="00:21:01.200" video="mainVideo" id=subtitle]] +[[!template text="It's only this part." start="00:21:07.200" video="mainVideo" id=subtitle]] +[[!template text="So if it's a function call" start="00:21:12.373" video="mainVideo" id=subtitle]] +[[!template text="where the name of the function is defun," start="00:21:16.456" video="mainVideo" id=subtitle]] +[[!template text="then we highlight the defun as a keyword," start="00:21:19.679" video="mainVideo" id=subtitle]] +[[!template text="and then the first string element," start="00:21:24.240" video="mainVideo" id=subtitle]] +[[!template text="we highlight it as a function name." start="00:21:26.923" video="mainVideo" id=subtitle]] +[[!template new="1" text="Since the language objects are actually native code," start="00:21:35.360" video="mainVideo" id=subtitle]] +[[!template text="they have to be compiled for each platform" start="00:21:39.280" video="mainVideo" id=subtitle]] +[[!template text="that we want to support." start="00:21:41.459" video="mainVideo" id=subtitle]] +[[!template text="This will become a big obstacle for tree-sitter adoption." start="00:21:43.440" video="mainVideo" id=subtitle]] +[[!template text="Therefore, I've created a language bundle package, tree-sitter-langs," start="00:21:48.159" video="mainVideo" id=subtitle]] +[[!template text="that takes care of pre-compiling the grammars," start="00:21:52.960" video="mainVideo" id=subtitle]] +[[!template text="the most common grammars for all three major platforms." start="00:21:55.773" video="mainVideo" id=subtitle]] +[[!template text="It also takes care of distributing these binaries" start="00:22:01.600" video="mainVideo" id=subtitle]] +[[!template text="and provides some highlighting queries" start="00:22:05.360" video="mainVideo" id=subtitle]] +[[!template text="for some of the languages." start="00:22:08.080" video="mainVideo" id=subtitle]] +[[!template text="It should be noted that this package" start="00:22:11.440" video="mainVideo" id=subtitle]] +[[!template text="should be treated as a temporary distribution mechanism only," start="00:22:13.760" video="mainVideo" id=subtitle]] +[[!template text="to help with bootstrapping tree-sitter adoption." start="00:22:19.919" video="mainVideo" id=subtitle]] +[[!template text="The plan is that eventually these files" start="00:22:24.720" video="mainVideo" id=subtitle]] +[[!template text="should be provided by" start="00:22:27.760" video="mainVideo" id=subtitle]] +[[!template text="the language major modes themselves." start="00:22:29.156" video="mainVideo" id=subtitle]] +[[!template text="But in order to do that, we need better tooling," start="00:22:32.480" video="mainVideo" id=subtitle]] +[[!template text="so we're not there yet." start="00:22:36.320" video="mainVideo" id=subtitle]] +[[!template new="1" text="Since the core already works reasonably well," start="00:22:40.240" video="mainVideo" id=subtitle]] +[[!template text="there are several areas that would benefit" start="00:22:43.280" video="mainVideo" id=subtitle]] +[[!template text="from the community's contribution." start="00:22:45.289" video="mainVideo" id=subtitle]] +[[!template text="So tree-sitter's upstream language repositories" start="00:22:49.120" video="mainVideo" id=subtitle]] +[[!template text="already contain highlighting queries on their own." start="00:22:52.640" video="mainVideo" id=subtitle]] +[[!template text="However, they are pretty basic," start="00:22:55.679" video="mainVideo" id=subtitle]] +[[!template text="and they may not fit well with existing Emacs conventions." start="00:22:57.573" video="mainVideo" id=subtitle]] +[[!template text="Therefore, the language bundle has its own set of highlighting queries." start="00:23:02.559" video="mainVideo" id=subtitle]] +[[!template text="This requires maintenance until language major modes adopt tree-sitter" start="00:23:07.120" video="mainVideo" id=subtitle]] +[[!template text="and maintain the queries on their own." start="00:23:12.556" video="mainVideo" id=subtitle]] +[[!template text="The queries are actually quite easy to write," start="00:23:16.640" video="mainVideo" id=subtitle]] +[[!template text="as you've already seen." start="00:23:19.056" video="mainVideo" id=subtitle]] +[[!template text="You just need to be familiar with the language," start="00:23:22.000" video="mainVideo" id=subtitle]] +[[!template text="familiar enough to come up with sensible highlighting patterns." start="00:23:25.360" video="mainVideo" id=subtitle]] +[[!template text="And if you are a maintainer of a language major mode," start="00:23:35.200" video="mainVideo" id=subtitle]] +[[!template text="you may want to consider integrating tree-sitter into your mode," start="00:23:39.679" video="mainVideo" id=subtitle]] +[[!template text="initially maybe as an optional feature." start="00:23:44.189" video="mainVideo" id=subtitle]] +[[!template text="The integration is actually pretty straightforward," start="00:23:48.573" video="mainVideo" id=subtitle]] +[[!template text="especially for syntax highlighting." start="00:23:53.279" video="mainVideo" id=subtitle]] +[[!template text="Or alternatively," start="00:23:56.640" video="mainVideo" id=subtitle]] +[[!template text="you can also try writing a new major mode from scratch" start="00:24:01.520" video="mainVideo" id=subtitle]] +[[!template text="that relies on tree-sitter" start="00:24:05.760" video="mainVideo" id=subtitle]] +[[!template text="from the very beginning." start="00:24:08.000" video="mainVideo" id=subtitle]] +[[!template text="The code for such a major mode is quite simple." start="00:24:12.559" video="mainVideo" id=subtitle]] +[[!template text="For example, this is the proposed" start="00:24:17.523" video="mainVideo" id=subtitle]] +[[!template text="wat-mode for web assembly." start="00:24:23.200" video="mainVideo" id=subtitle]] +[[!template text="The code is just one page of code, not a lot." start="00:24:26.240" video="mainVideo" id=subtitle]] +[[!template text="You can also try writing new minor modes" start="00:24:39.520" video="mainVideo" id=subtitle]] +[[!template text="or writing integration packages." start="00:24:42.720" video="mainVideo" id=subtitle]] +[[!template text="For example, a lot of packages" start="00:24:46.559" video="mainVideo" id=subtitle]] +[[!template text="may benefit from tree-sitter integration," start="00:24:50.880" video="mainVideo" id=subtitle]] +[[!template text="but no one has written the integration yet." start="00:24:54.559" video="mainVideo" id=subtitle]] +[[!template new="1" text="If you are interested in tree-sitter," start="00:25:02.960" video="mainVideo" id=subtitle]] +[[!template text="you can use these links to learn more about it." start="00:25:04.836" video="mainVideo" id=subtitle]] +[[!template text="I think that's it for me today." start="00:25:08.023" video="mainVideo" id=subtitle]] +[[!template text="I'm happy to answer any questions." start="00:25:11.440" video="mainVideo" id=subtitle]] |