1 files changed, 553 insertions, 0 deletions

diff --git a/2020/subtitles/emacsconf-2020--25-traverse-complex-json-structures-with-live-feedback-counsel-jq--zen-monk-alain-m-lafon.vtt b/2020/subtitles/emacsconf-2020--25-traverse-complex-json-structures-with-live-feedback-counsel-jq--zen-monk-alain-m-lafon.vtt
new file mode 100644
index 00000000..c453575d
--- /dev/null
+++ b/2020/subtitles/emacsconf-2020--25-traverse-complex-json-structures-with-live-feedback-counsel-jq--zen-monk-alain-m-lafon.vtt
@@ -0,0 +1,553 @@
+WEBVTT
+
+00:00:00.799 --> 00:00:05.520
+Hello, everyone, and welcome to this
+short lightning talk:
+
+00:00:05.520 --> 00:00:09.519
+"Traverse Complex JSON Structures with
+Live Feedback."
+
+00:00:09.519 --> 00:00:18.000
+This is a pre-recorded talk and part of
+the EmacsConf 2020 schedule.
+
+00:00:18.000 --> 00:00:19.439
+This is what we're going to do.
+
+00:00:19.439 --> 00:00:22.320
+I'll make a quick introduction to the
+topic at hand.
+
+00:00:22.320 --> 00:00:24.400
+I'll give you a demonstration of some
+tools,
+
+00:00:24.400 --> 00:00:29.199
+and then we'll leave you
+with the links to said tools.
+
+00:00:29.199 --> 00:00:31.679
+Before that, just a little bit about me.
+
+00:00:31.679 --> 00:00:40.399
+I am the CEO and co-founder of a company
+based in the Swiss mountains called 200ok.ch.
+
+00:00:40.399 --> 00:00:44.879
+We are a product incubator and
+service consultancy,
+
+00:00:44.879 --> 00:00:50.000
+but we like to spend most or at least as
+much time as we can
+
+00:00:50.000 --> 00:00:52.719
+building free software.
+
+00:00:52.719 --> 00:00:56.879
+I'm also an ordained Zen monk and abbot
+of the Lambda Zen temple.
+
+00:00:56.879 --> 00:01:04.159
+You can reach me anytime on questions
+regarding Emacs, for example,
+
+00:01:04.159 --> 00:01:07.200
+at alain@200ok.ch.
+
+00:01:07.200 --> 00:01:09.439
+But back to the topic at hand.
+
+00:01:09.439 --> 00:01:11.760
+The proposition is as following:
+
+00:01:11.760 --> 00:01:14.000
+most work on the computer is based on
+either
+
+00:01:14.000 --> 00:01:16.479
+text processing or text consumption.
+
+00:01:16.479 --> 00:01:22.799
+And very often, the text which you need
+to process is in a structured format,
+
+00:01:22.799 --> 00:01:24.560
+for example, in JSON.
+
+00:01:24.560 --> 00:01:28.560
+That might even be if your job is not
+programming per se.
+
+00:01:28.560 --> 00:01:33.119
+Reading through such a bigger chunk of
+JSON can be non-trivial, however,
+
+00:01:33.119 --> 00:01:36.479
+while just reading and understanding it
+
+00:01:36.479 --> 00:01:40.320
+will be essential to getting your job
+done.
+
+00:01:40.320 --> 00:01:44.479
+So let's quickly check out an example
+JSON file.
+
+00:01:44.479 --> 00:01:47.200
+This is from the Github API,
+
+00:01:47.200 --> 00:01:52.079
+which is a request--sorry, the
+response to a request
+
+00:01:52.079 --> 00:01:54.640
+for a specific issue on the github API.
+
+00:01:54.640 --> 00:01:58.799
+So let's quickly check that one out.
+
+00:01:58.799 --> 00:02:01.920
+Okay. So here it is open, and we can
+already see
+
+00:02:01.920 --> 00:02:05.439
+that there is lots of stuff
+going on here.
+
+00:02:05.439 --> 00:02:07.360
+It's 200 lines.
+
+00:02:07.360 --> 00:02:09.200
+It's not going to be very easy
+
+00:02:09.200 --> 00:02:11.840
+just to find out what are the top level
+things in here,
+
+00:02:11.840 --> 00:02:13.360
+what are the top level attributes.
+
+00:02:13.360 --> 00:02:17.840
+Of course I can do this, and maybe do it
+by hand, but that doesn't scale.
+
+00:02:17.840 --> 00:02:21.599
+I can use cool Emacs facilities like the
+hideshow-mode
+
+00:02:21.599 --> 00:02:24.720
+and try to fold all the things that are
+top level,
+
+00:02:24.720 --> 00:02:27.200
+but that also doesn't really scale.
+
+00:02:27.200 --> 00:02:29.360
+There must be a better way.
+
+00:02:29.360 --> 00:02:32.000
+Of course there is. There is prior art.
+
+00:02:32.000 --> 00:02:34.080
+There is a tool called jq.
+
+00:02:34.080 --> 00:02:37.760
+I'm going to quote the USP (unique selling proposition) from their website:
+
+00:02:37.760 --> 00:02:42.000
+jq is like sed for JSON data.
+
+00:02:42.000 --> 00:02:46.319
+you can use it to slice and filter and
+map and transform structured data
+
+00:02:46.319 --> 00:02:47.840
+with the same ease that
+
+00:02:47.840 --> 00:02:54.000
+sed, awk, grep, and friends let you
+play with text.
+
+00:02:54.000 --> 00:02:56.879
+Let me give you a quick demonstration of
+it.
+
+00:02:56.879 --> 00:02:59.040
+By the way, it's written in portable C.
+
+00:02:59.040 --> 00:03:03.519
+It has zero runtime dependency, so it's
+very easy to get started with it
+
+00:03:03.519 --> 00:03:09.840
+and use it on pretty much any UNIX-based
+computer.
+
+00:03:09.840 --> 00:03:14.000
+Sorry, no, Linux-based computer,
+apologies.
+
+00:03:14.000 --> 00:03:18.720
+Okay, so let's explore a
+JSON file with it.
+
+00:03:18.720 --> 00:03:20.000
+It's a command line tool,
+
+00:03:20.000 --> 00:03:24.000
+and it has a very simple command
+line syntax.
+
+00:03:24.000 --> 00:03:29.840
+So you call the binary and then you give
+it a query and a file,
+
+00:03:29.840 --> 00:03:32.560
+and then it will return its answer.
+
+00:03:32.560 --> 00:03:35.440
+So, for example, if I want the top
+level keys,
+
+00:03:35.440 --> 00:03:38.000
+I will just say jq keys the file
+
+00:03:38.000 --> 00:03:39.840
+and it will return the keys.
+
+00:03:39.840 --> 00:03:44.400
+Simple as that. So let's check this out
+in a real shell.
+
+00:03:44.400 --> 00:03:46.879
+Here I am in eshell.
+
+00:03:46.879 --> 00:03:51.440
+Let's run jq keys on the Github issue comment.
+
+00:03:51.440 --> 00:03:58.799
+We can see that we have actually
+received a list back here
+
+00:03:58.799 --> 00:04:00.319
+with the top-level things.
+
+00:04:00.319 --> 00:04:02.879
+So this issue... It looks very interesting.
+
+00:04:02.879 --> 00:04:07.360
+Let's ask it to give me more information on this issue.
+
+00:04:07.360 --> 00:04:11.360
+Then it's hairy again. That's a lot of stuff.
+
+00:04:11.360 --> 00:04:14.560
+I mean, lucky for us, we are in Emacs here,
+
+00:04:14.560 --> 00:04:16.720
+so we can use nice shortcuts.
+
+00:04:16.720 --> 00:04:22.000
+We can copy this. We can go in here, just select that,
+
+00:04:22.000 --> 00:04:24.160
+get that out or something like this.
+
+00:04:24.160 --> 00:04:32.320
+But still, this is not really the best way to do that, right?
+
+00:04:32.320 --> 00:04:34.080
+it gets kind of tedious.
+
+00:04:34.080 --> 00:04:37.680
+At this point the output can be humongous.
+
+00:04:37.680 --> 00:04:41.919
+The shell is not really the best place to read through such big output.
+
+00:04:41.919 --> 00:04:45.759
+I mean, eshell is probably one of the better shells for this,
+
+00:04:45.759 --> 00:04:47.919
+because it's just a regular Emacs buffer,
+
+00:04:47.919 --> 00:04:50.720
+but still, it's not really the best tool.
+
+00:04:50.720 --> 00:04:53.680
+I need to repeat the command all the time
+
+00:04:53.680 --> 00:04:56.000
+until I finally build the right query.
+
+00:04:56.000 --> 00:04:59.840
+And all the time, I lose my focus,
+
+00:04:59.840 --> 00:05:02.800
+I lose what I'm currently looking at.
+
+00:05:02.800 --> 00:05:05.520
+I'm seeing the new result.
+
+00:05:05.520 --> 00:05:08.160
+It would be so much nicer to have live feedback.
+
+00:05:08.160 --> 00:05:10.720
+When working with Emacs, we're quite used to that.
+
+00:05:10.720 --> 00:05:12.320
+So there should be an option.
+
+00:05:12.320 --> 00:05:15.120
+And of course there is. It's Emacs, right,
+
+00:05:15.120 --> 00:05:17.759
+so you can do anything.
+
+00:05:17.759 --> 00:05:22.960
+There is various good tools for completion in Emacs.
+
+00:05:22.960 --> 00:05:26.000
+I used ivy for this.
+
+00:05:26.000 --> 00:05:29.039
+I'm going to quote the USP for ivy.
+
+00:05:29.039 --> 00:05:32.639
+ivy is a generic completion mechanism for Emacs.
+
+00:05:32.639 --> 00:05:37.919
+While it operates similarly to other completion schemes such as icomplete mode,
+
+00:05:37.919 --> 00:05:42.160
+ivy aims to be more efficient, smaller, simpler, and smoother to use,
+
+00:05:42.160 --> 00:05:45.199
+yet highly customizable.
+
+00:05:45.199 --> 00:05:46.479
+And that's true.
+
+00:05:46.479 --> 00:05:49.440
+One of the cool things of ivy
+
+00:05:49.440 --> 00:05:54.320
+compared to other completion mechanisms in Emacs
+
+00:05:54.320 --> 00:05:59.120
+is that it can be used on dynamic data.
+
+00:05:59.120 --> 00:06:02.400
+So usually completion works on a static input.
+
+00:06:02.400 --> 00:06:05.360
+For example, you're in a buffer, a text buffer,
+
+00:06:05.360 --> 00:06:09.600
+and you use isearch maybe with ido-mode,
+
+00:06:09.600 --> 00:06:13.360
+and you find your results. That's all nice.
+
+00:06:13.360 --> 00:06:19.600
+However, if I want to search on dynamic data,
+
+00:06:19.600 --> 00:06:20.720
+that doesn't work.
+
+00:06:20.720 --> 00:06:24.880
+So whenever I type in my query for jq,
+
+00:06:24.880 --> 00:06:28.000
+I actually need to call the jq binary,
+
+00:06:28.000 --> 00:06:30.720
+and it will give a different result set back.
+
+00:06:30.720 --> 00:06:36.160
+So it's a really dynamic mechanism that we need here.
+
+00:06:36.160 --> 00:06:38.240
+It's much more like a search engine.
+
+00:06:38.240 --> 00:06:41.440
+ivy luckily has something built in,
+
+00:06:41.440 --> 00:06:43.520
+and it's called counsel.
+
+00:06:43.520 --> 00:06:47.360
+So I used counsel and jq and combined them,
+
+00:06:47.360 --> 00:06:49.199
+and built a new package
+
+00:06:49.199 --> 00:06:52.960
+with which we can use Emacs and jq
+
+00:06:52.960 --> 00:06:56.000
+to have live feedback.
+
+00:06:56.000 --> 00:06:57.759
+It's very easy to use.
+
+00:06:57.759 --> 00:06:59.840
+So you just call counsel-jq
+
+00:06:59.840 --> 00:07:02.160
+on a buffer containing JSON.
+
+00:07:02.160 --> 00:07:04.319
+For example, the one we have here.
+
+00:07:04.319 --> 00:07:06.800
+Let's call counsel-jq on it,
+
+00:07:06.800 --> 00:07:10.080
+and we already get a default query,
+
+00:07:10.080 --> 00:07:14.639
+the dot query, which just gives us the same file.
+
+00:07:14.639 --> 00:07:16.240
+But now we can change it.
+
+00:07:16.240 --> 00:07:18.639
+For example, find all the keys in here.
+
+00:07:18.639 --> 00:07:20.319
+And then we see I had this issue.
+
+00:07:20.319 --> 00:07:22.800
+This was the one that we were interested in.
+
+00:07:22.800 --> 00:07:25.599
+So let's find more information on the issue.
+
+00:07:25.599 --> 00:07:28.720
+What keys does it have actually have?
+
+00:07:28.720 --> 00:07:31.680
+It has assignees. That interests me.
+
+00:07:31.680 --> 00:07:34.800
+So let's check out the assignees in here.
+
+00:07:34.800 --> 00:07:39.759
+There's two of them, but I'm only interested in the first one.
+
+00:07:39.759 --> 00:07:43.599
+I'm making stuff up as I go here, of course.
+
+00:07:43.599 --> 00:07:47.039
+Whenever I hit enter, I get a new buffer
+
+00:07:47.039 --> 00:07:52.639
+which just shows me this particular result
+
+00:07:52.639 --> 00:07:55.599
+for the particular query that I entered.
+
+00:07:55.599 --> 00:07:57.680
+So let me do that again.
+
+00:07:57.680 --> 00:08:04.000
+We are in here. We are looking at a JSON file.
+
+00:08:04.000 --> 00:08:05.840
+This can be very, very big.
+
+00:08:05.840 --> 00:08:07.280
+Doesn't also need to be a file.
+
+00:08:07.280 --> 00:08:09.520
+Just needs to be a buffer.
+
+00:08:09.520 --> 00:08:11.360
+You call counsel-jq on it,
+
+00:08:11.360 --> 00:08:14.319
+and you can do any kind of query on it.
+
+00:08:14.319 --> 00:08:18.080
+For example, let's see if there is a URL here.
+
+00:08:18.080 --> 00:08:19.440
+Yes, there's a URL.
+
+00:08:19.440 --> 00:08:22.827
+Let's see if there's a repository here.
+
+00:08:22.827 --> 00:08:24.639
+Repository. No, there isn't.
+
+00:08:24.639 --> 00:08:33.440
+What was it called? Issue. Keys. Repository URL, it was called.
+
+00:08:33.440 --> 00:08:38.240
+So let's see issue repository URL,
+
+00:08:38.240 --> 00:08:39.519
+and then we see.
+
+00:08:39.519 --> 00:08:44.800
+So apparently this issue comment is for a repository called organice.
+
+00:08:44.800 --> 00:08:47.839
+I wonder what that might be.
+
+00:08:47.839 --> 00:08:52.640
+Okay. So that was a very short introduction to counsel-jq.
+
+00:08:52.640 --> 00:08:54.240
+You can see the timer here.
+
+00:08:54.240 --> 00:08:57.440
+I only have one minute left to go, so I'm going to leave
+
+00:08:57.440 --> 00:09:02.880
+with a very, very short introduction to the counsel-jq code.
+
+00:09:02.880 --> 00:09:06.000
+It's not even 60 lines of elisp,
+
+00:09:06.000 --> 00:09:09.600
+so building something like this is very, very easy.
+
+00:09:09.600 --> 00:09:14.560
+I would encourage you to go and read through the code in your own time,
+
+00:09:14.560 --> 00:09:17.519
+if you're interested in building something like this.
+
+00:09:17.519 --> 00:09:22.720
+If you're interested in just using jq or you're done,
+
+00:09:22.720 --> 00:09:24.320
+these are the links to all the tools.
+
+00:09:24.320 --> 00:09:28.240
+counsel-jq, of course, is readily available on MELPA.
+
+00:09:28.240 --> 00:09:32.959
+Also developed under the AGPL license on Github.
+
+00:09:32.959 --> 00:09:36.080
+And this organice thing, by the way, it's
+
+00:09:36.080 --> 00:09:38.560
+Org Mode for mobile and desktop browsers.
+
+00:09:38.560 --> 00:09:43.120
+Also a great free software tool maybe that interests you.
+
+00:09:43.120 --> 00:09:46.240
+Thank you for listening. Have a great time.
+
+00:09:46.240 --> 00:09:49.360
+10 seconds left. I am going to stop this now.
+
+00:09:49.360 --> 00:09:53.920
+Enjoy EmacsConf. Have a great day.