From b98df6fbe2a5c48013cfca81a95a5af41e202d07 Mon Sep 17 00:00:00 2001 From: Sacha Chua Date: Sun, 13 Dec 2020 00:06:32 -0500 Subject: Actually post subtitles, I think --- ...-counsel-jq--zen-monk-alain-m-lafon-autogen.vtt | 757 +++++++++++++++++++++ 1 file changed, 757 insertions(+) create mode 100644 2020/subtitles/emacsconf-2020--25-traverse-complex-json-structures-with-live-feedback-counsel-jq--zen-monk-alain-m-lafon-autogen.vtt (limited to '2020/subtitles/emacsconf-2020--25-traverse-complex-json-structures-with-live-feedback-counsel-jq--zen-monk-alain-m-lafon-autogen.vtt') diff --git a/2020/subtitles/emacsconf-2020--25-traverse-complex-json-structures-with-live-feedback-counsel-jq--zen-monk-alain-m-lafon-autogen.vtt b/2020/subtitles/emacsconf-2020--25-traverse-complex-json-structures-with-live-feedback-counsel-jq--zen-monk-alain-m-lafon-autogen.vtt new file mode 100644 index 00000000..e7eada5b --- /dev/null +++ b/2020/subtitles/emacsconf-2020--25-traverse-complex-json-structures-with-live-feedback-counsel-jq--zen-monk-alain-m-lafon-autogen.vtt @@ -0,0 +1,757 @@ +WEBVTT + +00:00:00.799 --> 00:00:04.000 +hello everyone and welcome to this short + +00:00:04.000 --> 00:00:05.520 +lightning talk + +00:00:05.520 --> 00:00:08.160 +traverse complex json structures with + +00:00:08.160 --> 00:00:09.519 +live feedback + +00:00:09.519 --> 00:00:13.040 +this is a pre-recorded talk and part of + +00:00:13.040 --> 00:00:18.000 +the e-max conf 2020 schedule + +00:00:18.000 --> 00:00:19.920 +this is what we're going to do I'll make + +00:00:19.920 --> 00:00:21.840 +a quick introduction to the topic at + +00:00:21.840 --> 00:00:22.320 +hand + +00:00:22.320 --> 00:00:23.920 +I'll give you a demonstration of some + +00:00:23.920 --> 00:00:25.760 +tools and then we'll leave you + +00:00:25.760 --> 00:00:29.199 +with the links to set tools + +00:00:29.199 --> 00:00:31.920 +before that just a little bit about me I + +00:00:31.920 --> 00:00:32.399 +am the + +00:00:32.399 --> 00:00:36.079 +ceo and co-founder of a company + +00:00:36.079 --> 00:00:39.520 +based in the swiss mountains called 200 + +00:00:39.520 --> 00:00:40.399 +okay + +00:00:40.399 --> 00:00:43.600 +we are a product incubator and + +00:00:43.600 --> 00:00:46.160 +service consultancy but we like to spend + +00:00:46.160 --> 00:00:47.200 +most or at least + +00:00:47.200 --> 00:00:50.719 +as much time as we can building free + +00:00:50.719 --> 00:00:52.719 +software + +00:00:52.719 --> 00:00:55.520 +I'm also an ordained zen monk and abbott + +00:00:55.520 --> 00:00:56.879 +of the lambda zen temple + +00:00:56.879 --> 00:01:00.160 +you can reach me anytime on questions + +00:01:00.160 --> 00:01:01.359 +regarding Emacs + +00:01:01.359 --> 00:01:05.860 +for example at ala at 200ok.ch + +00:01:05.860 --> 00:01:07.200 +[Music] + +00:01:07.200 --> 00:01:09.600 +but back to the topic at hand the + +00:01:09.600 --> 00:01:11.760 +proposition is as following + +00:01:11.760 --> 00:01:13.680 +most work on the computer is based on + +00:01:13.680 --> 00:01:15.520 +either text processing or text + +00:01:15.520 --> 00:01:16.479 +consumption + +00:01:16.479 --> 00:01:19.920 +and very often the text which you need + +00:01:19.920 --> 00:01:20.880 +to process + +00:01:20.880 --> 00:01:23.520 +is in a structured format for example in + +00:01:23.520 --> 00:01:24.560 +json + +00:01:24.560 --> 00:01:26.640 +that might even be if your job is not + +00:01:26.640 --> 00:01:28.560 +programming per se + +00:01:28.560 --> 00:01:30.400 +and reading through such a bigger chunk + +00:01:30.400 --> 00:01:33.119 +of json can be non-trivial however + +00:01:33.119 --> 00:01:36.479 +while just reading and understanding it + +00:01:36.479 --> 00:01:39.119 +will be essential to getting your job + +00:01:39.119 --> 00:01:40.320 +done + +00:01:40.320 --> 00:01:43.680 +so let's quickly check out an example + +00:01:43.680 --> 00:01:47.200 +json file this is from the github api + +00:01:47.200 --> 00:01:50.560 +where which is a request sorry the + +00:01:50.560 --> 00:01:53.119 +response to a request for a specific + +00:01:53.119 --> 00:01:55.280 +issue on the github api so let's + +00:01:55.280 --> 00:01:58.799 +quickly check that one out + +00:01:58.799 --> 00:02:01.280 +okay so here it is open and we can + +00:02:01.280 --> 00:02:03.280 +already see that there is lots of stuff + +00:02:03.280 --> 00:02:05.439 +going on here + +00:02:05.439 --> 00:02:08.319 +it's uh 200 lines it's not going to be + +00:02:08.319 --> 00:02:10.319 +very easy just just to find out what are + +00:02:10.319 --> 00:02:12.160 +the top level things in here what are + +00:02:12.160 --> 00:02:13.920 +the top level attributes of course I can + +00:02:13.920 --> 00:02:14.560 +do this + +00:02:14.560 --> 00:02:16.480 +and maybe it would by hand but that + +00:02:16.480 --> 00:02:17.840 +doesn't scale + +00:02:17.840 --> 00:02:20.560 +I can use cool Emacs facilities like the + +00:02:20.560 --> 00:02:22.560 +height show mode and try to + +00:02:22.560 --> 00:02:24.720 +fold all the things that are top level + +00:02:24.720 --> 00:02:27.200 +but that also doesn't really scale + +00:02:27.200 --> 00:02:29.760 +there must be a better way and of course + +00:02:29.760 --> 00:02:32.000 +there is there is prior art + +00:02:32.000 --> 00:02:34.959 +there is a tool called jq I'm going to + +00:02:34.959 --> 00:02:36.080 +quote the usp + +00:02:36.080 --> 00:02:39.120 +from their website jq is like + +00:02:39.120 --> 00:02:42.959 +sed said for json data you can use it to + +00:02:42.959 --> 00:02:44.720 +slice and filter and map + +00:02:44.720 --> 00:02:46.720 +and transform structured data with the + +00:02:46.720 --> 00:02:47.840 +same ease that + +00:02:47.840 --> 00:02:51.519 +z awk grep and friends let you play with + +00:02:51.519 --> 00:02:54.000 +text + +00:02:54.000 --> 00:02:56.160 +let me give you a quick demonstration of + +00:02:56.160 --> 00:02:57.519 +it by the way + +00:02:57.519 --> 00:02:59.680 +it's written in portable c it has zero + +00:02:59.680 --> 00:03:01.920 +runtime dependency so it's very easy + +00:03:01.920 --> 00:03:04.959 +to get started with it and use it on + +00:03:04.959 --> 00:03:09.040 +pretty much any unix-based + +00:03:09.040 --> 00:03:12.800 +computer sorry no linux based computer + +00:03:12.800 --> 00:03:16.159 +apologies okay so let's explore a + +00:03:16.159 --> 00:03:19.599 +json file with it it's a command line + +00:03:19.599 --> 00:03:21.120 +tool and it has a very + +00:03:21.120 --> 00:03:24.640 +simple command line syntax so you + +00:03:24.640 --> 00:03:26.799 +call the binary and then you give it a + +00:03:26.799 --> 00:03:27.840 +query + +00:03:27.840 --> 00:03:31.680 +and a file and then we'll return + +00:03:31.680 --> 00:03:33.840 +its answer so for example if I want the + +00:03:33.840 --> 00:03:35.440 +top level keys + +00:03:35.440 --> 00:03:38.319 +I will just say jq keys the file and it + +00:03:38.319 --> 00:03:39.840 +will return the keys + +00:03:39.840 --> 00:03:42.319 +simple as that so let's check this out + +00:03:42.319 --> 00:03:43.519 +in a real + +00:03:43.519 --> 00:03:46.879 +shell here I am in eshel + +00:03:46.879 --> 00:03:50.080 +let's run jq keys on the github + +00:03:50.080 --> 00:03:53.200 +issue comment and we can see + +00:03:53.200 --> 00:03:56.959 +that we have actually received a list + +00:03:56.959 --> 00:04:00.000 +back here with the top level + +00:04:00.000 --> 00:04:02.000 +things okay so this issue it looks very + +00:04:02.000 --> 00:04:04.319 +very interesting so let's uh ask it to + +00:04:04.319 --> 00:04:06.239 +give me more information on this + +00:04:06.239 --> 00:04:10.080 +issue then it's hairy again that's a lot + +00:04:10.080 --> 00:04:11.360 +of stuff + +00:04:11.360 --> 00:04:14.319 +I mean lucky for for us we are in Emacs + +00:04:14.319 --> 00:04:17.759 +here so we can use nice shortcuts we can + +00:04:17.759 --> 00:04:21.040 +copy this we go can go in here just + +00:04:21.040 --> 00:04:22.000 +select that + +00:04:22.000 --> 00:04:24.400 +get that out or something like this but + +00:04:24.400 --> 00:04:25.600 +still + +00:04:25.600 --> 00:04:28.320 +this is not really it's not really the + +00:04:28.320 --> 00:04:29.600 +best way + +00:04:29.600 --> 00:04:34.080 +to do that right it gets kind of tedious + +00:04:34.080 --> 00:04:36.639 +at this point the output can be + +00:04:36.639 --> 00:04:37.680 +humongous + +00:04:37.680 --> 00:04:39.360 +the shell is not really the best place + +00:04:39.360 --> 00:04:42.479 +to read through such big output I mean e + +00:04:42.479 --> 00:04:43.600 +shell is + +00:04:43.600 --> 00:04:45.440 +probably one of the better shells for + +00:04:45.440 --> 00:04:47.440 +this because it's just a regular Emacs + +00:04:47.440 --> 00:04:48.560 +buffer but still + +00:04:48.560 --> 00:04:51.199 +it's not really the best tool and I need + +00:04:51.199 --> 00:04:52.720 +to repeat the command + +00:04:52.720 --> 00:04:55.280 +all the time until I finally build the + +00:04:55.280 --> 00:04:56.000 +right query + +00:04:56.000 --> 00:04:59.120 +and all the time I use I lose my + +00:04:59.120 --> 00:05:02.080 +focus I lose what I'm currently looking + +00:05:02.080 --> 00:05:02.800 +at + +00:05:02.800 --> 00:05:06.080 +I'm seeing the new result it would be so + +00:05:06.080 --> 00:05:08.160 +much nicer to have live feedback + +00:05:08.160 --> 00:05:10.080 +and when working with Emacs we're quite + +00:05:10.080 --> 00:05:11.600 +used to that so there should be an + +00:05:11.600 --> 00:05:12.320 +option + +00:05:12.320 --> 00:05:15.120 +and of course there is it's Emacs right + +00:05:15.120 --> 00:05:17.759 +so you can do anything + +00:05:17.759 --> 00:05:20.880 +there is various good tools for + +00:05:20.880 --> 00:05:22.960 +completion in Emacs + +00:05:22.960 --> 00:05:26.000 +I used ivy for this + +00:05:26.000 --> 00:05:29.039 +I'm going to code the usp for ivy + +00:05:29.039 --> 00:05:31.840 +iv is a generic completion mechanism for + +00:05:31.840 --> 00:05:32.639 +Emacs + +00:05:32.639 --> 00:05:34.639 +while it operates similarly to other + +00:05:34.639 --> 00:05:35.919 +completion schemes + +00:05:35.919 --> 00:05:38.960 +such as icomplete mode iv aims to be + +00:05:38.960 --> 00:05:40.400 +more efficient smaller + +00:05:40.400 --> 00:05:43.120 +simpler and smoother to use yet highly + +00:05:43.120 --> 00:05:45.199 +customizable + +00:05:45.199 --> 00:05:47.840 +and that's true one of the cool things + +00:05:47.840 --> 00:05:49.440 +of iv + +00:05:49.440 --> 00:05:52.479 +compared to other completion mechanisms + +00:05:52.479 --> 00:05:54.320 +in Emacs + +00:05:54.320 --> 00:05:58.160 +is that it can be used on dynamic + +00:05:58.160 --> 00:06:01.600 +data so usually completion works on a + +00:06:01.600 --> 00:06:03.520 +static input for example you're in a + +00:06:03.520 --> 00:06:05.360 +buffer a text buffer + +00:06:05.360 --> 00:06:08.800 +and you use I search maybe with idle + +00:06:08.800 --> 00:06:09.600 +mode + +00:06:09.600 --> 00:06:12.400 +and you find your results that's all + +00:06:12.400 --> 00:06:13.360 +nice + +00:06:13.360 --> 00:06:16.960 +however if I + +00:06:16.960 --> 00:06:19.840 +want to search on dynamic data that + +00:06:19.840 --> 00:06:20.720 +doesn't work + +00:06:20.720 --> 00:06:23.759 +so whenever I type in my query + +00:06:23.759 --> 00:06:27.039 +for jq I actually need to call the jq + +00:06:27.039 --> 00:06:28.000 +binary + +00:06:28.000 --> 00:06:30.160 +and it will give a different result set + +00:06:30.160 --> 00:06:32.840 +back so it's a really + +00:06:32.840 --> 00:06:36.319 +dynamic mechanism that we need here it's + +00:06:36.319 --> 00:06:38.240 +much more like a search engine + +00:06:38.240 --> 00:06:41.440 +and ivy luckily has something built in + +00:06:41.440 --> 00:06:42.560 +and it's called + +00:06:42.560 --> 00:06:46.000 +console so I used council and jq + +00:06:46.000 --> 00:06:48.160 +and combined them and build a new + +00:06:48.160 --> 00:06:49.199 +package + +00:06:49.199 --> 00:06:52.960 +with which we can use Emacs and jq + +00:06:52.960 --> 00:06:56.000 +to have live feedback + +00:06:56.000 --> 00:06:58.800 +it's very easy to use so you just call + +00:06:58.800 --> 00:06:59.840 +console.jq + +00:06:59.840 --> 00:07:02.800 +on a buffer containing json for example + +00:07:02.800 --> 00:07:04.319 +the one we have here + +00:07:04.319 --> 00:07:07.280 +let's call console.jq on it and we + +00:07:07.280 --> 00:07:08.319 +already + +00:07:08.319 --> 00:07:11.280 +get a default query the dot query which + +00:07:11.280 --> 00:07:13.039 +just gives us + +00:07:13.039 --> 00:07:16.080 +the same file but now we can change it + +00:07:16.080 --> 00:07:17.280 +and for example + +00:07:17.280 --> 00:07:19.039 +find all the keys in here and then we + +00:07:19.039 --> 00:07:21.039 +see I had this issue this was the one + +00:07:21.039 --> 00:07:22.800 +that we were interested in + +00:07:22.800 --> 00:07:24.639 +so let's find more information on the + +00:07:24.639 --> 00:07:26.479 +issue what + +00:07:26.479 --> 00:07:28.800 +keys does it have actually have uh it + +00:07:28.800 --> 00:07:31.680 +has assignees that that interests me + +00:07:31.680 --> 00:07:34.800 +so let's check out the assignees in here + +00:07:34.800 --> 00:07:37.039 +there's two of them but I'm only + +00:07:37.039 --> 00:07:39.759 +interested in the first one + +00:07:39.759 --> 00:07:42.000 +I'm making stuff up as I go here of + +00:07:42.000 --> 00:07:43.599 +course + +00:07:43.599 --> 00:07:46.000 +and whenever I hit enter I get a new + +00:07:46.000 --> 00:07:47.039 +buffer + +00:07:47.039 --> 00:07:50.160 +which just shows me this + +00:07:50.160 --> 00:07:53.520 +particular result for the particular + +00:07:53.520 --> 00:07:55.599 +query that I entered + +00:07:55.599 --> 00:07:59.199 +so let me do that again we are in here + +00:07:59.199 --> 00:08:03.199 +we are looking at a + +00:08:03.199 --> 00:08:05.840 +json file this can be very very big + +00:08:05.840 --> 00:08:07.520 +doesn't also need to be a file just + +00:08:07.520 --> 00:08:09.520 +needs to be a buffer + +00:08:09.520 --> 00:08:12.160 +you call console.jq on it and you can do + +00:08:12.160 --> 00:08:13.599 +any kind of + +00:08:13.599 --> 00:08:16.479 +query on it for example let's see if + +00:08:16.479 --> 00:08:18.080 +there is a url here + +00:08:18.080 --> 00:08:20.319 +yes there's a url let's see if there's a + +00:08:20.319 --> 00:08:23.759 +repository here repository + +00:08:23.759 --> 00:08:26.879 +no there isn't what was it called issue + +00:08:26.879 --> 00:08:33.440 +keys repository url it was called okay + +00:08:33.440 --> 00:08:36.640 +so let's see issue pause + +00:08:36.640 --> 00:08:40.240 +url and then we see so + +00:08:40.240 --> 00:08:42.959 +apparently this issue command is for a + +00:08:42.959 --> 00:08:44.159 +repository called + +00:08:44.159 --> 00:08:47.839 +organize I wonder what that might be + +00:08:47.839 --> 00:08:50.320 +okay so that was a very short + +00:08:50.320 --> 00:08:52.640 +introduction to console jq + +00:08:52.640 --> 00:08:54.800 +you can see the timer here I only have + +00:08:54.800 --> 00:08:56.800 +one minute left to go so I'm going to + +00:08:56.800 --> 00:08:57.440 +leave + +00:08:57.440 --> 00:08:59.760 +with a very very short introduction to + +00:08:59.760 --> 00:09:01.519 +the console gq + +00:09:01.519 --> 00:09:05.040 +code it's not even 60 lines + +00:09:05.040 --> 00:09:07.519 +of e-lisp so building something like + +00:09:07.519 --> 00:09:09.600 +this is very very easy + +00:09:09.600 --> 00:09:12.800 +I would encourage you to go and read + +00:09:12.800 --> 00:09:14.560 +through the code in your own time + +00:09:14.560 --> 00:09:16.160 +if you're interested in building + +00:09:16.160 --> 00:09:18.320 +something like this if you're interested + +00:09:18.320 --> 00:09:19.600 +in just using + +00:09:19.600 --> 00:09:22.720 +jq or you're done + +00:09:22.720 --> 00:09:24.320 +these are the links to all the tools + +00:09:24.320 --> 00:09:25.839 +console.jq of course + +00:09:25.839 --> 00:09:28.800 +is readily available on melpah also + +00:09:28.800 --> 00:09:29.519 +developed + +00:09:29.519 --> 00:09:32.959 +under the agpl license on github + +00:09:32.959 --> 00:09:36.080 +and disorganized thing by the way it's + +00:09:36.080 --> 00:09:37.839 +orig mode for mobile and desktop + +00:09:37.839 --> 00:09:39.839 +browsers also a great + +00:09:39.839 --> 00:09:42.399 +free software tool maybe that interests + +00:09:42.399 --> 00:09:43.120 +you + +00:09:43.120 --> 00:09:46.240 +thank you for listening have great time + +00:09:46.240 --> 00:09:48.800 +10 seconds left I am going to stop this + +00:09:48.800 --> 00:09:49.360 +now + +00:09:49.360 --> 00:09:53.920 +enjoy EmacsConf have a great day -- cgit v1.2.3