From 8baff7beaf0e052f8369cb3dcf477b454744a567 Mon Sep 17 00:00:00 2001 From: Sacha Chua Date: Thu, 18 Mar 2021 23:52:45 -0400 Subject: Update transcript for #25 --- ...feedback-counsel-jq--zen-monk-alain-m-lafon.vtt | 553 +++++++++++++++++++++ 1 file changed, 553 insertions(+) create mode 100644 2020/subtitles/emacsconf-2020--25-traverse-complex-json-structures-with-live-feedback-counsel-jq--zen-monk-alain-m-lafon.vtt (limited to '2020/subtitles/emacsconf-2020--25-traverse-complex-json-structures-with-live-feedback-counsel-jq--zen-monk-alain-m-lafon.vtt') diff --git a/2020/subtitles/emacsconf-2020--25-traverse-complex-json-structures-with-live-feedback-counsel-jq--zen-monk-alain-m-lafon.vtt b/2020/subtitles/emacsconf-2020--25-traverse-complex-json-structures-with-live-feedback-counsel-jq--zen-monk-alain-m-lafon.vtt new file mode 100644 index 00000000..c453575d --- /dev/null +++ b/2020/subtitles/emacsconf-2020--25-traverse-complex-json-structures-with-live-feedback-counsel-jq--zen-monk-alain-m-lafon.vtt @@ -0,0 +1,553 @@ +WEBVTT + +00:00:00.799 --> 00:00:05.520 +Hello, everyone, and welcome to this +short lightning talk: + +00:00:05.520 --> 00:00:09.519 +"Traverse Complex JSON Structures with +Live Feedback." + +00:00:09.519 --> 00:00:18.000 +This is a pre-recorded talk and part of +the EmacsConf 2020 schedule. + +00:00:18.000 --> 00:00:19.439 +This is what we're going to do. + +00:00:19.439 --> 00:00:22.320 +I'll make a quick introduction to the +topic at hand. + +00:00:22.320 --> 00:00:24.400 +I'll give you a demonstration of some +tools, + +00:00:24.400 --> 00:00:29.199 +and then we'll leave you +with the links to said tools. + +00:00:29.199 --> 00:00:31.679 +Before that, just a little bit about me. + +00:00:31.679 --> 00:00:40.399 +I am the CEO and co-founder of a company +based in the Swiss mountains called 200ok.ch. + +00:00:40.399 --> 00:00:44.879 +We are a product incubator and +service consultancy, + +00:00:44.879 --> 00:00:50.000 +but we like to spend most or at least as +much time as we can + +00:00:50.000 --> 00:00:52.719 +building free software. + +00:00:52.719 --> 00:00:56.879 +I'm also an ordained Zen monk and abbot +of the Lambda Zen temple. + +00:00:56.879 --> 00:01:04.159 +You can reach me anytime on questions +regarding Emacs, for example, + +00:01:04.159 --> 00:01:07.200 +at alain@200ok.ch. + +00:01:07.200 --> 00:01:09.439 +But back to the topic at hand. + +00:01:09.439 --> 00:01:11.760 +The proposition is as following: + +00:01:11.760 --> 00:01:14.000 +most work on the computer is based on +either + +00:01:14.000 --> 00:01:16.479 +text processing or text consumption. + +00:01:16.479 --> 00:01:22.799 +And very often, the text which you need +to process is in a structured format, + +00:01:22.799 --> 00:01:24.560 +for example, in JSON. + +00:01:24.560 --> 00:01:28.560 +That might even be if your job is not +programming per se. + +00:01:28.560 --> 00:01:33.119 +Reading through such a bigger chunk of +JSON can be non-trivial, however, + +00:01:33.119 --> 00:01:36.479 +while just reading and understanding it + +00:01:36.479 --> 00:01:40.320 +will be essential to getting your job +done. + +00:01:40.320 --> 00:01:44.479 +So let's quickly check out an example +JSON file. + +00:01:44.479 --> 00:01:47.200 +This is from the Github API, + +00:01:47.200 --> 00:01:52.079 +which is a request--sorry, the +response to a request + +00:01:52.079 --> 00:01:54.640 +for a specific issue on the github API. + +00:01:54.640 --> 00:01:58.799 +So let's quickly check that one out. + +00:01:58.799 --> 00:02:01.920 +Okay. So here it is open, and we can +already see + +00:02:01.920 --> 00:02:05.439 +that there is lots of stuff +going on here. + +00:02:05.439 --> 00:02:07.360 +It's 200 lines. + +00:02:07.360 --> 00:02:09.200 +It's not going to be very easy + +00:02:09.200 --> 00:02:11.840 +just to find out what are the top level +things in here, + +00:02:11.840 --> 00:02:13.360 +what are the top level attributes. + +00:02:13.360 --> 00:02:17.840 +Of course I can do this, and maybe do it +by hand, but that doesn't scale. + +00:02:17.840 --> 00:02:21.599 +I can use cool Emacs facilities like the +hideshow-mode + +00:02:21.599 --> 00:02:24.720 +and try to fold all the things that are +top level, + +00:02:24.720 --> 00:02:27.200 +but that also doesn't really scale. + +00:02:27.200 --> 00:02:29.360 +There must be a better way. + +00:02:29.360 --> 00:02:32.000 +Of course there is. There is prior art. + +00:02:32.000 --> 00:02:34.080 +There is a tool called jq. + +00:02:34.080 --> 00:02:37.760 +I'm going to quote the USP (unique selling proposition) from their website: + +00:02:37.760 --> 00:02:42.000 +jq is like sed for JSON data. + +00:02:42.000 --> 00:02:46.319 +you can use it to slice and filter and +map and transform structured data + +00:02:46.319 --> 00:02:47.840 +with the same ease that + +00:02:47.840 --> 00:02:54.000 +sed, awk, grep, and friends let you +play with text. + +00:02:54.000 --> 00:02:56.879 +Let me give you a quick demonstration of +it. + +00:02:56.879 --> 00:02:59.040 +By the way, it's written in portable C. + +00:02:59.040 --> 00:03:03.519 +It has zero runtime dependency, so it's +very easy to get started with it + +00:03:03.519 --> 00:03:09.840 +and use it on pretty much any UNIX-based +computer. + +00:03:09.840 --> 00:03:14.000 +Sorry, no, Linux-based computer, +apologies. + +00:03:14.000 --> 00:03:18.720 +Okay, so let's explore a +JSON file with it. + +00:03:18.720 --> 00:03:20.000 +It's a command line tool, + +00:03:20.000 --> 00:03:24.000 +and it has a very simple command +line syntax. + +00:03:24.000 --> 00:03:29.840 +So you call the binary and then you give +it a query and a file, + +00:03:29.840 --> 00:03:32.560 +and then it will return its answer. + +00:03:32.560 --> 00:03:35.440 +So, for example, if I want the top +level keys, + +00:03:35.440 --> 00:03:38.000 +I will just say jq keys the file + +00:03:38.000 --> 00:03:39.840 +and it will return the keys. + +00:03:39.840 --> 00:03:44.400 +Simple as that. So let's check this out +in a real shell. + +00:03:44.400 --> 00:03:46.879 +Here I am in eshell. + +00:03:46.879 --> 00:03:51.440 +Let's run jq keys on the Github issue comment. + +00:03:51.440 --> 00:03:58.799 +We can see that we have actually +received a list back here + +00:03:58.799 --> 00:04:00.319 +with the top-level things. + +00:04:00.319 --> 00:04:02.879 +So this issue... It looks very interesting. + +00:04:02.879 --> 00:04:07.360 +Let's ask it to give me more information on this issue. + +00:04:07.360 --> 00:04:11.360 +Then it's hairy again. That's a lot of stuff. + +00:04:11.360 --> 00:04:14.560 +I mean, lucky for us, we are in Emacs here, + +00:04:14.560 --> 00:04:16.720 +so we can use nice shortcuts. + +00:04:16.720 --> 00:04:22.000 +We can copy this. We can go in here, just select that, + +00:04:22.000 --> 00:04:24.160 +get that out or something like this. + +00:04:24.160 --> 00:04:32.320 +But still, this is not really the best way to do that, right? + +00:04:32.320 --> 00:04:34.080 +it gets kind of tedious. + +00:04:34.080 --> 00:04:37.680 +At this point the output can be humongous. + +00:04:37.680 --> 00:04:41.919 +The shell is not really the best place to read through such big output. + +00:04:41.919 --> 00:04:45.759 +I mean, eshell is probably one of the better shells for this, + +00:04:45.759 --> 00:04:47.919 +because it's just a regular Emacs buffer, + +00:04:47.919 --> 00:04:50.720 +but still, it's not really the best tool. + +00:04:50.720 --> 00:04:53.680 +I need to repeat the command all the time + +00:04:53.680 --> 00:04:56.000 +until I finally build the right query. + +00:04:56.000 --> 00:04:59.840 +And all the time, I lose my focus, + +00:04:59.840 --> 00:05:02.800 +I lose what I'm currently looking at. + +00:05:02.800 --> 00:05:05.520 +I'm seeing the new result. + +00:05:05.520 --> 00:05:08.160 +It would be so much nicer to have live feedback. + +00:05:08.160 --> 00:05:10.720 +When working with Emacs, we're quite used to that. + +00:05:10.720 --> 00:05:12.320 +So there should be an option. + +00:05:12.320 --> 00:05:15.120 +And of course there is. It's Emacs, right, + +00:05:15.120 --> 00:05:17.759 +so you can do anything. + +00:05:17.759 --> 00:05:22.960 +There is various good tools for completion in Emacs. + +00:05:22.960 --> 00:05:26.000 +I used ivy for this. + +00:05:26.000 --> 00:05:29.039 +I'm going to quote the USP for ivy. + +00:05:29.039 --> 00:05:32.639 +ivy is a generic completion mechanism for Emacs. + +00:05:32.639 --> 00:05:37.919 +While it operates similarly to other completion schemes such as icomplete mode, + +00:05:37.919 --> 00:05:42.160 +ivy aims to be more efficient, smaller, simpler, and smoother to use, + +00:05:42.160 --> 00:05:45.199 +yet highly customizable. + +00:05:45.199 --> 00:05:46.479 +And that's true. + +00:05:46.479 --> 00:05:49.440 +One of the cool things of ivy + +00:05:49.440 --> 00:05:54.320 +compared to other completion mechanisms in Emacs + +00:05:54.320 --> 00:05:59.120 +is that it can be used on dynamic data. + +00:05:59.120 --> 00:06:02.400 +So usually completion works on a static input. + +00:06:02.400 --> 00:06:05.360 +For example, you're in a buffer, a text buffer, + +00:06:05.360 --> 00:06:09.600 +and you use isearch maybe with ido-mode, + +00:06:09.600 --> 00:06:13.360 +and you find your results. That's all nice. + +00:06:13.360 --> 00:06:19.600 +However, if I want to search on dynamic data, + +00:06:19.600 --> 00:06:20.720 +that doesn't work. + +00:06:20.720 --> 00:06:24.880 +So whenever I type in my query for jq, + +00:06:24.880 --> 00:06:28.000 +I actually need to call the jq binary, + +00:06:28.000 --> 00:06:30.720 +and it will give a different result set back. + +00:06:30.720 --> 00:06:36.160 +So it's a really dynamic mechanism that we need here. + +00:06:36.160 --> 00:06:38.240 +It's much more like a search engine. + +00:06:38.240 --> 00:06:41.440 +ivy luckily has something built in, + +00:06:41.440 --> 00:06:43.520 +and it's called counsel. + +00:06:43.520 --> 00:06:47.360 +So I used counsel and jq and combined them, + +00:06:47.360 --> 00:06:49.199 +and built a new package + +00:06:49.199 --> 00:06:52.960 +with which we can use Emacs and jq + +00:06:52.960 --> 00:06:56.000 +to have live feedback. + +00:06:56.000 --> 00:06:57.759 +It's very easy to use. + +00:06:57.759 --> 00:06:59.840 +So you just call counsel-jq + +00:06:59.840 --> 00:07:02.160 +on a buffer containing JSON. + +00:07:02.160 --> 00:07:04.319 +For example, the one we have here. + +00:07:04.319 --> 00:07:06.800 +Let's call counsel-jq on it, + +00:07:06.800 --> 00:07:10.080 +and we already get a default query, + +00:07:10.080 --> 00:07:14.639 +the dot query, which just gives us the same file. + +00:07:14.639 --> 00:07:16.240 +But now we can change it. + +00:07:16.240 --> 00:07:18.639 +For example, find all the keys in here. + +00:07:18.639 --> 00:07:20.319 +And then we see I had this issue. + +00:07:20.319 --> 00:07:22.800 +This was the one that we were interested in. + +00:07:22.800 --> 00:07:25.599 +So let's find more information on the issue. + +00:07:25.599 --> 00:07:28.720 +What keys does it have actually have? + +00:07:28.720 --> 00:07:31.680 +It has assignees. That interests me. + +00:07:31.680 --> 00:07:34.800 +So let's check out the assignees in here. + +00:07:34.800 --> 00:07:39.759 +There's two of them, but I'm only interested in the first one. + +00:07:39.759 --> 00:07:43.599 +I'm making stuff up as I go here, of course. + +00:07:43.599 --> 00:07:47.039 +Whenever I hit enter, I get a new buffer + +00:07:47.039 --> 00:07:52.639 +which just shows me this particular result + +00:07:52.639 --> 00:07:55.599 +for the particular query that I entered. + +00:07:55.599 --> 00:07:57.680 +So let me do that again. + +00:07:57.680 --> 00:08:04.000 +We are in here. We are looking at a JSON file. + +00:08:04.000 --> 00:08:05.840 +This can be very, very big. + +00:08:05.840 --> 00:08:07.280 +Doesn't also need to be a file. + +00:08:07.280 --> 00:08:09.520 +Just needs to be a buffer. + +00:08:09.520 --> 00:08:11.360 +You call counsel-jq on it, + +00:08:11.360 --> 00:08:14.319 +and you can do any kind of query on it. + +00:08:14.319 --> 00:08:18.080 +For example, let's see if there is a URL here. + +00:08:18.080 --> 00:08:19.440 +Yes, there's a URL. + +00:08:19.440 --> 00:08:22.827 +Let's see if there's a repository here. + +00:08:22.827 --> 00:08:24.639 +Repository. No, there isn't. + +00:08:24.639 --> 00:08:33.440 +What was it called? Issue. Keys. Repository URL, it was called. + +00:08:33.440 --> 00:08:38.240 +So let's see issue repository URL, + +00:08:38.240 --> 00:08:39.519 +and then we see. + +00:08:39.519 --> 00:08:44.800 +So apparently this issue comment is for a repository called organice. + +00:08:44.800 --> 00:08:47.839 +I wonder what that might be. + +00:08:47.839 --> 00:08:52.640 +Okay. So that was a very short introduction to counsel-jq. + +00:08:52.640 --> 00:08:54.240 +You can see the timer here. + +00:08:54.240 --> 00:08:57.440 +I only have one minute left to go, so I'm going to leave + +00:08:57.440 --> 00:09:02.880 +with a very, very short introduction to the counsel-jq code. + +00:09:02.880 --> 00:09:06.000 +It's not even 60 lines of elisp, + +00:09:06.000 --> 00:09:09.600 +so building something like this is very, very easy. + +00:09:09.600 --> 00:09:14.560 +I would encourage you to go and read through the code in your own time, + +00:09:14.560 --> 00:09:17.519 +if you're interested in building something like this. + +00:09:17.519 --> 00:09:22.720 +If you're interested in just using jq or you're done, + +00:09:22.720 --> 00:09:24.320 +these are the links to all the tools. + +00:09:24.320 --> 00:09:28.240 +counsel-jq, of course, is readily available on MELPA. + +00:09:28.240 --> 00:09:32.959 +Also developed under the AGPL license on Github. + +00:09:32.959 --> 00:09:36.080 +And this organice thing, by the way, it's + +00:09:36.080 --> 00:09:38.560 +Org Mode for mobile and desktop browsers. + +00:09:38.560 --> 00:09:43.120 +Also a great free software tool maybe that interests you. + +00:09:43.120 --> 00:09:46.240 +Thank you for listening. Have a great time. + +00:09:46.240 --> 00:09:49.360 +10 seconds left. I am going to stop this now. + +00:09:49.360 --> 00:09:53.920 +Enjoy EmacsConf. Have a great day. -- cgit v1.2.3