diff options
Diffstat (limited to '2020/info/25.md')
| -rw-r--r-- | 2020/info/25.md | 181 |
1 files changed, 179 insertions, 2 deletions
diff --git a/2020/info/25.md b/2020/info/25.md index 090cb22e..4e49cdc6 100644 --- a/2020/info/25.md +++ b/2020/info/25.md @@ -1,8 +1,9 @@ # Traverse complex JSON structures with live feedback Zen Monk Alain M. Lafon -[[!template id=vid src="https://mirror.csclub.uwaterloo.ca/emacsconf/2020/emacsconf-2020--25-traverse-complex-json-structures-with-live-feedback-counsel-jq--zen-monk-alain-m-lafon.webm"]] -[Download compressed .webm video (15.8M)](https://mirror.csclub.uwaterloo.ca/emacsconf/2020/smaller/emacsconf-2020--25-traverse-complex-json-structures-with-live-feedback-counsel-jq--zen-monk-alain-m-lafon--vp9-q56-video-original-audio.webm) +[[!template vidid="mainVideo" id=vid src="https://mirror.csclub.uwaterloo.ca/emacsconf/2020/emacsconf-2020--25-traverse-complex-json-structures-with-live-feedback-counsel-jq--zen-monk-alain-m-lafon.webm" subtitles="/2020/subtitles/emacsconf-2020--25-traverse-complex-json-structures-with-live-feedback-counsel-jq--zen-monk-alain-m-lafon.vtt"]] +[Download compressed .webm video (15.8M)](https://mirror.csclub.uwaterloo.ca/emacsconf/2020/smaller/emacsconf-2020--25-traverse-complex-json-structures-with-live-feedback-counsel-jq--zen-monk-alain-m-lafon--vp9-q56-video-original-audio.webm) +[View transcript](#transcript) If you are working with complex nested JSON structures, you are probably familiar with jq which is like sed for JSON data and great at @@ -69,3 +70,179 @@ queries in jq should be valid in jq. Hence, I'm inclined to say - counsel-jq: <https://github.com/200ok-ch/counsel-jq>. - organice: <https://github.com/200ok-ch/organice>. - Entire presentation inside Emacs, with a count down reminder. + +<a name="transcript"></a> +# Transcript + +[[!template new="1" text="Hello, everyone, and welcome to this short lightning talk:" start="00:00:00.799" video="mainVideo" id=subtitle]] +[[!template text=""Traverse Complex JSON Structures with Live Feedback."" start="00:00:05.520" video="mainVideo" id=subtitle]] +[[!template text="This is a pre-recorded talk and part of the EmacsConf 2020 schedule." start="00:00:09.519" video="mainVideo" id=subtitle]] +[[!template text="This is what we're going to do." start="00:00:18.000" video="mainVideo" id=subtitle]] +[[!template text="I'll make a quick introduction to the topic at hand." start="00:00:19.439" video="mainVideo" id=subtitle]] +[[!template text="I'll give you a demonstration of some tools," start="00:00:22.320" video="mainVideo" id=subtitle]] +[[!template text="and then we'll leave you with the links to said tools." start="00:00:24.400" video="mainVideo" id=subtitle]] +[[!template new="1" text="Before that, just a little bit about me." start="00:00:29.199" video="mainVideo" id=subtitle]] +[[!template text="I am the CEO and co-founder of a company based in the Swiss mountains called 200ok.ch." start="00:00:31.679" video="mainVideo" id=subtitle]] +[[!template text="We are a product incubator and service consultancy," start="00:00:40.399" video="mainVideo" id=subtitle]] +[[!template text="but we like to spend most or at least as much time as we can" start="00:00:44.879" video="mainVideo" id=subtitle]] +[[!template text="building free software." start="00:00:50.000" video="mainVideo" id=subtitle]] +[[!template text="I'm also an ordained Zen monk and abbot of the Lambda Zen temple." start="00:00:52.719" video="mainVideo" id=subtitle]] +[[!template text="You can reach me anytime on questions regarding Emacs, for example," start="00:00:56.879" video="mainVideo" id=subtitle]] +[[!template text="at alain@200ok.ch." start="00:01:04.159" video="mainVideo" id=subtitle]] +[[!template new="1" text="But back to the topic at hand." start="00:01:07.200" video="mainVideo" id=subtitle]] +[[!template text="The proposition is as following:" start="00:01:09.439" video="mainVideo" id=subtitle]] +[[!template text="most work on the computer is based on either" start="00:01:11.760" video="mainVideo" id=subtitle]] +[[!template text="text processing or text consumption." start="00:01:14.000" video="mainVideo" id=subtitle]] +[[!template text="And very often, the text which you need to process is in a structured format," start="00:01:16.479" video="mainVideo" id=subtitle]] +[[!template text="for example, in JSON." start="00:01:22.799" video="mainVideo" id=subtitle]] +[[!template text="That might even be if your job is not programming per se." start="00:01:24.560" video="mainVideo" id=subtitle]] +[[!template text="Reading through such a bigger chunk of JSON can be non-trivial, however," start="00:01:28.560" video="mainVideo" id=subtitle]] +[[!template text="while just reading and understanding it" start="00:01:33.119" video="mainVideo" id=subtitle]] +[[!template text="will be essential to getting your job done." start="00:01:36.479" video="mainVideo" id=subtitle]] +[[!template new="1" text="So let's quickly check out an example JSON file." start="00:01:40.320" video="mainVideo" id=subtitle]] +[[!template text="This is from the Github API," start="00:01:44.479" video="mainVideo" id=subtitle]] +[[!template text="which is a request--sorry, the response to a request" start="00:01:47.200" video="mainVideo" id=subtitle]] +[[!template text="for a specific issue on the github API." start="00:01:52.079" video="mainVideo" id=subtitle]] +[[!template text="So let's quickly check that one out." start="00:01:54.640" video="mainVideo" id=subtitle]] +[[!template text="Okay. So here it is open, and we can already see" start="00:01:58.799" video="mainVideo" id=subtitle]] +[[!template text="that there is lots of stuff going on here." start="00:02:01.920" video="mainVideo" id=subtitle]] +[[!template text="It's 200 lines." start="00:02:05.439" video="mainVideo" id=subtitle]] +[[!template text="It's not going to be very easy" start="00:02:07.360" video="mainVideo" id=subtitle]] +[[!template text="just to find out what are the top level things in here," start="00:02:09.200" video="mainVideo" id=subtitle]] +[[!template text="what are the top level attributes." start="00:02:11.840" video="mainVideo" id=subtitle]] +[[!template text="Of course I can do this, and maybe do it by hand, but that doesn't scale." start="00:02:13.360" video="mainVideo" id=subtitle]] +[[!template text="I can use cool Emacs facilities like the hideshow-mode" start="00:02:17.840" video="mainVideo" id=subtitle]] +[[!template text="and try to fold all the things that are top level," start="00:02:21.599" video="mainVideo" id=subtitle]] +[[!template text="but that also doesn't really scale." start="00:02:24.720" video="mainVideo" id=subtitle]] +[[!template text="There must be a better way." start="00:02:27.200" video="mainVideo" id=subtitle]] +[[!template text="Of course there is. There is prior art." start="00:02:29.360" video="mainVideo" id=subtitle]] +[[!template new="1" text="There is a tool called jq." start="00:02:32.000" video="mainVideo" id=subtitle]] +[[!template text="I'm going to quote the USP (unique selling proposition) from their website:" start="00:02:34.080" video="mainVideo" id=subtitle]] +[[!template text="jq is like sed for JSON data." start="00:02:37.760" video="mainVideo" id=subtitle]] +[[!template text="you can use it to slice and filter and map and transform structured data" start="00:02:42.000" video="mainVideo" id=subtitle]] +[[!template text="with the same ease that" start="00:02:46.319" video="mainVideo" id=subtitle]] +[[!template text="sed, awk, grep, and friends let you play with text." start="00:02:47.840" video="mainVideo" id=subtitle]] +[[!template text="Let me give you a quick demonstration of it." start="00:02:54.000" video="mainVideo" id=subtitle]] +[[!template text="By the way, it's written in portable C." start="00:02:56.879" video="mainVideo" id=subtitle]] +[[!template text="It has zero runtime dependency, so it's very easy to get started with it" start="00:02:59.040" video="mainVideo" id=subtitle]] +[[!template text="and use it on pretty much any UNIX-based computer." start="00:03:03.519" video="mainVideo" id=subtitle]] +[[!template text="Sorry, no, Linux-based computer, apologies." start="00:03:09.840" video="mainVideo" id=subtitle]] +[[!template new="1" text="Okay, so let's explore a JSON file with it." start="00:03:14.000" video="mainVideo" id=subtitle]] +[[!template text="It's a command line tool," start="00:03:18.720" video="mainVideo" id=subtitle]] +[[!template text="and it has a very simple command line syntax." start="00:03:20.000" video="mainVideo" id=subtitle]] +[[!template text="So you call the binary and then you give it a query and a file," start="00:03:24.000" video="mainVideo" id=subtitle]] +[[!template text="and then it will return its answer." start="00:03:29.840" video="mainVideo" id=subtitle]] +[[!template text="So, for example, if I want the top level keys," start="00:03:32.560" video="mainVideo" id=subtitle]] +[[!template text="I will just say jq keys the file" start="00:03:35.440" video="mainVideo" id=subtitle]] +[[!template text="and it will return the keys." start="00:03:38.000" video="mainVideo" id=subtitle]] +[[!template text="Simple as that. So let's check this out in a real shell." start="00:03:39.840" video="mainVideo" id=subtitle]] +[[!template text="Here I am in eshell." start="00:03:44.400" video="mainVideo" id=subtitle]] +[[!template text="Let's run jq keys on the Github issue comment." start="00:03:46.879" video="mainVideo" id=subtitle]] +[[!template text="We can see that we have actually received a list back here" start="00:03:51.440" video="mainVideo" id=subtitle]] +[[!template text="with the top-level things." start="00:03:58.799" video="mainVideo" id=subtitle]] +[[!template text="So this issue... It looks very interesting." start="00:04:00.319" video="mainVideo" id=subtitle]] +[[!template text="Let's ask it to give me more information on this issue." start="00:04:02.879" video="mainVideo" id=subtitle]] +[[!template text="Then it's hairy again. That's a lot of stuff." start="00:04:07.360" video="mainVideo" id=subtitle]] +[[!template new="1" text="I mean, lucky for us, we are in Emacs here," start="00:04:11.360" video="mainVideo" id=subtitle]] +[[!template text="so we can use nice shortcuts." start="00:04:14.560" video="mainVideo" id=subtitle]] +[[!template text="We can copy this. We can go in here, just select that," start="00:04:16.720" video="mainVideo" id=subtitle]] +[[!template text="get that out or something like this." start="00:04:22.000" video="mainVideo" id=subtitle]] +[[!template text="But still, this is not really the best way to do that, right?" start="00:04:24.160" video="mainVideo" id=subtitle]] +[[!template text="it gets kind of tedious." start="00:04:32.320" video="mainVideo" id=subtitle]] +[[!template text="At this point the output can be humongous." start="00:04:34.080" video="mainVideo" id=subtitle]] +[[!template text="The shell is not really the best place to read through such big output." start="00:04:37.680" video="mainVideo" id=subtitle]] +[[!template text="I mean, eshell is probably one of the better shells for this," start="00:04:41.919" video="mainVideo" id=subtitle]] +[[!template text="because it's just a regular Emacs buffer," start="00:04:45.759" video="mainVideo" id=subtitle]] +[[!template text="but still, it's not really the best tool." start="00:04:47.919" video="mainVideo" id=subtitle]] +[[!template text="I need to repeat the command all the time" start="00:04:50.720" video="mainVideo" id=subtitle]] +[[!template text="until I finally build the right query." start="00:04:53.680" video="mainVideo" id=subtitle]] +[[!template text="And all the time, I lose my focus," start="00:04:56.000" video="mainVideo" id=subtitle]] +[[!template text="I lose what I'm currently looking at." start="00:04:59.840" video="mainVideo" id=subtitle]] +[[!template text="I'm seeing the new result." start="00:05:02.800" video="mainVideo" id=subtitle]] +[[!template new="1" text="It would be so much nicer to have live feedback." start="00:05:05.520" video="mainVideo" id=subtitle]] +[[!template text="When working with Emacs, we're quite used to that." start="00:05:08.160" video="mainVideo" id=subtitle]] +[[!template text="So there should be an option." start="00:05:10.720" video="mainVideo" id=subtitle]] +[[!template text="And of course there is. It's Emacs, right," start="00:05:12.320" video="mainVideo" id=subtitle]] +[[!template text="so you can do anything." start="00:05:15.120" video="mainVideo" id=subtitle]] +[[!template new="1" text="There is various good tools for completion in Emacs." start="00:05:17.759" video="mainVideo" id=subtitle]] +[[!template text="I used ivy for this." start="00:05:22.960" video="mainVideo" id=subtitle]] +[[!template text="I'm going to quote the USP for ivy." start="00:05:26.000" video="mainVideo" id=subtitle]] +[[!template text="ivy is a generic completion mechanism for Emacs." start="00:05:29.039" video="mainVideo" id=subtitle]] +[[!template text="While it operates similarly to other completion schemes such as icomplete mode," start="00:05:32.639" video="mainVideo" id=subtitle]] +[[!template text="ivy aims to be more efficient, smaller, simpler, and smoother to use," start="00:05:37.919" video="mainVideo" id=subtitle]] +[[!template text="yet highly customizable." start="00:05:42.160" video="mainVideo" id=subtitle]] +[[!template text="And that's true." start="00:05:45.199" video="mainVideo" id=subtitle]] +[[!template new="1" text="One of the cool things of ivy" start="00:05:46.479" video="mainVideo" id=subtitle]] +[[!template text="compared to other completion mechanisms in Emacs" start="00:05:49.440" video="mainVideo" id=subtitle]] +[[!template text="is that it can be used on dynamic data." start="00:05:54.320" video="mainVideo" id=subtitle]] +[[!template text="So usually completion works on a static input." start="00:05:59.120" video="mainVideo" id=subtitle]] +[[!template text="For example, you're in a buffer, a text buffer," start="00:06:02.400" video="mainVideo" id=subtitle]] +[[!template text="and you use isearch maybe with ido-mode," start="00:06:05.360" video="mainVideo" id=subtitle]] +[[!template text="and you find your results. That's all nice." start="00:06:09.600" video="mainVideo" id=subtitle]] +[[!template text="However, if I want to search on dynamic data," start="00:06:13.360" video="mainVideo" id=subtitle]] +[[!template text="that doesn't work." start="00:06:19.600" video="mainVideo" id=subtitle]] +[[!template text="So whenever I type in my query for jq," start="00:06:20.720" video="mainVideo" id=subtitle]] +[[!template text="I actually need to call the jq binary," start="00:06:24.880" video="mainVideo" id=subtitle]] +[[!template text="and it will give a different result set back." start="00:06:28.000" video="mainVideo" id=subtitle]] +[[!template text="So it's a really dynamic mechanism that we need here." start="00:06:30.720" video="mainVideo" id=subtitle]] +[[!template text="It's much more like a search engine." start="00:06:36.160" video="mainVideo" id=subtitle]] +[[!template new="1" text="ivy luckily has something built in," start="00:06:38.240" video="mainVideo" id=subtitle]] +[[!template text="and it's called counsel." start="00:06:41.440" video="mainVideo" id=subtitle]] +[[!template text="So I used counsel and jq and combined them," start="00:06:43.520" video="mainVideo" id=subtitle]] +[[!template text="and built a new package" start="00:06:47.360" video="mainVideo" id=subtitle]] +[[!template text="with which we can use Emacs and jq" start="00:06:49.199" video="mainVideo" id=subtitle]] +[[!template text="to have live feedback." start="00:06:52.960" video="mainVideo" id=subtitle]] +[[!template text="It's very easy to use." start="00:06:56.000" video="mainVideo" id=subtitle]] +[[!template text="So you just call counsel-jq" start="00:06:57.759" video="mainVideo" id=subtitle]] +[[!template text="on a buffer containing JSON." start="00:06:59.840" video="mainVideo" id=subtitle]] +[[!template text="For example, the one we have here." start="00:07:02.160" video="mainVideo" id=subtitle]] +[[!template text="Let's call counsel-jq on it," start="00:07:04.319" video="mainVideo" id=subtitle]] +[[!template text="and we already get a default query," start="00:07:06.800" video="mainVideo" id=subtitle]] +[[!template text="the dot query, which just gives us the same file." start="00:07:10.080" video="mainVideo" id=subtitle]] +[[!template text="But now we can change it." start="00:07:14.639" video="mainVideo" id=subtitle]] +[[!template text="For example, find all the keys in here." start="00:07:16.240" video="mainVideo" id=subtitle]] +[[!template text="And then we see I had this issue." start="00:07:18.639" video="mainVideo" id=subtitle]] +[[!template text="This was the one that we were interested in." start="00:07:20.319" video="mainVideo" id=subtitle]] +[[!template text="So let's find more information on the issue." start="00:07:22.800" video="mainVideo" id=subtitle]] +[[!template text="What keys does it have actually have?" start="00:07:25.599" video="mainVideo" id=subtitle]] +[[!template text="It has assignees. That interests me." start="00:07:28.720" video="mainVideo" id=subtitle]] +[[!template text="So let's check out the assignees in here." start="00:07:31.680" video="mainVideo" id=subtitle]] +[[!template text="There's two of them, but I'm only interested in the first one." start="00:07:34.800" video="mainVideo" id=subtitle]] +[[!template text="I'm making stuff up as I go here, of course." start="00:07:39.759" video="mainVideo" id=subtitle]] +[[!template text="Whenever I hit enter, I get a new buffer" start="00:07:43.599" video="mainVideo" id=subtitle]] +[[!template text="which just shows me this particular result" start="00:07:47.039" video="mainVideo" id=subtitle]] +[[!template text="for the particular query that I entered." start="00:07:52.639" video="mainVideo" id=subtitle]] +[[!template new="1" text="So let me do that again." start="00:07:55.599" video="mainVideo" id=subtitle]] +[[!template text="We are in here. We are looking at a JSON file." start="00:07:57.680" video="mainVideo" id=subtitle]] +[[!template text="This can be very, very big." start="00:08:04.000" video="mainVideo" id=subtitle]] +[[!template text="Doesn't also need to be a file." start="00:08:05.840" video="mainVideo" id=subtitle]] +[[!template text="Just needs to be a buffer." start="00:08:07.280" video="mainVideo" id=subtitle]] +[[!template text="You call counsel-jq on it," start="00:08:09.520" video="mainVideo" id=subtitle]] +[[!template text="and you can do any kind of query on it." start="00:08:11.360" video="mainVideo" id=subtitle]] +[[!template text="For example, let's see if there is a URL here." start="00:08:14.319" video="mainVideo" id=subtitle]] +[[!template text="Yes, there's a URL." start="00:08:18.080" video="mainVideo" id=subtitle]] +[[!template text="Let's see if there's a repository here." start="00:08:19.440" video="mainVideo" id=subtitle]] +[[!template text="Repository. No, there isn't." start="00:08:22.827" video="mainVideo" id=subtitle]] +[[!template text="What was it called? Issue. Keys. Repository URL, it was called." start="00:08:24.639" video="mainVideo" id=subtitle]] +[[!template text="So let's see issue repository URL," start="00:08:33.440" video="mainVideo" id=subtitle]] +[[!template text="and then we see." start="00:08:38.240" video="mainVideo" id=subtitle]] +[[!template text="So apparently this issue comment is for a repository called organice." start="00:08:39.519" video="mainVideo" id=subtitle]] +[[!template text="I wonder what that might be." start="00:08:44.800" video="mainVideo" id=subtitle]] +[[!template new="1" text="Okay. So that was a very short introduction to counsel-jq." start="00:08:47.839" video="mainVideo" id=subtitle]] +[[!template text="You can see the timer here." start="00:08:52.640" video="mainVideo" id=subtitle]] +[[!template text="I only have one minute left to go, so I'm going to leave" start="00:08:54.240" video="mainVideo" id=subtitle]] +[[!template text="with a very, very short introduction to the counsel-jq code." start="00:08:57.440" video="mainVideo" id=subtitle]] +[[!template text="It's not even 60 lines of elisp," start="00:09:02.880" video="mainVideo" id=subtitle]] +[[!template text="so building something like this is very, very easy." start="00:09:06.000" video="mainVideo" id=subtitle]] +[[!template text="I would encourage you to go and read through the code in your own time," start="00:09:09.600" video="mainVideo" id=subtitle]] +[[!template text="if you're interested in building something like this." start="00:09:14.560" video="mainVideo" id=subtitle]] +[[!template text="If you're interested in just using jq or you're done," start="00:09:17.519" video="mainVideo" id=subtitle]] +[[!template text="these are the links to all the tools." start="00:09:22.720" video="mainVideo" id=subtitle]] +[[!template text="counsel-jq, of course, is readily available on MELPA." start="00:09:24.320" video="mainVideo" id=subtitle]] +[[!template text="Also developed under the AGPL license on Github." start="00:09:28.240" video="mainVideo" id=subtitle]] +[[!template text="And this organice thing, by the way, it's" start="00:09:32.959" video="mainVideo" id=subtitle]] +[[!template text="Org Mode for mobile and desktop browsers." start="00:09:36.080" video="mainVideo" id=subtitle]] +[[!template text="Also a great free software tool maybe that interests you." start="00:09:38.560" video="mainVideo" id=subtitle]] +[[!template text="Thank you for listening. Have a great time." start="00:09:43.120" video="mainVideo" id=subtitle]] +[[!template text="10 seconds left. I am going to stop this now." start="00:09:46.240" video="mainVideo" id=subtitle]] +[[!template text="Enjoy EmacsConf. Have a great day." start="00:09:49.360" video="mainVideo" id=subtitle]] |