path: root/2020/info/25.md

# Traverse complex JSON structures with live feedback
Zen Monk Alain M. Lafon

[[!template vidid="mainVideo" id=vid src="https://mirror.csclub.uwaterloo.ca/emacsconf/2020/emacsconf-2020--25-traverse-complex-json-structures-with-live-feedback-counsel-jq--zen-monk-alain-m-lafon.webm" subtitles="/2020/subtitles/emacsconf-2020--25-traverse-complex-json-structures-with-live-feedback-counsel-jq--zen-monk-alain-m-lafon.vtt"]] 
[Download compressed .webm video (18.1M)](https://media.emacsconf.org/2020/emacsconf-2020--25-traverse-complex-json-structures-with-live-feedback-counsel-jq--zen-monk-alain-m-lafon--compressed32.webm)  
[Download compressed .webm video (15.8M, highly compressed)](https://mirror.csclub.uwaterloo.ca/emacsconf/2020/smaller/emacsconf-2020--25-traverse-complex-json-structures-with-live-feedback-counsel-jq--zen-monk-alain-m-lafon--vp9-q56-video-original-audio.webm)  
[View transcript](#transcript)

If you are working with complex nested JSON structures, you are
probably familiar with jq which is like sed for JSON data and great at
what it does.  However, being a command-line tool like sed, the
feedback for writing queries and seeing their results is a discrete
process and not live.

When working with Emacs, we are used to good auto-completion and live
feedback.  Formerly, this was mostly done with static input, but with
modern completion frameworks like Ivy and Counsel, this can be done
with dynamic inputs, as well.

counsel-jq is a package with which you can quickly test queries and
traverse a complex JSON structure whilst having live feedback.  Just
call `M-x counsel-jq` in a buffer containing JSON, then start writing
your `jq` query string and see the output appear live in the message
area.  Whenever you're happy, hit `RET` and the results will be
displayed to you in the buffer `*jq-json*`.

In this lightning talk, I'll give a quick overview on how to use
counsel-jq and how to build similar completion functionality.

## Resources

<https://200ok.ch/posts/2020-11-30_emacsconf_traverse_complex_json_structures_with_live_feedback_with_counseljq.html>

# Questions

## Q4: Any plans for counsel-yq and/or -xq? ;-)
counsel-jq currently just shells out to jq. Adding tools build on top
of jq (at least yq is afaik) would be very easy. We could employ a
strategy pattern to find the right tool based on the current
major-mode with a configurable fallback. Here's the place where the
shellout happens:
<https://github.com/200ok-ch/counsel-jq/blob/master/counsel-jq.el#L23>

Would you be interested in making a PR for that? (;

## Q3: Why repository_url did not autocomplete in addition to the result (I know that it is ivy thing but possible to configure?)
There's no autocompletion for the search query, but that would be a
great addition. That would theoretically be possibe by employing jq to
look ahead in the current tree and providing options for
autocomplete. I'm not certain if Ivy does have autocomplete for search
queries, though.

If somebody has more knowledge on that and would like to ping me up or
provide a (draft) PR, I'd be happy to help out in that endeavour!

## Q2: Is it difficult to provide autocompletion for the JSON query in the minibuffer?
Good question. I'd be curious, too. It's the same question as Q2 where
I went into a possibe scenario.

## Q1: Is it possible to search in arbitrary deep objects? E.g., an AST represented in JSON.
counsel-jq uses jq under the hood, so all queries that are valid
queries in jq should be valid in jq. Hence, I'm inclined to say
'yes' (;

# Notes
- 200ok GmbH (<https://200ok.ch>).
- Play Emacs like an instrument:
  <https://www.youtube.com/watch?v=gfZDwYeBlO4>.
- jq: <https://stedolan.github.io/jq/>.
- ivy supports dynamic sources. So does helm I guess.
- counsel-jq: <https://github.com/200ok-ch/counsel-jq>.
- organice: <https://github.com/200ok-ch/organice>.
- Entire presentation inside Emacs, with a count down reminder.

<a name="transcript"></a>
# Transcript

[[!template new="1" text="Hello, everyone, and welcome to this short lightning talk:" start="00:00:00.799" video="mainVideo" id=subtitle]]
[[!template text="&quot;Traverse Complex JSON Structures with Live Feedback.&quot;" start="00:00:05.520" video="mainVideo" id=subtitle]]
[[!template text="This is a pre-recorded talk and part of the EmacsConf 2020 schedule." start="00:00:09.519" video="mainVideo" id=subtitle]]
[[!template text="This is what we're going to do." start="00:00:18.000" video="mainVideo" id=subtitle]]
[[!template text="I'll make a quick introduction to the topic at hand." start="00:00:19.439" video="mainVideo" id=subtitle]]
[[!template text="I'll give you a demonstration of some tools," start="00:00:22.320" video="mainVideo" id=subtitle]]
[[!template text="and then we'll leave you with the links to said tools." start="00:00:24.400" video="mainVideo" id=subtitle]]
[[!template new="1" text="Before that, just a little bit about me." start="00:00:29.199" video="mainVideo" id=subtitle]]
[[!template text="I am the CEO and co-founder of a company based in the Swiss mountains called 200ok.ch." start="00:00:31.679" video="mainVideo" id=subtitle]]
[[!template text="We are a product incubator and service consultancy," start="00:00:40.399" video="mainVideo" id=subtitle]]
[[!template text="but we like to spend most or at least as much time as we can" start="00:00:44.879" video="mainVideo" id=subtitle]]
[[!template text="building free software." start="00:00:50.000" video="mainVideo" id=subtitle]]
[[!template text="I'm also an ordained Zen monk and abbot of the Lambda Zen temple." start="00:00:52.719" video="mainVideo" id=subtitle]]
[[!template text="You can reach me anytime on questions regarding Emacs, for example," start="00:00:56.879" video="mainVideo" id=subtitle]]
[[!template text="at alain@200ok.ch." start="00:01:04.159" video="mainVideo" id=subtitle]]
[[!template new="1" text="But back to the topic at hand." start="00:01:07.200" video="mainVideo" id=subtitle]]
[[!template text="The proposition is as following:" start="00:01:09.439" video="mainVideo" id=subtitle]]
[[!template text="most work on the computer is based on either" start="00:01:11.760" video="mainVideo" id=subtitle]]
[[!template text="text processing or text consumption." start="00:01:14.000" video="mainVideo" id=subtitle]]
[[!template text="And very often, the text which you need to process is in a structured format," start="00:01:16.479" video="mainVideo" id=subtitle]]
[[!template text="for example, in JSON." start="00:01:22.799" video="mainVideo" id=subtitle]]
[[!template text="That might even be if your job is not programming per se." start="00:01:24.560" video="mainVideo" id=subtitle]]
[[!template text="Reading through such a bigger chunk of JSON can be non-trivial, however," start="00:01:28.560" video="mainVideo" id=subtitle]]
[[!template text="while just reading and understanding it" start="00:01:33.119" video="mainVideo" id=subtitle]]
[[!template text="will be essential to getting your job done." start="00:01:36.479" video="mainVideo" id=subtitle]]
[[!template new="1" text="So let's quickly check out an example JSON file." start="00:01:40.320" video="mainVideo" id=subtitle]]
[[!template text="This is from the Github API," start="00:01:44.479" video="mainVideo" id=subtitle]]
[[!template text="which is a request--sorry, the response to a request" start="00:01:47.200" video="mainVideo" id=subtitle]]
[[!template text="for a specific issue on the github API." start="00:01:52.079" video="mainVideo" id=subtitle]]
[[!template text="So let's quickly check that one out." start="00:01:54.640" video="mainVideo" id=subtitle]]
[[!template text="Okay. So here it is open, and we can already see" start="00:01:58.799" video="mainVideo" id=subtitle]]
[[!template text="that there is lots of stuff going on here." start="00:02:01.920" video="mainVideo" id=subtitle]]
[[!template text="It's 200 lines." start="00:02:05.439" video="mainVideo" id=subtitle]]
[[!template text="It's not going to be very easy" start="00:02:07.360" video="mainVideo" id=subtitle]]
[[!template text="just to find out what are the top level things in here," start="00:02:09.200" video="mainVideo" id=subtitle]]
[[!template text="what are the top level attributes." start="00:02:11.840" video="mainVideo" id=subtitle]]
[[!template text="Of course I can do this, and maybe do it by hand, but that doesn't scale." start="00:02:13.360" video="mainVideo" id=subtitle]]
[[!template text="I can use cool Emacs facilities like the hideshow-mode" start="00:02:17.840" video="mainVideo" id=subtitle]]
[[!template text="and try to fold all the things that are top level," start="00:02:21.599" video="mainVideo" id=subtitle]]
[[!template text="but that also doesn't really scale." start="00:02:24.720" video="mainVideo" id=subtitle]]
[[!template text="There must be a better way." start="00:02:27.200" video="mainVideo" id=subtitle]]
[[!template text="Of course there is. There is prior art." start="00:02:29.360" video="mainVideo" id=subtitle]]
[[!template new="1" text="There is a tool called jq." start="00:02:32.000" video="mainVideo" id=subtitle]]
[[!template text="I'm going to quote the USP (unique selling proposition) from their website:" start="00:02:34.080" video="mainVideo" id=subtitle]]
[[!template text="jq is like sed for JSON data." start="00:02:37.760" video="mainVideo" id=subtitle]]
[[!template text="you can use it to slice and filter and map and transform structured data" start="00:02:42.000" video="mainVideo" id=subtitle]]
[[!template text="with the same ease that" start="00:02:46.319" video="mainVideo" id=subtitle]]
[[!template text="sed, awk, grep, and friends let you play with text." start="00:02:47.840" video="mainVideo" id=subtitle]]
[[!template text="Let me give you a quick demonstration of it." start="00:02:54.000" video="mainVideo" id=subtitle]]
[[!template text="By the way, it's written in portable C." start="00:02:56.879" video="mainVideo" id=subtitle]]
[[!template text="It has zero runtime dependency, so it's very easy to get started with it" start="00:02:59.040" video="mainVideo" id=subtitle]]
[[!template text="and use it on pretty much any UNIX-based computer." start="00:03:03.519" video="mainVideo" id=subtitle]]
[[!template text="Sorry, no, Linux-based computer, apologies." start="00:03:09.840" video="mainVideo" id=subtitle]]
[[!template new="1" text="Okay, so let's explore a JSON file with it." start="00:03:14.000" video="mainVideo" id=subtitle]]
[[!template text="It's a command line tool," start="00:03:18.720" video="mainVideo" id=subtitle]]
[[!template text="and it has a very simple command line syntax." start="00:03:20.000" video="mainVideo" id=subtitle]]
[[!template text="So you call the binary and then you give it a query and a file," start="00:03:24.000" video="mainVideo" id=subtitle]]
[[!template text="and then it will return its answer." start="00:03:29.840" video="mainVideo" id=subtitle]]
[[!template text="So, for example, if I want the top level keys," start="00:03:32.560" video="mainVideo" id=subtitle]]
[[!template text="I will just say jq keys the file" start="00:03:35.440" video="mainVideo" id=subtitle]]
[[!template text="and it will return the keys." start="00:03:38.000" video="mainVideo" id=subtitle]]
[[!template text="Simple as that. So let's check this out in a real shell." start="00:03:39.840" video="mainVideo" id=subtitle]]
[[!template text="Here I am in eshell." start="00:03:44.400" video="mainVideo" id=subtitle]]
[[!template text="Let's run jq keys on the Github issue comment." start="00:03:46.879" video="mainVideo" id=subtitle]]
[[!template text="We can see that we have actually received a list back here" start="00:03:51.440" video="mainVideo" id=subtitle]]
[[!template text="with the top-level things." start="00:03:58.799" video="mainVideo" id=subtitle]]
[[!template text="So this issue... It looks very interesting." start="00:04:00.319" video="mainVideo" id=subtitle]]
[[!template text="Let's ask it to give me more information on this issue." start="00:04:02.879" video="mainVideo" id=subtitle]]
[[!template text="Then it's hairy again. That's a lot of stuff." start="00:04:07.360" video="mainVideo" id=subtitle]]
[[!template new="1" text="I mean, lucky for us, we are in Emacs here," start="00:04:11.360" video="mainVideo" id=subtitle]]
[[!template text="so we can use nice shortcuts." start="00:04:14.560" video="mainVideo" id=subtitle]]
[[!template text="We can copy this. We can go in here, just select that," start="00:04:16.720" video="mainVideo" id=subtitle]]
[[!template text="get that out or something like this." start="00:04:22.000" video="mainVideo" id=subtitle]]
[[!template text="But still, this is not really the best way to do that, right?" start="00:04:24.160" video="mainVideo" id=subtitle]]
[[!template text="it gets kind of tedious." start="00:04:32.320" video="mainVideo" id=subtitle]]
[[!template text="At this point the output can be humongous." start="00:04:34.080" video="mainVideo" id=subtitle]]
[[!template text="The shell is not really the best place to read through such big output." start="00:04:37.680" video="mainVideo" id=subtitle]]
[[!template text="I mean, eshell is probably one of the better shells for this," start="00:04:41.919" video="mainVideo" id=subtitle]]
[[!template text="because it's just a regular Emacs buffer," start="00:04:45.759" video="mainVideo" id=subtitle]]
[[!template text="but still, it's not really the best tool." start="00:04:47.919" video="mainVideo" id=subtitle]]
[[!template text="I need to repeat the command all the time" start="00:04:50.720" video="mainVideo" id=subtitle]]
[[!template text="until I finally build the right query." start="00:04:53.680" video="mainVideo" id=subtitle]]
[[!template text="And all the time, I lose my focus," start="00:04:56.000" video="mainVideo" id=subtitle]]
[[!template text="I lose what I'm currently looking at." start="00:04:59.840" video="mainVideo" id=subtitle]]
[[!template text="I'm seeing the new result." start="00:05:02.800" video="mainVideo" id=subtitle]]
[[!template new="1" text="It would be so much nicer to have live feedback." start="00:05:05.520" video="mainVideo" id=subtitle]]
[[!template text="When working with Emacs, we're quite used to that." start="00:05:08.160" video="mainVideo" id=subtitle]]
[[!template text="So there should be an option." start="00:05:10.720" video="mainVideo" id=subtitle]]
[[!template text="And of course there is. It's Emacs, right," start="00:05:12.320" video="mainVideo" id=subtitle]]
[[!template text="so you can do anything." start="00:05:15.120" video="mainVideo" id=subtitle]]
[[!template new="1" text="There is various good tools for completion in Emacs." start="00:05:17.759" video="mainVideo" id=subtitle]]
[[!template text="I used ivy for this." start="00:05:22.960" video="mainVideo" id=subtitle]]
[[!template text="I'm going to quote the USP for ivy." start="00:05:26.000" video="mainVideo" id=subtitle]]
[[!template text="ivy is a generic completion mechanism for Emacs." start="00:05:29.039" video="mainVideo" id=subtitle]]
[[!template text="While it operates similarly to other completion schemes such as icomplete mode," start="00:05:32.639" video="mainVideo" id=subtitle]]
[[!template text="ivy aims to be more efficient, smaller, simpler, and smoother to use," start="00:05:37.919" video="mainVideo" id=subtitle]]
[[!template text="yet highly customizable." start="00:05:42.160" video="mainVideo" id=subtitle]]
[[!template text="And that's true." start="00:05:45.199" video="mainVideo" id=subtitle]]
[[!template new="1" text="One of the cool things of ivy" start="00:05:46.479" video="mainVideo" id=subtitle]]
[[!template text="compared to other completion mechanisms in Emacs" start="00:05:49.440" video="mainVideo" id=subtitle]]
[[!template text="is that it can be used on dynamic data." start="00:05:54.320" video="mainVideo" id=subtitle]]
[[!template text="So usually completion works on a static input." start="00:05:59.120" video="mainVideo" id=subtitle]]
[[!template text="For example, you're in a buffer, a text buffer," start="00:06:02.400" video="mainVideo" id=subtitle]]
[[!template text="and you use isearch maybe with ido-mode," start="00:06:05.360" video="mainVideo" id=subtitle]]
[[!template text="and you find your results. That's all nice." start="00:06:09.600" video="mainVideo" id=subtitle]]
[[!template text="However, if I want to search on dynamic data," start="00:06:13.360" video="mainVideo" id=subtitle]]
[[!template text="that doesn't work." start="00:06:19.600" video="mainVideo" id=subtitle]]
[[!template text="So whenever I type in my query for jq," start="00:06:20.720" video="mainVideo" id=subtitle]]
[[!template text="I actually need to call the jq binary," start="00:06:24.880" video="mainVideo" id=subtitle]]
[[!template text="and it will give a different result set back." start="00:06:28.000" video="mainVideo" id=subtitle]]
[[!template text="So it's a really dynamic mechanism that we need here." start="00:06:30.720" video="mainVideo" id=subtitle]]
[[!template text="It's much more like a search engine." start="00:06:36.160" video="mainVideo" id=subtitle]]
[[!template new="1" text="ivy luckily has something built in," start="00:06:38.240" video="mainVideo" id=subtitle]]
[[!template text="and it's called counsel." start="00:06:41.440" video="mainVideo" id=subtitle]]
[[!template text="So I used counsel and jq and combined them," start="00:06:43.520" video="mainVideo" id=subtitle]]
[[!template text="and built a new package" start="00:06:47.360" video="mainVideo" id=subtitle]]
[[!template text="with which we can use Emacs and jq" start="00:06:49.199" video="mainVideo" id=subtitle]]
[[!template text="to have live feedback." start="00:06:52.960" video="mainVideo" id=subtitle]]
[[!template text="It's very easy to use." start="00:06:56.000" video="mainVideo" id=subtitle]]
[[!template text="So you just call counsel-jq" start="00:06:57.759" video="mainVideo" id=subtitle]]
[[!template text="on a buffer containing JSON." start="00:06:59.840" video="mainVideo" id=subtitle]]
[[!template text="For example, the one we have here." start="00:07:02.160" video="mainVideo" id=subtitle]]
[[!template text="Let's call counsel-jq on it," start="00:07:04.319" video="mainVideo" id=subtitle]]
[[!template text="and we already get a default query," start="00:07:06.800" video="mainVideo" id=subtitle]]
[[!template text="the dot query, which just gives us the same file." start="00:07:10.080" video="mainVideo" id=subtitle]]
[[!template text="But now we can change it." start="00:07:14.639" video="mainVideo" id=subtitle]]
[[!template text="For example, find all the keys in here." start="00:07:16.240" video="mainVideo" id=subtitle]]
[[!template text="And then we see I had this issue." start="00:07:18.639" video="mainVideo" id=subtitle]]
[[!template text="This was the one that we were interested in." start="00:07:20.319" video="mainVideo" id=subtitle]]
[[!template text="So let's find more information on the issue." start="00:07:22.800" video="mainVideo" id=subtitle]]
[[!template text="What keys does it have actually have?" start="00:07:25.599" video="mainVideo" id=subtitle]]
[[!template text="It has assignees. That interests me." start="00:07:28.720" video="mainVideo" id=subtitle]]
[[!template text="So let's check out the assignees in here." start="00:07:31.680" video="mainVideo" id=subtitle]]
[[!template text="There's two of them, but I'm only interested in the first one." start="00:07:34.800" video="mainVideo" id=subtitle]]
[[!template text="I'm making stuff up as I go here, of course." start="00:07:39.759" video="mainVideo" id=subtitle]]
[[!template text="Whenever I hit enter, I get a new buffer" start="00:07:43.599" video="mainVideo" id=subtitle]]
[[!template text="which just shows me this particular result" start="00:07:47.039" video="mainVideo" id=subtitle]]
[[!template text="for the particular query that I entered." start="00:07:52.639" video="mainVideo" id=subtitle]]
[[!template new="1" text="So let me do that again." start="00:07:55.599" video="mainVideo" id=subtitle]]
[[!template text="We are in here. We are looking at a JSON file." start="00:07:57.680" video="mainVideo" id=subtitle]]
[[!template text="This can be very, very big." start="00:08:04.000" video="mainVideo" id=subtitle]]
[[!template text="Doesn't also need to be a file." start="00:08:05.840" video="mainVideo" id=subtitle]]
[[!template text="Just needs to be a buffer." start="00:08:07.280" video="mainVideo" id=subtitle]]
[[!template text="You call counsel-jq on it," start="00:08:09.520" video="mainVideo" id=subtitle]]
[[!template text="and you can do any kind of query on it." start="00:08:11.360" video="mainVideo" id=subtitle]]
[[!template text="For example, let's see if there is a URL here." start="00:08:14.319" video="mainVideo" id=subtitle]]
[[!template text="Yes, there's a URL." start="00:08:18.080" video="mainVideo" id=subtitle]]
[[!template text="Let's see if there's a repository here." start="00:08:19.440" video="mainVideo" id=subtitle]]
[[!template text="Repository. No, there isn't." start="00:08:22.827" video="mainVideo" id=subtitle]]
[[!template text="What was it called? Issue. Keys. Repository URL, it was called." start="00:08:24.639" video="mainVideo" id=subtitle]]
[[!template text="So let's see issue repository URL," start="00:08:33.440" video="mainVideo" id=subtitle]]
[[!template text="and then we see." start="00:08:38.240" video="mainVideo" id=subtitle]]
[[!template text="So apparently this issue comment is for a repository called organice." start="00:08:39.519" video="mainVideo" id=subtitle]]
[[!template text="I wonder what that might be." start="00:08:44.800" video="mainVideo" id=subtitle]]
[[!template new="1" text="Okay. So that was a very short introduction to counsel-jq." start="00:08:47.839" video="mainVideo" id=subtitle]]
[[!template text="You can see the timer here." start="00:08:52.640" video="mainVideo" id=subtitle]]
[[!template text="I only have one minute left to go, so I'm going to leave" start="00:08:54.240" video="mainVideo" id=subtitle]]
[[!template text="with a very, very short introduction to the counsel-jq code." start="00:08:57.440" video="mainVideo" id=subtitle]]
[[!template text="It's not even 60 lines of elisp," start="00:09:02.880" video="mainVideo" id=subtitle]]
[[!template text="so building something like this is very, very easy." start="00:09:06.000" video="mainVideo" id=subtitle]]
[[!template text="I would encourage you to go and read through the code in your own time," start="00:09:09.600" video="mainVideo" id=subtitle]]
[[!template text="if you're interested in building something like this." start="00:09:14.560" video="mainVideo" id=subtitle]]
[[!template text="If you're interested in just using jq or you're done," start="00:09:17.519" video="mainVideo" id=subtitle]]
[[!template text="these are the links to all the tools." start="00:09:22.720" video="mainVideo" id=subtitle]]
[[!template text="counsel-jq, of course, is readily available on MELPA." start="00:09:24.320" video="mainVideo" id=subtitle]]
[[!template text="Also developed under the AGPL license on Github." start="00:09:28.240" video="mainVideo" id=subtitle]]
[[!template text="And this organice thing, by the way, it's" start="00:09:32.959" video="mainVideo" id=subtitle]]
[[!template text="Org Mode for mobile and desktop browsers." start="00:09:36.080" video="mainVideo" id=subtitle]]
[[!template text="Also a great free software tool maybe that interests you." start="00:09:38.560" video="mainVideo" id=subtitle]]
[[!template text="Thank you for listening. Have a great time." start="00:09:43.120" video="mainVideo" id=subtitle]]
[[!template text="10 seconds left. I am going to stop this now." start="00:09:46.240" video="mainVideo" id=subtitle]]
[[!template text="Enjoy EmacsConf. Have a great day." start="00:09:49.360" video="mainVideo" id=subtitle]]