Hello, everyone, and welcome to this short lightning talk:
"Traverse Complex JSON Structures with Live Feedback."
This is a pre-recorded talk and part of the EmacsConf 2020 schedule.
This is what we're going to do.
I'll make a quick introduction to the topic at hand.
I'll give you a demonstration of some tools,
and then we'll leave you with the links to said tools.
Before that, just a little bit about me.
I am the CEO and co-founder of a company based in the Swiss mountains called 200ok.ch.
We are a product incubator and service consultancy,
but we like to spend most or at least as much time as we can
building free software.
I'm also an ordained Zen monk and abbot of the Lambda Zen temple.
You can reach me anytime on questions regarding Emacs, for example,
at alain@200ok.ch.
But back to the topic at hand.
The proposition is as following:
most work on the computer is based on either
text processing or text consumption.
And very often, the text which you need to process is in a structured format,
for example, in JSON.
That might even be if your job is not programming per se.
Reading through such a bigger chunk of JSON can be non-trivial, however,
while just reading and understanding it
will be essential to getting your job done.
So let's quickly check out an example JSON file.
This is from the Github API,
which is a request--sorry, the response to a request
for a specific issue on the github API.
So let's quickly check that one out.
Okay. So here it is open, and we can already see
that there is lots of stuff going on here.
It's 200 lines.
It's not going to be very easy
just to find out what are the top level things in here,
what are the top level attributes.
Of course I can do this, and maybe do it by hand, but that doesn't scale.
I can use cool Emacs facilities like the hideshow-mode
and try to fold all the things that are top level,
but that also doesn't really scale.
There must be a better way.
Of course there is. There is prior art.
There is a tool called jq.
I'm going to quote the USP (unique selling proposition) from their website:
jq is like sed for JSON data.
you can use it to slice and filter and map and transform structured data
with the same ease that
sed, awk, grep, and friends let you play with text.
Let me give you a quick demonstration of it.
By the way, it's written in portable C.
It has zero runtime dependency, so it's very easy to get started with it
and use it on pretty much any UNIX-based computer.
Sorry, no, Linux-based computer, apologies.
Okay, so let's explore a JSON file with it.
It's a command line tool,
and it has a very simple command line syntax.
So you call the binary and then you give it a query and a file,
and then it will return its answer.
So, for example, if I want the top level keys,
I will just say jq keys the file
and it will return the keys.
Simple as that. So let's check this out in a real shell.
Here I am in eshell.
Let's run jq keys on the Github issue comment.
We can see that we have actually received a list back here
with the top-level things.
So this issue... It looks very interesting.
Let's ask it to give me more information on this issue.
Then it's hairy again. That's a lot of stuff.
I mean, lucky for us, we are in Emacs here,
so we can use nice shortcuts.
We can copy this. We can go in here, just select that,
get that out or something like this.
But still, this is not really the best way to do that, right?
it gets kind of tedious.
At this point the output can be humongous.
The shell is not really the best place to read through such big output.
I mean, eshell is probably one of the better shells for this,
because it's just a regular Emacs buffer,
but still, it's not really the best tool.
I need to repeat the command all the time
until I finally build the right query.
And all the time, I lose my focus,
I lose what I'm currently looking at.
I'm seeing the new result.
It would be so much nicer to have live feedback.
When working with Emacs, we're quite used to that.
So there should be an option.
And of course there is. It's Emacs, right,
so you can do anything.
There is various good tools for completion in Emacs.
I used ivy for this.
I'm going to quote the USP for ivy.
ivy is a generic completion mechanism for Emacs.
While it operates similarly to other completion schemes such as icomplete mode,
ivy aims to be more efficient, smaller, simpler, and smoother to use,
yet highly customizable.
And that's true.
One of the cool things of ivy
compared to other completion mechanisms in Emacs
is that it can be used on dynamic data.
So usually completion works on a static input.
For example, you're in a buffer, a text buffer,
and you use isearch maybe with ido-mode,
and you find your results. That's all nice.
However, if I want to search on dynamic data,
that doesn't work.
So whenever I type in my query for jq,
I actually need to call the jq binary,
and it will give a different result set back.
So it's a really dynamic mechanism that we need here.
It's much more like a search engine.
ivy luckily has something built in,
and it's called counsel.
So I used counsel and jq and combined them,
and built a new package
with which we can use Emacs and jq
to have live feedback.
It's very easy to use.
So you just call counsel-jq
on a buffer containing JSON.
For example, the one we have here.
Let's call counsel-jq on it,
and we already get a default query,
the dot query, which just gives us the same file.
But now we can change it.
For example, find all the keys in here.
And then we see I had this issue.
This was the one that we were interested in.
So let's find more information on the issue.
What keys does it have actually have?
It has assignees. That interests me.
So let's check out the assignees in here.
There's two of them, but I'm only interested in the first one.
I'm making stuff up as I go here, of course.
Whenever I hit enter, I get a new buffer
which just shows me this particular result
for the particular query that I entered.
So let me do that again.
We are in here. We are looking at a JSON file.
This can be very, very big.
Doesn't also need to be a file.
Just needs to be a buffer.
You call counsel-jq on it,
and you can do any kind of query on it.
For example, let's see if there is a URL here.
Yes, there's a URL.
Let's see if there's a repository here.
Repository. No, there isn't.
What was it called? Issue. Keys. Repository URL, it was called.
So let's see issue repository URL,
and then we see.
So apparently this issue comment is for a repository called organice.
I wonder what that might be.
Okay. So that was a very short introduction to counsel-jq.
You can see the timer here.
I only have one minute left to go, so I'm going to leave
with a very, very short introduction to the counsel-jq code.
It's not even 60 lines of elisp,
so building something like this is very, very easy.
I would encourage you to go and read through the code in your own time,
if you're interested in building something like this.
If you're interested in just using jq or you're done,
these are the links to all the tools.
counsel-jq, of course, is readily available on MELPA.
Also developed under the AGPL license on Github.
And this organice thing, by the way, it's
Org Mode for mobile and desktop browsers.
Also a great free software tool maybe that interests you.
Thank you for listening. Have a great time.
10 seconds left. I am going to stop this now.
Enjoy EmacsConf. Have a great day.