path: root/2020/subtitles/emacsconf-2020--25-traverse-complex-json-structures-with-live-feedback-counsel-jq--zen-monk-alain-m-lafon.vtt

WEBVTT

00:00:00.799 --> 00:00:05.520
Hello, everyone, and welcome to this
short lightning talk:

00:00:05.520 --> 00:00:09.519
"Traverse Complex JSON Structures with
Live Feedback."

00:00:09.519 --> 00:00:18.000
This is a pre-recorded talk and part of
the EmacsConf 2020 schedule.

00:00:18.000 --> 00:00:19.439
This is what we're going to do.

00:00:19.439 --> 00:00:22.320
I'll make a quick introduction to the
topic at hand.

00:00:22.320 --> 00:00:24.400
I'll give you a demonstration of some
tools,

00:00:24.400 --> 00:00:29.199
and then we'll leave you
with the links to said tools.

00:00:29.199 --> 00:00:31.679
Before that, just a little bit about me.

00:00:31.679 --> 00:00:40.399
I am the CEO and co-founder of a company
based in the Swiss mountains called 200ok.ch.

00:00:40.399 --> 00:00:44.879
We are a product incubator and
service consultancy,

00:00:44.879 --> 00:00:50.000
but we like to spend most or at least as
much time as we can

00:00:50.000 --> 00:00:52.719
building free software.

00:00:52.719 --> 00:00:56.879
I'm also an ordained Zen monk and abbot
of the Lambda Zen temple.

00:00:56.879 --> 00:01:04.159
You can reach me anytime on questions
regarding Emacs, for example,

00:01:04.159 --> 00:01:07.200
at alain@200ok.ch.

00:01:07.200 --> 00:01:09.439
But back to the topic at hand.

00:01:09.439 --> 00:01:11.760
The proposition is as following:

00:01:11.760 --> 00:01:14.000
most work on the computer is based on
either

00:01:14.000 --> 00:01:16.479
text processing or text consumption.

00:01:16.479 --> 00:01:22.799
And very often, the text which you need
to process is in a structured format,

00:01:22.799 --> 00:01:24.560
for example, in JSON.

00:01:24.560 --> 00:01:28.560
That might even be if your job is not
programming per se.

00:01:28.560 --> 00:01:33.119
Reading through such a bigger chunk of
JSON can be non-trivial, however,

00:01:33.119 --> 00:01:36.479
while just reading and understanding it

00:01:36.479 --> 00:01:40.320
will be essential to getting your job
done.

00:01:40.320 --> 00:01:44.479
So let's quickly check out an example
JSON file.

00:01:44.479 --> 00:01:47.200
This is from the Github API,

00:01:47.200 --> 00:01:52.079
which is a request--sorry, the
response to a request

00:01:52.079 --> 00:01:54.640
for a specific issue on the github API.

00:01:54.640 --> 00:01:58.799
So let's quickly check that one out.

00:01:58.799 --> 00:02:01.920
Okay. So here it is open, and we can
already see

00:02:01.920 --> 00:02:05.439
that there is lots of stuff
going on here.

00:02:05.439 --> 00:02:07.360
It's 200 lines.

00:02:07.360 --> 00:02:09.200
It's not going to be very easy

00:02:09.200 --> 00:02:11.840
just to find out what are the top level
things in here,

00:02:11.840 --> 00:02:13.360
what are the top level attributes.

00:02:13.360 --> 00:02:17.840
Of course I can do this, and maybe do it
by hand, but that doesn't scale.

00:02:17.840 --> 00:02:21.599
I can use cool Emacs facilities like the
hideshow-mode

00:02:21.599 --> 00:02:24.720
and try to fold all the things that are
top level,

00:02:24.720 --> 00:02:27.200
but that also doesn't really scale.

00:02:27.200 --> 00:02:29.360
There must be a better way.

00:02:29.360 --> 00:02:32.000
Of course there is. There is prior art.

00:02:32.000 --> 00:02:34.080
There is a tool called jq.

00:02:34.080 --> 00:02:37.760
I'm going to quote the USP (unique selling proposition) from their website:

00:02:37.760 --> 00:02:42.000
jq is like sed for JSON data.

00:02:42.000 --> 00:02:46.319
you can use it to slice and filter and
map and transform structured data

00:02:46.319 --> 00:02:47.840
with the same ease that

00:02:47.840 --> 00:02:54.000
sed, awk, grep, and friends let you
play with text.

00:02:54.000 --> 00:02:56.879
Let me give you a quick demonstration of
it.

00:02:56.879 --> 00:02:59.040
By the way, it's written in portable C.

00:02:59.040 --> 00:03:03.519
It has zero runtime dependency, so it's
very easy to get started with it

00:03:03.519 --> 00:03:09.840
and use it on pretty much any UNIX-based
computer.

00:03:09.840 --> 00:03:14.000
Sorry, no, Linux-based computer,
apologies.

00:03:14.000 --> 00:03:18.720
Okay, so let's explore a
JSON file with it.

00:03:18.720 --> 00:03:20.000
It's a command line tool,

00:03:20.000 --> 00:03:24.000
and it has a very simple command
line syntax.

00:03:24.000 --> 00:03:29.840
So you call the binary and then you give
it a query and a file,

00:03:29.840 --> 00:03:32.560
and then it will return its answer.

00:03:32.560 --> 00:03:35.440
So, for example, if I want the top
level keys,

00:03:35.440 --> 00:03:38.000
I will just say jq keys the file

00:03:38.000 --> 00:03:39.840
and it will return the keys.

00:03:39.840 --> 00:03:44.400
Simple as that. So let's check this out
in a real shell.

00:03:44.400 --> 00:03:46.879
Here I am in eshell.

00:03:46.879 --> 00:03:51.440
Let's run jq keys on the Github issue comment.

00:03:51.440 --> 00:03:58.799
We can see that we have actually
received a list back here

00:03:58.799 --> 00:04:00.319
with the top-level things.

00:04:00.319 --> 00:04:02.879
So this issue... It looks very interesting.

00:04:02.879 --> 00:04:07.360
Let's ask it to give me more information on this issue.

00:04:07.360 --> 00:04:11.360
Then it's hairy again. That's a lot of stuff.

00:04:11.360 --> 00:04:14.560
I mean, lucky for us, we are in Emacs here,

00:04:14.560 --> 00:04:16.720
so we can use nice shortcuts.

00:04:16.720 --> 00:04:22.000
We can copy this. We can go in here, just select that,

00:04:22.000 --> 00:04:24.160
get that out or something like this.

00:04:24.160 --> 00:04:32.320
But still, this is not really the best way to do that, right?

00:04:32.320 --> 00:04:34.080
it gets kind of tedious.

00:04:34.080 --> 00:04:37.680
At this point the output can be humongous.

00:04:37.680 --> 00:04:41.919
The shell is not really the best place to read through such big output.

00:04:41.919 --> 00:04:45.759
I mean, eshell is probably one of the better shells for this,

00:04:45.759 --> 00:04:47.919
because it's just a regular Emacs buffer,

00:04:47.919 --> 00:04:50.720
but still, it's not really the best tool.

00:04:50.720 --> 00:04:53.680
I need to repeat the command all the time

00:04:53.680 --> 00:04:56.000
until I finally build the right query.

00:04:56.000 --> 00:04:59.840
And all the time, I lose my focus,

00:04:59.840 --> 00:05:02.800
I lose what I'm currently looking at.

00:05:02.800 --> 00:05:05.520
I'm seeing the new result.

00:05:05.520 --> 00:05:08.160
It would be so much nicer to have live feedback.

00:05:08.160 --> 00:05:10.720
When working with Emacs, we're quite used to that.

00:05:10.720 --> 00:05:12.320
So there should be an option.

00:05:12.320 --> 00:05:15.120
And of course there is. It's Emacs, right,

00:05:15.120 --> 00:05:17.759
so you can do anything.

00:05:17.759 --> 00:05:22.960
There is various good tools for completion in Emacs.

00:05:22.960 --> 00:05:26.000
I used ivy for this.

00:05:26.000 --> 00:05:29.039
I'm going to quote the USP for ivy.

00:05:29.039 --> 00:05:32.639
ivy is a generic completion mechanism for Emacs.

00:05:32.639 --> 00:05:37.919
While it operates similarly to other completion schemes such as icomplete mode,

00:05:37.919 --> 00:05:42.160
ivy aims to be more efficient, smaller, simpler, and smoother to use,

00:05:42.160 --> 00:05:45.199
yet highly customizable.

00:05:45.199 --> 00:05:46.479
And that's true.

00:05:46.479 --> 00:05:49.440
One of the cool things of ivy

00:05:49.440 --> 00:05:54.320
compared to other completion mechanisms in Emacs

00:05:54.320 --> 00:05:59.120
is that it can be used on dynamic data.

00:05:59.120 --> 00:06:02.400
So usually completion works on a static input.

00:06:02.400 --> 00:06:05.360
For example, you're in a buffer, a text buffer,

00:06:05.360 --> 00:06:09.600
and you use isearch maybe with ido-mode,

00:06:09.600 --> 00:06:13.360
and you find your results. That's all nice.

00:06:13.360 --> 00:06:19.600
However, if I want to search on dynamic data,

00:06:19.600 --> 00:06:20.720
that doesn't work.

00:06:20.720 --> 00:06:24.880
So whenever I type in my query for jq,

00:06:24.880 --> 00:06:28.000
I actually need to call the jq binary,

00:06:28.000 --> 00:06:30.720
and it will give a different result set back.

00:06:30.720 --> 00:06:36.160
So it's a really dynamic mechanism that we need here.

00:06:36.160 --> 00:06:38.240
It's much more like a search engine.

00:06:38.240 --> 00:06:41.440
ivy luckily has something built in,

00:06:41.440 --> 00:06:43.520
and it's called counsel.

00:06:43.520 --> 00:06:47.360
So I used counsel and jq and combined them,

00:06:47.360 --> 00:06:49.199
and built a new package

00:06:49.199 --> 00:06:52.960
with which we can use Emacs and jq

00:06:52.960 --> 00:06:56.000
to have live feedback.

00:06:56.000 --> 00:06:57.759
It's very easy to use.

00:06:57.759 --> 00:06:59.840
So you just call counsel-jq

00:06:59.840 --> 00:07:02.160
on a buffer containing JSON.

00:07:02.160 --> 00:07:04.319
For example, the one we have here.

00:07:04.319 --> 00:07:06.800
Let's call counsel-jq on it,

00:07:06.800 --> 00:07:10.080
and we already get a default query,

00:07:10.080 --> 00:07:14.639
the dot query, which just gives us the same file.

00:07:14.639 --> 00:07:16.240
But now we can change it.

00:07:16.240 --> 00:07:18.639
For example, find all the keys in here.

00:07:18.639 --> 00:07:20.319
And then we see I had this issue.

00:07:20.319 --> 00:07:22.800
This was the one that we were interested in.

00:07:22.800 --> 00:07:25.599
So let's find more information on the issue.

00:07:25.599 --> 00:07:28.720
What keys does it have actually have?

00:07:28.720 --> 00:07:31.680
It has assignees. That interests me.

00:07:31.680 --> 00:07:34.800
So let's check out the assignees in here.

00:07:34.800 --> 00:07:39.759
There's two of them, but I'm only interested in the first one.

00:07:39.759 --> 00:07:43.599
I'm making stuff up as I go here, of course.

00:07:43.599 --> 00:07:47.039
Whenever I hit enter, I get a new buffer

00:07:47.039 --> 00:07:52.639
which just shows me this particular result

00:07:52.639 --> 00:07:55.599
for the particular query that I entered.

00:07:55.599 --> 00:07:57.680
So let me do that again.

00:07:57.680 --> 00:08:04.000
We are in here. We are looking at a JSON file.

00:08:04.000 --> 00:08:05.840
This can be very, very big.

00:08:05.840 --> 00:08:07.280
Doesn't also need to be a file.

00:08:07.280 --> 00:08:09.520
Just needs to be a buffer.

00:08:09.520 --> 00:08:11.360
You call counsel-jq on it,

00:08:11.360 --> 00:08:14.319
and you can do any kind of query on it.

00:08:14.319 --> 00:08:18.080
For example, let's see if there is a URL here.

00:08:18.080 --> 00:08:19.440
Yes, there's a URL.

00:08:19.440 --> 00:08:22.827
Let's see if there's a repository here.

00:08:22.827 --> 00:08:24.639
Repository. No, there isn't.

00:08:24.639 --> 00:08:33.440
What was it called? Issue. Keys. Repository URL, it was called.

00:08:33.440 --> 00:08:38.240
So let's see issue repository URL,

00:08:38.240 --> 00:08:39.519
and then we see.

00:08:39.519 --> 00:08:44.800
So apparently this issue comment is for a repository called organice.

00:08:44.800 --> 00:08:47.839
I wonder what that might be.

00:08:47.839 --> 00:08:52.640
Okay. So that was a very short introduction to counsel-jq.

00:08:52.640 --> 00:08:54.240
You can see the timer here.

00:08:54.240 --> 00:08:57.440
I only have one minute left to go, so I'm going to leave

00:08:57.440 --> 00:09:02.880
with a very, very short introduction to the counsel-jq code.

00:09:02.880 --> 00:09:06.000
It's not even 60 lines of elisp,

00:09:06.000 --> 00:09:09.600
so building something like this is very, very easy.

00:09:09.600 --> 00:09:14.560
I would encourage you to go and read through the code in your own time,

00:09:14.560 --> 00:09:17.519
if you're interested in building something like this.

00:09:17.519 --> 00:09:22.720
If you're interested in just using jq or you're done,

00:09:22.720 --> 00:09:24.320
these are the links to all the tools.

00:09:24.320 --> 00:09:28.240
counsel-jq, of course, is readily available on MELPA.

00:09:28.240 --> 00:09:32.959
Also developed under the AGPL license on Github.

00:09:32.959 --> 00:09:36.080
And this organice thing, by the way, it's

00:09:36.080 --> 00:09:38.560
Org Mode for mobile and desktop browsers.

00:09:38.560 --> 00:09:43.120
Also a great free software tool maybe that interests you.

00:09:43.120 --> 00:09:46.240
Thank you for listening. Have a great time.

00:09:46.240 --> 00:09:49.360
10 seconds left. I am going to stop this now.

00:09:49.360 --> 00:09:53.920
Enjoy EmacsConf. Have a great day.