path: root/2020/subtitles/emacsconf-2020--25-traverse-complex-json-structures-with-live-feedback-counsel-jq--zen-monk-alain-m-lafon-autogen.sbv

0:00:00.799,0:00:05.520
hello everyone and welcome to this short

0:00:04.000,0:00:08.160
lightning talk

0:00:05.520,0:00:09.519
traverse complex json structures with

0:00:08.160,0:00:13.040
live feedback

0:00:09.519,0:00:18.000
this is a pre-recorded talk and part of

0:00:13.040,0:00:19.920
the e-max conf 2020 schedule

0:00:18.000,0:00:21.840
this is what we're going to do i'll make

0:00:19.920,0:00:22.320
a quick introduction to the topic at

0:00:21.840,0:00:23.920
hand

0:00:22.320,0:00:25.760
i'll give you a demonstration of some

0:00:23.920,0:00:29.199
tools and then we'll leave you

0:00:25.760,0:00:31.920
with the links to set tools

0:00:29.199,0:00:32.399
before that just a little bit about me i

0:00:31.920,0:00:36.079
am the

0:00:32.399,0:00:39.520
ceo and co-founder of a company

0:00:36.079,0:00:40.399
based in the swiss mountains called 200

0:00:39.520,0:00:43.600
okay

0:00:40.399,0:00:46.160
we are a product incubator and

0:00:43.600,0:00:47.200
service consultancy but we like to spend

0:00:46.160,0:00:50.719
most or at least

0:00:47.200,0:00:52.719
as much time as we can building free

0:00:50.719,0:00:55.520
software

0:00:52.719,0:00:56.879
i'm also an ordained zen monk and abbott

0:00:55.520,0:01:00.160
of the lambda zen temple

0:00:56.879,0:01:01.359
you can reach me anytime on questions

0:01:00.160,0:01:05.860
regarding emacs

0:01:01.359,0:01:07.200
for example at ala at 200ok.ch

0:01:05.860,0:01:09.600
[Music]

0:01:07.200,0:01:11.760
but back to the topic at hand the

0:01:09.600,0:01:13.680
proposition is as following

0:01:11.760,0:01:15.520
most work on the computer is based on

0:01:13.680,0:01:16.479
either text processing or text

0:01:15.520,0:01:19.920
consumption

0:01:16.479,0:01:20.880
and very often the text which you need

0:01:19.920,0:01:23.520
to process

0:01:20.880,0:01:24.560
is in a structured format for example in

0:01:23.520,0:01:26.640
json

0:01:24.560,0:01:28.560
that might even be if your job is not

0:01:26.640,0:01:30.400
programming per se

0:01:28.560,0:01:33.119
and reading through such a bigger chunk

0:01:30.400,0:01:36.479
of json can be non-trivial however

0:01:33.119,0:01:39.119
while just reading and understanding it

0:01:36.479,0:01:40.320
will be essential to getting your job

0:01:39.119,0:01:43.680
done

0:01:40.320,0:01:47.200
so let's quickly check out an example

0:01:43.680,0:01:50.560
json file this is from the github api

0:01:47.200,0:01:53.119
where which is a request sorry the

0:01:50.560,0:01:55.280
response to a request for a specific

0:01:53.119,0:01:58.560
issue on the github api so let's

0:01:55.280,0:01:58.560
quickly check that one out

0:01:58.799,0:02:03.280
okay so here it is open and we can

0:02:01.280,0:02:05.439
already see that there is lots of stuff

0:02:03.280,0:02:08.319
going on here

0:02:05.439,0:02:10.319
it's uh 200 lines it's not going to be

0:02:08.319,0:02:12.160
very easy just just to find out what are

0:02:10.319,0:02:13.920
the top level things in here what are

0:02:12.160,0:02:14.560
the top level attributes of course i can

0:02:13.920,0:02:16.480
do this

0:02:14.560,0:02:17.840
and maybe it would by hand but that

0:02:16.480,0:02:20.560
doesn't scale

0:02:17.840,0:02:22.560
i can use cool emacs facilities like the

0:02:20.560,0:02:24.720
height show mode and try to

0:02:22.560,0:02:27.200
fold all the things that are top level

0:02:24.720,0:02:29.760
but that also doesn't really scale

0:02:27.200,0:02:32.000
there must be a better way and of course

0:02:29.760,0:02:34.959
there is there is prior art

0:02:32.000,0:02:36.080
there is a tool called jq i'm going to

0:02:34.959,0:02:39.120
quote the usp

0:02:36.080,0:02:42.959
from their website jq is like

0:02:39.120,0:02:44.720
sed said for json data you can use it to

0:02:42.959,0:02:46.720
slice and filter and map

0:02:44.720,0:02:47.840
and transform structured data with the

0:02:46.720,0:02:51.519
same ease that

0:02:47.840,0:02:53.920
z awk grep and friends let you play with

0:02:51.519,0:02:53.920
text

0:02:54.000,0:02:57.519
let me give you a quick demonstration of

0:02:56.160,0:02:59.680
it by the way

0:02:57.519,0:03:01.920
it's written in portable c it has zero

0:02:59.680,0:03:04.959
runtime dependency so it's very easy

0:03:01.920,0:03:09.040
to get started with it and use it on

0:03:04.959,0:03:12.800
pretty much any unix-based

0:03:09.040,0:03:16.159
computer sorry no linux based computer

0:03:12.800,0:03:19.599
apologies okay so let's explore a

0:03:16.159,0:03:21.120
json file with it it's a command line

0:03:19.599,0:03:24.640
tool and it has a very

0:03:21.120,0:03:26.799
simple command line syntax so you

0:03:24.640,0:03:27.840
call the binary and then you give it a

0:03:26.799,0:03:31.680
query

0:03:27.840,0:03:33.840
and a file and then we'll return

0:03:31.680,0:03:35.440
its answer so for example if i want the

0:03:33.840,0:03:38.319
top level keys

0:03:35.440,0:03:39.840
i will just say jq keys the file and it

0:03:38.319,0:03:42.319
will return the keys

0:03:39.840,0:03:43.519
simple as that so let's check this out

0:03:42.319,0:03:46.879
in a real

0:03:43.519,0:03:50.080
shell here i am in eshel

0:03:46.879,0:03:53.200
let's run jq keys on the github

0:03:50.080,0:03:56.959
issue comment and we can see

0:03:53.200,0:04:00.000
that we have actually received a list

0:03:56.959,0:04:02.000
back here with the top level

0:04:00.000,0:04:04.319
things okay so this issue it looks very

0:04:02.000,0:04:06.239
very interesting so let's uh ask it to

0:04:04.319,0:04:10.080
give me more information on this

0:04:06.239,0:04:11.360
issue then it's hairy again that's a lot

0:04:10.080,0:04:14.319
of stuff

0:04:11.360,0:04:17.759
i mean lucky for for us we are in emacs

0:04:14.319,0:04:21.040
here so we can use nice shortcuts we can

0:04:17.759,0:04:22.000
copy this we go can go in here just

0:04:21.040,0:04:24.400
select that

0:04:22.000,0:04:25.600
get that out or something like this but

0:04:24.400,0:04:28.320
still

0:04:25.600,0:04:29.600
this is not really it's not really the

0:04:28.320,0:04:34.080
best way

0:04:29.600,0:04:36.639
to do that right it gets kind of tedious

0:04:34.080,0:04:37.680
at this point the output can be

0:04:36.639,0:04:39.360
humongous

0:04:37.680,0:04:42.479
the shell is not really the best place

0:04:39.360,0:04:43.600
to read through such big output i mean e

0:04:42.479,0:04:45.440
shell is

0:04:43.600,0:04:47.440
probably one of the better shells for

0:04:45.440,0:04:48.560
this because it's just a regular emacs

0:04:47.440,0:04:51.199
buffer but still

0:04:48.560,0:04:52.720
it's not really the best tool and i need

0:04:51.199,0:04:55.280
to repeat the command

0:04:52.720,0:04:56.000
all the time until i finally build the

0:04:55.280,0:04:59.120
right query

0:04:56.000,0:05:02.080
and all the time i use i lose my

0:04:59.120,0:05:02.800
focus i lose what i'm currently looking

0:05:02.080,0:05:06.080
at

0:05:02.800,0:05:08.160
i'm seeing the new result it would be so

0:05:06.080,0:05:10.080
much nicer to have live feedback

0:05:08.160,0:05:11.600
and when working with emacs we're quite

0:05:10.080,0:05:12.320
used to that so there should be an

0:05:11.600,0:05:15.120
option

0:05:12.320,0:05:17.759
and of course there is it's emacs right

0:05:15.120,0:05:20.880
so you can do anything

0:05:17.759,0:05:22.960
there is various good tools for

0:05:20.880,0:05:26.000
completion in emacs

0:05:22.960,0:05:29.039
i used ivy for this

0:05:26.000,0:05:31.840
i'm going to code the usp for ivy

0:05:29.039,0:05:32.639
iv is a generic completion mechanism for

0:05:31.840,0:05:34.639
emacs

0:05:32.639,0:05:35.919
while it operates similarly to other

0:05:34.639,0:05:38.960
completion schemes

0:05:35.919,0:05:40.400
such as icomplete mode iv aims to be

0:05:38.960,0:05:43.120
more efficient smaller

0:05:40.400,0:05:45.199
simpler and smoother to use yet highly

0:05:43.120,0:05:47.840
customizable

0:05:45.199,0:05:49.440
and that's true one of the cool things

0:05:47.840,0:05:52.479
of iv

0:05:49.440,0:05:54.320
compared to other completion mechanisms

0:05:52.479,0:05:58.160
in emacs

0:05:54.320,0:06:01.600
is that it can be used on dynamic

0:05:58.160,0:06:03.520
data so usually completion works on a

0:06:01.600,0:06:05.360
static input for example you're in a

0:06:03.520,0:06:08.800
buffer a text buffer

0:06:05.360,0:06:09.600
and you use i search maybe with idle

0:06:08.800,0:06:12.400
mode

0:06:09.600,0:06:13.360
and you find your results that's all

0:06:12.400,0:06:16.960
nice

0:06:13.360,0:06:19.840
however if i

0:06:16.960,0:06:20.720
want to search on dynamic data that

0:06:19.840,0:06:23.759
doesn't work

0:06:20.720,0:06:27.039
so whenever i type in my query

0:06:23.759,0:06:28.000
for jq i actually need to call the jq

0:06:27.039,0:06:30.160
binary

0:06:28.000,0:06:32.840
and it will give a different result set

0:06:30.160,0:06:36.319
back so it's a really

0:06:32.840,0:06:38.240
dynamic mechanism that we need here it's

0:06:36.319,0:06:41.440
much more like a search engine

0:06:38.240,0:06:42.560
and ivy luckily has something built in

0:06:41.440,0:06:46.000
and it's called

0:06:42.560,0:06:48.160
console so i used council and jq

0:06:46.000,0:06:49.199
and combined them and build a new

0:06:48.160,0:06:52.960
package

0:06:49.199,0:06:56.000
with which we can use emacs and jq

0:06:52.960,0:06:58.800
to have live feedback

0:06:56.000,0:06:59.840
it's very easy to use so you just call

0:06:58.800,0:07:02.800
console.jq

0:06:59.840,0:07:04.319
on a buffer containing json for example

0:07:02.800,0:07:07.280
the one we have here

0:07:04.319,0:07:08.319
let's call console.jq on it and we

0:07:07.280,0:07:11.280
already

0:07:08.319,0:07:13.039
get a default query the dot query which

0:07:11.280,0:07:16.080
just gives us

0:07:13.039,0:07:17.280
the same file but now we can change it

0:07:16.080,0:07:19.039
and for example

0:07:17.280,0:07:21.039
find all the keys in here and then we

0:07:19.039,0:07:22.800
see i had this issue this was the one

0:07:21.039,0:07:24.639
that we were interested in

0:07:22.800,0:07:26.479
so let's find more information on the

0:07:24.639,0:07:28.800
issue what

0:07:26.479,0:07:31.680
keys does it have actually have uh it

0:07:28.800,0:07:34.800
has assignees that that interests me

0:07:31.680,0:07:37.039
so let's check out the assignees in here

0:07:34.800,0:07:39.759
there's two of them but i'm only

0:07:37.039,0:07:42.000
interested in the first one

0:07:39.759,0:07:43.599
i'm making stuff up as i go here of

0:07:42.000,0:07:46.000
course

0:07:43.599,0:07:47.039
and whenever i hit enter i get a new

0:07:46.000,0:07:50.160
buffer

0:07:47.039,0:07:53.520
which just shows me this

0:07:50.160,0:07:55.599
particular result for the particular

0:07:53.520,0:07:59.199
query that i entered

0:07:55.599,0:08:03.199
so let me do that again we are in here

0:07:59.199,0:08:05.840
we are looking at a

0:08:03.199,0:08:07.520
json file this can be very very big

0:08:05.840,0:08:09.520
doesn't also need to be a file just

0:08:07.520,0:08:12.160
needs to be a buffer

0:08:09.520,0:08:13.599
you call console.jq on it and you can do

0:08:12.160,0:08:16.479
any kind of

0:08:13.599,0:08:18.080
query on it for example let's see if

0:08:16.479,0:08:20.319
there is a url here

0:08:18.080,0:08:23.759
yes there's a url let's see if there's a

0:08:20.319,0:08:26.879
repository here repository

0:08:23.759,0:08:33.360
no there isn't what was it called issue

0:08:26.879,0:08:33.360
keys repository url it was called okay

0:08:33.440,0:08:40.240
so let's see issue pause

0:08:36.640,0:08:42.959
url and then we see so

0:08:40.240,0:08:44.159
apparently this issue command is for a

0:08:42.959,0:08:47.839
repository called

0:08:44.159,0:08:50.320
organize i wonder what that might be

0:08:47.839,0:08:52.640
okay so that was a very short

0:08:50.320,0:08:54.800
introduction to console jq

0:08:52.640,0:08:56.800
you can see the timer here i only have

0:08:54.800,0:08:57.440
one minute left to go so i'm going to

0:08:56.800,0:08:59.760
leave

0:08:57.440,0:09:01.519
with a very very short introduction to

0:08:59.760,0:09:05.040
the console gq

0:09:01.519,0:09:07.519
code it's not even 60 lines

0:09:05.040,0:09:09.600
of e-lisp so building something like

0:09:07.519,0:09:12.800
this is very very easy

0:09:09.600,0:09:14.560
i would encourage you to go and read

0:09:12.800,0:09:16.160
through the code in your own time

0:09:14.560,0:09:18.320
if you're interested in building

0:09:16.160,0:09:19.600
something like this if you're interested

0:09:18.320,0:09:22.720
in just using

0:09:19.600,0:09:24.320
jq or you're done

0:09:22.720,0:09:25.839
these are the links to all the tools

0:09:24.320,0:09:28.800
console.jq of course

0:09:25.839,0:09:29.519
is readily available on melpah also

0:09:28.800,0:09:32.959
developed

0:09:29.519,0:09:36.080
under the agpl license on github

0:09:32.959,0:09:37.839
and disorganized thing by the way it's

0:09:36.080,0:09:39.839
orig mode for mobile and desktop

0:09:37.839,0:09:42.399
browsers also a great

0:09:39.839,0:09:43.120
free software tool maybe that interests

0:09:42.399,0:09:46.240
you

0:09:43.120,0:09:48.800
thank you for listening have great time

0:09:46.240,0:09:49.360
10 seconds left i am going to stop this

0:09:48.800,0:09:53.920
now

0:09:49.360,0:09:53.920
enjoy emacs conf have a great day