WEBVTT

00:00:00.799 --> 00:00:04.000
hello everyone and welcome to this short

00:00:04.000 --> 00:00:05.520
lightning talk

00:00:05.520 --> 00:00:08.160
traverse complex json structures with

00:00:08.160 --> 00:00:09.519
live feedback

00:00:09.519 --> 00:00:13.040
this is a pre-recorded talk and part of

00:00:13.040 --> 00:00:18.000
the e-max conf 2020 schedule

00:00:18.000 --> 00:00:19.920
this is what we're going to do I'll make

00:00:19.920 --> 00:00:21.840
a quick introduction to the topic at

00:00:21.840 --> 00:00:22.320
hand

00:00:22.320 --> 00:00:23.920
I'll give you a demonstration of some

00:00:23.920 --> 00:00:25.760
tools and then we'll leave you

00:00:25.760 --> 00:00:29.199
with the links to set tools

00:00:29.199 --> 00:00:31.920
before that just a little bit about me I

00:00:31.920 --> 00:00:32.399
am the

00:00:32.399 --> 00:00:36.079
ceo and co-founder of a company

00:00:36.079 --> 00:00:39.520
based in the swiss mountains called 200

00:00:39.520 --> 00:00:40.399
okay

00:00:40.399 --> 00:00:43.600
we are a product incubator and

00:00:43.600 --> 00:00:46.160
service consultancy but we like to spend

00:00:46.160 --> 00:00:47.200
most or at least

00:00:47.200 --> 00:00:50.719
as much time as we can building free

00:00:50.719 --> 00:00:52.719
software

00:00:52.719 --> 00:00:55.520
I'm also an ordained zen monk and abbott

00:00:55.520 --> 00:00:56.879
of the lambda zen temple

00:00:56.879 --> 00:01:00.160
you can reach me anytime on questions

00:01:00.160 --> 00:01:01.359
regarding Emacs

00:01:01.359 --> 00:01:05.860
for example at ala at 200ok.ch

00:01:05.860 --> 00:01:07.200
[Music]

00:01:07.200 --> 00:01:09.600
but back to the topic at hand the

00:01:09.600 --> 00:01:11.760
proposition is as following

00:01:11.760 --> 00:01:13.680
most work on the computer is based on

00:01:13.680 --> 00:01:15.520
either text processing or text

00:01:15.520 --> 00:01:16.479
consumption

00:01:16.479 --> 00:01:19.920
and very often the text which you need

00:01:19.920 --> 00:01:20.880
to process

00:01:20.880 --> 00:01:23.520
is in a structured format for example in

00:01:23.520 --> 00:01:24.560
json

00:01:24.560 --> 00:01:26.640
that might even be if your job is not

00:01:26.640 --> 00:01:28.560
programming per se

00:01:28.560 --> 00:01:30.400
and reading through such a bigger chunk

00:01:30.400 --> 00:01:33.119
of json can be non-trivial however

00:01:33.119 --> 00:01:36.479
while just reading and understanding it

00:01:36.479 --> 00:01:39.119
will be essential to getting your job

00:01:39.119 --> 00:01:40.320
done

00:01:40.320 --> 00:01:43.680
so let's quickly check out an example

00:01:43.680 --> 00:01:47.200
json file this is from the github api

00:01:47.200 --> 00:01:50.560
where which is a request sorry the

00:01:50.560 --> 00:01:53.119
response to a request for a specific

00:01:53.119 --> 00:01:55.280
issue on the github api so let's

00:01:55.280 --> 00:01:58.799
quickly check that one out

00:01:58.799 --> 00:02:01.280
okay so here it is open and we can

00:02:01.280 --> 00:02:03.280
already see that there is lots of stuff

00:02:03.280 --> 00:02:05.439
going on here

00:02:05.439 --> 00:02:08.319
it's uh 200 lines it's not going to be

00:02:08.319 --> 00:02:10.319
very easy just just to find out what are

00:02:10.319 --> 00:02:12.160
the top level things in here what are

00:02:12.160 --> 00:02:13.920
the top level attributes of course I can

00:02:13.920 --> 00:02:14.560
do this

00:02:14.560 --> 00:02:16.480
and maybe it would by hand but that

00:02:16.480 --> 00:02:17.840
doesn't scale

00:02:17.840 --> 00:02:20.560
I can use cool Emacs facilities like the

00:02:20.560 --> 00:02:22.560
height show mode and try to

00:02:22.560 --> 00:02:24.720
fold all the things that are top level

00:02:24.720 --> 00:02:27.200
but that also doesn't really scale

00:02:27.200 --> 00:02:29.760
there must be a better way and of course

00:02:29.760 --> 00:02:32.000
there is there is prior art

00:02:32.000 --> 00:02:34.959
there is a tool called jq I'm going to

00:02:34.959 --> 00:02:36.080
quote the usp

00:02:36.080 --> 00:02:39.120
from their website jq is like

00:02:39.120 --> 00:02:42.959
sed said for json data you can use it to

00:02:42.959 --> 00:02:44.720
slice and filter and map

00:02:44.720 --> 00:02:46.720
and transform structured data with the

00:02:46.720 --> 00:02:47.840
same ease that

00:02:47.840 --> 00:02:51.519
z awk grep and friends let you play with

00:02:51.519 --> 00:02:54.000
text

00:02:54.000 --> 00:02:56.160
let me give you a quick demonstration of

00:02:56.160 --> 00:02:57.519
it by the way

00:02:57.519 --> 00:02:59.680
it's written in portable c it has zero

00:02:59.680 --> 00:03:01.920
runtime dependency so it's very easy

00:03:01.920 --> 00:03:04.959
to get started with it and use it on

00:03:04.959 --> 00:03:09.040
pretty much any unix-based

00:03:09.040 --> 00:03:12.800
computer sorry no linux based computer

00:03:12.800 --> 00:03:16.159
apologies okay so let's explore a

00:03:16.159 --> 00:03:19.599
json file with it it's a command line

00:03:19.599 --> 00:03:21.120
tool and it has a very

00:03:21.120 --> 00:03:24.640
simple command line syntax so you

00:03:24.640 --> 00:03:26.799
call the binary and then you give it a

00:03:26.799 --> 00:03:27.840
query

00:03:27.840 --> 00:03:31.680
and a file and then we'll return

00:03:31.680 --> 00:03:33.840
its answer so for example if I want the

00:03:33.840 --> 00:03:35.440
top level keys

00:03:35.440 --> 00:03:38.319
I will just say jq keys the file and it

00:03:38.319 --> 00:03:39.840
will return the keys

00:03:39.840 --> 00:03:42.319
simple as that so let's check this out

00:03:42.319 --> 00:03:43.519
in a real

00:03:43.519 --> 00:03:46.879
shell here I am in eshel

00:03:46.879 --> 00:03:50.080
let's run jq keys on the github

00:03:50.080 --> 00:03:53.200
issue comment and we can see

00:03:53.200 --> 00:03:56.959
that we have actually received a list

00:03:56.959 --> 00:04:00.000
back here with the top level

00:04:00.000 --> 00:04:02.000
things okay so this issue it looks very

00:04:02.000 --> 00:04:04.319
very interesting so let's uh ask it to

00:04:04.319 --> 00:04:06.239
give me more information on this

00:04:06.239 --> 00:04:10.080
issue then it's hairy again that's a lot

00:04:10.080 --> 00:04:11.360
of stuff

00:04:11.360 --> 00:04:14.319
I mean lucky for for us we are in Emacs

00:04:14.319 --> 00:04:17.759
here so we can use nice shortcuts we can

00:04:17.759 --> 00:04:21.040
copy this we go can go in here just

00:04:21.040 --> 00:04:22.000
select that

00:04:22.000 --> 00:04:24.400
get that out or something like this but

00:04:24.400 --> 00:04:25.600
still

00:04:25.600 --> 00:04:28.320
this is not really it's not really the

00:04:28.320 --> 00:04:29.600
best way

00:04:29.600 --> 00:04:34.080
to do that right it gets kind of tedious

00:04:34.080 --> 00:04:36.639
at this point the output can be

00:04:36.639 --> 00:04:37.680
humongous

00:04:37.680 --> 00:04:39.360
the shell is not really the best place

00:04:39.360 --> 00:04:42.479
to read through such big output I mean e

00:04:42.479 --> 00:04:43.600
shell is

00:04:43.600 --> 00:04:45.440
probably one of the better shells for

00:04:45.440 --> 00:04:47.440
this because it's just a regular Emacs

00:04:47.440 --> 00:04:48.560
buffer but still

00:04:48.560 --> 00:04:51.199
it's not really the best tool and I need

00:04:51.199 --> 00:04:52.720
to repeat the command

00:04:52.720 --> 00:04:55.280
all the time until I finally build the

00:04:55.280 --> 00:04:56.000
right query

00:04:56.000 --> 00:04:59.120
and all the time I use I lose my

00:04:59.120 --> 00:05:02.080
focus I lose what I'm currently looking

00:05:02.080 --> 00:05:02.800
at

00:05:02.800 --> 00:05:06.080
I'm seeing the new result it would be so

00:05:06.080 --> 00:05:08.160
much nicer to have live feedback

00:05:08.160 --> 00:05:10.080
and when working with Emacs we're quite

00:05:10.080 --> 00:05:11.600
used to that so there should be an

00:05:11.600 --> 00:05:12.320
option

00:05:12.320 --> 00:05:15.120
and of course there is it's Emacs right

00:05:15.120 --> 00:05:17.759
so you can do anything

00:05:17.759 --> 00:05:20.880
there is various good tools for

00:05:20.880 --> 00:05:22.960
completion in Emacs

00:05:22.960 --> 00:05:26.000
I used ivy for this

00:05:26.000 --> 00:05:29.039
I'm going to code the usp for ivy

00:05:29.039 --> 00:05:31.840
iv is a generic completion mechanism for

00:05:31.840 --> 00:05:32.639
Emacs

00:05:32.639 --> 00:05:34.639
while it operates similarly to other

00:05:34.639 --> 00:05:35.919
completion schemes

00:05:35.919 --> 00:05:38.960
such as icomplete mode iv aims to be

00:05:38.960 --> 00:05:40.400
more efficient smaller

00:05:40.400 --> 00:05:43.120
simpler and smoother to use yet highly

00:05:43.120 --> 00:05:45.199
customizable

00:05:45.199 --> 00:05:47.840
and that's true one of the cool things

00:05:47.840 --> 00:05:49.440
of iv

00:05:49.440 --> 00:05:52.479
compared to other completion mechanisms

00:05:52.479 --> 00:05:54.320
in Emacs

00:05:54.320 --> 00:05:58.160
is that it can be used on dynamic

00:05:58.160 --> 00:06:01.600
data so usually completion works on a

00:06:01.600 --> 00:06:03.520
static input for example you're in a

00:06:03.520 --> 00:06:05.360
buffer a text buffer

00:06:05.360 --> 00:06:08.800
and you use I search maybe with idle

00:06:08.800 --> 00:06:09.600
mode

00:06:09.600 --> 00:06:12.400
and you find your results that's all

00:06:12.400 --> 00:06:13.360
nice

00:06:13.360 --> 00:06:16.960
however if I

00:06:16.960 --> 00:06:19.840
want to search on dynamic data that

00:06:19.840 --> 00:06:20.720
doesn't work

00:06:20.720 --> 00:06:23.759
so whenever I type in my query

00:06:23.759 --> 00:06:27.039
for jq I actually need to call the jq

00:06:27.039 --> 00:06:28.000
binary

00:06:28.000 --> 00:06:30.160
and it will give a different result set

00:06:30.160 --> 00:06:32.840
back so it's a really

00:06:32.840 --> 00:06:36.319
dynamic mechanism that we need here it's

00:06:36.319 --> 00:06:38.240
much more like a search engine

00:06:38.240 --> 00:06:41.440
and ivy luckily has something built in

00:06:41.440 --> 00:06:42.560
and it's called

00:06:42.560 --> 00:06:46.000
console so I used council and jq

00:06:46.000 --> 00:06:48.160
and combined them and build a new

00:06:48.160 --> 00:06:49.199
package

00:06:49.199 --> 00:06:52.960
with which we can use Emacs and jq

00:06:52.960 --> 00:06:56.000
to have live feedback

00:06:56.000 --> 00:06:58.800
it's very easy to use so you just call

00:06:58.800 --> 00:06:59.840
console.jq

00:06:59.840 --> 00:07:02.800
on a buffer containing json for example

00:07:02.800 --> 00:07:04.319
the one we have here

00:07:04.319 --> 00:07:07.280
let's call console.jq on it and we

00:07:07.280 --> 00:07:08.319
already

00:07:08.319 --> 00:07:11.280
get a default query the dot query which

00:07:11.280 --> 00:07:13.039
just gives us

00:07:13.039 --> 00:07:16.080
the same file but now we can change it

00:07:16.080 --> 00:07:17.280
and for example

00:07:17.280 --> 00:07:19.039
find all the keys in here and then we

00:07:19.039 --> 00:07:21.039
see I had this issue this was the one

00:07:21.039 --> 00:07:22.800
that we were interested in

00:07:22.800 --> 00:07:24.639
so let's find more information on the

00:07:24.639 --> 00:07:26.479
issue what

00:07:26.479 --> 00:07:28.800
keys does it have actually have uh it

00:07:28.800 --> 00:07:31.680
has assignees that that interests me

00:07:31.680 --> 00:07:34.800
so let's check out the assignees in here

00:07:34.800 --> 00:07:37.039
there's two of them but I'm only

00:07:37.039 --> 00:07:39.759
interested in the first one

00:07:39.759 --> 00:07:42.000
I'm making stuff up as I go here of

00:07:42.000 --> 00:07:43.599
course

00:07:43.599 --> 00:07:46.000
and whenever I hit enter I get a new

00:07:46.000 --> 00:07:47.039
buffer

00:07:47.039 --> 00:07:50.160
which just shows me this

00:07:50.160 --> 00:07:53.520
particular result for the particular

00:07:53.520 --> 00:07:55.599
query that I entered

00:07:55.599 --> 00:07:59.199
so let me do that again we are in here

00:07:59.199 --> 00:08:03.199
we are looking at a

00:08:03.199 --> 00:08:05.840
json file this can be very very big

00:08:05.840 --> 00:08:07.520
doesn't also need to be a file just

00:08:07.520 --> 00:08:09.520
needs to be a buffer

00:08:09.520 --> 00:08:12.160
you call console.jq on it and you can do

00:08:12.160 --> 00:08:13.599
any kind of

00:08:13.599 --> 00:08:16.479
query on it for example let's see if

00:08:16.479 --> 00:08:18.080
there is a url here

00:08:18.080 --> 00:08:20.319
yes there's a url let's see if there's a

00:08:20.319 --> 00:08:23.759
repository here repository

00:08:23.759 --> 00:08:26.879
no there isn't what was it called issue

00:08:26.879 --> 00:08:33.440
keys repository url it was called okay

00:08:33.440 --> 00:08:36.640
so let's see issue pause

00:08:36.640 --> 00:08:40.240
url and then we see so

00:08:40.240 --> 00:08:42.959
apparently this issue command is for a

00:08:42.959 --> 00:08:44.159
repository called

00:08:44.159 --> 00:08:47.839
organize I wonder what that might be

00:08:47.839 --> 00:08:50.320
okay so that was a very short

00:08:50.320 --> 00:08:52.640
introduction to console jq

00:08:52.640 --> 00:08:54.800
you can see the timer here I only have

00:08:54.800 --> 00:08:56.800
one minute left to go so I'm going to

00:08:56.800 --> 00:08:57.440
leave

00:08:57.440 --> 00:08:59.760
with a very very short introduction to

00:08:59.760 --> 00:09:01.519
the console gq

00:09:01.519 --> 00:09:05.040
code it's not even 60 lines

00:09:05.040 --> 00:09:07.519
of e-lisp so building something like

00:09:07.519 --> 00:09:09.600
this is very very easy

00:09:09.600 --> 00:09:12.800
I would encourage you to go and read

00:09:12.800 --> 00:09:14.560
through the code in your own time

00:09:14.560 --> 00:09:16.160
if you're interested in building

00:09:16.160 --> 00:09:18.320
something like this if you're interested

00:09:18.320 --> 00:09:19.600
in just using

00:09:19.600 --> 00:09:22.720
jq or you're done

00:09:22.720 --> 00:09:24.320
these are the links to all the tools

00:09:24.320 --> 00:09:25.839
console.jq of course

00:09:25.839 --> 00:09:28.800
is readily available on melpah also

00:09:28.800 --> 00:09:29.519
developed

00:09:29.519 --> 00:09:32.959
under the agpl license on github

00:09:32.959 --> 00:09:36.080
and disorganized thing by the way it's

00:09:36.080 --> 00:09:37.839
orig mode for mobile and desktop

00:09:37.839 --> 00:09:39.839
browsers also a great

00:09:39.839 --> 00:09:42.399
free software tool maybe that interests

00:09:42.399 --> 00:09:43.120
you

00:09:43.120 --> 00:09:46.240
thank you for listening have great time

00:09:46.240 --> 00:09:48.800
10 seconds left I am going to stop this

00:09:48.800 --> 00:09:49.360
now

00:09:49.360 --> 00:09:53.920
enjoy EmacsConf have a great day