blob: e7eada5b8257ab51e544a07667d88bfa0d4a32e1 (
plain) (
tree)
|
|
WEBVTT
00:00:00.799 --> 00:00:04.000
hello everyone and welcome to this short
00:00:04.000 --> 00:00:05.520
lightning talk
00:00:05.520 --> 00:00:08.160
traverse complex json structures with
00:00:08.160 --> 00:00:09.519
live feedback
00:00:09.519 --> 00:00:13.040
this is a pre-recorded talk and part of
00:00:13.040 --> 00:00:18.000
the e-max conf 2020 schedule
00:00:18.000 --> 00:00:19.920
this is what we're going to do I'll make
00:00:19.920 --> 00:00:21.840
a quick introduction to the topic at
00:00:21.840 --> 00:00:22.320
hand
00:00:22.320 --> 00:00:23.920
I'll give you a demonstration of some
00:00:23.920 --> 00:00:25.760
tools and then we'll leave you
00:00:25.760 --> 00:00:29.199
with the links to set tools
00:00:29.199 --> 00:00:31.920
before that just a little bit about me I
00:00:31.920 --> 00:00:32.399
am the
00:00:32.399 --> 00:00:36.079
ceo and co-founder of a company
00:00:36.079 --> 00:00:39.520
based in the swiss mountains called 200
00:00:39.520 --> 00:00:40.399
okay
00:00:40.399 --> 00:00:43.600
we are a product incubator and
00:00:43.600 --> 00:00:46.160
service consultancy but we like to spend
00:00:46.160 --> 00:00:47.200
most or at least
00:00:47.200 --> 00:00:50.719
as much time as we can building free
00:00:50.719 --> 00:00:52.719
software
00:00:52.719 --> 00:00:55.520
I'm also an ordained zen monk and abbott
00:00:55.520 --> 00:00:56.879
of the lambda zen temple
00:00:56.879 --> 00:01:00.160
you can reach me anytime on questions
00:01:00.160 --> 00:01:01.359
regarding Emacs
00:01:01.359 --> 00:01:05.860
for example at ala at 200ok.ch
00:01:05.860 --> 00:01:07.200
[Music]
00:01:07.200 --> 00:01:09.600
but back to the topic at hand the
00:01:09.600 --> 00:01:11.760
proposition is as following
00:01:11.760 --> 00:01:13.680
most work on the computer is based on
00:01:13.680 --> 00:01:15.520
either text processing or text
00:01:15.520 --> 00:01:16.479
consumption
00:01:16.479 --> 00:01:19.920
and very often the text which you need
00:01:19.920 --> 00:01:20.880
to process
00:01:20.880 --> 00:01:23.520
is in a structured format for example in
00:01:23.520 --> 00:01:24.560
json
00:01:24.560 --> 00:01:26.640
that might even be if your job is not
00:01:26.640 --> 00:01:28.560
programming per se
00:01:28.560 --> 00:01:30.400
and reading through such a bigger chunk
00:01:30.400 --> 00:01:33.119
of json can be non-trivial however
00:01:33.119 --> 00:01:36.479
while just reading and understanding it
00:01:36.479 --> 00:01:39.119
will be essential to getting your job
00:01:39.119 --> 00:01:40.320
done
00:01:40.320 --> 00:01:43.680
so let's quickly check out an example
00:01:43.680 --> 00:01:47.200
json file this is from the github api
00:01:47.200 --> 00:01:50.560
where which is a request sorry the
00:01:50.560 --> 00:01:53.119
response to a request for a specific
00:01:53.119 --> 00:01:55.280
issue on the github api so let's
00:01:55.280 --> 00:01:58.799
quickly check that one out
00:01:58.799 --> 00:02:01.280
okay so here it is open and we can
00:02:01.280 --> 00:02:03.280
already see that there is lots of stuff
00:02:03.280 --> 00:02:05.439
going on here
00:02:05.439 --> 00:02:08.319
it's uh 200 lines it's not going to be
00:02:08.319 --> 00:02:10.319
very easy just just to find out what are
00:02:10.319 --> 00:02:12.160
the top level things in here what are
00:02:12.160 --> 00:02:13.920
the top level attributes of course I can
00:02:13.920 --> 00:02:14.560
do this
00:02:14.560 --> 00:02:16.480
and maybe it would by hand but that
00:02:16.480 --> 00:02:17.840
doesn't scale
00:02:17.840 --> 00:02:20.560
I can use cool Emacs facilities like the
00:02:20.560 --> 00:02:22.560
height show mode and try to
00:02:22.560 --> 00:02:24.720
fold all the things that are top level
00:02:24.720 --> 00:02:27.200
but that also doesn't really scale
00:02:27.200 --> 00:02:29.760
there must be a better way and of course
00:02:29.760 --> 00:02:32.000
there is there is prior art
00:02:32.000 --> 00:02:34.959
there is a tool called jq I'm going to
00:02:34.959 --> 00:02:36.080
quote the usp
00:02:36.080 --> 00:02:39.120
from their website jq is like
00:02:39.120 --> 00:02:42.959
sed said for json data you can use it to
00:02:42.959 --> 00:02:44.720
slice and filter and map
00:02:44.720 --> 00:02:46.720
and transform structured data with the
00:02:46.720 --> 00:02:47.840
same ease that
00:02:47.840 --> 00:02:51.519
z awk grep and friends let you play with
00:02:51.519 --> 00:02:54.000
text
00:02:54.000 --> 00:02:56.160
let me give you a quick demonstration of
00:02:56.160 --> 00:02:57.519
it by the way
00:02:57.519 --> 00:02:59.680
it's written in portable c it has zero
00:02:59.680 --> 00:03:01.920
runtime dependency so it's very easy
00:03:01.920 --> 00:03:04.959
to get started with it and use it on
00:03:04.959 --> 00:03:09.040
pretty much any unix-based
00:03:09.040 --> 00:03:12.800
computer sorry no linux based computer
00:03:12.800 --> 00:03:16.159
apologies okay so let's explore a
00:03:16.159 --> 00:03:19.599
json file with it it's a command line
00:03:19.599 --> 00:03:21.120
tool and it has a very
00:03:21.120 --> 00:03:24.640
simple command line syntax so you
00:03:24.640 --> 00:03:26.799
call the binary and then you give it a
00:03:26.799 --> 00:03:27.840
query
00:03:27.840 --> 00:03:31.680
and a file and then we'll return
00:03:31.680 --> 00:03:33.840
its answer so for example if I want the
00:03:33.840 --> 00:03:35.440
top level keys
00:03:35.440 --> 00:03:38.319
I will just say jq keys the file and it
00:03:38.319 --> 00:03:39.840
will return the keys
00:03:39.840 --> 00:03:42.319
simple as that so let's check this out
00:03:42.319 --> 00:03:43.519
in a real
00:03:43.519 --> 00:03:46.879
shell here I am in eshel
00:03:46.879 --> 00:03:50.080
let's run jq keys on the github
00:03:50.080 --> 00:03:53.200
issue comment and we can see
00:03:53.200 --> 00:03:56.959
that we have actually received a list
00:03:56.959 --> 00:04:00.000
back here with the top level
00:04:00.000 --> 00:04:02.000
things okay so this issue it looks very
00:04:02.000 --> 00:04:04.319
very interesting so let's uh ask it to
00:04:04.319 --> 00:04:06.239
give me more information on this
00:04:06.239 --> 00:04:10.080
issue then it's hairy again that's a lot
00:04:10.080 --> 00:04:11.360
of stuff
00:04:11.360 --> 00:04:14.319
I mean lucky for for us we are in Emacs
00:04:14.319 --> 00:04:17.759
here so we can use nice shortcuts we can
00:04:17.759 --> 00:04:21.040
copy this we go can go in here just
00:04:21.040 --> 00:04:22.000
select that
00:04:22.000 --> 00:04:24.400
get that out or something like this but
00:04:24.400 --> 00:04:25.600
still
00:04:25.600 --> 00:04:28.320
this is not really it's not really the
00:04:28.320 --> 00:04:29.600
best way
00:04:29.600 --> 00:04:34.080
to do that right it gets kind of tedious
00:04:34.080 --> 00:04:36.639
at this point the output can be
00:04:36.639 --> 00:04:37.680
humongous
00:04:37.680 --> 00:04:39.360
the shell is not really the best place
00:04:39.360 --> 00:04:42.479
to read through such big output I mean e
00:04:42.479 --> 00:04:43.600
shell is
00:04:43.600 --> 00:04:45.440
probably one of the better shells for
00:04:45.440 --> 00:04:47.440
this because it's just a regular Emacs
00:04:47.440 --> 00:04:48.560
buffer but still
00:04:48.560 --> 00:04:51.199
it's not really the best tool and I need
00:04:51.199 --> 00:04:52.720
to repeat the command
00:04:52.720 --> 00:04:55.280
all the time until I finally build the
00:04:55.280 --> 00:04:56.000
right query
00:04:56.000 --> 00:04:59.120
and all the time I use I lose my
00:04:59.120 --> 00:05:02.080
focus I lose what I'm currently looking
00:05:02.080 --> 00:05:02.800
at
00:05:02.800 --> 00:05:06.080
I'm seeing the new result it would be so
00:05:06.080 --> 00:05:08.160
much nicer to have live feedback
00:05:08.160 --> 00:05:10.080
and when working with Emacs we're quite
00:05:10.080 --> 00:05:11.600
used to that so there should be an
00:05:11.600 --> 00:05:12.320
option
00:05:12.320 --> 00:05:15.120
and of course there is it's Emacs right
00:05:15.120 --> 00:05:17.759
so you can do anything
00:05:17.759 --> 00:05:20.880
there is various good tools for
00:05:20.880 --> 00:05:22.960
completion in Emacs
00:05:22.960 --> 00:05:26.000
I used ivy for this
00:05:26.000 --> 00:05:29.039
I'm going to code the usp for ivy
00:05:29.039 --> 00:05:31.840
iv is a generic completion mechanism for
00:05:31.840 --> 00:05:32.639
Emacs
00:05:32.639 --> 00:05:34.639
while it operates similarly to other
00:05:34.639 --> 00:05:35.919
completion schemes
00:05:35.919 --> 00:05:38.960
such as icomplete mode iv aims to be
00:05:38.960 --> 00:05:40.400
more efficient smaller
00:05:40.400 --> 00:05:43.120
simpler and smoother to use yet highly
00:05:43.120 --> 00:05:45.199
customizable
00:05:45.199 --> 00:05:47.840
and that's true one of the cool things
00:05:47.840 --> 00:05:49.440
of iv
00:05:49.440 --> 00:05:52.479
compared to other completion mechanisms
00:05:52.479 --> 00:05:54.320
in Emacs
00:05:54.320 --> 00:05:58.160
is that it can be used on dynamic
00:05:58.160 --> 00:06:01.600
data so usually completion works on a
00:06:01.600 --> 00:06:03.520
static input for example you're in a
00:06:03.520 --> 00:06:05.360
buffer a text buffer
00:06:05.360 --> 00:06:08.800
and you use I search maybe with idle
00:06:08.800 --> 00:06:09.600
mode
00:06:09.600 --> 00:06:12.400
and you find your results that's all
00:06:12.400 --> 00:06:13.360
nice
00:06:13.360 --> 00:06:16.960
however if I
00:06:16.960 --> 00:06:19.840
want to search on dynamic data that
00:06:19.840 --> 00:06:20.720
doesn't work
00:06:20.720 --> 00:06:23.759
so whenever I type in my query
00:06:23.759 --> 00:06:27.039
for jq I actually need to call the jq
00:06:27.039 --> 00:06:28.000
binary
00:06:28.000 --> 00:06:30.160
and it will give a different result set
00:06:30.160 --> 00:06:32.840
back so it's a really
00:06:32.840 --> 00:06:36.319
dynamic mechanism that we need here it's
00:06:36.319 --> 00:06:38.240
much more like a search engine
00:06:38.240 --> 00:06:41.440
and ivy luckily has something built in
00:06:41.440 --> 00:06:42.560
and it's called
00:06:42.560 --> 00:06:46.000
console so I used council and jq
00:06:46.000 --> 00:06:48.160
and combined them and build a new
00:06:48.160 --> 00:06:49.199
package
00:06:49.199 --> 00:06:52.960
with which we can use Emacs and jq
00:06:52.960 --> 00:06:56.000
to have live feedback
00:06:56.000 --> 00:06:58.800
it's very easy to use so you just call
00:06:58.800 --> 00:06:59.840
console.jq
00:06:59.840 --> 00:07:02.800
on a buffer containing json for example
00:07:02.800 --> 00:07:04.319
the one we have here
00:07:04.319 --> 00:07:07.280
let's call console.jq on it and we
00:07:07.280 --> 00:07:08.319
already
00:07:08.319 --> 00:07:11.280
get a default query the dot query which
00:07:11.280 --> 00:07:13.039
just gives us
00:07:13.039 --> 00:07:16.080
the same file but now we can change it
00:07:16.080 --> 00:07:17.280
and for example
00:07:17.280 --> 00:07:19.039
find all the keys in here and then we
00:07:19.039 --> 00:07:21.039
see I had this issue this was the one
00:07:21.039 --> 00:07:22.800
that we were interested in
00:07:22.800 --> 00:07:24.639
so let's find more information on the
00:07:24.639 --> 00:07:26.479
issue what
00:07:26.479 --> 00:07:28.800
keys does it have actually have uh it
00:07:28.800 --> 00:07:31.680
has assignees that that interests me
00:07:31.680 --> 00:07:34.800
so let's check out the assignees in here
00:07:34.800 --> 00:07:37.039
there's two of them but I'm only
00:07:37.039 --> 00:07:39.759
interested in the first one
00:07:39.759 --> 00:07:42.000
I'm making stuff up as I go here of
00:07:42.000 --> 00:07:43.599
course
00:07:43.599 --> 00:07:46.000
and whenever I hit enter I get a new
00:07:46.000 --> 00:07:47.039
buffer
00:07:47.039 --> 00:07:50.160
which just shows me this
00:07:50.160 --> 00:07:53.520
particular result for the particular
00:07:53.520 --> 00:07:55.599
query that I entered
00:07:55.599 --> 00:07:59.199
so let me do that again we are in here
00:07:59.199 --> 00:08:03.199
we are looking at a
00:08:03.199 --> 00:08:05.840
json file this can be very very big
00:08:05.840 --> 00:08:07.520
doesn't also need to be a file just
00:08:07.520 --> 00:08:09.520
needs to be a buffer
00:08:09.520 --> 00:08:12.160
you call console.jq on it and you can do
00:08:12.160 --> 00:08:13.599
any kind of
00:08:13.599 --> 00:08:16.479
query on it for example let's see if
00:08:16.479 --> 00:08:18.080
there is a url here
00:08:18.080 --> 00:08:20.319
yes there's a url let's see if there's a
00:08:20.319 --> 00:08:23.759
repository here repository
00:08:23.759 --> 00:08:26.879
no there isn't what was it called issue
00:08:26.879 --> 00:08:33.440
keys repository url it was called okay
00:08:33.440 --> 00:08:36.640
so let's see issue pause
00:08:36.640 --> 00:08:40.240
url and then we see so
00:08:40.240 --> 00:08:42.959
apparently this issue command is for a
00:08:42.959 --> 00:08:44.159
repository called
00:08:44.159 --> 00:08:47.839
organize I wonder what that might be
00:08:47.839 --> 00:08:50.320
okay so that was a very short
00:08:50.320 --> 00:08:52.640
introduction to console jq
00:08:52.640 --> 00:08:54.800
you can see the timer here I only have
00:08:54.800 --> 00:08:56.800
one minute left to go so I'm going to
00:08:56.800 --> 00:08:57.440
leave
00:08:57.440 --> 00:08:59.760
with a very very short introduction to
00:08:59.760 --> 00:09:01.519
the console gq
00:09:01.519 --> 00:09:05.040
code it's not even 60 lines
00:09:05.040 --> 00:09:07.519
of e-lisp so building something like
00:09:07.519 --> 00:09:09.600
this is very very easy
00:09:09.600 --> 00:09:12.800
I would encourage you to go and read
00:09:12.800 --> 00:09:14.560
through the code in your own time
00:09:14.560 --> 00:09:16.160
if you're interested in building
00:09:16.160 --> 00:09:18.320
something like this if you're interested
00:09:18.320 --> 00:09:19.600
in just using
00:09:19.600 --> 00:09:22.720
jq or you're done
00:09:22.720 --> 00:09:24.320
these are the links to all the tools
00:09:24.320 --> 00:09:25.839
console.jq of course
00:09:25.839 --> 00:09:28.800
is readily available on melpah also
00:09:28.800 --> 00:09:29.519
developed
00:09:29.519 --> 00:09:32.959
under the agpl license on github
00:09:32.959 --> 00:09:36.080
and disorganized thing by the way it's
00:09:36.080 --> 00:09:37.839
orig mode for mobile and desktop
00:09:37.839 --> 00:09:39.839
browsers also a great
00:09:39.839 --> 00:09:42.399
free software tool maybe that interests
00:09:42.399 --> 00:09:43.120
you
00:09:43.120 --> 00:09:46.240
thank you for listening have great time
00:09:46.240 --> 00:09:48.800
10 seconds left I am going to stop this
00:09:48.800 --> 00:09:49.360
now
00:09:49.360 --> 00:09:53.920
enjoy EmacsConf have a great day
|
