path: root/2020/subtitles/emacsconf-2020--13-experience-report-steps-to-emacs-hyper-notebooks--joseph-corneli-raymond-puzio-cameron-ray-smith-autogen.sbv

0:00:00.320,0:00:05.920
um so hi I'm joe corneli

0:00:03.280,0:00:06.879
uh this is work I did with ray puzio and

0:00:05.920,0:00:10.559
cameron smith

0:00:06.879,0:00:11.599
um and they're the main protagonists in

0:00:10.559,0:00:14.960
this story they

0:00:11.599,0:00:17.840
are uh researchers who've been working

0:00:14.960,0:00:20.160
on theoretical biology um so in a

0:00:17.840,0:00:23.760
typical project they may use manxima

0:00:20.160,0:00:24.800
and julia their work for biology physics

0:00:23.760,0:00:27.199
and computer science

0:00:24.800,0:00:29.439
computer science and the latest work in

0:00:27.199,0:00:30.800
progress is on branching processes for

0:00:29.439,0:00:34.719
cancer modeling so

0:00:30.800,0:00:37.360
how can um Emacs possibly help

0:00:34.719,0:00:38.399
let's let's have a look uh moving code

0:00:37.360,0:00:39.680
and data between these different

0:00:38.399,0:00:42.000
programs by hand is

0:00:39.680,0:00:43.200
annoying on separate workflows for

0:00:42.000,0:00:44.399
writing up notes and preparing

0:00:43.200,0:00:46.000
publications

0:00:44.399,0:00:48.640
is perhaps even more annoying all of

0:00:46.000,0:00:52.000
it's time consuming and error-prone

0:00:48.640,0:00:53.760
um so what about maybe using jupiter

0:00:52.000,0:00:55.199
uh we found something called script of

0:00:53.760,0:00:58.399
scripts and it solves some of those

0:00:55.199,0:01:01.120
problems because you can use

0:00:58.399,0:01:02.640
maximum and julia together but we were

0:01:01.120,0:01:05.199
quite happy to explore emacs based

0:01:02.640,0:01:05.840
solutions being emax enthusiasts and we

0:01:05.199,0:01:07.760
even

0:01:05.840,0:01:09.200
got cameron to be enthusiastic about

0:01:07.760,0:01:11.600
doing emacs so that

0:01:09.200,0:01:12.400
went nice so just here's a little

0:01:11.600,0:01:15.360
feature grid

0:01:12.400,0:01:16.400
of emacs org versus just sort of your

0:01:15.360,0:01:19.040
generic

0:01:16.400,0:01:20.960
um tools that are in a different more

0:01:19.040,0:01:21.520
general ecosystem so as you can see it's

0:01:20.960,0:01:23.360
quite

0:01:21.520,0:01:25.520
feature complete you've got your maximo

0:01:23.360,0:01:27.119
mode julia mode you can use both of them

0:01:25.520,0:01:28.720
inside of org mode

0:01:27.119,0:01:30.240
you can present things with word tree

0:01:28.720,0:01:33.280
slide you can set up a

0:01:30.240,0:01:33.680
wiki inside of orgrome this is one I

0:01:33.280,0:01:35.759
found

0:01:33.680,0:01:37.759
rather recently you can even use

0:01:35.759,0:01:38.799
compatibly with orgrome something called

0:01:37.759,0:01:41.520
log seek

0:01:38.799,0:01:42.159
which is in the browser um so that's

0:01:41.520,0:01:44.320
nice

0:01:42.159,0:01:45.840
um you can do real-time collaborative

0:01:44.320,0:01:47.840
editing um

0:01:45.840,0:01:49.280
either in a kind of pairing style or in

0:01:47.840,0:01:51.520
a more etherpad style

0:01:49.280,0:01:54.159
obviously you can manage your references

0:01:51.520,0:01:55.759
you can typeset whatever you want um you

0:01:54.159,0:01:57.439
can publish work in progress on a blog

0:01:55.759,0:01:58.159
and the fern is another one of these

0:01:57.439,0:02:00.560
external

0:01:58.159,0:02:01.360
org mode tools it's not actually any mac

0:02:00.560,0:02:03.680
but works with

0:02:01.360,0:02:04.399
org mode stuff and you know so we're

0:02:03.680,0:02:06.640
good to go

0:02:04.399,0:02:08.000
uh with all of that so what does that

0:02:06.640,0:02:09.679
look like well here's

0:02:08.000,0:02:11.599
a little example from before they were

0:02:09.679,0:02:12.800
doing um

0:02:11.599,0:02:14.560
before we started really thinking

0:02:12.800,0:02:15.599
seriously about this stuff so this is

0:02:14.560,0:02:17.440
just maxima

0:02:15.599,0:02:19.280
well maximo doesn't have a long running

0:02:17.440,0:02:20.160
process by default if you've ever used

0:02:19.280,0:02:22.480
python

0:02:20.160,0:02:23.920
uh you have something called sessions uh

0:02:22.480,0:02:24.959
they don't have that for maxim at least

0:02:23.920,0:02:27.599
not by default

0:02:24.959,0:02:29.360
um so how what was the workaround

0:02:27.599,0:02:32.480
there's this thing called uh

0:02:29.360,0:02:34.879
solve for you here and um

0:02:32.480,0:02:36.319
that shows up down below again in these

0:02:34.879,0:02:37.360
angle brackets which you've seen maybe

0:02:36.319,0:02:40.480
in someone else's talk

0:02:37.360,0:02:41.920
uh which means go to the previous uh

0:02:40.480,0:02:43.920
thing that was named soul for you

0:02:41.920,0:02:45.280
and do that all over again so they do

0:02:43.920,0:02:47.440
that over again

0:02:45.280,0:02:49.519
and here's the little maxima code for

0:02:47.440,0:02:50.560
defining you saw so you've now defined

0:02:49.519,0:02:51.920
you saw

0:02:50.560,0:02:54.000
and then you can use it in the next

0:02:51.920,0:02:56.560
expression you get out a nice juicy

0:02:54.000,0:02:57.200
zero at the end but it's a little bit a

0:02:56.560,0:02:58.959
little bit

0:02:57.200,0:03:00.640
like cartridge in a pear tree to have to

0:02:58.959,0:03:02.560
redefine everything every time

0:03:00.640,0:03:04.159
so this is clearly at the level of work

0:03:02.560,0:03:06.879
around um maybe just

0:03:04.159,0:03:07.920
one more time going looking through that

0:03:06.879,0:03:11.599
um

0:03:07.920,0:03:13.760
that stuff um

0:03:11.599,0:03:15.280
sorry so uh looking through that stuff

0:03:13.760,0:03:17.200
this is we're going to need something

0:03:15.280,0:03:19.599
like that probably for stitching

0:03:17.200,0:03:20.319
maxima and julie julia together so it's

0:03:19.599,0:03:21.680
good to

0:03:20.319,0:03:23.920
look a little bit about how that might

0:03:21.680,0:03:25.680
work so first of all you can cache

0:03:23.920,0:03:26.480
results so if you wanted to save the

0:03:25.680,0:03:28.480
date

0:03:26.480,0:03:29.920
out of block one at a certain time and

0:03:28.480,0:03:32.239
then use

0:03:29.920,0:03:33.280
use it again later so at the time when I

0:03:32.239,0:03:34.640
ran this code

0:03:33.280,0:03:36.720
you can see I've got two slightly

0:03:34.640,0:03:38.560
different time stamps down below one's

0:03:36.720,0:03:40.319
the cached result and the other was the

0:03:38.560,0:03:42.640
result of reevaluating

0:03:40.319,0:03:43.760
the block so you can move things around

0:03:42.640,0:03:46.000
um and that's

0:03:43.760,0:03:47.280
that's going to be useful but you know

0:03:46.000,0:03:48.080
that's not really the main problem the

0:03:47.280,0:03:50.799
main problem is

0:03:48.080,0:03:51.440
making maxima long running so kind of

0:03:50.799,0:03:53.920
the

0:03:51.440,0:03:56.400
core of this talk is a new observant

0:03:53.920,0:03:59.280
facility which is a general purpose

0:03:56.400,0:04:00.560
way to do that kind of thing which

0:03:59.280,0:04:04.239
involves a very simple

0:04:00.560,0:04:05.360
change to obcor uh so we'll give a quick

0:04:04.239,0:04:08.480
overview of that and show

0:04:05.360,0:04:11.760
an example um so here's the example

0:04:08.480,0:04:13.040
um a very simple sort of silly example

0:04:11.760,0:04:14.640
uh what does it mean to have a long

0:04:13.040,0:04:15.920
running process here I've set this

0:04:14.640,0:04:18.560
display2d

0:04:15.920,0:04:19.440
to be false um which just means that

0:04:18.560,0:04:22.320
things are going to come

0:04:19.440,0:04:23.040
come across in 1d and then I ask it to

0:04:22.320,0:04:25.199
expand

0:04:23.040,0:04:27.280
uh something and I get latex by default

0:04:25.199,0:04:28.639
so so that's what it means is I've sent

0:04:27.280,0:04:30.240
something in and it's going to come

0:04:28.639,0:04:32.080
across in one view which is great

0:04:30.240,0:04:34.560
um maybe you'll also notice that there's

0:04:32.080,0:04:36.720
no semicolon if you're a maxima fan

0:04:34.560,0:04:38.400
um and things are coming across as tech

0:04:36.720,0:04:40.320
so those were some little bonus features

0:04:38.400,0:04:41.040
and I'll show you how that works later

0:04:40.320,0:04:45.440
so

0:04:41.040,0:04:48.880
um the change to obcor is as follows uh

0:04:45.440,0:04:51.520
um we uh actually this should say uh

0:04:48.880,0:04:52.479
instead of stream here it should say um

0:04:51.520,0:04:54.800
servant

0:04:52.479,0:04:56.160
sorry uh we tried an experimental

0:04:54.800,0:04:58.160
version what's called stream so now it's

0:04:56.160,0:05:01.520
called servant but all it does is it

0:04:58.160,0:05:02.639
overrides uh or babel execute laying for

0:05:01.520,0:05:05.919
arbitrary laying

0:05:02.639,0:05:07.840
if you have um a servant

0:05:05.919,0:05:09.759
in your in your params so that's the

0:05:07.840,0:05:11.919
change that hasn't been

0:05:09.759,0:05:13.759
pushed out or sent as a patch to anybody

0:05:11.919,0:05:16.960
but it's a pretty minor change

0:05:13.759,0:05:19.080
um here's an overview without the code

0:05:16.960,0:05:20.720
of just a high level overview of

0:05:19.080,0:05:22.160
observant.el so

0:05:20.720,0:05:24.479
it stores information about these

0:05:22.160,0:05:26.080
processes in a hash table

0:05:24.479,0:05:27.600
it can do pre-processing and

0:05:26.080,0:05:29.759
post-processing

0:05:27.600,0:05:30.720
um it does all these things it stores

0:05:29.759,0:05:32.479
the output

0:05:30.720,0:05:34.080
I mentioned here that in principle we

0:05:32.479,0:05:35.280
could store lots of output and have a

0:05:34.080,0:05:36.560
kind of browsable

0:05:35.280,0:05:38.880
history although we don't do that

0:05:36.560,0:05:40.639
presently um but that's what observant

0:05:38.880,0:05:41.440
does is it does what you might expect

0:05:40.639,0:05:45.440
and here's the

0:05:41.440,0:05:48.160
here's the maxima kind of um

0:05:45.440,0:05:49.600
on-ramp uh to get maxima brought in so

0:05:48.160,0:05:51.360
you have to obviously have a maximum

0:05:49.600,0:05:54.960
process you can call

0:05:51.360,0:05:57.840
um put hash uh this is the preprocessing

0:05:54.960,0:05:58.960
thing I mentioned adding in some tech um

0:05:57.840,0:06:01.520
and adding in

0:05:58.960,0:06:03.120
uh or deleting rather a substring and

0:06:01.520,0:06:03.759
here here is why you delete the

0:06:03.120,0:06:06.960
substring

0:06:03.759,0:06:08.240
is because um maxima thinks it's a good

0:06:06.960,0:06:10.080
idea to tell you false

0:06:08.240,0:06:11.759
once you once you run check on things

0:06:10.080,0:06:13.680
you've got to delete that back out to

0:06:11.759,0:06:16.960
get something kind of coherent out of it

0:06:13.680,0:06:19.280
but so this is how to set up maximal um

0:06:16.960,0:06:20.000
that's enough really of the demo is not

0:06:19.280,0:06:21.919
really a demos for

0:06:20.000,0:06:23.600
show and tell but uh this is an

0:06:21.919,0:06:25.440
experience report I wanted to talk about

0:06:23.600,0:06:28.080
the experience of doing this

0:06:25.440,0:06:30.160
so some some negatives like we tried to

0:06:28.080,0:06:33.199
get emacs jupiter working

0:06:30.160,0:06:36.000
prior to prior to working on the um

0:06:33.199,0:06:37.919
observant and we couldn't get it doing

0:06:36.000,0:06:40.160
everything we wanted despite a bit of

0:06:37.919,0:06:41.840
heavy lifting and debugging and stuff so

0:06:40.160,0:06:42.880
that's not not finished that was a bit

0:06:41.840,0:06:45.360
difficult

0:06:42.880,0:06:47.759
um on the other hand working on

0:06:45.360,0:06:48.479
observing was fun and pretty lightweight

0:06:47.759,0:06:50.400
and easy

0:06:48.479,0:06:52.400
um we got some experience co-editing

0:06:50.400,0:06:54.479
things with these real-time tools

0:06:52.400,0:06:55.919
obviously the stack is somewhat work in

0:06:54.479,0:06:58.000
progress um

0:06:55.919,0:07:00.800
so I just wanted to give a shout out to

0:06:58.000,0:07:03.919
crdt which was really fun

0:07:00.800,0:07:07.039
and champion was making bug fiction

0:07:03.919,0:07:08.960
bug fixes for that as we go similarly

0:07:07.039,0:07:10.160
for fern and log seek the maintainers

0:07:08.960,0:07:12.960
were really responsive

0:07:10.160,0:07:14.560
um and so that was nice we did try to

0:07:12.960,0:07:15.840
get emacs running in the browser

0:07:14.560,0:07:16.880
thinking it would be really nice for

0:07:15.840,0:07:19.520
people who

0:07:16.880,0:07:21.120
um didn't want to install it to get a

0:07:19.520,0:07:23.120
chance to just try it

0:07:21.120,0:07:24.479
uh but actually browsers capture things

0:07:23.120,0:07:27.120
like uh control n

0:07:24.479,0:07:28.479
and so that's that was a bit annoying

0:07:27.120,0:07:29.759
but we did get lots of great feedback

0:07:28.479,0:07:31.599
and interaction with people including

0:07:29.759,0:07:33.759
around this conference so thank you

0:07:31.599,0:07:35.599
to those who we've had discussions with

0:07:33.759,0:07:37.680
um

0:07:35.599,0:07:38.639
so future work okay so maybe you

0:07:37.680,0:07:41.039
remember I didn't

0:07:38.639,0:07:42.400
give a talk a few years back on arcana

0:07:41.039,0:07:44.479
so what what

0:07:42.400,0:07:45.919
might this have to do with org mode and

0:07:44.479,0:07:47.039
that's always the question one asks

0:07:45.919,0:07:48.720
about arcata

0:07:47.039,0:07:50.319
well so arcana one of the things it does

0:07:48.720,0:07:51.680
is transclusions and so that could be

0:07:50.319,0:07:54.000
actually very helpful

0:07:51.680,0:07:55.520
in connection with this combined notes

0:07:54.000,0:07:56.479
and write-up workflow so you might have

0:07:55.520,0:07:58.400
a

0:07:56.479,0:08:00.800
forward mode uh and some of these

0:07:58.400,0:08:03.199
results we got back as raw results

0:08:00.800,0:08:03.919
could go right into your write up in a

0:08:03.199,0:08:07.520
kind of

0:08:03.919,0:08:09.039
convenient way at a level above um

0:08:07.520,0:08:11.280
transparently a level above the notebook

0:08:09.039,0:08:13.440
so you'd have the notebook alongside the

0:08:11.280,0:08:15.599
write-up in that case

0:08:13.440,0:08:16.800
which is a sort of variation on the

0:08:15.599,0:08:19.120
literate programming

0:08:16.800,0:08:20.000
workflow this is speculative who knows

0:08:19.120,0:08:22.080
the other thought is

0:08:20.000,0:08:23.520
it just relates to the idea of network

0:08:22.080,0:08:24.800
programming so we can imagine these

0:08:23.520,0:08:26.639
networks of

0:08:24.800,0:08:28.800
computational nodes maybe sitting inside

0:08:26.639,0:08:31.199
of orgrome calling each other

0:08:28.800,0:08:32.560
um and you would want to maintain some

0:08:31.199,0:08:34.959
kind of model of that

0:08:32.560,0:08:36.640
process and a sort of general question

0:08:34.959,0:08:38.320
is how do we have a remote control for

0:08:36.640,0:08:40.240
long-running processes you could do that

0:08:38.320,0:08:42.479
in lisp or closure but

0:08:40.240,0:08:44.080
maybe we could have something a little

0:08:42.479,0:08:45.839
bit like that here

0:08:44.080,0:08:48.080
um so conclusions what have we actually

0:08:45.839,0:08:50.080
addressed well uh we addressed uh

0:08:48.080,0:08:51.600
accessing any long-running process with

0:08:50.080,0:08:52.959
a simple or mode interface

0:08:51.600,0:08:54.880
obviously we're not the only people to

0:08:52.959,0:08:56.880
think about notebooks but we think that

0:08:54.880,0:08:58.880
emacs has some advantages

0:08:56.880,0:09:01.120
related to reproducible research and

0:08:58.880,0:09:01.680
interdisciplinary collaboration so let's

0:09:01.120,0:09:03.120
just

0:09:01.680,0:09:04.880
say that we think something is

0:09:03.120,0:09:07.200
reproducible if it's actually teachable

0:09:04.880,0:09:09.279
to someone new and they can do it

0:09:07.200,0:09:11.680
org mode seems very useful for that many

0:09:09.279,0:09:14.399
of the other talks have touched on this

0:09:11.680,0:09:16.000
interdisciplinary collaboration is great

0:09:14.399,0:09:17.839
this was an interdisciplinary

0:09:16.000,0:09:19.680
collaboration on some level but

0:09:17.839,0:09:21.600
what about you know future work for

0:09:19.680,0:09:22.320
bringing in scenario planners simulation

0:09:21.600,0:09:24.480
scientists

0:09:22.320,0:09:25.519
and local farmers and building something

0:09:24.480,0:09:27.200
that they can all

0:09:25.519,0:09:29.200
use that's more than the sum of the

0:09:27.200,0:09:30.720
parts so a little

0:09:29.200,0:09:32.160
future work for everybody else here so

0:09:30.720,0:09:33.120
we think science should be widely

0:09:32.160,0:09:34.560
teachable

0:09:33.120,0:09:36.720
shareable semi-automated

0:09:34.560,0:09:40.399
transdisciplinary and real time

0:09:36.720,0:09:43.279
like EmacsConf so you can get in touch

0:09:40.399,0:09:44.560
uh via these methods and the code which

0:09:43.279,0:09:45.200
is very much early stage work in

0:09:44.560,0:09:46.640
progress

0:09:45.200,0:09:48.800
as this was meant to be an experience

0:09:46.640,0:09:49.920
report not a it's all done here it is

0:09:48.800,0:09:51.680
polished report

0:09:49.920,0:09:53.360
um it's also online if you'd like to

0:09:51.680,0:09:54.560
have a look so that's the end of the

0:09:53.360,0:09:55.279
talk and I don't know if there's time

0:09:54.560,0:09:57.440
for questions

0:09:55.279,0:10:00.240
or not but um I'm at your disposal now

0:09:57.440,0:10:04.079
thank you you are now unmuted

0:10:00.240,0:10:07.120
um many thanks for the tough job

0:10:04.079,0:10:08.880
uh let's see we have about I think four

0:10:07.120,0:10:10.880
minutes four questions

0:10:08.880,0:10:12.320
um and we have a couple of questions on

0:10:10.880,0:10:14.240
the pad would you like to read them

0:10:12.320,0:10:16.560
yourself or should I read them to you

0:10:14.240,0:10:18.079
uh just for sake of easy management why

0:10:16.560,0:10:21.279
don't you read them out if that's okay

0:10:18.079,0:10:22.240
yeah sure uh so they ask have you looked

0:10:21.279,0:10:25.440
into trying

0:10:22.240,0:10:33.760
sage math I've long uh long wanted

0:10:25.440,0:10:36.959
nothing org files

0:10:33.760,0:10:39.279
right and I and I wrote the answer that

0:10:36.959,0:10:42.079
we should be possible because one can

0:10:39.279,0:10:42.079
call it from a command

0:10:44.839,0:10:48.640
okay

0:10:46.399,0:10:50.079
and I see there's another sagemath

0:10:48.640,0:10:52.100
question that you seem to have answered

0:10:50.079,0:10:54.880
as well so I guess I won't repeat that

0:10:52.100,0:10:56.959
[Music]

0:10:54.880,0:11:00.640
there's let's not forget about embedded

0:10:56.959,0:11:03.680
calc in emacs

0:11:00.640,0:11:05.040
so so the first demos actually were with

0:11:03.680,0:11:06.640
calc so that's that's

0:11:05.040,0:11:07.839
useful uh although I think it was a

0:11:06.640,0:11:08.240
different kind of a different command

0:11:07.839,0:11:11.839
line

0:11:08.240,0:11:13.839
well that was the next calc yeah

0:11:11.839,0:11:15.680
so sure there is there is calc so that

0:11:13.839,0:11:19.120
would be calculus and

0:11:15.680,0:11:21.839
calc is already in org mode yeah

0:11:19.120,0:11:21.839
yeah yeah

0:11:25.680,0:11:29.839
um still looking for questions

0:11:39.760,0:11:44.320
okay I think that's about it I don't see

0:11:41.760,0:11:47.440
any questions on the ether pad

0:11:44.320,0:11:50.480
and let's see

0:11:47.440,0:11:50.480
anything on irc

0:11:53.040,0:11:56.720
um nothing but phrases and everyone

0:11:55.760,0:11:59.120
thanking you

0:11:56.720,0:12:00.240
thank you all right you're welcome

0:11:59.120,0:12:02.880
thanks a lot uh

0:12:00.240,0:12:06.800
we'll see you guys around then cheers

0:12:02.880,0:12:06.800
and see you around