path: root/2020/subtitles/emacsconf-2020--13-experience-report-steps-to-emacs-hyper-notebooks--joseph-corneli-raymond-puzio-cameron-ray-smith.vtt

WEBVTT

00:00:00.320 --> 00:00:03.280
Joe: Hi, I'm Joe Corneli.

00:00:03.280 --> 00:00:06.879
This is work I did with Ray Puzio and
Cameron Smith.

00:00:06.879 --> 00:00:11.123
They're the main protagonists in
this story.

00:00:11.123 --> 00:00:14.960
They are researchers who've been working

00:00:14.960 --> 00:00:17.490
on theoretical biology.

00:00:17.490 --> 00:00:21.357
In a typical project, they may use
Maxima and Julia.

00:00:21.357 --> 00:00:24.800
Their work combines biology, physics and
computer science.

00:00:24.800 --> 00:00:29.439
The latest work-in-progress is on
branching processes for

00:00:29.439 --> 00:00:30.800
cancer modeling.

00:00:30.800 --> 00:00:34.719
How can Emacs possibly help?

00:00:34.719 --> 00:00:37.360
Let's have a look. Moving code

00:00:37.360 --> 00:00:38.399
and data between these different

00:00:38.399 --> 00:00:39.680
programs by hand is

00:00:39.680 --> 00:00:42.000
annoying. Separate workflows for

00:00:42.000 --> 00:00:44.399
writing up notes and preparing
publications

00:00:44.399 --> 00:00:46.000
is perhaps even more annoying. All of

00:00:46.000 --> 00:00:48.640
it is time consuming and error-prone.

00:00:48.640 --> 00:00:52.000
So what about maybe using Jupyter?

00:00:52.000 --> 00:00:53.760
We found something called Script of

00:00:53.760 --> 00:00:55.199
Scripts. It solves some of those

00:00:55.199 --> 00:00:58.399
problems because you can use

00:00:58.399 --> 00:01:01.120
Maxima and Julia together, but we were

00:01:01.120 --> 00:01:02.640
quite happy to explore Emacs-based

00:01:02.640 --> 00:01:04.890
solutions, being Emcas enthusiasts

00:01:04.890 --> 00:01:07.760
We even got Cameron to be
enthusiastic about

00:01:07.760 --> 00:01:10.057
doing Emacs, so that went nice.

00:01:10.057 --> 00:01:12.400
Here's a little feature grid

00:01:12.400 --> 00:01:15.360
of Emacs + Org versus your

00:01:15.360 --> 00:01:18.390
generic tools that are in a

00:01:18.390 --> 00:01:20.123
different, more general ecosystem.

00:01:20.123 --> 00:01:21.520
As you can see, it's quite

00:01:21.520 --> 00:01:22.957
feature-complete. You've got your

00:01:22.957 --> 00:01:25.520
maxima-mode, julia-mode. You can use both of them

00:01:25.520 --> 00:01:27.119
inside of org-mode.

00:01:27.119 --> 00:01:28.720
You can present things with org-tree-slide.

00:01:28.720 --> 00:01:31.490
You can set up a wiki inside of org-roam.

00:01:31.490 --> 00:01:33.680
This is one I found

00:01:33.680 --> 00:01:35.759
rather recently. You can even use

00:01:35.759 --> 00:01:37.759
compatibly with org-roam, something called

00:01:37.759 --> 00:01:42.159
logseq, which is in the browser, so that's
nice.

00:01:42.159 --> 00:01:44.957
You can do real-time collaborative
editing,

00:01:44.957 --> 00:01:47.657
either in a kind of pairing style

00:01:47.657 --> 00:01:49.280
or in a more Etherpad style.

00:01:49.280 --> 00:01:51.520
Obviously, you can manage your references.

00:01:51.520 --> 00:01:53.457
You can typeset whatever you want.

00:01:53.457 --> 00:01:55.759
You can publish work in progress on
a blog.

00:01:55.759 --> 00:01:57.439
Firn is another one of these

00:01:57.439 --> 00:01:59.790
external Org Mode tools.

00:01:59.790 --> 00:02:00.560
It's not actually in Emacs,

00:02:00.560 --> 00:02:03.557
but works with Org Mode stuff. And, you know...

00:02:03.557 --> 00:02:05.657
So we're good to go with all of that.

00:02:05.657 --> 00:02:07.423
So what does that look like? Well,

00:02:07.423 --> 00:02:09.423
here's a little example from before

00:02:09.423 --> 00:02:12.800
they were doing... before we started
really thinking

00:02:12.800 --> 00:02:13.890
seriously about this stuff.

00:02:13.890 --> 00:02:15.599
So this is just Maxima.

00:02:15.599 --> 00:02:17.440
Well, Maxima doesn't have a long running

00:02:17.440 --> 00:02:19.280
process by default. If you've ever used

00:02:19.280 --> 00:02:22.480
Python, you have something called
sessions.

00:02:22.480 --> 00:02:23.920
They don't have that for Maxima, at least

00:02:23.920 --> 00:02:24.959
not by default.

00:02:24.959 --> 00:02:27.599
So how... What was the workaround?

00:02:27.599 --> 00:02:29.360
There's this thing called

00:02:29.360 --> 00:02:32.480
solve-for-u here

00:02:32.480 --> 00:02:34.879
that shows up down below again in these

00:02:34.879 --> 00:02:36.319
angle brackets, which you've seen maybe

00:02:36.319 --> 00:02:37.360
in someone else's talk,

00:02:37.360 --> 00:02:40.480
which means go to the previous

00:02:40.480 --> 00:02:41.920
thing that was named solve-for-u

00:02:41.920 --> 00:02:43.920
and do that all over again, so they do

00:02:43.920 --> 00:02:45.280
that over again.

00:02:45.280 --> 00:02:47.440
Here's the little Maxima code for

00:02:47.440 --> 00:02:48.990
defining usol, so you've now

00:02:48.990 --> 00:02:51.723
defined usol, and then you can use it

00:02:51.723 --> 00:02:54.000
in the next expression. You get out a
nice juicy

00:02:54.000 --> 00:02:56.560
zero at the end. It's a little bit

00:02:57.200 --> 00:02:58.959
like a partridge in a pear tree to have to

00:02:58.959 --> 00:03:00.640
redefine everything every time.

00:03:00.640 --> 00:03:02.490
So this is clearly at the level of

00:03:02.490 --> 00:03:04.159
work-around. Maybe just

00:03:04.159 --> 00:03:06.223
one more time looking through

00:03:06.223 --> 00:03:11.599
that stuff.

00:03:11.599 --> 00:03:13.760
Sorry. So, looking through that stuff,

00:03:13.760 --> 00:03:15.280
this is... We're going to need something

00:03:15.280 --> 00:03:17.200
like that, probably, for stitching

00:03:17.200 --> 00:03:19.599
Maxima and Julia together. so it's

00:03:19.599 --> 00:03:20.890
good to look a little bit

00:03:20.890 --> 00:03:22.590
about how that might work.

00:03:22.590 --> 00:03:23.920
First of all, you can cache

00:03:23.920 --> 00:03:25.680
results, so if you wanted to save the

00:03:25.680 --> 00:03:28.480
date out of block one at a certain
time and

00:03:28.480 --> 00:03:30.990
then use it again later...

00:03:30.990 --> 00:03:33.280
At the time when I ran this code,

00:03:33.280 --> 00:03:34.640
you can see I've got two slightly

00:03:34.640 --> 00:03:36.623
different time stamps down below.

00:03:36.623 --> 00:03:38.323
One's the cached result, and the other

00:03:38.323 --> 00:03:40.319
was the result of reevaluating

00:03:40.319 --> 00:03:42.640
the block. So you can move things around.

00:03:43.760 --> 00:03:46.000
That's going to be useful. But you know,

00:03:46.000 --> 00:03:46.923
that's not really the main problem.

00:03:46.923 --> 00:03:48.080
The main problem is

00:03:48.080 --> 00:03:50.799
making Maxima long-running.

00:03:50.799 --> 00:03:53.920
The core of this talk is a new observant

00:03:53.920 --> 00:03:56.400
facility, which is a general purpose

00:03:56.400 --> 00:03:59.280
way to do that kind of thing, which

00:03:59.280 --> 00:04:01.823
involves a very simple change to ob-core.

00:04:01.823 --> 00:04:04.239
We'll give a quick

00:04:04.239 --> 00:04:05.360
overview of that and show

00:04:05.360 --> 00:04:08.480
an example. So here's the example,

00:04:08.480 --> 00:04:11.760
a very simple sort of silly example.

00:04:11.760 --> 00:04:12.957
What does it mean to have

00:04:12.957 --> 00:04:13.990
a long-running process?

00:04:13.990 --> 00:04:14.640
Here, I've set this

00:04:14.640 --> 00:04:18.560
display2d to be false, which just
means that

00:04:18.560 --> 00:04:19.440
things are going to come

00:04:19.440 --> 00:04:22.320
come across in 1d. Then I ask it to

00:04:22.320 --> 00:04:25.199
expand something. I get LaTeX
by default.

00:04:25.199 --> 00:04:27.280
So that's what it means. It's that I've sent

00:04:27.280 --> 00:04:28.639
something in and it's going to come

00:04:28.639 --> 00:04:30.240
across in one view, which is great.

00:04:30.240 --> 00:04:32.080
Maybe you'll also notice that there's

00:04:32.080 --> 00:04:34.560
no semicolon, if you're a Maxima fan,

00:04:34.560 --> 00:04:36.720
and things are coming across as TeX.

00:04:36.720 --> 00:04:38.400
So those were some little bonus features.

00:04:38.400 --> 00:04:40.320
I'll show you how that works later.

00:04:41.040 --> 00:04:45.440
The change to ob-core is as follows.

00:04:45.440 --> 00:04:48.880
Actually, this should say...

00:04:48.880 --> 00:04:51.520
Instead of stream here, it should say

00:04:51.520 --> 00:04:54.800
servant. Sorry. We tried an experimental

00:04:54.800 --> 00:04:56.160
version which was called stream, so now it's

00:04:56.160 --> 00:04:58.160
called servant. But all it does is it

00:04:58.160 --> 00:05:01.520
overrides org-babel-execute lang for

00:05:01.520 --> 00:05:02.639
arbitrary lang

00:05:02.639 --> 00:05:05.919
if you have a servant

00:05:05.919 --> 00:05:07.840
in your params. So that's the

00:05:07.840 --> 00:05:09.759
change that hasn't been

00:05:09.759 --> 00:05:11.919
pushed out or sent as a patch to anybody,

00:05:11.919 --> 00:05:13.759
but it's a pretty minor change.

00:05:13.759 --> 00:05:16.960
Here's an overview without the code.

00:05:16.960 --> 00:05:19.080
Just a high level overview of

00:05:19.080 --> 00:05:20.720
observant.el.

00:05:20.720 --> 00:05:22.160
It stores information about these

00:05:22.160 --> 00:05:24.479
processes in a hash table.

00:05:24.479 --> 00:05:26.080
It can do pre-processing and

00:05:26.080 --> 00:05:27.600
post-processing.

00:05:27.600 --> 00:05:29.759
It does all these things. It stores

00:05:29.759 --> 00:05:30.720
the output.

00:05:30.720 --> 00:05:32.479
I mentioned here that, in principle, we

00:05:32.479 --> 00:05:34.080
could store lots of output and have a

00:05:34.080 --> 00:05:35.657
kind of browsable history,

00:05:35.657 --> 00:05:37.790
although we don't do that
presently.

00:05:37.790 --> 00:05:38.790
But that's what observant does.

00:05:38.790 --> 00:05:40.639
It does what you might expect.

00:05:41.440 --> 00:05:46.190
Here's the Maxima on-ramp

00:05:46.190 --> 00:05:48.160
to get Maxima brought in.

00:05:48.160 --> 00:05:49.257
You have to obviously have

00:05:49.257 --> 00:05:51.360
a Maxima process you can call.

00:05:51.360 --> 00:05:54.960
puthash... this is the preprocessing

00:05:54.960 --> 00:05:57.840
thing I mentioned, adding in some Tex

00:05:57.840 --> 00:05:58.960
and adding in--

00:05:58.960 --> 00:06:01.520
or deleting, rather--a substring.

00:06:01.520 --> 00:06:03.759
Here is why you delete the
substring.

00:06:03.759 --> 00:06:06.960
It's because Maxima thinks it's a good

00:06:06.960 --> 00:06:08.240
idea to tell you false

00:06:08.240 --> 00:06:10.080
once you run check on things.

00:06:10.080 --> 00:06:11.759
You've got to delete that back out to

00:06:11.759 --> 00:06:13.680
get something coherent out of it.

00:06:13.680 --> 00:06:16.960
So this is how to set up Maxima.

00:06:16.960 --> 00:06:19.157
That's enough, really, of the demo.

00:06:19.157 --> 00:06:20.000
It's not really a demo for

00:06:20.000 --> 00:06:21.919
show and tell, but as this is an

00:06:21.919 --> 00:06:23.600
experience report, I wanted to talk about

00:06:23.600 --> 00:06:25.440
the experience of doing this.

00:06:25.440 --> 00:06:28.080
Some negatives, like we tried to

00:06:28.080 --> 00:06:30.160
get Emacs Jupyter working

00:06:30.160 --> 00:06:34.190
prior to working on observant.

00:06:34.190 --> 00:06:36.000
We couldn't get it doing

00:06:36.000 --> 00:06:37.919
everything we wanted, despite a bit of

00:06:37.919 --> 00:06:40.160
heavy lifting and debugging and stuff.

00:06:40.160 --> 00:06:42.880
So that's not finished. That was a bit
difficult.

00:06:42.880 --> 00:06:45.360
On the other hand, working on

00:06:45.360 --> 00:06:47.759
observant was fun, pretty lightweight,

00:06:47.759 --> 00:06:48.479
and easy.

00:06:48.479 --> 00:06:50.400
We got some experience co-editing

00:06:50.400 --> 00:06:52.400
things with these real-time tools.

00:06:52.400 --> 00:06:55.919
Obviously, the stack is somewhat work in
progress.

00:06:55.919 --> 00:06:58.000
I just wanted to give a shout out to

00:06:58.000 --> 00:07:00.800
crdt which was really fun,

00:07:00.800 --> 00:07:03.919
and Qiantan was making

00:07:03.919 --> 00:07:06.057
bug fixes for that as we go.

00:07:06.057 --> 00:07:08.960
Similarly, for firn and logseq,
the maintainers

00:07:08.960 --> 00:07:10.160
were really responsive,

00:07:10.160 --> 00:07:12.960
so that was nice. We did try to

00:07:12.960 --> 00:07:14.560
get Emacs running in the browser,

00:07:14.560 --> 00:07:15.840
thinking it would be really nice for

00:07:15.840 --> 00:07:19.290
people who didn't want to install it

00:07:19.290 --> 00:07:21.120
to get a chance to just try it,

00:07:21.120 --> 00:07:23.120
but actually, browsers capture things

00:07:23.120 --> 00:07:27.120
like C-n, so that was a bit annoying.

00:07:27.120 --> 00:07:28.479
But we did get lots of great feedback

00:07:28.479 --> 00:07:29.759
and interaction with people, including

00:07:29.759 --> 00:07:31.599
around this conference. Thank you

00:07:31.599 --> 00:07:33.759
to those who we've had discussions with.

00:07:35.599 --> 00:07:37.680
So, future work. Okay, so... Maybe you

00:07:37.680 --> 00:07:41.039
remember, I gave a talk a few years back
on Arxana.

00:07:41.039 --> 00:07:44.479
What might this have to do with Org
Mode?

00:07:44.479 --> 00:07:45.919
That's always the question one asks

00:07:45.919 --> 00:07:47.039
about Arxana.

00:07:47.039 --> 00:07:48.720
Arxana... One of the things it does

00:07:48.720 --> 00:07:50.319
is transclusions, and so that could be

00:07:50.319 --> 00:07:51.680
actually very helpful

00:07:51.680 --> 00:07:54.000
in connection with this "combined notes

00:07:54.000 --> 00:07:55.520
and write-up" workflow. So you might have

00:07:55.520 --> 00:07:58.400
an Org Mode. Some of these

00:07:58.400 --> 00:08:00.800
results we got back as raw results

00:08:00.800 --> 00:08:03.199
could go right into your write-up in a

00:08:03.919 --> 00:08:07.520
convenient way, at a level above--

00:08:07.520 --> 00:08:09.039
transparently, a level above the notebook.

00:08:09.039 --> 00:08:11.280
So you'd have the notebook alongside the

00:08:11.280 --> 00:08:13.440
write-up in that case,

00:08:13.440 --> 00:08:15.599
which is a variation on the

00:08:15.599 --> 00:08:17.423
literate programming workflow.

00:08:17.423 --> 00:08:19.120
This is speculative. Who knows?

00:08:19.120 --> 00:08:20.000
The other thought is,

00:08:20.000 --> 00:08:22.080
it just relates to the idea of network

00:08:22.080 --> 00:08:23.520
programming. So we can imagine these

00:08:23.520 --> 00:08:25.990
networks of computational nodes

00:08:25.990 --> 00:08:27.623
sitting inside of org-roam,

00:08:27.623 --> 00:08:28.800
calling each other.

00:08:28.800 --> 00:08:31.199
You would want to maintain some

00:08:31.199 --> 00:08:33.357
kind of model of that process.

00:08:33.357 --> 00:08:36.640
A general question is: how do we have a
remote control for

00:08:36.640 --> 00:08:37.957
long-running processes?

00:08:37.957 --> 00:08:38.320
You could do that

00:08:38.320 --> 00:08:39.857
in Lisp or Clojure,

00:08:39.857 --> 00:08:41.657
but maybe we could have something

00:08:41.657 --> 00:08:44.080
a little bit like that here.

00:08:44.080 --> 00:08:45.839
Conclusions: what have we actually

00:08:45.839 --> 00:08:48.080
addressed? Well, we addressed

00:08:48.080 --> 00:08:50.080
accessing any long-running process with

00:08:50.080 --> 00:08:51.600
a simple Org Mode interface.

00:08:51.600 --> 00:08:52.959
Obviously, we're not the only people to

00:08:52.959 --> 00:08:54.880
think about notebooks, but we think that

00:08:54.880 --> 00:08:56.880
Emacs has some advantages

00:08:56.880 --> 00:08:58.880
related to reproducible research and

00:08:58.880 --> 00:09:00.757
interdisciplinary collaboration.

00:09:00.757 --> 00:09:02.590
Let's just say that we think

00:09:02.590 --> 00:09:04.880
something is reproducible if it's
actually teachable

00:09:04.880 --> 00:09:07.200
to someone new and they can do it.

00:09:07.200 --> 00:09:09.190
Org Mode seems very useful for that.

00:09:09.190 --> 00:09:11.680
Many of the other talks have touched
on this.

00:09:11.680 --> 00:09:14.399
Interdisciplinary collaboration is great.

00:09:14.399 --> 00:09:16.000
This was an interdisciplinary

00:09:16.000 --> 00:09:17.839
collaboration on some level, but

00:09:17.839 --> 00:09:19.680
what about future work for

00:09:19.680 --> 00:09:21.190
bringing in scenario planners,

00:09:21.190 --> 00:09:22.320
simulation scientists,

00:09:22.320 --> 00:09:24.480
and local farmers, and building something

00:09:24.480 --> 00:09:26.023
that they can all use

00:09:26.023 --> 00:09:27.857
that's more than the sum of the
parts?

00:09:27.857 --> 00:09:30.720
So a little future work for everybody
else here.

00:09:30.720 --> 00:09:32.160
We think science should be widely

00:09:32.160 --> 00:09:34.560
teachable, shareable, semi-automated,

00:09:34.560 --> 00:09:36.720
transdisciplinary, and real-time

00:09:36.720 --> 00:09:40.399
like EmacsConf. So you can get in touch

00:09:40.399 --> 00:09:43.279
via these methods. The code--which

00:09:43.279 --> 00:09:45.200
is very much early stage work in
progress,

00:09:45.200 --> 00:09:46.640
as this was meant to be an experience

00:09:46.640 --> 00:09:48.800
report, not a "it's all done, here, it is

00:09:48.800 --> 00:09:49.920
polished" report--

00:09:49.920 --> 00:09:51.680
it's also online if you'd like to

00:09:51.680 --> 00:09:53.360
have a look. That's the end of the

00:09:53.360 --> 00:09:54.560
talk. I don't know if there's time

00:09:54.560 --> 00:09:57.440
for questions or not, but um I'm at your
disposal now.

00:09:57.440 --> 00:10:00.240
Thank you.

00:10:00.240 --> 00:10:04.079
(Amin: Many thanks for the tough job.

00:10:04.079 --> 00:10:07.120
Let's see. We have about I think four

00:10:07.120 --> 00:10:08.880
minutes for questions,

00:10:08.880 --> 00:10:10.880
and we have a couple of questions on

00:10:10.880 --> 00:10:12.320
the pad. Would you like to read them

00:10:12.320 --> 00:10:14.240
yourself or should I read them to you?)

00:10:14.240 --> 00:10:16.423
Just for the sake of easy management

00:10:16.423 --> 00:10:18.079
why don't you read them out, if
that's okay?

00:10:18.079 --> 00:10:21.279
(Amin: yeah, sure. They ask, "Have you looked

00:10:21.279 --> 00:10:25.440
into trying Sage Math? I've long wanted

00:10:25.440 --> 00:10:33.760
to use Sage Math in Org files.")

00:10:33.760 --> 00:10:36.959
Ray: Right. I wrote the answer that

00:10:36.959 --> 00:10:39.279
it should be possible because one can

00:10:39.279 --> 00:10:44.839
call it from a command.

00:10:44.839 --> 00:10:48.190
(Amin: okay, and I see there's

00:10:48.190 --> 00:10:50.079
another Sage Math question that you seem
to have answered

00:10:50.079 --> 00:10:52.100
as well, so I guess I won't repeat that.

00:10:54.880 --> 00:10:56.959
There's... "Let's not forget about embedded

00:10:56.959 --> 00:11:00.640
Calc in Emacs.")

00:11:00.640 --> 00:11:05.040
Joe: So the first demos actually were with
Calc.

00:11:05.040 --> 00:11:06.640
That's useful. Although I think it was a

00:11:06.640 --> 00:11:08.240
different--kind of a different command
line.

00:11:08.240 --> 00:11:11.839
Ray: Well, that was UNIX Calc.

00:11:11.839 --> 00:11:13.839
Joe: So, sure, there is calc, so that...

00:11:15.680 --> 00:11:19.120
Ray: Calc is already in Org Mode.

00:11:25.680 --> 00:11:39.760
(Amin: Still looking for questions.

00:11:39.760 --> 00:11:41.760
Okay, I think that's about it. I don't see

00:11:41.760 --> 00:11:44.320
any questions on the Etherpad.

00:11:44.320 --> 00:11:47.440
And let's see...

00:11:47.440 --> 00:11:53.040
Anything on irc?

00:11:53.040 --> 00:11:55.760
Nothing but praises and everyone

00:11:55.760 --> 00:11:57.290
thanking you. Thank you.)

00:11:57.290 --> 00:11:59.120
Ray: all right, you're welcome.

00:11:59.120 --> 00:12:00.240
Joe: Thanks a lot!

00:12:00.240 --> 00:12:01.923
We'll see you guys around then.

00:12:01.923 --> 00:12:06.800
Amin: Cheers, and see you around!