Joe: Hi, I'm Joe Corneli.
This is work I did with Ray Puzio and Cameron Smith.
They're the main protagonists in this story.
They are researchers who've been working
on theoretical biology.
In a typical project, they may use Maxima and Julia.
Their work combines biology, physics and computer science.
The latest work-in-progress is on branching processes for
cancer modeling.
How can Emacs possibly help?
Let's have a look. Moving code
and data between these different
programs by hand is
annoying. Separate workflows for
writing up notes and preparing publications
is perhaps even more annoying. All of
it is time consuming and error-prone.
So what about maybe using Jupyter?
We found something called Script of
Scripts. It solves some of those
problems because you can use
Maxima and Julia together, but we were
quite happy to explore Emacs-based
solutions, being Emcas enthusiasts
We even got Cameron to be enthusiastic about
doing Emacs, so that went nice.
Here's a little feature grid
of Emacs + Org versus your
generic tools that are in a
different, more general ecosystem.
As you can see, it's quite
feature-complete. You've got your
maxima-mode, julia-mode. You can use both of them
inside of org-mode.
You can present things with org-tree-slide.
You can set up a wiki inside of org-roam.
This is one I found
rather recently. You can even use
compatibly with org-roam, something called
logseq, which is in the browser, so that's nice.
You can do real-time collaborative editing,
either in a kind of pairing style
or in a more Etherpad style.
Obviously, you can manage your references.
You can typeset whatever you want.
You can publish work in progress on a blog.
Firn is another one of these
external Org Mode tools.
It's not actually in Emacs,
but works with Org Mode stuff. And, you know...
So we're good to go with all of that.
So what does that look like? Well,
here's a little example from before
they were doing... before we started really thinking
seriously about this stuff.
So this is just Maxima.
Well, Maxima doesn't have a long running
process by default. If you've ever used
Python, you have something called sessions.
They don't have that for Maxima, at least
not by default.
So how... What was the workaround?
There's this thing called
solve-for-u here
that shows up down below again in these
angle brackets, which you've seen maybe
in someone else's talk,
which means go to the previous
thing that was named solve-for-u
and do that all over again, so they do
that over again.
Here's the little Maxima code for
defining usol, so you've now
defined usol, and then you can use it
in the next expression. You get out a nice juicy
zero at the end. It's a little bit
like a partridge in a pear tree to have to
redefine everything every time.
So this is clearly at the level of
work-around. Maybe just
one more time looking through
that stuff.
Sorry. So, looking through that stuff,
this is... We're going to need something
like that, probably, for stitching
Maxima and Julia together. so it's
good to look a little bit
about how that might work.
First of all, you can cache
results, so if you wanted to save the
date out of block one at a certain time and
then use it again later...
At the time when I ran this code,
you can see I've got two slightly
different time stamps down below.
One's the cached result, and the other
was the result of reevaluating
the block. So you can move things around.
That's going to be useful. But you know,
that's not really the main problem.
The main problem is
making Maxima long-running.
The core of this talk is a new observant
facility, which is a general purpose
way to do that kind of thing, which
involves a very simple change to ob-core.
We'll give a quick
overview of that and show
an example. So here's the example,
a very simple sort of silly example.
What does it mean to have
a long-running process?
Here, I've set this
display2d to be false, which just means that
things are going to come
come across in 1d. Then I ask it to
expand something. I get LaTeX by default.
So that's what it means. It's that I've sent
something in and it's going to come
across in one view, which is great.
Maybe you'll also notice that there's
no semicolon, if you're a Maxima fan,
and things are coming across as TeX.
So those were some little bonus features.
I'll show you how that works later.
The change to ob-core is as follows.
Actually, this should say...
Instead of stream here, it should say
servant. Sorry. We tried an experimental
version which was called stream, so now it's
called servant. But all it does is it
overrides org-babel-execute lang for
arbitrary lang
if you have a servant
in your params. So that's the
change that hasn't been
pushed out or sent as a patch to anybody,
but it's a pretty minor change.
Here's an overview without the code.
Just a high level overview of
observant.el.
It stores information about these
processes in a hash table.
It can do pre-processing and
post-processing.
It does all these things. It stores
the output.
I mentioned here that, in principle, we
could store lots of output and have a
kind of browsable history,
although we don't do that presently.
But that's what observant does.
It does what you might expect.
Here's the Maxima on-ramp
to get Maxima brought in.
You have to obviously have
a Maxima process you can call.
puthash... this is the preprocessing
thing I mentioned, adding in some Tex
and adding in--
or deleting, rather--a substring.
Here is why you delete the substring.
It's because Maxima thinks it's a good
idea to tell you false
once you run check on things.
You've got to delete that back out to
get something coherent out of it.
So this is how to set up Maxima.
That's enough, really, of the demo.
It's not really a demo for
show and tell, but as this is an
experience report, I wanted to talk about
the experience of doing this.
Some negatives, like we tried to
get Emacs Jupyter working
prior to working on observant.
We couldn't get it doing
everything we wanted, despite a bit of
heavy lifting and debugging and stuff.
So that's not finished. That was a bit difficult.
On the other hand, working on
observant was fun, pretty lightweight,
and easy.
We got some experience co-editing
things with these real-time tools.
Obviously, the stack is somewhat work in progress.
I just wanted to give a shout out to
crdt which was really fun,
and Qiantan was making
bug fixes for that as we go.
Similarly, for firn and logseq, the maintainers
were really responsive,
so that was nice. We did try to
get Emacs running in the browser,
thinking it would be really nice for
people who didn't want to install it
to get a chance to just try it,
but actually, browsers capture things
like C-n, so that was a bit annoying.
But we did get lots of great feedback
and interaction with people, including
around this conference. Thank you
to those who we've had discussions with.
So, future work. Okay, so... Maybe you
remember, I gave a talk a few years back on Arxana.
What might this have to do with Org Mode?
That's always the question one asks
about Arxana.
Arxana... One of the things it does
is transclusions, and so that could be
actually very helpful
in connection with this "combined notes
and write-up" workflow. So you might have
an Org Mode. Some of these
results we got back as raw results
could go right into your write-up in a
convenient way, at a level above--
transparently, a level above the notebook.
So you'd have the notebook alongside the
write-up in that case,
which is a variation on the
literate programming workflow.
This is speculative. Who knows?
The other thought is,
it just relates to the idea of network
programming. So we can imagine these
networks of computational nodes
sitting inside of org-roam,
calling each other.
You would want to maintain some
kind of model of that process.
A general question is: how do we have a remote control for
long-running processes?
You could do that
in Lisp or Clojure,
but maybe we could have something
a little bit like that here.
Conclusions: what have we actually
addressed? Well, we addressed
accessing any long-running process with
a simple Org Mode interface.
Obviously, we're not the only people to
think about notebooks, but we think that
Emacs has some advantages
related to reproducible research and
interdisciplinary collaboration.
Let's just say that we think
something is reproducible if it's actually teachable
to someone new and they can do it.
Org Mode seems very useful for that.
Many of the other talks have touched on this.
Interdisciplinary collaboration is great.
This was an interdisciplinary
collaboration on some level, but
what about future work for
bringing in scenario planners,
simulation scientists,
and local farmers, and building something
that they can all use
that's more than the sum of the parts?
So a little future work for everybody else here.
We think science should be widely
teachable, shareable, semi-automated,
transdisciplinary, and real-time
like EmacsConf. So you can get in touch
via these methods. The code--which
is very much early stage work in progress,
as this was meant to be an experience
report, not a "it's all done, here, it is
polished" report--
it's also online if you'd like to
have a look. That's the end of the
talk. I don't know if there's time
for questions or not, but um I'm at your disposal now.
Thank you.
(Amin: Many thanks for the tough job.
Let's see. We have about I think four
minutes for questions,
and we have a couple of questions on
the pad. Would you like to read them
yourself or should I read them to you?)
Just for the sake of easy management
why don't you read them out, if that's okay?
(Amin: yeah, sure. They ask, "Have you looked
into trying Sage Math? I've long wanted
to use Sage Math in Org files.")
Ray: Right. I wrote the answer that
it should be possible because one can
call it from a command.
(Amin: okay, and I see there's
another Sage Math question that you seem to have answered
as well, so I guess I won't repeat that.
There's... "Let's not forget about embedded
Calc in Emacs.")
Joe: So the first demos actually were with Calc.
That's useful. Although I think it was a
different--kind of a different command line.
Ray: Well, that was UNIX Calc.
Joe: So, sure, there is calc, so that...
Ray: Calc is already in Org Mode.
(Amin: Still looking for questions.
Okay, I think that's about it. I don't see
any questions on the Etherpad.
And let's see...
Anything on irc?
Nothing but praises and everyone
thanking you. Thank you.)
Ray: all right, you're welcome.
Joe: Thanks a lot!
We'll see you guys around then.
Amin: Cheers, and see you around!