diff options
| -rw-r--r-- | 2020/info/13.md | 379 |
1 files changed, 378 insertions, 1 deletions
diff --git a/2020/info/13.md b/2020/info/13.md index 3140b467..f98ef404 100644 --- a/2020/info/13.md +++ b/2020/info/13.md @@ -2,7 +2,8 @@ Joseph Corneli, Raymond Puzio, and Cameron Ray Smith [[!template id=vid src="https://mirror.csclub.uwaterloo.ca/emacsconf/2020/emacsconf-2020--13-experience-report-steps-to-emacs-hyper-notebooks--joseph-corneli-raymond-puzio-cameron-ray-smith.webm" subtitles="/2020/subtitles/emacsconf-2020--13-experience-report-steps-to-emacs-hyper-notebooks--joseph-corneli-raymond-puzio-cameron-ray-smith.vtt"]] -[Download compressed .webm video (8.6M)](https://mirror.csclub.uwaterloo.ca/emacsconf/2020/smaller/emacsconf-2020--13-experience-report-steps-to-emacs-hyper-notebooks--joseph-corneli-raymond-puzio-cameron-ray-smith--vp9-q56-video-original-audio.webm) +[Download compressed .webm video (8.6M)](https://mirror.csclub.uwaterloo.ca/emacsconf/2020/smaller/emacsconf-2020--13-experience-report-steps-to-emacs-hyper-notebooks--joseph-corneli-raymond-puzio-cameron-ray-smith--vp9-q56-video-original-audio.webm) +[View transcript](#transcript) We present a short experience report from the perspective of two long-time Emacs users and one relative newcomer. Our motivations @@ -52,3 +53,379 @@ I think he used org-tree-slides, like some earlier presentations. # Notes - <https://github.com/exp2exp/ob-servant> + +<a name="transcript"></a> +# Transcripts + +00:00:00.320 --> 00:00:30.800 +Joe: Hi, I'm Joe Corneli. This is work I +did with Ray Puzio and Cameron Smith. +They're the main protagonists in this +story. They are researchers who've been +working on theoretical biology. In a +typical project, they may use Maxima and +Julia. Their work combines biology, +physics and computer science. The latest +work-in-progress is on branching +processes for cancer modeling. + +00:00:30.800 --> 00:00:48.640 +How can Emacs possibly help? Let's have +a look. Moving code and data between +these different programs by hand is +annoying. Separate workflows for writing +up notes and preparing publications is +perhaps even more annoying. All of it is +time consuming and error-prone. + +00:00:48.640 --> 00:01:10.057 +So what about maybe using Jupyter? We +found something called Script of +Scripts. It solves some of those +problems because you can use Maxima and +Julia together, but we were quite happy +to explore Emacs-based solutions, being +Emacs enthusiasts. We even got Cameron to +be enthusiastic about doing Emacs, so +that went nice. + +00:01:10.057 --> 00:02:05.657 +Here's a little feature grid of Emacs + +Org versus your generic tools that are +in a different, more general ecosystem. +As you can see, it's quite +feature-complete. You've got your +maxima-mode, julia-mode. You can use +both of them inside of org-mode. You can +present things with org-tree-slide. You +can set up a wiki inside of org-roam. +This is one I found rather recently. You +can even use compatibly with org-roam, +something called logseq, which is in the +browser, so that's nice. You can do +real-time collaborative editing, either +in a kind of pairing style or in a more +Etherpad style. Obviously, you can +manage your references. You can typeset +whatever you want. You can publish work +in progress on a blog. Firn is another +one of these external Org Mode tools. +It's not actually in Emacs, but works +with Org Mode stuff. And, you know... So +we're good to go with all of that. + +00:02:05.657 --> 00:02:13.890 +So what does that look like? Well, +here's a little example from before they +were doing... before we started really +thinking seriously about this stuff. + +00:02:13.890 --> 00:02:45.280 +So this is just Maxima. Well, Maxima +doesn't have a long running process by +default. If you've ever used Python, you +have something called sessions. They +don't have that for Maxima, at least not +by default. So how... What was the +workaround? There's this thing called +solve-for-u here that shows up down +below again in these angle brackets, +which you've seen maybe in someone +else's talk, which means go to the +previous thing that was named +solve-for-u and do that all over again, +so they do that over again. + +00:02:45.280 --> 00:03:00.640 +Here's the little Maxima code for +defining usol, so you've now defined +usol, and then you can use it in the +next expression. You get out a nice +juicy zero at the end. It's a little bit +like a partridge in a pear tree to have +to redefine everything every time. + +00:03:00.640 --> 00:03:22.590 +So this is clearly at the level of +work-around. Maybe just one more time +looking through that stuff. Sorry. So, +looking through that stuff, this is... +We're going to need something like that, +probably, for stitching Maxima and Julia +together. so it's good to look a little +bit about how that might work. + +00:03:22.590 --> 00:03:46.923 +First of all, you can cache results, so +if you wanted to save the date out of +block one at a certain time and then use +it again later... At the time when I ran +this code, you can see I've got two +slightly different time stamps down +below. One's the cached result, and the +other was the result of reevaluating the +block. So you can move things around. +That's going to be useful. But you know, +that's not really the main problem. + +00:03:46.923 --> 00:04:11.760 +The main problem is making Maxima +long-running. The core of this talk is a +new observant facility, which is a +general purpose way to do that kind of +thing, which involves a very simple +change to ob-core. We'll give a quick +overview of that and show an example. So +here's the example, a very simple sort +of silly example. + +00:04:11.760 --> 00:04:30.240 +What does it mean to have a long-running +process? Here, I've set this display2d +to be false, which just means that +things are going to come come across in +1d. Then I ask it to expand something. I +get LaTeX by default. So that's what it +means. It's that I've sent something in +and it's going to come across in one +view, which is great. + +00:04:30.240 --> 00:04:40.320 +Maybe you'll also notice that there's no +semicolon, if you're a Maxima fan, and +things are coming across as TeX. So +those were some little bonus features. +I'll show you how that works later. + +00:04:41.040 --> 00:05:13.759 +The change to ob-core is as follows. +Actually, this should say... Instead of +stream here, it should say servant. +Sorry. We tried an experimental version +which was called stream, so now it's +called servant. But all it does is it +overrides org-babel-execute lang for +arbitrary lang if you have a servant in +your params. So that's the change that +hasn't been pushed out or sent as a +patch to anybody, but it's a pretty +minor change. + +00:05:13.759 --> 00:05:30.720 +Here's an overview without the code. +Just a high level overview of +observant.el. It stores information +about these processes in a hash table. +It can do pre-processing and +post-processing. It does all these +things. It stores the output. + +00:05:30.720 --> 00:05:40.639 +I mentioned here that, in principle, we +could store lots of output and have a +kind of browsable history, although we +don't do that presently. But that's what +observant does. It does what you might +expect. + +00:05:41.440 --> 00:06:16.960 +Here's the Maxima on-ramp to get Maxima +brought in. You have to obviously have a +Maxima process you can call. puthash... +this is the preprocessing thing I +mentioned, adding in some Tex and adding +in-- or deleting, rather--a substring. +Here is why you delete the substring. +It's because Maxima thinks it's a good +idea to tell you false once you run +check on things. You've got to delete +that back out to get something coherent +out of it. So this is how to set up +Maxima. + +00:06:16.960 --> 00:06:25.440 +That's enough, really, of the demo. It's +not really a demo for show and tell, but +as this is an experience report, I +wanted to talk about the experience of +doing this. + +00:06:25.440 --> 00:06:42.880 +Some negatives, like we tried to get +Emacs Jupyter working prior to working +on observant. We couldn't get it doing +everything we wanted, despite a bit of +heavy lifting and debugging and stuff. +So that's not finished. That was a bit +difficult. + +00:06:42.880 --> 00:07:11.695 +On the other hand, working on observant +was fun, pretty lightweight, and easy. +We got some experience co-editing things +with these real-time tools. Obviously, +the stack is somewhat work in progress. +I just wanted to give a shout out to +crdt which was really fun, and Qiantan +was making bug fixes for that as we go. +Similarly, for firn and logseq, the +maintainers were really responsive, so +that was nice. + +00:07:11.695 --> 00:07:27.120 +We did try to get Emacs running in the +browser, thinking it would be really +nice for people who didn't want to +install it to get a chance to just try +it, but actually, browsers capture +things like C-n, so that was a bit +annoying. + +00:07:27.120 --> 00:07:33.759 +But we did get lots of great feedback +and interaction with people, including +around this conference. Thank you to +those who we've had discussions with. + +00:07:35.599 --> 00:08:19.120 +So, future work. Okay, so... Maybe you +remember, I gave a talk a few years back +on Arxana. What might this have to do +with Org Mode? That's always the +question one asks about Arxana. +Arxana... One of the things it does is +transclusions, and so that could be +actually very helpful in connection with +this "combined notes and write-up" +workflow. So you might have an Org Mode. +Some of these results we got back as raw +results could go right into your +write-up in a convenient way, at a level +above-- transparently, a level above the +notebook. So you'd have the notebook +alongside the write-up in that case, +which is a variation on the literate +programming workflow. This is +speculative. Who knows? + +00:08:19.120 --> 00:08:33.357 +The other thought is, +it just relates to the idea of network +programming. So we can imagine these +networks of computational nodes +sitting inside of org-roam, +calling each other. +You would want to maintain some +kind of model of that process. + +00:08:33.357 --> 00:09:11.680 +A general question is: how do we have a +remote control for long-running +processes? You could do that in Lisp or +Clojure, but maybe we could have +something a little bit like that here. +Conclusions: what have we actually +addressed? Well, we addressed accessing +any long-running process with a simple +Org Mode interface. Obviously, we're not +the only people to think about +notebooks, but we think that Emacs has +some advantages related to reproducible +research and interdisciplinary +collaboration. Let's just say that we +think something is reproducible if it's +actually teachable to someone new and +they can do it. Org Mode seems very +useful for that. Many of the other talks +have touched on this. + +00:09:11.680 --> 00:09:27.857 +Interdisciplinary collaboration is +great. This was an interdisciplinary +collaboration on some level, but what +about future work for bringing in +scenario planners, simulation +scientists, and local farmers, and +building something that they can all use +that's more than the sum of the parts? + +00:09:27.857 --> 00:09:38.135 +So a little future work for everybody +else here. We think science should be +widely teachable, shareable, +semi-automated, transdisciplinary, and +real-time like EmacsConf. + +00:09:38.135 --> 00:10:00.240 +So you can get in touch via these +methods. The code--which is very much +early stage work in progress, as this +was meant to be an experience report, +not a "it's all done, here, it is +polished" report-- it's also online if +you'd like to have a look. That's the +end of the talk. I don't know if there's +time for questions or not, but um I'm at +your disposal now. Thank you. + +00:10:00.240 --> 00:10:14.240 +(Amin: Many thanks for the tough job. +Let's see. We have about I think four +minutes for questions, and we have a +couple of questions on the pad. Would +you like to read them yourself or should +I read them to you?) + +00:10:14.240 --> 00:10:18.079 +Just for the sake of easy management why +don't you read them out, if that's okay? + +00:10:18.079 --> 00:10:33.760 +(Amin: yeah, sure. They ask, "Have you +looked into trying Sage Math? I've long +wanted to use Sage Math in Org files.") + +00:10:33.760 --> 00:10:44.839 +Ray: Right. I wrote the answer that it +should be possible because one can call +it from a command. + +00:10:44.839 --> 00:11:00.640 +(Amin: okay, and I see there's another +Sage Math question that you seem to have +answered as well, so I guess I won't +repeat that. There's... "Let's not +forget about embedded Calc in Emacs.") + +00:11:00.640 --> 00:11:08.240 +Joe: So the first demos actually were +with Calc. That's useful. Although I +think it was a different--kind of a +different command line. + +00:11:08.240 --> 00:11:11.839 +Ray: Well, that was UNIX Calc. + +00:11:11.839 --> 00:11:13.839 +Joe: So, sure, there is calc, so that... + +00:11:15.680 --> 00:11:19.120 +Ray: Calc is already in Org Mode. + +00:11:25.680 --> 00:11:57.290 +(Amin: Still looking for questions. +Okay, I think that's about it. I don't +see any questions on the Etherpad. And +let's see... Anything on irc? Nothing +but praises and everyone thanking you. +Thank you.) + +00:11:57.290 --> 00:11:59.120 +Ray: all right, you're welcome. + +00:11:59.120 --> 00:12:01.923 +Joe: Thanks a lot! +We'll see you guys around then. + +00:12:01.923 --> 00:12:06.800 +Amin: Cheers, and see you around! |