WEBVTT 00:00:00.320 --> 00:00:03.280 Joe: Hi, I'm Joe Corneli. 00:00:03.280 --> 00:00:06.879 This is work I did with Ray Puzio and Cameron Smith. 00:00:06.879 --> 00:00:11.123 They're the main protagonists in this story. 00:00:11.123 --> 00:00:14.960 They are researchers who've been working 00:00:14.960 --> 00:00:17.490 on theoretical biology. 00:00:17.490 --> 00:00:21.357 In a typical project, they may use Maxima and Julia. 00:00:21.357 --> 00:00:24.800 Their work combines biology, physics and computer science. 00:00:24.800 --> 00:00:29.439 The latest work-in-progress is on branching processes for 00:00:29.439 --> 00:00:30.800 cancer modeling. 00:00:30.800 --> 00:00:34.719 How can Emacs possibly help? 00:00:34.719 --> 00:00:37.360 Let's have a look. Moving code 00:00:37.360 --> 00:00:38.399 and data between these different 00:00:38.399 --> 00:00:39.680 programs by hand is 00:00:39.680 --> 00:00:42.000 annoying. Separate workflows for 00:00:42.000 --> 00:00:44.399 writing up notes and preparing publications 00:00:44.399 --> 00:00:46.000 is perhaps even more annoying. All of 00:00:46.000 --> 00:00:48.640 it is time consuming and error-prone. 00:00:48.640 --> 00:00:52.000 So what about maybe using Jupyter? 00:00:52.000 --> 00:00:53.760 We found something called Script of 00:00:53.760 --> 00:00:55.199 Scripts. It solves some of those 00:00:55.199 --> 00:00:58.399 problems because you can use 00:00:58.399 --> 00:01:01.120 Maxima and Julia together, but we were 00:01:01.120 --> 00:01:02.640 quite happy to explore Emacs-based 00:01:02.640 --> 00:01:04.890 solutions, being Emcas enthusiasts 00:01:04.890 --> 00:01:07.760 We even got Cameron to be enthusiastic about 00:01:07.760 --> 00:01:10.057 doing Emacs, so that went nice. 00:01:10.057 --> 00:01:12.400 Here's a little feature grid 00:01:12.400 --> 00:01:15.360 of Emacs + Org versus your 00:01:15.360 --> 00:01:18.390 generic tools that are in a 00:01:18.390 --> 00:01:20.123 different, more general ecosystem. 00:01:20.123 --> 00:01:21.520 As you can see, it's quite 00:01:21.520 --> 00:01:22.957 feature-complete. You've got your 00:01:22.957 --> 00:01:25.520 maxima-mode, julia-mode. You can use both of them 00:01:25.520 --> 00:01:27.119 inside of org-mode. 00:01:27.119 --> 00:01:28.720 You can present things with org-tree-slide. 00:01:28.720 --> 00:01:31.490 You can set up a wiki inside of org-roam. 00:01:31.490 --> 00:01:33.680 This is one I found 00:01:33.680 --> 00:01:35.759 rather recently. You can even use 00:01:35.759 --> 00:01:37.759 compatibly with org-roam, something called 00:01:37.759 --> 00:01:42.159 logseq, which is in the browser, so that's nice. 00:01:42.159 --> 00:01:44.957 You can do real-time collaborative editing, 00:01:44.957 --> 00:01:47.657 either in a kind of pairing style 00:01:47.657 --> 00:01:49.280 or in a more Etherpad style. 00:01:49.280 --> 00:01:51.520 Obviously, you can manage your references. 00:01:51.520 --> 00:01:53.457 You can typeset whatever you want. 00:01:53.457 --> 00:01:55.759 You can publish work in progress on a blog. 00:01:55.759 --> 00:01:57.439 Firn is another one of these 00:01:57.439 --> 00:01:59.790 external Org Mode tools. 00:01:59.790 --> 00:02:00.560 It's not actually in Emacs, 00:02:00.560 --> 00:02:03.557 but works with Org Mode stuff. And, you know... 00:02:03.557 --> 00:02:05.657 So we're good to go with all of that. 00:02:05.657 --> 00:02:07.423 So what does that look like? Well, 00:02:07.423 --> 00:02:09.423 here's a little example from before 00:02:09.423 --> 00:02:12.800 they were doing... before we started really thinking 00:02:12.800 --> 00:02:13.890 seriously about this stuff. 00:02:13.890 --> 00:02:15.599 So this is just Maxima. 00:02:15.599 --> 00:02:17.440 Well, Maxima doesn't have a long running 00:02:17.440 --> 00:02:19.280 process by default. If you've ever used 00:02:19.280 --> 00:02:22.480 Python, you have something called sessions. 00:02:22.480 --> 00:02:23.920 They don't have that for Maxima, at least 00:02:23.920 --> 00:02:24.959 not by default. 00:02:24.959 --> 00:02:27.599 So how... What was the workaround? 00:02:27.599 --> 00:02:29.360 There's this thing called 00:02:29.360 --> 00:02:32.480 solve-for-u here 00:02:32.480 --> 00:02:34.879 that shows up down below again in these 00:02:34.879 --> 00:02:36.319 angle brackets, which you've seen maybe 00:02:36.319 --> 00:02:37.360 in someone else's talk, 00:02:37.360 --> 00:02:40.480 which means go to the previous 00:02:40.480 --> 00:02:41.920 thing that was named solve-for-u 00:02:41.920 --> 00:02:43.920 and do that all over again, so they do 00:02:43.920 --> 00:02:45.280 that over again. 00:02:45.280 --> 00:02:47.440 Here's the little Maxima code for 00:02:47.440 --> 00:02:48.990 defining usol, so you've now 00:02:48.990 --> 00:02:51.723 defined usol, and then you can use it 00:02:51.723 --> 00:02:54.000 in the next expression. You get out a nice juicy 00:02:54.000 --> 00:02:56.560 zero at the end. It's a little bit 00:02:57.200 --> 00:02:58.959 like a partridge in a pear tree to have to 00:02:58.959 --> 00:03:00.640 redefine everything every time. 00:03:00.640 --> 00:03:02.490 So this is clearly at the level of 00:03:02.490 --> 00:03:04.159 work-around. Maybe just 00:03:04.159 --> 00:03:06.223 one more time looking through 00:03:06.223 --> 00:03:11.599 that stuff. 00:03:11.599 --> 00:03:13.760 Sorry. So, looking through that stuff, 00:03:13.760 --> 00:03:15.280 this is... We're going to need something 00:03:15.280 --> 00:03:17.200 like that, probably, for stitching 00:03:17.200 --> 00:03:19.599 Maxima and Julia together. so it's 00:03:19.599 --> 00:03:20.890 good to look a little bit 00:03:20.890 --> 00:03:22.590 about how that might work. 00:03:22.590 --> 00:03:23.920 First of all, you can cache 00:03:23.920 --> 00:03:25.680 results, so if you wanted to save the 00:03:25.680 --> 00:03:28.480 date out of block one at a certain time and 00:03:28.480 --> 00:03:30.990 then use it again later... 00:03:30.990 --> 00:03:33.280 At the time when I ran this code, 00:03:33.280 --> 00:03:34.640 you can see I've got two slightly 00:03:34.640 --> 00:03:36.623 different time stamps down below. 00:03:36.623 --> 00:03:38.323 One's the cached result, and the other 00:03:38.323 --> 00:03:40.319 was the result of reevaluating 00:03:40.319 --> 00:03:42.640 the block. So you can move things around. 00:03:43.760 --> 00:03:46.000 That's going to be useful. But you know, 00:03:46.000 --> 00:03:46.923 that's not really the main problem. 00:03:46.923 --> 00:03:48.080 The main problem is 00:03:48.080 --> 00:03:50.799 making Maxima long-running. 00:03:50.799 --> 00:03:53.920 The core of this talk is a new observant 00:03:53.920 --> 00:03:56.400 facility, which is a general purpose 00:03:56.400 --> 00:03:59.280 way to do that kind of thing, which 00:03:59.280 --> 00:04:01.823 involves a very simple change to ob-core. 00:04:01.823 --> 00:04:04.239 We'll give a quick 00:04:04.239 --> 00:04:05.360 overview of that and show 00:04:05.360 --> 00:04:08.480 an example. So here's the example, 00:04:08.480 --> 00:04:11.760 a very simple sort of silly example. 00:04:11.760 --> 00:04:12.957 What does it mean to have 00:04:12.957 --> 00:04:13.990 a long-running process? 00:04:13.990 --> 00:04:14.640 Here, I've set this 00:04:14.640 --> 00:04:18.560 display2d to be false, which just means that 00:04:18.560 --> 00:04:19.440 things are going to come 00:04:19.440 --> 00:04:22.320 come across in 1d. Then I ask it to 00:04:22.320 --> 00:04:25.199 expand something. I get LaTeX by default. 00:04:25.199 --> 00:04:27.280 So that's what it means. It's that I've sent 00:04:27.280 --> 00:04:28.639 something in and it's going to come 00:04:28.639 --> 00:04:30.240 across in one view, which is great. 00:04:30.240 --> 00:04:32.080 Maybe you'll also notice that there's 00:04:32.080 --> 00:04:34.560 no semicolon, if you're a Maxima fan, 00:04:34.560 --> 00:04:36.720 and things are coming across as TeX. 00:04:36.720 --> 00:04:38.400 So those were some little bonus features. 00:04:38.400 --> 00:04:40.320 I'll show you how that works later. 00:04:41.040 --> 00:04:45.440 The change to ob-core is as follows. 00:04:45.440 --> 00:04:48.880 Actually, this should say... 00:04:48.880 --> 00:04:51.520 Instead of stream here, it should say 00:04:51.520 --> 00:04:54.800 servant. Sorry. We tried an experimental 00:04:54.800 --> 00:04:56.160 version which was called stream, so now it's 00:04:56.160 --> 00:04:58.160 called servant. But all it does is it 00:04:58.160 --> 00:05:01.520 overrides org-babel-execute lang for 00:05:01.520 --> 00:05:02.639 arbitrary lang 00:05:02.639 --> 00:05:05.919 if you have a servant 00:05:05.919 --> 00:05:07.840 in your params. So that's the 00:05:07.840 --> 00:05:09.759 change that hasn't been 00:05:09.759 --> 00:05:11.919 pushed out or sent as a patch to anybody, 00:05:11.919 --> 00:05:13.759 but it's a pretty minor change. 00:05:13.759 --> 00:05:16.960 Here's an overview without the code. 00:05:16.960 --> 00:05:19.080 Just a high level overview of 00:05:19.080 --> 00:05:20.720 observant.el. 00:05:20.720 --> 00:05:22.160 It stores information about these 00:05:22.160 --> 00:05:24.479 processes in a hash table. 00:05:24.479 --> 00:05:26.080 It can do pre-processing and 00:05:26.080 --> 00:05:27.600 post-processing. 00:05:27.600 --> 00:05:29.759 It does all these things. It stores 00:05:29.759 --> 00:05:30.720 the output. 00:05:30.720 --> 00:05:32.479 I mentioned here that, in principle, we 00:05:32.479 --> 00:05:34.080 could store lots of output and have a 00:05:34.080 --> 00:05:35.657 kind of browsable history, 00:05:35.657 --> 00:05:37.790 although we don't do that presently. 00:05:37.790 --> 00:05:38.790 But that's what observant does. 00:05:38.790 --> 00:05:40.639 It does what you might expect. 00:05:41.440 --> 00:05:46.190 Here's the Maxima on-ramp 00:05:46.190 --> 00:05:48.160 to get Maxima brought in. 00:05:48.160 --> 00:05:49.257 You have to obviously have 00:05:49.257 --> 00:05:51.360 a Maxima process you can call. 00:05:51.360 --> 00:05:54.960 puthash... this is the preprocessing 00:05:54.960 --> 00:05:57.840 thing I mentioned, adding in some Tex 00:05:57.840 --> 00:05:58.960 and adding in-- 00:05:58.960 --> 00:06:01.520 or deleting, rather--a substring. 00:06:01.520 --> 00:06:03.759 Here is why you delete the substring. 00:06:03.759 --> 00:06:06.960 It's because Maxima thinks it's a good 00:06:06.960 --> 00:06:08.240 idea to tell you false 00:06:08.240 --> 00:06:10.080 once you run check on things. 00:06:10.080 --> 00:06:11.759 You've got to delete that back out to 00:06:11.759 --> 00:06:13.680 get something coherent out of it. 00:06:13.680 --> 00:06:16.960 So this is how to set up Maxima. 00:06:16.960 --> 00:06:19.157 That's enough, really, of the demo. 00:06:19.157 --> 00:06:20.000 It's not really a demo for 00:06:20.000 --> 00:06:21.919 show and tell, but as this is an 00:06:21.919 --> 00:06:23.600 experience report, I wanted to talk about 00:06:23.600 --> 00:06:25.440 the experience of doing this. 00:06:25.440 --> 00:06:28.080 Some negatives, like we tried to 00:06:28.080 --> 00:06:30.160 get Emacs Jupyter working 00:06:30.160 --> 00:06:34.190 prior to working on observant. 00:06:34.190 --> 00:06:36.000 We couldn't get it doing 00:06:36.000 --> 00:06:37.919 everything we wanted, despite a bit of 00:06:37.919 --> 00:06:40.160 heavy lifting and debugging and stuff. 00:06:40.160 --> 00:06:42.880 So that's not finished. That was a bit difficult. 00:06:42.880 --> 00:06:45.360 On the other hand, working on 00:06:45.360 --> 00:06:47.759 observant was fun, pretty lightweight, 00:06:47.759 --> 00:06:48.479 and easy. 00:06:48.479 --> 00:06:50.400 We got some experience co-editing 00:06:50.400 --> 00:06:52.400 things with these real-time tools. 00:06:52.400 --> 00:06:55.919 Obviously, the stack is somewhat work in progress. 00:06:55.919 --> 00:06:58.000 I just wanted to give a shout out to 00:06:58.000 --> 00:07:00.800 crdt which was really fun, 00:07:00.800 --> 00:07:03.919 and Qiantan was making 00:07:03.919 --> 00:07:06.057 bug fixes for that as we go. 00:07:06.057 --> 00:07:08.960 Similarly, for firn and logseq, the maintainers 00:07:08.960 --> 00:07:10.160 were really responsive, 00:07:10.160 --> 00:07:12.960 so that was nice. We did try to 00:07:12.960 --> 00:07:14.560 get Emacs running in the browser, 00:07:14.560 --> 00:07:15.840 thinking it would be really nice for 00:07:15.840 --> 00:07:19.290 people who didn't want to install it 00:07:19.290 --> 00:07:21.120 to get a chance to just try it, 00:07:21.120 --> 00:07:23.120 but actually, browsers capture things 00:07:23.120 --> 00:07:27.120 like C-n, so that was a bit annoying. 00:07:27.120 --> 00:07:28.479 But we did get lots of great feedback 00:07:28.479 --> 00:07:29.759 and interaction with people, including 00:07:29.759 --> 00:07:31.599 around this conference. Thank you 00:07:31.599 --> 00:07:33.759 to those who we've had discussions with. 00:07:35.599 --> 00:07:37.680 So, future work. Okay, so... Maybe you 00:07:37.680 --> 00:07:41.039 remember, I gave a talk a few years back on Arxana. 00:07:41.039 --> 00:07:44.479 What might this have to do with Org Mode? 00:07:44.479 --> 00:07:45.919 That's always the question one asks 00:07:45.919 --> 00:07:47.039 about Arxana. 00:07:47.039 --> 00:07:48.720 Arxana... One of the things it does 00:07:48.720 --> 00:07:50.319 is transclusions, and so that could be 00:07:50.319 --> 00:07:51.680 actually very helpful 00:07:51.680 --> 00:07:54.000 in connection with this "combined notes 00:07:54.000 --> 00:07:55.520 and write-up" workflow. So you might have 00:07:55.520 --> 00:07:58.400 an Org Mode. Some of these 00:07:58.400 --> 00:08:00.800 results we got back as raw results 00:08:00.800 --> 00:08:03.199 could go right into your write-up in a 00:08:03.919 --> 00:08:07.520 convenient way, at a level above-- 00:08:07.520 --> 00:08:09.039 transparently, a level above the notebook. 00:08:09.039 --> 00:08:11.280 So you'd have the notebook alongside the 00:08:11.280 --> 00:08:13.440 write-up in that case, 00:08:13.440 --> 00:08:15.599 which is a variation on the 00:08:15.599 --> 00:08:17.423 literate programming workflow. 00:08:17.423 --> 00:08:19.120 This is speculative. Who knows? 00:08:19.120 --> 00:08:20.000 The other thought is, 00:08:20.000 --> 00:08:22.080 it just relates to the idea of network 00:08:22.080 --> 00:08:23.520 programming. So we can imagine these 00:08:23.520 --> 00:08:25.990 networks of computational nodes 00:08:25.990 --> 00:08:27.623 sitting inside of org-roam, 00:08:27.623 --> 00:08:28.800 calling each other. 00:08:28.800 --> 00:08:31.199 You would want to maintain some 00:08:31.199 --> 00:08:33.357 kind of model of that process. 00:08:33.357 --> 00:08:36.640 A general question is: how do we have a remote control for 00:08:36.640 --> 00:08:37.957 long-running processes? 00:08:37.957 --> 00:08:38.320 You could do that 00:08:38.320 --> 00:08:39.857 in Lisp or Clojure, 00:08:39.857 --> 00:08:41.657 but maybe we could have something 00:08:41.657 --> 00:08:44.080 a little bit like that here. 00:08:44.080 --> 00:08:45.839 Conclusions: what have we actually 00:08:45.839 --> 00:08:48.080 addressed? Well, we addressed 00:08:48.080 --> 00:08:50.080 accessing any long-running process with 00:08:50.080 --> 00:08:51.600 a simple Org Mode interface. 00:08:51.600 --> 00:08:52.959 Obviously, we're not the only people to 00:08:52.959 --> 00:08:54.880 think about notebooks, but we think that 00:08:54.880 --> 00:08:56.880 Emacs has some advantages 00:08:56.880 --> 00:08:58.880 related to reproducible research and 00:08:58.880 --> 00:09:00.757 interdisciplinary collaboration. 00:09:00.757 --> 00:09:02.590 Let's just say that we think 00:09:02.590 --> 00:09:04.880 something is reproducible if it's actually teachable 00:09:04.880 --> 00:09:07.200 to someone new and they can do it. 00:09:07.200 --> 00:09:09.190 Org Mode seems very useful for that. 00:09:09.190 --> 00:09:11.680 Many of the other talks have touched on this. 00:09:11.680 --> 00:09:14.399 Interdisciplinary collaboration is great. 00:09:14.399 --> 00:09:16.000 This was an interdisciplinary 00:09:16.000 --> 00:09:17.839 collaboration on some level, but 00:09:17.839 --> 00:09:19.680 what about future work for 00:09:19.680 --> 00:09:21.190 bringing in scenario planners, 00:09:21.190 --> 00:09:22.320 simulation scientists, 00:09:22.320 --> 00:09:24.480 and local farmers, and building something 00:09:24.480 --> 00:09:26.023 that they can all use 00:09:26.023 --> 00:09:27.857 that's more than the sum of the parts? 00:09:27.857 --> 00:09:30.720 So a little future work for everybody else here. 00:09:30.720 --> 00:09:32.160 We think science should be widely 00:09:32.160 --> 00:09:34.560 teachable, shareable, semi-automated, 00:09:34.560 --> 00:09:36.720 transdisciplinary, and real-time 00:09:36.720 --> 00:09:40.399 like EmacsConf. So you can get in touch 00:09:40.399 --> 00:09:43.279 via these methods. The code--which 00:09:43.279 --> 00:09:45.200 is very much early stage work in progress, 00:09:45.200 --> 00:09:46.640 as this was meant to be an experience 00:09:46.640 --> 00:09:48.800 report, not a "it's all done, here, it is 00:09:48.800 --> 00:09:49.920 polished" report-- 00:09:49.920 --> 00:09:51.680 it's also online if you'd like to 00:09:51.680 --> 00:09:53.360 have a look. That's the end of the 00:09:53.360 --> 00:09:54.560 talk. I don't know if there's time 00:09:54.560 --> 00:09:57.440 for questions or not, but um I'm at your disposal now. 00:09:57.440 --> 00:10:00.240 Thank you. 00:10:00.240 --> 00:10:04.079 (Amin: Many thanks for the tough job. 00:10:04.079 --> 00:10:07.120 Let's see. We have about I think four 00:10:07.120 --> 00:10:08.880 minutes for questions, 00:10:08.880 --> 00:10:10.880 and we have a couple of questions on 00:10:10.880 --> 00:10:12.320 the pad. Would you like to read them 00:10:12.320 --> 00:10:14.240 yourself or should I read them to you?) 00:10:14.240 --> 00:10:16.423 Just for the sake of easy management 00:10:16.423 --> 00:10:18.079 why don't you read them out, if that's okay? 00:10:18.079 --> 00:10:21.279 (Amin: yeah, sure. They ask, "Have you looked 00:10:21.279 --> 00:10:25.440 into trying Sage Math? I've long wanted 00:10:25.440 --> 00:10:33.760 to use Sage Math in Org files.") 00:10:33.760 --> 00:10:36.959 Ray: Right. I wrote the answer that 00:10:36.959 --> 00:10:39.279 it should be possible because one can 00:10:39.279 --> 00:10:44.839 call it from a command. 00:10:44.839 --> 00:10:48.190 (Amin: okay, and I see there's 00:10:48.190 --> 00:10:50.079 another Sage Math question that you seem to have answered 00:10:50.079 --> 00:10:52.100 as well, so I guess I won't repeat that. 00:10:54.880 --> 00:10:56.959 There's... "Let's not forget about embedded 00:10:56.959 --> 00:11:00.640 Calc in Emacs.") 00:11:00.640 --> 00:11:05.040 Joe: So the first demos actually were with Calc. 00:11:05.040 --> 00:11:06.640 That's useful. Although I think it was a 00:11:06.640 --> 00:11:08.240 different--kind of a different command line. 00:11:08.240 --> 00:11:11.839 Ray: Well, that was UNIX Calc. 00:11:11.839 --> 00:11:13.839 Joe: So, sure, there is calc, so that... 00:11:15.680 --> 00:11:19.120 Ray: Calc is already in Org Mode. 00:11:25.680 --> 00:11:39.760 (Amin: Still looking for questions. 00:11:39.760 --> 00:11:41.760 Okay, I think that's about it. I don't see 00:11:41.760 --> 00:11:44.320 any questions on the Etherpad. 00:11:44.320 --> 00:11:47.440 And let's see... 00:11:47.440 --> 00:11:53.040 Anything on irc? 00:11:53.040 --> 00:11:55.760 Nothing but praises and everyone 00:11:55.760 --> 00:11:57.290 thanking you. Thank you.) 00:11:57.290 --> 00:11:59.120 Ray: all right, you're welcome. 00:11:59.120 --> 00:12:00.240 Joe: Thanks a lot! 00:12:00.240 --> 00:12:01.923 We'll see you guys around then. 00:12:01.923 --> 00:12:06.800 Amin: Cheers, and see you around!