WEBVTT captioned by abhinav NOTE What are reactive notebooks? 00:00:01.120 --> 00:00:03.033 Hello, everyone. My name is Abhinav, 00:00:03.034 --> 00:00:03.900 and I'm going to talk about 00:00:03.901 --> 00:00:07.140 how to make Org Babel reactive. So reactivity here 00:00:07.240 --> 00:00:10.000 means reactivity in the sense of reactive notebooks. 00:00:10.001 --> 00:00:11.600 So if you used Org Babel, 00:00:11.601 --> 00:00:13.933 you might also have used Jupyter notebooks, 00:00:13.934 --> 00:00:16.100 which are basically notebooks primarily for 00:00:16.200 --> 00:00:16.933 Python programming, 00:00:16.934 --> 00:00:20.100 where you have these text and code blocks interleaved, 00:00:20.101 --> 00:00:23.157 and then you can execute every code block independently, 00:00:23.158 --> 00:00:25.858 and then you control the order of execution manually, 00:00:25.859 --> 00:00:27.199 or you can just run the code blocks 00:00:27.200 --> 00:00:29.699 from top to bottom. But with reactive notebooks, 00:00:29.700 --> 00:00:32.927 what happens is that there's another way of running 00:00:32.928 --> 00:00:35.329 which is basically by having all these 00:00:35.330 --> 00:00:36.900 dependent code blocks automatically get 00:00:37.000 --> 00:00:38.900 executed whenever you make a change. 00:00:38.901 --> 00:00:40.774 So for example, if you change a variable, 00:00:40.775 --> 00:00:42.060 everything else that's dependent on 00:00:42.160 --> 00:00:44.433 that variable will be executed automatically. 00:00:44.434 --> 00:00:49.041 I'll show you an example of what that looks like. NOTE Reactivity demo 00:00:49.042 --> 00:00:51.762 Right, here's an example reactive Notebook. 00:00:51.763 --> 00:00:53.460 So this is called Observable. 00:00:53.560 --> 00:00:54.863 Observable is this tool made by 00:00:54.864 --> 00:00:57.679 the creator of d3.js which is 00:00:57.680 --> 00:01:01.499 a famous JavaScript charting library. So here, the 00:01:01.500 --> 00:01:03.667 interface is very similar to Jupyter Notebook. 00:01:03.668 --> 00:01:06.407 You basically are having these cells 00:01:06.408 --> 00:01:08.508 and each cell could be a text cell, like here, 00:01:08.509 --> 00:01:09.588 this is a Markdown cell 00:01:09.589 --> 00:01:11.609 and then there are these code blocks. 00:01:11.610 --> 00:01:15.250 Now each code cell is basically defining a variable. 00:01:15.251 --> 00:01:17.740 This is important in reactive notebooks because 00:01:17.840 --> 00:01:21.140 each cell is connected to other cell via this variable 00:01:21.240 --> 00:01:23.552 usage. So here data is defined, 00:01:23.553 --> 00:01:25.012 then there is filtered which is defined 00:01:25.013 --> 00:01:27.620 which is dependent on data, and then this plot is 00:01:27.720 --> 00:01:29.133 dependent on filtered. 00:01:29.134 --> 00:01:31.153 So now, in a classical notebook, what I will do is 00:01:31.154 --> 00:01:34.394 if I change something here, let's say from 1 to 2, 00:01:34.395 --> 00:01:34.854 I will have to run this, and then run this plot block again 00:01:34.855 --> 00:01:40.335 to make the change be visible. 00:01:40.336 --> 00:01:42.055 But in a reactive notebook, what happens is 00:01:42.056 --> 00:01:44.396 I can just change this from some value 00:01:44.397 --> 00:01:46.256 to some value, and then execute, 00:01:46.257 --> 00:01:48.817 and then every descendant is also executed, 00:01:48.818 --> 00:01:50.940 because that's how the reactivity works. 00:01:51.040 --> 00:01:51.937 You change this variable, 00:01:51.938 --> 00:01:53.080 so this should also be changed, 00:01:53.081 --> 00:01:55.238 because this is dependent on this variable. 00:01:55.239 --> 00:01:56.858 Now this is really helpful 00:01:56.859 --> 00:01:58.900 if you have a very complex and messy notebook 00:01:59.000 --> 00:02:01.199 which is what actually happens in reality. 00:02:01.200 --> 00:02:03.480 You end up doing an exploratory analysis, 00:02:03.481 --> 00:02:05.959 and you have these code blocks lying here and there. 00:02:05.960 --> 00:02:07.101 Then you change something 00:02:07.102 --> 00:02:09.281 and then you have to keep something in your mind 00:02:09.282 --> 00:02:11.362 that if I change this, I need to run 00:02:11.363 --> 00:02:13.023 these five code blocks again 00:02:13.024 --> 00:02:15.604 to finally get to the result that I want to see. 00:02:15.605 --> 00:02:20.467 Stale state causes a lot of issues in Jupyter Notebooks. 00:02:20.468 --> 00:02:23.788 So this is really good for reactivity, sorry reproducibility, 00:02:23.789 --> 00:02:26.630 but this is also really good for 00:02:26.631 --> 00:02:28.599 just having this exploration 00:02:28.600 --> 00:02:30.117 that you're trying to do. For example, 00:02:30.118 --> 00:02:31.761 you're changing something and it's really easy 00:02:31.762 --> 00:02:34.887 to just see that change happening in real time 00:02:34.888 --> 00:02:38.498 in your outcome variables, right? NOTE Org-Babel 00:02:38.499 --> 00:02:41.920 So I was wondering how to introduce this reactivity in Org Mode. 00:02:41.921 --> 00:02:43.200 And here's how it will look like. 00:02:43.201 --> 00:02:46.302 So this is a demo Org Mode file. 00:02:46.303 --> 00:02:48.603 There are many Org Babel blocks here. 00:02:48.604 --> 00:02:49.563 So you start from here. 00:02:49.564 --> 00:02:52.085 Let's say this is a code block. It has a name. 00:02:52.086 --> 00:02:53.665 And then there's another code block, 00:02:53.666 --> 00:02:55.426 which is dependent on the previous one, 00:02:55.427 --> 00:02:57.807 as you can see here, and so on. 00:02:57.808 --> 00:02:59.368 And then finally, there's a plot here, 00:02:59.369 --> 00:03:00.889 which is a gnuplot code. 00:03:00.890 --> 00:03:02.550 And you can see the image here. 00:03:02.551 --> 00:03:04.131 Now, what happens usually is that 00:03:04.132 --> 00:03:05.196 if I change this value from, 00:03:05.197 --> 00:03:09.199 let's say, 113 to 112, nothing happens on its own right? 00:03:09.200 --> 00:03:12.199 There's an extra step of execution that I will have to do 00:03:12.200 --> 00:03:15.079 so I will do that, and then the value is changed. 00:03:15.080 --> 00:03:17.699 Now the problem is that only this value is changed and 00:03:17.700 --> 00:03:21.079 if I go down and see the image, nothing will have changed. NOTE Running the whole buffer 00:03:21.080 --> 00:03:23.079 So what I can do is basically, 00:03:23.080 --> 00:03:24.818 a really simple thing is that, 00:03:24.819 --> 00:03:26.500 a simple trick is to basically 00:03:26.600 --> 00:03:29.445 enable a hook, like, add a hook 00:03:29.446 --> 00:03:30.525 whenever you're saving the buffer, 00:03:30.526 --> 00:03:31.866 you just run the full buffer again, 00:03:31.867 --> 00:03:34.287 like run all the code blocks automatically. 00:03:34.288 --> 00:03:36.849 Now if you do that, you can basically make a change somewhere 00:03:36.850 --> 00:03:37.889 and then you can, you know, 00:03:37.890 --> 00:03:41.071 see how everything else is changing 00:03:41.072 --> 00:03:42.712 which gives you some sort of reactivity, 00:03:42.713 --> 00:03:43.972 but there's still a lot of computation 00:03:43.973 --> 00:03:45.973 that's being wasted. 00:03:45.974 --> 00:03:49.595 You might not want to change or run this code block again 00:03:49.596 --> 00:03:51.900 when something down there is changing. NOTE Caching 00:03:51.901 --> 00:03:54.567 So to counter that, you can actually add caching. 00:03:54.568 --> 00:03:57.133 So if you add caching to any code block, 00:03:57.134 --> 00:03:59.800 that code block will only be executed again 00:03:59.801 --> 00:04:02.300 if that code has changed or 00:04:02.400 --> 00:04:04.755 the input variables have changed. 00:04:04.756 --> 00:04:06.336 But the other problem is that 00:04:06.337 --> 00:04:08.659 you don't want caching to be enabled for a lot of cases 00:04:08.660 --> 00:04:10.840 where the code block is actually dependent on 00:04:10.841 --> 00:04:12.722 external state, like for example, 00:04:12.723 --> 00:04:15.024 some sort of randomness or time. 00:04:15.025 --> 00:04:17.433 So caching also is, you know, kind of, 00:04:17.434 --> 00:04:18.967 it's, like, an important thing to use, 00:04:18.968 --> 00:04:21.660 but it's probably not giving you the complete answer. NOTE Computation dependencies 00:04:21.760 --> 00:04:25.973 So what we can instead do is basically figure out 00:04:25.974 --> 00:04:28.554 the whole computation dependencies here. 00:04:28.555 --> 00:04:31.275 So let's say if I look at this buffer, 00:04:31.276 --> 00:04:35.076 here's how all the blocks are connected. 00:04:35.077 --> 00:04:37.656 So as you can see the plot code block 00:04:37.657 --> 00:04:40.117 is dependent on c and then legendpg, 00:04:40.118 --> 00:04:43.918 and they themselves are dependent on these other nodes. 00:04:43.919 --> 00:04:47.279 So when I make a change in b, I only want b to run 00:04:47.280 --> 00:04:50.844 and then c and then plot. I don't want anything else to run. 00:04:50.845 --> 00:04:54.267 So what I did was I wrote a small minor mode for Org Mode 00:04:54.268 --> 00:04:55.368 which does exactly this. 00:04:55.369 --> 00:04:57.769 So whenever you are in a code block 00:04:57.770 --> 00:04:59.871 and you are making a change and then you save it, 00:04:59.872 --> 00:05:01.913 it will just follow the trail from that code block 00:05:01.914 --> 00:05:05.355 to every other descendant which is going to be impacted, 00:05:05.356 --> 00:05:09.719 and it just runs all of them, and nothing else gets executed. 00:05:09.720 --> 00:05:13.119 So to see it in action, I will just enable that mode. 00:05:13.120 --> 00:05:17.021 Yeah, right. So now here, if I change this 113 to 112 00:05:17.022 --> 00:05:21.243 and I save, this code, this variable gets changed. 00:05:21.244 --> 00:05:23.744 It's the same value because I did not update it again. 00:05:23.745 --> 00:05:25.719 And you can also see b also got changed 00:05:25.720 --> 00:05:29.667 because it's just following all the execution order and so on. 00:05:29.668 --> 00:05:31.727 The plot also got updated. 00:05:31.728 --> 00:05:34.068 We will be able to see more clearly 00:05:34.069 --> 00:05:36.402 once I change something more substantial. 00:05:36.402 --> 00:05:36.402 So here's another variable. 00:05:36.403 --> 00:05:41.332 So I added a small toggle button here, 00:05:41.333 --> 00:05:43.468 which is again part of the minor mode. 00:05:43.469 --> 00:05:45.209 So since this is nil, if I toggle it, 00:05:45.210 --> 00:05:49.300 it will become true. And this variable dictates whether 00:05:49.400 --> 00:05:51.174 the plot will have the legend or not. 00:05:51.175 --> 00:05:54.457 So if I toggle it to be t, now it's t 00:05:54.458 --> 00:05:57.900 and you can see that the plot has legend that's visible. 00:05:57.901 --> 00:06:03.139 If I toggle it back again to nil, the legend is gone. 00:06:03.140 --> 00:06:04.533 Now this is nice, this... NOTE Making this even better 00:06:04.534 --> 00:06:06.380 This is already pretty helpful for me 00:06:06.480 --> 00:06:10.179 but what we can do is we can make it even better. 00:06:10.180 --> 00:06:11.400 So one of the nicer ideas 00:06:11.401 --> 00:06:13.015 from these reactive notebooks 00:06:13.016 --> 00:06:16.078 is this idea of having an infinite canvas 00:06:16.079 --> 00:06:19.022 where you don't look at the document model, 00:06:19.023 --> 00:06:20.623 you look at the whole document 00:06:20.624 --> 00:06:25.008 as a canvas of multiple connected documents. 00:06:25.009 --> 00:06:26.589 One good thing that happens there is that 00:06:26.590 --> 00:06:29.550 you can basically have a piece of code somewhere 00:06:29.551 --> 00:06:30.410 and then piece of code 00:06:30.411 --> 00:06:32.499 somewhere very different position in the document, 00:06:32.500 --> 00:06:34.732 but you can put them together in the canvas 00:06:34.733 --> 00:06:36.933 and then see them side by side. 00:06:36.934 --> 00:06:38.294 So here also, let's say 00:06:38.295 --> 00:06:41.996 if I want to just have this image shown up at the top, 00:06:41.997 --> 00:06:45.857 what I can do is like I can pop this out, 00:06:45.858 --> 00:06:49.938 which opens a child frame, and then I can just go here. 00:06:49.939 --> 00:06:52.460 This child frame is showing the same image. 00:06:52.461 --> 00:06:55.502 So there's no change. So if I toggle this variable here, 00:06:55.503 --> 00:06:58.423 you can see that the image is updated. 00:06:58.424 --> 00:07:02.199 If I toggle it back to nil, the image, the legend is gone. 00:07:02.200 --> 00:07:03.367 And you can obviously, you know, 00:07:03.368 --> 00:07:08.690 you can make a lot of things come up as child frames. 00:07:08.691 --> 00:07:09.430 This is the same image. 00:07:09.431 --> 00:07:11.291 So even if you go down to the document, 00:07:11.292 --> 00:07:13.810 you will see the same image. 00:07:13.811 --> 00:07:18.174 So yeah, this is what I have right now. 00:07:18.175 --> 00:07:21.956 I'm definitely looking forward to making it more useful, 00:07:21.957 --> 00:07:25.599 probably including more kinds of child frames, 00:07:25.600 --> 00:07:29.965 maybe like making the whole document an infinite canvas. NOTE Wrapping up 00:07:29.966 --> 00:07:32.099 Alright, so that's the talk. 00:07:32.100 --> 00:07:33.346 If you're interested in the codebase, 00:07:33.347 --> 00:07:34.446 here's the homepage 00:07:34.447 --> 00:07:35.546 for the project [https://dev.lepisma.xyz/git/ob-rx]. 00:07:35.547 --> 00:07:37.566 So the next steps for me are basically 00:07:37.567 --> 00:07:40.647 making my workflow easier in matplotlib, 00:07:40.648 --> 00:07:42.587 which is a Python-based library, 00:07:42.588 --> 00:07:45.348 and d3.js, which is for JavaScript. 00:07:45.349 --> 00:07:47.888 For the JS thing, I might have to add 00:07:47.889 --> 00:07:49.540 the interactive JS child frames, 00:07:49.640 --> 00:07:51.829 and I am also looking forward to building something 00:07:51.830 --> 00:07:53.969 which can replicate the work 00:07:53.970 --> 00:07:56.750 of the Observable's infinite canvas, 00:07:56.751 --> 00:07:57.490 because that's something 00:07:57.491 --> 00:08:00.619 which I found really useful in my work with 00:08:00.620 --> 00:08:02.240 just JS visualizations. 00:08:02.340 --> 00:08:05.540 So yeah, happy to take questions on Etherpad 00:08:05.560 --> 00:08:08.240 and thank you for your time.