update

2 files changed, 1179 insertions, 0 deletions

diff --git a/2024/captions/emacsconf-2024-transducers--transducers-finally-ergonomic-data-processing-for-emacs--colin-woodbury--main--chapters.vtt b/2024/captions/emacsconf-2024-transducers--transducers-finally-ergonomic-data-processing-for-emacs--colin-woodbury--main--chapters.vtt
new file mode 100644
index 00000000..cf0a6206
--- /dev/null
+++ b/2024/captions/emacsconf-2024-transducers--transducers-finally-ergonomic-data-processing-for-emacs--colin-woodbury--main--chapters.vtt
@@ -0,0 +1,38 @@
+WEBVTT
+
+
+00:00:00.000 --> 00:00:41.519
+Intro
+
+00:00:41.520 --> 00:03:27.589
+What are transducers?
+
+00:03:27.590 --> 00:05:47.279
+Common issues
+
+00:05:47.280 --> 00:07:35.279
+Transducers
+
+00:07:35.280 --> 00:09:52.624
+Using transducers
+
+00:09:52.625 --> 00:11:49.332
+A more involved example with comp
+
+00:11:49.333 --> 00:14:29.468
+In Emacs
+
+00:14:29.469 --> 00:14:58.039
+Hash tables
+
+00:14:58.040 --> 00:15:55.799
+Clarity
+
+00:15:55.800 --> 00:19:57.679
+How do transducers work?
+
+00:20:00.520 --> 00:26:03.239
+Transducers in the wild - CSV
+
+00:26:03.240 --> 00:26:51.240
+Issues and next steps
diff --git a/2024/captions/emacsconf-2024-transducers--transducers-finally-ergonomic-data-processing-for-emacs--colin-woodbury--main.vtt b/2024/captions/emacsconf-2024-transducers--transducers-finally-ergonomic-data-processing-for-emacs--colin-woodbury--main.vtt
new file mode 100644
index 00000000..b0083b86
--- /dev/null
+++ b/2024/captions/emacsconf-2024-transducers--transducers-finally-ergonomic-data-processing-for-emacs--colin-woodbury--main.vtt
@@ -0,0 +1,1141 @@
+WEBVTT captioned by sachac
+
+NOTE Intro
+
+00:00:00.000 --> 00:00:10.799
+Hi everyone, this is EmacsConf 2024. I'm Colin, and today
+
+00:00:10.800 --> 00:00:17.319
+I'll be talking about transducers.
+
+00:00:17.320 --> 00:00:21.879
+After introducing them, I'll share a bit of history about
+
+00:00:21.880 --> 00:00:25.359
+transducers and the problems that they solve, some basics
+
+00:00:25.360 --> 00:00:28.879
+about how we can use them, how they work, like how they're
+
+00:00:28.880 --> 00:00:32.399
+implemented, some demonstrations of how we can actually
+
+00:00:32.400 --> 00:00:36.959
+use them in the wild, and then some other discussions about
+
+00:00:36.960 --> 00:00:41.519
+issues that they have.
+
+NOTE What are transducers?
+
+00:00:41.520 --> 00:00:46.399
+Okay, let's get right in. What are transducers?
+
+00:00:46.400 --> 00:00:49.679
+Transducers are a way to do streaming iteration with a
+
+00:00:49.680 --> 00:00:55.679
+modern API.
+
+00:00:55.680 --> 00:01:00.359
+Who are transducers for, and thereby, who is
+
+00:01:00.360 --> 00:01:05.599
+this talk for? Well, it's for people who want to do streamed
+
+00:01:05.600 --> 00:01:10.519
+data processing in Emacs. It's for people who perhaps
+
+00:01:10.520 --> 00:01:14.199
+aren't satisfied with the existing APIs, for example, the
+
+00:01:14.200 --> 00:01:19.359
+seq API, or some other common libraries that provide
+
+00:01:19.360 --> 00:01:23.719
+similar functionality. Maybe you're not a fan of the loop
+
+00:01:23.720 --> 00:01:29.079
+macro. Some people find it difficult to understand. Or
+
+00:01:29.080 --> 00:01:32.719
+maybe you've done a bunch of Clojure before, and you'd like
+
+00:01:32.720 --> 00:01:36.879
+more aspects of Clojure in your Emacs Lisp. Or maybe you're
+
+00:01:36.880 --> 00:01:40.239
+just interested in transducers in general, because the
+
+00:01:40.240 --> 00:01:48.839
+pattern has now been ported to multiple different Lisps.
+
+00:01:48.840 --> 00:01:55.039
+So I'm Colin. I'm fosskers on everything online, and I do
+
+00:01:55.040 --> 00:01:58.519
+mainly back-end programming work and a lot of open source
+
+00:01:58.520 --> 00:02:05.159
+software. I wrote Haskell for a long time, both as a hobbyist
+
+00:02:05.160 --> 00:02:09.079
+and professionally. Since the COVID years, I've been
+
+00:02:09.080 --> 00:02:13.439
+writing Rust, both open source and professionally. But now
+
+00:02:13.440 --> 00:02:19.719
+I find that in my spare time, I'm mostly writing Common Lisp.
+
+00:02:19.720 --> 00:02:22.719
+Some things I learned from my years of Haskell was that a lot
+
+00:02:22.720 --> 00:02:27.519
+of programming is just altering the shape of data. You know,
+
+00:02:27.520 --> 00:02:31.359
+sometimes we work through our algorithm line by line. We're
+
+00:02:31.360 --> 00:02:36.239
+trying to just tell the computer exactly what to do. But if we
+
+00:02:36.240 --> 00:02:39.639
+step back, a lot of the time we're just getting in data of some
+
+00:02:39.640 --> 00:02:44.119
+shape, changing it, and then passing it along. A lot of
+
+00:02:44.120 --> 00:02:49.279
+these patterns are common, identified
+
+00:02:49.280 --> 00:02:53.639
+decades ago. For instance, we have some collection, and we
+
+00:02:53.640 --> 00:02:56.999
+want to transform every element of that collection and then
+
+00:02:57.000 --> 00:03:01.199
+pass it on. Or maybe we're trying to filter out bad elements
+
+00:03:01.200 --> 00:03:04.799
+in that collection. Or maybe we're looking for a specific
+
+00:03:04.800 --> 00:03:07.759
+element in that collection. Yes, you could write all that
+
+00:03:07.760 --> 00:03:11.839
+with for loops, but these kind of common patterns were
+
+00:03:11.840 --> 00:03:18.559
+identified and given names decades ago. So why not use them?
+
+00:03:18.560 --> 00:03:21.879
+They say that there are two major problems in computer
+
+00:03:21.880 --> 00:03:25.759
+science, one being cache validation and the other being
+
+00:03:25.760 --> 00:03:27.589
+naming things.
+
+NOTE Common issues
+
+00:03:27.590 --> 00:03:29.799
+I've identified five other problems that
+
+00:03:29.800 --> 00:03:33.199
+come up when we're trying to deal with collections of data,
+
+00:03:33.200 --> 00:03:40.599
+or big streams of data. One is that if we were trying to
+
+00:03:40.600 --> 00:03:45.279
+load a file all into memory all at once and process the whole
+
+00:03:45.280 --> 00:03:48.279
+thing, sometimes we can have memory problems. You've
+
+00:03:48.280 --> 00:03:54.999
+probably seen out-of-memory errors or such things.
+
+00:03:55.000 --> 00:03:58.199
+A second issue that comes up is that if we were looking at a
+
+00:03:58.200 --> 00:04:01.799
+giant for loop, in particular a nested for loop or such
+
+00:04:01.800 --> 00:04:06.079
+things, it can be hard to tell just by looking at the code what
+
+00:04:06.080 --> 00:04:11.039
+it's trying to do, what it intends. If we don't go character
+
+00:04:11.040 --> 00:04:16.439
+by character or line by line, it can be hard to understand it.
+
+00:04:16.440 --> 00:04:20.039
+Furthermore, and this is particularly an issue with Emacs
+
+00:04:20.040 --> 00:04:26.399
+Lisp, is that if one call, for instance, to seq-map, then
+
+00:04:26.400 --> 00:04:29.319
+piped into seq-filter, for instance, will have an
+
+00:04:29.320 --> 00:04:33.599
+intermediate allocation, the map will take the source
+
+00:04:33.600 --> 00:04:37.639
+container, allocate a new one, and then the filter will
+
+00:04:37.640 --> 00:04:40.319
+operate over the second one. This is wasteful.
+
+00:04:40.320 --> 00:04:48.879
+Furthermore, it can often be difficult to abort a stream.
+
+00:04:48.880 --> 00:04:53.199
+For instance, if we were filtering through our collection,
+
+00:04:53.200 --> 00:04:57.319
+but we knew we only wanted to go halfway, for instance, for
+
+00:04:57.320 --> 00:05:01.759
+some reason, we have no way to stop it halfway through. We
+
+00:05:01.760 --> 00:05:05.479
+just have to process the whole thing, even if we know we don't
+
+00:05:05.480 --> 00:05:11.919
+need to. Another issue is that for languages that have
+
+00:05:11.920 --> 00:05:18.039
+traits, or in Haskell they're called type classes, if you
+
+00:05:18.040 --> 00:05:22.399
+are defining what it means to map over something, you often
+
+00:05:22.400 --> 00:05:27.039
+have to redefine that for every kind of container or thing
+
+00:05:27.040 --> 00:05:31.239
+that you're iterating over. Wouldn't it be nice if we could
+
+00:05:31.240 --> 00:05:34.719
+define things like map just once and then reuse them
+
+00:05:34.720 --> 00:05:39.839
+everywhere? Now, transducers solve all five of these,
+
+00:05:39.840 --> 00:05:44.039
+without the addition of new language features, and with
+
+00:05:44.040 --> 00:05:47.279
+little more than plain old function composition.
+
+NOTE Transducers
+
+00:05:47.280 --> 00:05:53.119
+If this is your first time hearing of transducers, yeah,
+
+00:05:53.120 --> 00:05:57.439
+no problem. They were originally invented in Clojure by
+
+00:05:57.440 --> 00:06:01.039
+Rich Hickey, and this is a quote from him. He thinks
+
+00:06:01.040 --> 00:06:05.439
+transducers are a fundamental primitive that decouple
+
+00:06:05.440 --> 00:06:10.079
+critical logic from list or sequence processing, and if he
+
+00:06:10.080 --> 00:06:13.999
+had to do Clojure all over, he'd put them at the bottom, at the
+
+00:06:14.000 --> 00:06:19.279
+very bottom of all the fundamental primitives. Now, that's
+
+00:06:19.280 --> 00:06:24.599
+Rich speaking quite highly of them. And I think he has a point
+
+00:06:24.600 --> 00:06:25.159
+here.
+
+00:06:25.160 --> 00:06:32.399
+They were invented originally in Clojure. In more
+
+00:06:32.400 --> 00:06:34.772
+recent years, they were brought over to Scheme
+
+00:06:34.773 --> 00:06:38.774
+via SRFI 171. That's where I found them
+
+00:06:38.775 --> 00:06:41.521
+when I was learning the Guile language.
+
+00:06:41.522 --> 00:06:43.919
+In the process of submitting a patch, I realized
+
+00:06:43.920 --> 00:06:48.199
+that there were other things to be improved. So I ported the
+
+00:06:48.200 --> 00:06:51.399
+pattern to Common Lisp, then Fennel, and then more
+
+00:06:51.400 --> 00:06:56.639
+recently, Emacs Lisp. The Common Lisp and Emacs Lisp APIs
+
+00:06:56.640 --> 00:07:01.199
+are identical. And the Fennel one is not identical, but
+
+00:07:01.200 --> 00:07:05.799
+fairly similar. Overall, everywhere you find
+
+00:07:05.800 --> 00:07:10.279
+transducers, they should basically be fairly uniform.
+
+00:07:10.280 --> 00:07:15.759
+When I originally made the Common Lisp variant first, I
+
+00:07:15.760 --> 00:07:18.799
+sampled the APIs from a number of different languages and
+
+00:07:18.800 --> 00:07:23.439
+came up with what I believed to be a representative sample of
+
+00:07:23.440 --> 00:07:27.959
+what most people would want out of such a library. I gave
+
+00:07:27.960 --> 00:07:32.439
+functions their common modern names. For instance, map
+
+00:07:32.440 --> 00:07:35.279
+is map and filter is filter and so on.
+
+NOTE Using transducers
+
+00:07:35.280 --> 00:07:42.599
+What does the usage of transducers look like? Well,
+
+00:07:42.600 --> 00:07:48.959
+these examples will all be the Emacs Lisp variant, but the
+
+00:07:48.960 --> 00:07:52.359
+Common Lisp will look basically exactly the same, minus
+
+00:07:52.360 --> 00:07:54.079
+this little t- prefix.
+
+00:07:54.080 --> 00:08:00.919
+Running transducers requires three things. It requires a
+
+00:08:00.920 --> 00:08:06.439
+source. This could be an obvious thing like a list or a
+
+00:08:06.440 --> 00:08:11.479
+vector, but it could be other things like a file, or in Emacs
+
+00:08:11.480 --> 00:08:16.348
+list in particular, a buffer.
+
+00:08:16.349 --> 00:08:20.112
+A reducer is a function. It's something like
+
+00:08:20.113 --> 00:08:22.639
+the + operator or the * operator,
+
+00:08:22.640 --> 00:08:26.785
+or certain constructors of various containers.
+
+00:08:26.786 --> 00:08:32.125
+It takes values and collates them into some final version.
+
+00:08:32.126 --> 00:08:33.946
+Now, finally, we have what we're calling here
+
+00:08:33.947 --> 00:08:37.567
+a transducer chain. This could be one transducer function
+
+00:08:37.568 --> 00:08:43.479
+or it could be multiple composed together. These are the
+
+00:08:43.480 --> 00:08:47.079
+functions that actually take data and transform them
+
+00:08:47.080 --> 00:08:55.279
+somehow. For instance, this. We have a list of three
+
+00:08:55.280 --> 00:09:04.199
+elements. We want to reduce it into a vector. How we are
+
+00:09:04.200 --> 00:09:07.519
+going to transform the elements along the way: we are doing
+
+00:09:07.520 --> 00:09:13.359
+plus one to each of them. If this syntax is new to you, just
+
+00:09:13.360 --> 00:09:18.039
+know that this #' just means that this thing that
+
+00:09:18.040 --> 00:09:22.079
+comes after it is the name of the function. In Common Lisp and
+
+00:09:22.080 --> 00:09:26.079
+Emacs Lisp, this is necessary, but for Clojure and Scheme,
+
+00:09:26.080 --> 00:09:32.719
+it is not. So we can see here that just this example is not much
+
+00:09:32.720 --> 00:09:36.119
+different than any other normal map call you might see made,
+
+00:09:36.120 --> 00:09:40.239
+but if nothing else, it's a handy way to convert a list to a
+
+00:09:40.240 --> 00:09:44.999
+vector or anything else. There are many, many reducers
+
+00:09:45.000 --> 00:09:48.239
+available and many different forms that we can
+
+00:09:48.240 --> 00:09:52.624
+collate the final value into.
+
+NOTE A more involved example with comp
+
+00:09:52.625 --> 00:09:55.086
+Let's see a more involved example.
+
+00:09:55.087 --> 00:09:58.049
+Okay, now we've got some more meat here.
+
+00:09:58.050 --> 00:10:01.772
+Here we can see usage of the comp function
+
+00:10:01.773 --> 00:10:05.255
+and a custom source, ints.
+
+00:10:05.256 --> 00:10:11.079
+Ints is an infinite generator of integer values. That's not
+
+00:10:11.080 --> 00:10:14.783
+like a list or a file. It will generate infinitely.
+
+00:10:14.784 --> 00:10:19.439
+Comp is letting us compose multiple transducer functions
+
+00:10:19.440 --> 00:10:23.759
+together. Notice that this is the opposite order of what
+
+00:10:23.760 --> 00:10:28.079
+we'd usually be used to from a function like comp. The order
+
+00:10:28.080 --> 00:10:32.679
+here is top to bottom, basically, so that the map goes first,
+
+00:10:32.680 --> 00:10:37.839
+then the filter, and then the take. So effectively is what
+
+00:10:37.840 --> 00:10:40.919
+we're doing is taking all the integers that exist,
+
+00:10:40.920 --> 00:10:45.399
+positive, adding one to them, filtering out only the even
+
+00:10:45.400 --> 00:10:50.039
+ones, but then just taking 10. Cons here is a function that
+
+00:10:50.040 --> 00:10:57.039
+just produces the ending result as a list. So what happens
+
+00:10:57.040 --> 00:11:00.479
+here specifically is how we are avoiding intermediate
+
+00:11:00.480 --> 00:11:04.238
+allocations. First, the number 0 will come through.
+
+00:11:04.239 --> 00:11:07.879
+It will be pulled out of this source internally by transduce.
+
+00:11:07.880 --> 00:11:10.919
+It will make its way into the map. The map will add it. Then it
+
+00:11:10.920 --> 00:11:15.799
+will immediately go into this filter step. So it's not like
+
+00:11:15.800 --> 00:11:19.119
+all the maps occur, and then all the filters occur. We do
+
+00:11:19.120 --> 00:11:24.039
+everything for each element. So the 0 comes in, now it's 1.
+
+00:11:24.040 --> 00:11:27.559
+The filter would occur. Well, it's going to fail that
+
+00:11:27.560 --> 00:11:31.119
+because it's not even, so it will just bail there. Now we'll
+
+00:11:31.120 --> 00:11:35.239
+go to the next one. Now 1 will come, it will become 2, then
+
+00:11:35.240 --> 00:11:39.119
+it will be saved by this evenp call, and then the take will
+
+00:11:39.120 --> 00:11:42.599
+capture it, because we only want 10 values here. You can
+
+00:11:42.600 --> 00:11:45.239
+see 2, 4, 6, 8, and so on is the result that we
+
+00:11:45.240 --> 00:11:49.332
+expect. So let's play around a little bit.
+
+NOTE In Emacs
+
+00:11:49.333 --> 00:11:53.336
+Let's jump into Emacs and see what we can do.
+
+00:11:53.337 --> 00:11:58.500
+Alright, you should see my Emacs screen here.
+
+00:11:58.501 --> 00:12:04.359
+These are the actual notes for the actual
+
+00:12:04.360 --> 00:12:08.959
+presentation done in Org Mode. I'll boost that up in size for
+
+00:12:08.960 --> 00:12:12.639
+a little bit. That should be more than big enough for you.
+
+00:12:12.640 --> 00:12:17.719
+Just by changing the reducer, we can change the result.
+
+00:12:17.720 --> 00:12:21.079
+Okay, now it's a vector. Well, what else can we do to it? Well,
+
+00:12:21.080 --> 00:12:25.959
+let's just add up the results. Maybe we just want to count the
+
+00:12:25.960 --> 00:12:30.919
+results. Oh, indeed, there were 10. What if we want to find
+
+00:12:30.920 --> 00:12:36.959
+the average of the results? What if we want to find the median
+
+00:12:36.960 --> 00:12:40.959
+of the results? And so on. Here's some more interesting
+
+00:12:40.960 --> 00:12:45.839
+things that we could do. We could add different steps. So
+
+00:12:45.840 --> 00:12:51.239
+here we have all the integers. Let's add, hmm, okay, we'll
+
+00:12:51.240 --> 00:12:57.399
+keep that. We're going to add t-enumerate. What enumerate does
+
+00:12:57.400 --> 00:13:00.879
+is for each item that comes through, it is
+
+00:13:00.880 --> 00:13:06.039
+going to add a sort of index to it and make it a pair. In this
+
+00:13:06.040 --> 00:13:08.719
+case, it's going to be equal to what came in here. Well, we can
+
+00:13:08.720 --> 00:13:12.399
+change it. If we start this at 1, now it will be different.
+
+00:13:12.400 --> 00:13:15.519
+1 will be paired with 0, and then 2 would be paired
+
+00:13:15.520 --> 00:13:19.559
+with 1, and so on. We'll accept that the even call will change
+
+00:13:19.560 --> 00:13:24.039
+that a little bit. Why we're doing this is because we want
+
+00:13:24.040 --> 00:13:27.279
+to form a hash table. Let's move that down to 3, maybe
+
+00:13:27.280 --> 00:13:31.439
+we'll get a better result. What do we see? Okay, here now the
+
+00:13:31.440 --> 00:13:37.359
+result is a hash table. What are its values? Well, 0 seems
+
+00:13:37.360 --> 00:13:40.479
+to have... The key of 0 seems to be paired with 2, the key of
+
+00:13:40.480 --> 00:13:42.909
+1 seems to be paired with 4,
+
+00:13:42.910 --> 00:13:47.411
+and 2 seems to be paired with 6.
+
+00:13:47.412 --> 00:13:51.293
+Maybe let's jazz that up even a little bit more.
+
+00:13:51.294 --> 00:13:52.973
+We're going to start from a string
+
+00:13:52.974 --> 00:13:57.943
+and we'll call it hello.
+
+00:13:57.944 --> 00:13:59.564
+That's not going to work anymore
+
+00:13:59.565 --> 00:14:02.585
+and neither is that, but what we could do is
+
+00:14:02.586 --> 00:14:05.498
+we could say t-map #'string.
+
+00:14:05.499 --> 00:14:08.627
+I believe we'll do that.
+
+00:14:08.628 --> 00:14:08.959
+Let's see if that works. It did. So that's
+
+00:14:08.960 --> 00:14:13.589
+going to convert a character into a string.
+
+00:14:13.590 --> 00:14:14.679
+Let's just go two
+
+00:14:14.680 --> 00:14:18.399
+just to make it a little easier. Now you can see that we've
+
+00:14:18.400 --> 00:14:21.919
+constructed a hash table here. The key of 0 is mapped to the
+
+00:14:21.920 --> 00:14:27.079
+string of h and 1 is mapped to e. Now, I really like having
+
+00:14:27.080 --> 00:14:29.468
+this reducer in particular.
+
+NOTE Hash tables
+
+00:14:29.469 --> 00:14:30.639
+Know that hash tables are
+
+00:14:30.640 --> 00:14:34.199
+also legal sources. I find that both in Emacs Lisp and in
+
+00:14:34.200 --> 00:14:37.119
+Common Lisp, dealing with hash tables--like creating them
+
+00:14:37.120 --> 00:14:41.599
+and altering them--can be a bit of a pain. Having them
+
+00:14:41.600 --> 00:14:45.679
+immediately available like this with transducers is very
+
+00:14:45.680 --> 00:14:49.079
+handy, I find. We can work with something that wasn't a hash
+
+00:14:49.080 --> 00:14:53.279
+table. We can construct it in a way that makes it amenable to
+
+00:14:53.280 --> 00:14:56.199
+that, and then reduce it down into a hash table, and here you
+
+00:14:56.200 --> 00:14:58.039
+go. Very handy.
+
+NOTE Clarity
+
+00:14:58.040 --> 00:15:06.399
+One last point is that you can see very clearly what
+
+00:15:06.400 --> 00:15:10.479
+this is attempting to do, as opposed to, say, a for loop. It's
+
+00:15:10.480 --> 00:15:12.719
+very clear what that step is doing, and then you can see what
+
+00:15:12.720 --> 00:15:15.119
+that is doing, and you know that the result is going to be two.
+
+00:15:15.120 --> 00:15:18.559
+Each line is kind of its own declarative step, and it should
+
+00:15:18.560 --> 00:15:22.159
+be clear, just by staring at this, basically what you're
+
+00:15:22.160 --> 00:15:25.399
+going to get out. This is one main difference from other
+
+00:15:25.400 --> 00:15:29.599
+languages that have things--say, for instance, Rust's
+
+00:15:29.600 --> 00:15:35.439
+iterator API--is the difference between the transducers
+
+00:15:35.440 --> 00:15:41.639
+and the reducers. If we go up here, for example, the
+
+00:15:41.640 --> 00:15:44.679
+difference between the transducers and the reducers and
+
+00:15:44.680 --> 00:15:48.119
+the sources is not explicitly laid out, whereas with
+
+00:15:48.120 --> 00:15:53.119
+transducers, it is. You have to be aware of how these things
+
+00:15:53.120 --> 00:15:55.799
+are different. I think that that helps clarity.
+
+NOTE How do transducers work?
+
+00:15:55.800 --> 00:16:01.999
+Moving on. How do transducers work? Well,
+
+00:16:02.000 --> 00:16:09.857
+we want to go see the README.
+
+00:16:09.858 --> 00:16:11.399
+So, what we're going to do is
+
+00:16:11.400 --> 00:16:19.102
+we're going to go to here.
+
+00:16:19.103 --> 00:16:21.959
+You should still be able to see this.
+
+00:16:21.960 --> 00:16:28.583
+This is the CL example, actually.
+
+00:16:28.584 --> 00:16:32.279
+Let's go to transducers.el.
+
+00:16:32.280 --> 00:16:37.744
+Their APIs and READMEs are the same,
+
+00:16:37.745 --> 00:16:39.919
+but just for the sake of it, we will go see
+
+00:16:39.920 --> 00:16:45.726
+how this looks on the Emacs side,
+
+00:16:45.727 --> 00:16:48.046
+just so that nothing is a surprise.
+
+00:16:48.047 --> 00:16:50.239
+But recall that the APIs are essentially the same
+
+00:16:50.240 --> 00:16:53.679
+between the two. If you go to this section, writing your
+
+00:16:53.680 --> 00:16:56.839
+own primitives, you can read about how transducers are
+
+00:16:56.840 --> 00:17:00.999
+actually formed, whether or not you want to write them
+
+00:17:01.000 --> 00:17:06.799
+yourself or not. We can see here t-map. We accept the
+
+00:17:06.800 --> 00:17:10.239
+function that you want to operate with. Then you've got
+
+00:17:10.240 --> 00:17:13.319
+this extra little lambda here that's coming in, and it's
+
+00:17:13.320 --> 00:17:17.079
+receiving a thing that is named reducer. Now, while here
+
+00:17:17.080 --> 00:17:20.439
+we're calling it reducer, it's actually the chain of all the
+
+00:17:20.440 --> 00:17:25.159
+composed functions together. It's all those main
+
+00:17:25.160 --> 00:17:28.479
+transducer steps. Finally, it's the reducer all
+
+00:17:28.480 --> 00:17:31.879
+composed together with normal function composition.
+
+00:17:31.880 --> 00:17:35.877
+That will matter very soon. Now here's the actual meat.
+
+00:17:35.878 --> 00:17:40.519
+We can see the accumulative result that's coming in with the
+
+00:17:40.520 --> 00:17:45.739
+current element. Now we need to operate on this.
+
+00:17:45.740 --> 00:17:47.840
+Were it normally mapped, we would see us
+
+00:17:47.841 --> 00:17:49.919
+applying the F to the input.
+
+00:17:49.920 --> 00:17:53.519
+But here, you can see us applying the F to the input and then
+
+00:17:53.520 --> 00:17:58.679
+continuing on. So us calling the rest of the composed chain
+
+00:17:58.680 --> 00:18:03.159
+here is the effect of, in the previous slide, moving to the
+
+00:18:03.160 --> 00:18:07.156
+next step. We could ignore this line for now.
+
+00:18:07.157 --> 00:18:13.819
+If you're curious, please read the README in detail.
+
+00:18:13.820 --> 00:18:15.579
+Now, what about reducers?
+
+00:18:15.580 --> 00:18:18.879
+What do those look like? Well, let's just scroll
+
+00:18:18.880 --> 00:18:22.439
+down here. Recall that a reducer is a function that's
+
+00:18:22.440 --> 00:18:26.959
+consuming a stream, right? Zoom that up for you a little bit.
+
+00:18:26.960 --> 00:18:33.919
+Now, in the case of count, recall that this is how it's
+
+00:18:33.920 --> 00:18:37.679
+working, how we saw a moment ago. So clearly this list of five
+
+00:18:37.680 --> 00:18:42.199
+elements only has five things in it. Well, a reducer by
+
+00:18:42.200 --> 00:18:47.599
+structure is a function of two, one, or zero arguments. So we
+
+00:18:47.600 --> 00:18:50.639
+can see here in the case of two, this is the normal iterative
+
+00:18:50.640 --> 00:18:54.519
+case. We don't care about the input for count, we just care
+
+00:18:54.520 --> 00:18:58.559
+about the current accumulated count that we're doing, and
+
+00:18:58.560 --> 00:19:02.879
+we add one to it, and that's it. This then goes back to
+
+00:19:02.880 --> 00:19:06.359
+the loop and the whole process starts again with the next
+
+00:19:06.360 --> 00:19:10.879
+element. In this kind of done case, this is used internal to
+
+00:19:10.880 --> 00:19:16.879
+that sort of the supervising function transduce. It's just
+
+00:19:16.880 --> 00:19:19.639
+confirming the final result. Sometimes some
+
+00:19:19.640 --> 00:19:21.839
+post-processing is necessary here, but in the case of
+
+00:19:21.840 --> 00:19:26.039
+count, as it is so simple, that is not necessary. And now
+
+00:19:26.040 --> 00:19:29.359
+here's the base case. This is also used within that
+
+00:19:29.360 --> 00:19:34.319
+supervising transduce function at the very top. Well, if
+
+00:19:34.320 --> 00:19:36.679
+you're counting, you have to start from somewhere, right?
+
+00:19:36.680 --> 00:19:37.349
+In this case, well, what you're starting with is zero.
+
+00:19:37.350 --> 00:19:40.251
+In the case of cons, you'd be starting with an empty list.
+
+00:19:40.252 --> 00:19:44.434
+In the case of vector, you'd be starting
+
+00:19:44.435 --> 00:19:53.999
+with an empty vector and so on.
+
+00:19:54.000 --> 00:19:56.799
+Once again, if you are more curious, please take a look at
+
+00:19:56.800 --> 00:19:57.679
+the README.
+
+NOTE Transducers in the wild - CSV
+
+00:20:00.520 --> 00:20:06.039
+Okay, transducers in the wild. Well, let's go take a look at
+
+00:20:06.040 --> 00:20:07.639
+processing some CSV data.
+
+00:20:07.640 --> 00:20:21.319
+We're going to open up a new Emacs Lisp bracket here. So I have
+
+00:20:21.320 --> 00:20:28.839
+a file. And in this file, let's just go look at C-x b right
+
+00:20:28.840 --> 00:20:34.839
+there, you will see that we've got some bank transaction
+
+00:20:34.840 --> 00:20:37.879
+information. It's got these transactions from a whole
+
+00:20:37.880 --> 00:20:40.199
+bunch of different people into different accounts,
+
+00:20:40.200 --> 00:20:43.879
+whether it's money coming in, money going out, and then a
+
+00:20:43.880 --> 00:20:47.839
+basic description. How's your Latin? But for this little
+
+00:20:47.840 --> 00:20:53.679
+test, what we want to do is we want to find Bob's final bank
+
+00:20:53.680 --> 00:20:59.679
+balance. Let's get on to it. First of all, let's
+
+00:20:59.680 --> 00:21:04.444
+just confirm, let's do some basic stuff.
+
+00:21:04.445 --> 00:21:10.844
+with-current-buffer, find-file-noselect.
+
+00:21:10.845 --> 00:21:15.542
+What's the name of that file?
+
+00:21:15.543 --> 00:21:17.439
+This is pre-organized, so you
+
+00:21:17.440 --> 00:21:20.879
+will just see it right here.
+
+00:21:20.880 --> 00:21:26.999
+t-transduce and t-comp. We don't know what we're going to comp
+
+00:21:27.000 --> 00:21:33.039
+yet. Actually, I'll just pass to show you. And then we will
+
+00:21:33.040 --> 00:21:36.999
+see, let's just do a little t-count just to confirm. What's
+
+00:21:37.000 --> 00:21:45.112
+our source? Well, our source is a buffer, t-buffer-read.
+
+00:21:45.113 --> 00:21:50.153
+And note that because we're using with-current-buffer,
+
+00:21:50.154 --> 00:21:55.079
+if we go like this, if we go current-buffer, this will just work. So
+
+00:21:55.080 --> 00:21:59.919
+now let's... Well, that was odd. I should have done it like
+
+00:21:59.920 --> 00:22:02.159
+that. There we go. So now we should make that a little smaller
+
+00:22:02.160 --> 00:22:04.799
+so you can see what it is. Now if we hit RET, we should get the
+
+00:22:04.800 --> 00:22:09.559
+right result. Okay, so there are 50,001 lines in this file,
+
+00:22:09.560 --> 00:22:13.516
+but the one extra one is the name of the headers, right?
+
+00:22:13.517 --> 00:22:18.079
+We want to process this file in more detail. So how can we do
+
+00:22:18.080 --> 00:22:22.079
+that? Well, let's start by just automatically
+
+00:22:22.080 --> 00:22:28.799
+interpreting the results as CSV. If we do that, okay, well
+
+00:22:28.800 --> 00:22:31.559
+now we only have 50,000 entries as we expected, right?
+
+00:22:31.560 --> 00:22:36.759
+Because it's going to pull out the header line. If we now say
+
+00:22:36.760 --> 00:22:42.679
+we want to just filter out, you know, We only want Bob, right?
+
+00:22:42.680 --> 00:22:53.679
+So if... gethash, it was in the row of name. Each line here is
+
+00:22:53.680 --> 00:22:57.079
+made into, at least by default, is made into a hash map. So if
+
+00:22:57.080 --> 00:23:02.759
+we go like this, we should see that. Okay, so 12,000 of these
+
+00:23:02.760 --> 00:23:05.639
+lines or thereabout belong to Bob.
+
+00:23:05.640 --> 00:23:13.839
+Let's just move that over a little bit. Actually, I suppose we don't even
+
+00:23:13.840 --> 00:23:17.799
+need that anymore. I'll just keep that full size for you.
+
+00:23:17.800 --> 00:23:24.399
+Okay, so all right, there's about 12,000 results for Bob of
+
+00:23:24.400 --> 00:23:32.479
+the 50,000. What's next? Well, we want to confirm,
+
+00:23:32.480 --> 00:23:40.039
+we want to pull out everything,
+
+00:23:40.040 --> 00:23:43.079
+all of the in and the out entries.
+
+00:23:43.080 --> 00:23:56.279
+Thank you. So, string to number, because we know that
+
+00:23:56.280 --> 00:24:01.239
+everything came in as strings. Unfortunately, the from-csv
+
+00:24:01.240 --> 00:24:03.799
+doesn't try to be smart at all, it's just pulling everything
+
+00:24:03.800 --> 00:24:09.479
+in as string values. If you want actual things to be
+
+00:24:09.480 --> 00:24:13.399
+numbers or whatever, that is up to you to do the parsing
+
+00:24:13.400 --> 00:24:20.679
+yourself. Okay, so we have those two values now. We know
+
+00:24:20.680 --> 00:24:23.879
+that we saw from the data just a moment ago that you're only
+
+00:24:23.880 --> 00:24:26.999
+going to have a value in one column or the other. It's either
+
+00:24:27.000 --> 00:24:29.119
+going to be 0 in the empty one, or you're going to have some
+
+00:24:29.120 --> 00:24:32.159
+number in the other. So we know that we can just naively add
+
+00:24:32.160 --> 00:24:35.479
+them. If it was in, it would always be positive. So we'll just
+
+00:24:35.480 --> 00:24:41.519
+add that. But in the negative case, we want to just make it
+
+00:24:41.520 --> 00:24:45.279
+negative really briefly before we add them all together.
+
+00:24:45.280 --> 00:24:50.519
+let's now just prove to ourselves that we are sane here. What
+
+00:24:50.520 --> 00:24:52.479
+we're going to do is we're going to quickly go say take
+
+00:24:52.480 --> 00:24:57.039
+5 just to convince ourselves, and we'll go cons, and let's
+
+00:24:57.040 --> 00:24:59.839
+see if we get kind of results that make sense. Okay, these
+
+00:24:59.840 --> 00:25:02.799
+sort of make sense. It looks like you know Bob's got some big
+
+00:25:02.800 --> 00:25:07.679
+expenses here. If we take say 15, does it look any better?
+
+00:25:07.680 --> 00:25:10.319
+Okay, looks like he had a payday. All right, good job Bob.
+
+00:25:10.320 --> 00:25:15.439
+Let's get back in there. Now we only really care about
+
+00:25:15.440 --> 00:25:20.119
+adding the final result, right? So there we go. Add that all
+
+00:25:20.120 --> 00:25:24.559
+together and we'll see what we get in a moment. Okay, wow,
+
+00:25:24.560 --> 00:25:27.519
+Bob's rich. Okay, so it looks like in his 12,000
+
+00:25:27.520 --> 00:25:32.279
+transaction, Bob has an overall net worth of $8.5 million.
+
+00:25:32.280 --> 00:25:34.439
+Looking pretty good.
+
+00:25:34.440 --> 00:25:38.999
+So here's an example of how you can, particularly in Emacs
+
+00:25:39.000 --> 00:25:42.959
+Lisp, how you can very easily just get a file, consider it the
+
+00:25:42.960 --> 00:25:45.879
+current buffer, and then just do whatever you want to it.
+
+00:25:45.880 --> 00:25:50.359
+Note that there is sort of first-class support for both CSV
+
+00:25:50.360 --> 00:25:54.359
+and JSON, and then you have, and both of those bring in their
+
+00:25:54.360 --> 00:25:57.719
+values as hash maps, and then you're just free to do whatever
+
+00:25:57.720 --> 00:26:00.439
+you want and process them, potentially both writing them
+
+00:26:00.440 --> 00:26:03.239
+back out as CSV or JSON once again.
+
+NOTE Issues and next steps
+
+00:26:03.240 --> 00:26:10.719
+Some issues with transducers that can come up is
+
+00:26:10.720 --> 00:26:14.919
+that one, a zip operator is missing, but I'm working on it.
+
+00:26:14.920 --> 00:26:19.399
+Two is that performance, particularly in Emacs Lisp, isn't
+
+00:26:19.400 --> 00:26:24.119
+that great. It could be due to the sort of nested lambda calls
+
+00:26:24.120 --> 00:26:27.759
+that have to occur internally, but the common Lisp
+
+00:26:27.760 --> 00:26:32.239
+implementation is quite good. and there's yet no support
+
+00:26:32.240 --> 00:26:35.399
+for parallelism. You can imagine that a lot of those steps
+
+00:26:35.400 --> 00:26:38.559
+you could potentially perform in parallel depending on the
+
+00:26:38.560 --> 00:26:44.399
+platform, but research has not yet gotten that far. Okay,
+
+00:26:44.400 --> 00:26:47.639
+that's all. Thank you very much. If you have any questions,
+
+00:26:47.640 --> 00:26:51.240
+please contact me.