summaryrefslogblamecommitdiffstats
path: root/2024/captions/emacsconf-2024-transducers--transducers-finally-ergonomic-data-processing-for-emacs--colin-woodbury--answers.vtt
blob: 375cad2a6e9a6460f3c25f4ac1b23f1e9afaf013 (plain) (tree)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032







































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































                                                                                                                                                                                                                                                                                                                                 
WEBVTT chapters by sachac

00:00:00.000 --> 00:00:07.559
Hopefully the internet goes well. It's a nice Monday

00:00:07.560 --> 00:00:31.999
morning here in Tokyo.

00:00:32.000 --> 00:00:37.879
Are we connected all right?

00:00:37.880 --> 00:00:40.879
Okay, I seem to be struggling still with my audio. 1 2nd

00:00:40.880 --> 00:00:44.519
calling. Yeah, you were muted for a moment there. Okay,

00:00:44.520 --> 00:00:49.959
there we are. Okay. All right. Sorry about that. I got a mute

00:00:49.960 --> 00:00:55.119
out my, my back office chatter. That's kind of distracting

00:00:55.120 --> 00:00:58.079
me a little bit. All right. Sorry. I may have lost the plot a

00:00:58.080 --> 00:01:04.919
little bit. I think I did. However, find the 1st question.

00:01:04.920 --> 00:01:09.919
I got pretty distracted by conversation backstage. Yeah,

NOTE Q: When I tried comparing transducers.el to cl-lib and dash (benchmark-compiled), I got the following results

00:01:09.920 --> 00:01:15.879
no problem. So the first question here, someone's asking,

00:01:15.880 --> 00:01:22.279
when they first tried comparing transducers.el, the cl-lib

00:01:22.280 --> 00:01:27.959
and Dash bookmark compiled, and they give some detailed

00:01:27.960 --> 00:01:32.479
results we're sharing on the stream. Um, they expected

00:01:32.480 --> 00:01:36.679
transducers to be slower than CL loop, but faster than CL lib

00:01:36.680 --> 00:01:41.119
or dash. However, this isn't the case, any idea why. And so

00:01:41.120 --> 00:01:43.639
I'll, I'll come back into their data to show there's they're

00:01:43.640 --> 00:01:48.279
showing, um, you know, there's not a lot of detail on the, on

00:01:48.280 --> 00:01:52.199
the, on the use case here. We could certainly click through

00:01:52.200 --> 00:02:02.559
it, do it.

00:02:02.560 --> 00:02:06.999
Oh, I should've waited to zoom until I find my spot here.

00:02:07.000 --> 00:02:13.639
There we are.

00:02:13.640 --> 00:02:18.599
All right, so there's our example.

00:02:18.600 --> 00:02:23.759
Looks like we are doing a simple map and a sum.

00:02:23.760 --> 00:02:29.239
Mm-hmm. Yeah, that's right. Yeah, question about

00:02:29.240 --> 00:02:36.279
performance. So a case like this, a simple, I just want to rip

00:02:36.280 --> 00:02:40.279
through a collection of numbers and sum them all. That's a

00:02:40.280 --> 00:02:44.679
case where basically loop is always going to win because

00:02:44.680 --> 00:02:51.319
loop is optimized. This is true in both Emacs Lisp and in

00:02:51.320 --> 00:02:56.039
Common Lisp. For a case like this where you're not really

00:02:56.040 --> 00:03:02.399
doing two nested of chained calls, like you don't have many

00:03:02.400 --> 00:03:05.839
sort of what I was compositional steps. If you're just

00:03:05.840 --> 00:03:09.999
ripping through a collection of numbers, loop is always

00:03:10.000 --> 00:03:15.559
going to win. Transducers kind of shines when you have to do

00:03:15.560 --> 00:03:19.639
things that loop can't in terms of expressing yourself. So

00:03:19.640 --> 00:03:22.559
there are lots of different transducers that you can chain

00:03:22.560 --> 00:03:27.079
together. And in that case, you're kind of prioritizing

00:03:27.080 --> 00:03:33.039
developer time and developer happiness because you're

00:03:33.040 --> 00:03:36.399
able to yourself more clearly, whereas sometimes those

00:03:36.400 --> 00:03:40.679
kind of algorithms can get very hairy if you're just using

00:03:40.680 --> 00:03:45.399
loop. Now that sounds like I'm moving the goalposts, and

00:03:45.400 --> 00:03:48.639
there's really no excuse for these things not being as

00:03:48.640 --> 00:03:54.559
performant as possible. In this specific case, my guess is

00:03:54.560 --> 00:03:57.759
that the transducers is slower because it has to do a whole

00:03:57.760 --> 00:04:03.239
bunch of like inner function calls in order to actually do

00:04:03.240 --> 00:04:09.239
the adding and the collecting. So there's a lot of stuff that

00:04:09.240 --> 00:04:12.119
just the raw loop doesn't have to do, which transducers

00:04:12.120 --> 00:04:20.439
does. And so in this case, that's why it would be slower.

00:04:20.440 --> 00:04:29.079
All right, makes sense.

00:04:29.080 --> 00:04:36.239
Um... I cannot comment against Dash. And also a reminder

00:04:36.240 --> 00:04:40.159
that transducers both in CL and in Emacs Lisp here doesn't

00:04:40.160 --> 00:04:44.919
attempt to do any, you know, fun, you know, inner rewriting

00:04:44.920 --> 00:04:48.239
or, you know, what's called an Haskell fusion. Like if you

00:04:48.240 --> 00:04:51.359
have two different map steps, like in a row, it's not gonna

00:04:51.360 --> 00:04:55.159
see that and somehow fuse them internally. It's a fairly, in

00:04:55.160 --> 00:04:59.679
that sense, the implementation is just as is.

00:04:59.680 --> 00:05:04.159
to make it you know as raw fast as possible. The idea being

00:05:04.160 --> 00:05:12.839
that ergonomics is more important up front. Yeah, that's

00:05:12.840 --> 00:05:17.519
kind of a whole fascinating sub-panel, right? My theme this

00:05:17.520 --> 00:05:19.799
conference has been, oh, all these different things we

00:05:19.800 --> 00:05:24.039
should try to get sub-panels going for and use that. Maybe

00:05:24.040 --> 00:05:29.039
fill in the dev track or even have a third track or whatever.

00:05:29.040 --> 00:05:31.519
I'm not that concerned about the logistics of squeezing

00:05:31.520 --> 00:05:38.519
into the schedule so much. But anyway, interesting, I mean,

00:05:38.520 --> 00:05:40.839
to say.

NOTE Q: Do you know of any theoretical texts on transducers?

00:05:40.840 --> 00:05:47.799
Did we already speak to theoretical texts? No, right? No,

00:05:47.800 --> 00:05:53.399
let's continue. Okay, so another question from the group.

00:05:53.400 --> 00:05:58.879
Do you know of any theoretical texts on transducers? My

00:05:58.880 --> 00:06:01.959
readme, particularly of the Common Lisp implementation,

00:06:01.960 --> 00:06:06.159
is the theoretical text on transducers. Rich Hickey has

00:06:06.160 --> 00:06:10.439
some YouTube videos which also come close. I mean, he

00:06:10.440 --> 00:06:14.799
invented the things. But in terms of having a full

00:06:14.800 --> 00:06:21.559
explanation of everything, it's my readme and it's also

00:06:21.560 --> 00:06:23.319
the...

00:06:23.320 --> 00:06:28.559
The info manual of Guile Scheme, their documentation on

00:06:28.560 --> 00:06:34.199
Surfy 171 is what I used to learn transducers and to

00:06:34.200 --> 00:06:38.399
re-implement them in other LISPs. So if you just want like a

00:06:38.400 --> 00:06:41.639
document explaining them, MyReadMe is actually the

00:06:41.640 --> 00:06:46.959
clearest that I've found. Awesome. Okay, next question.

00:06:46.960 --> 00:06:50.119
And I'm sorry, you gave a name, you referred to somebody's

00:06:50.120 --> 00:06:55.439
videos. Rich Hickey, the inventor of Clojure. Rich Hickey,

00:06:55.440 --> 00:07:00.399
thank you. Hope I got the spelling right, and maybe somebody

00:07:00.400 --> 00:07:04.719
can catch that and fix it. If not, I'll reach on. Thank you.

NOTE Q: Did you think about [compiler features, macros] viz your cl, fennel, elisp, porting of your transducers?

00:07:04.720 --> 00:07:08.239
Reach on to the next question. Waters (Lazy Series in

00:07:08.240 --> 00:07:12.799
Lisp, late 70s) said this should have been done as an

00:07:12.800 --> 00:07:16.799
additional compiler feature in compilers, but if not, must

00:07:16.800 --> 00:07:21.439
be a macro package. Do you think about that vis your CL,

00:07:21.440 --> 00:07:27.519
Fennel, Elisp, porting of transducers? I think that

00:07:27.520 --> 00:07:28.519
there's definitely

00:07:28.520 --> 00:07:36.519
some Galaxy Brain Lisp author out there is probably smart

00:07:36.520 --> 00:07:40.599
enough to turn a bunch of this stuff into macros. I believe

00:07:40.600 --> 00:07:47.119
that's how the common Lisp library series works. It sees

00:07:47.120 --> 00:07:52.079
that you were calling map or whatever, and it actually knows

00:07:52.080 --> 00:07:56.639
that that's a special macro key. in order to be fast. I did not

00:07:56.640 --> 00:08:01.839
do that. The implementation as I have it is very simple and

00:08:01.840 --> 00:08:05.759
simplicity shouldn't be underestimated.

00:08:05.760 --> 00:08:13.559
I love it. What a nice succinct answer. Even I can manage to

00:08:13.560 --> 00:08:16.578
type that out as I scroll us to the next question.

NOTE Q: Does t-buffer-read provide a lazy stream that's linewise, or charwise, or do something else entirely?

00:08:16.579 --> 00:08:24.079
So, does t-buffer-read provide a lazy stream

00:08:24.080 --> 00:08:28.359
that's line-wise or character-wise or do something else

00:08:28.360 --> 00:08:29.018
entirely?

00:08:29.019 --> 00:08:31.587
Okay, there are two functions. I showed

00:08:31.588 --> 00:08:35.073
t-buffer-read. There's also one called t-file-read,

00:08:35.074 --> 00:08:38.682
which does that. You actually have the buffer open,

00:08:38.683 --> 00:08:40.239
it's much more clever.

00:08:40.240 --> 00:08:45.999
t-buffer-read, I believe, is simpler. As long as you have an

00:08:46.000 --> 00:08:52.079
Emacs list, what is called the current buffer active. I'm

00:08:52.080 --> 00:08:56.679
fairly sure you're able to just call next-line on it. I don't

00:08:56.680 --> 00:08:59.479
believe that I'm doing anything fancy there, looking for

00:08:59.480 --> 00:09:03.999
line ends. I believe I'm just grabbing the next line and then

00:09:04.000 --> 00:09:09.423
processing that line-wise. Very good.

NOTE Q: Can the Elisp library be combined with the stream.el API or seq in general?

00:09:09.424 --> 00:09:17.303
Can the Elisp library be combined with the stream.el API

00:09:17.304 --> 00:09:22.830
or seq in general? I would say that these libraries

00:09:22.831 --> 00:09:27.596
are completely orthogonal. You saw that everything

00:09:27.597 --> 00:09:29.279
was prefixed by t-.

00:09:29.280 --> 00:09:36.879
Basically, transducer is its own zone. However, one thing

00:09:36.880 --> 00:09:40.239
that I do in the common lisp, which is theoretically

00:09:40.240 --> 00:09:44.359
possible for the Emacs Lisp as well, is kind of like little

00:09:44.360 --> 00:09:48.919
shim libraries. So I provide, at least for Common Lisp, for a

00:09:48.920 --> 00:09:51.799
number of, you know, popular sort of third-party

00:09:51.800 --> 00:09:55.239
collection types, I provide an ability to use them as

00:09:55.240 --> 00:09:59.559
sources. Maybe that's what you mean. Like

00:09:59.560 --> 00:10:04.439
the built-in containers for Emacs Lisp are already

00:10:04.440 --> 00:10:06.519
supported. So, you know, a vector hash table and so on.

00:10:06.520 --> 00:10:13.719
make sense so i think what i heard there is yeah go ahead

00:10:13.720 --> 00:10:17.879
please sorry in terms of mixing like you know like for

00:10:17.880 --> 00:10:22.599
instance you know like seq-map used in transducers

00:10:22.600 --> 00:10:28.119
we'll put it that way

00:10:28.120 --> 00:10:31.879
i was just gonna say i think it um it it sounds like you're

00:10:31.880 --> 00:10:37.199
saying Yeah, probably they are actually. We don't know yet

00:10:37.200 --> 00:10:41.239
about any places where they don't play nicely together. So

00:10:41.240 --> 00:10:45.399
quite possibly so. We can use sequence and transducers

00:10:45.400 --> 00:10:49.959
together, for example. As a source potentially, yeah. It's

00:10:49.960 --> 00:10:54.159
very easy because that just uses defgeneric. As long as you

00:10:54.160 --> 00:10:57.719
have a new, like if you have a new collection type, as long as

00:10:57.720 --> 00:11:01.519
you implement a def method for it somewhere, it'll just

00:11:01.520 --> 00:11:12.159
magically work with this library. That's the magic of...

00:11:12.160 --> 00:11:18.439
Yeah, as an Emacs user enjoying, you know, sort of the

00:11:18.440 --> 00:11:21.959
renaissance of new features it's had, or sorry, Emacs ERC

00:11:21.960 --> 00:11:27.799
user for chat. I've seen a lot of awesome stuff get done in the

00:11:27.800 --> 00:11:32.119
last couple of years with generic set. JP never was working

00:11:32.120 --> 00:11:36.679
on that. And like, that's just making me my eyes pop and go,

00:11:36.680 --> 00:11:39.279
wow, that does make a whole lot of things simpler, doesn't

00:11:39.280 --> 00:11:44.279
it? I think we're a lot of us running into generics and how

00:11:44.280 --> 00:11:47.542
that solves problems in Emacs.

NOTE Q: How does one debug a t-comp expression? Can you single step and see intermediate results of the different statements you declare?

00:11:47.543 --> 00:11:50.279
How does one debug a t-comp

00:11:50.280 --> 00:11:55.119
expression? Can you talk in terms of single step,

00:11:55.120 --> 00:11:58.479
step-by-step, intermediate results of the different

00:11:58.480 --> 00:12:08.759
statements you declare? Yes. So in Common Lisp, this is

00:12:08.760 --> 00:12:12.919
and sly stickers and things like that. In Emacs Lisp, it's a

00:12:12.920 --> 00:12:19.559
little bit, shall we say, more difficult. For step

00:12:19.560 --> 00:12:20.479
debugging,

00:12:20.480 --> 00:12:25.679
so what comp does is comp internally, it should be a macro,

00:12:25.680 --> 00:12:28.839
but currently it's not, although there's work to improve

00:12:28.840 --> 00:12:33.559
that. It's doing an internal reduce and it's turning into

00:12:33.560 --> 00:12:37.479
one giant kind of composed lambda inside. So I don't know if

00:12:37.480 --> 00:12:42.999
step debugging would work there. However, we do have one

00:12:43.000 --> 00:12:47.439
function called log, which lets you inspect intermediate

00:12:47.440 --> 00:12:50.759
results. So you could technically use that to inject

00:12:50.760 --> 00:12:54.279
yourself somewhere into the transduction chain and, you

00:12:54.280 --> 00:12:57.239
know, halt or, you know, inspect the current value, et

00:12:57.240 --> 00:13:01.119
cetera. So you get a bunch of questions lined up. I think

00:13:01.120 --> 00:13:04.199
we're coming up, uh, within our last five minutes, uh,

00:13:04.200 --> 00:13:07.919
before some declared, uh, reset time that we have

00:13:07.920 --> 00:13:11.919
internally to just roll our closing credits, so to speak.

00:13:11.920 --> 00:13:14.839
Um, not that I would want to cut the question and answer

00:13:14.840 --> 00:13:18.399
short, but I might have to step away personally. But, um, as

00:13:18.400 --> 00:13:21.519
we discussed before, you can just kind of run the QA, however

00:13:21.520 --> 00:13:24.879
you want here. Um, or, or take questions offline if you'd

00:13:24.880 --> 00:13:27.999
like to answer them off the pad. And I just want to say one more

00:13:28.000 --> 00:13:30.959
time. Kitt said it managed later. Thanks again for your talk

00:13:30.960 --> 00:13:35.759
for dedicating the time to this live QA. And I think we can see

00:13:35.760 --> 00:13:40.279
by the many questions that are here. So I'll try to kind of

00:13:40.280 --> 00:13:42.959
flip us through as many of them as I can with our last couple of

00:13:42.960 --> 00:13:48.399
minutes, if that sounds good. Alternately, this might be a

00:13:48.400 --> 00:13:52.079
good time if you have kind of wrap it up, final thoughts, as

00:13:52.080 --> 00:13:58.399
Leo Sopanda saying. By all means, have at. Sure, thanks a

00:13:58.400 --> 00:14:01.639
lot. I'd say that if you are still curious, check out the

00:14:01.640 --> 00:14:05.159
read-me's because those have a lot of information,

00:14:05.160 --> 00:14:09.519
including a full description of the API and everything

00:14:09.520 --> 00:14:10.719
that's available.

00:14:10.720 --> 00:14:16.599
Otherwise, just give them a shot. Using these things is the

00:14:16.600 --> 00:14:21.639
best way to learn them, of course. I use them everywhere,

00:14:21.640 --> 00:14:24.719
basically, all across my Emacs list and all across my common

00:14:24.720 --> 00:14:29.839
list now. They get a lot of mileage. All right. You're

00:14:29.840 --> 00:14:33.639
speaking our language now. As Emacs users, all our ears poke

00:14:33.640 --> 00:14:36.039
up when you say, I'm getting a lot of mileage. I'm using it

00:14:36.040 --> 00:14:39.879
across everything. Every Emacs user has a story that

00:14:39.880 --> 00:14:42.494
harmonizes with that, I think.

NOTE Q: Is there a path for transducers to enable elisp processing of otherwise overly large datasets as if just normal Emacs \"buffers\" (i.e. just pulling one thing at a time so essentially stream-like under the hood but buffer-like in interface), with none of the usual perf issues with a traditional buffer structure?

00:14:42.495 --> 00:14:44.519
So our next question, is

00:14:44.520 --> 00:14:48.599
there a path for transducers to enable Elisp processing or

00:14:48.600 --> 00:14:53.999
otherwise overly large data sets as if just normal Emacs

00:14:54.000 --> 00:14:56.959
buffers, i.e. just pulling one thing at a time. So

00:14:56.960 --> 00:15:00.719
essentially stream like under the hood, but buffer like an

00:15:00.720 --> 00:15:03.519
interface. I think that makes sense to me. with none of the

00:15:03.520 --> 00:15:07.799
usual performance issues, like as if, you know, the history

00:15:07.800 --> 00:15:11.399
with long files is what that brings to mind, I guess. Yes, so

00:15:11.400 --> 00:15:15.799
as you saw before, the withBufferRead sort of stream

00:15:15.800 --> 00:15:19.879
function does have to have the actual buffer in memory, and

00:15:19.880 --> 00:15:22.679
then you can go really fast. But there's another one with

00:15:22.680 --> 00:15:26.839
file read. Now, again, I haven't tried to optimize that yet.

00:15:26.840 --> 00:15:30.119
But in theory, it is able to read right from the underlying

00:15:30.120 --> 00:15:32.839
file without having to open it as a buffer first.

00:15:32.840 --> 00:15:39.199
Awesome. Ari, the performance issues mentioned, and that

00:15:39.200 --> 00:15:43.479
popped up recently in the list and forums, to what extent

00:15:43.480 --> 00:15:46.959
does tail call optimization and other mechanisms like

00:15:46.960 --> 00:15:50.159
inlining, garbage collection friendliness, and so on,

00:15:50.160 --> 00:15:55.159
could these alleviate issues, enable their use at little to

00:15:55.160 --> 00:15:58.439
no extra costs? I feel like we're leading the witness here,

00:15:58.440 --> 00:16:01.279
but I'm sure you see where we're going. Yeah, no problem. So

00:16:01.280 --> 00:16:03.799
in terms of tail optimization, that's already happening

00:16:03.800 --> 00:16:09.199
because the internal loop mechanism is using CL labels. And

00:16:09.200 --> 00:16:12.199
in Emacs Lisp, CL labels is just a macro that is like

00:16:12.200 --> 00:16:16.079
extremely tail recursive. So that's very, very fast. It's

00:16:16.080 --> 00:16:19.039
not tail recursive, but it's using like goto. So it's

00:16:19.040 --> 00:16:22.519
extremely, extremely fast, like the raw looping of it. So,

00:16:22.520 --> 00:16:24.359
okay, well then where does the slowness come from? It's

00:16:24.360 --> 00:16:26.439
probably coming from those lambdas and it's probably

00:16:26.440 --> 00:16:32.399
coming from, uh, like extra consing, extra allocation

00:16:32.400 --> 00:16:35.999
somewhere, which is, um, sort of what you were, what you're

00:16:36.000 --> 00:16:38.519
referring to with the GC friendliness. So perhaps there's

00:16:38.520 --> 00:16:45.199
some, um, um, yeah, some, like some fusion that I can do to

00:16:45.200 --> 00:16:51.199
speed it up. Yeah, that just sounds fascinating endlessly.

NOTE Q: Is there an option to read a csv/json and produce an alist or plist instead of a hash table for an entry?

00:16:51.200 --> 00:16:55.559
Are there options to like read from a CSV, JSON, produce an

00:16:55.560 --> 00:17:01.679
alist or plist instead of hash table? Absolutely.

00:17:01.680 --> 00:17:06.239
Yes, I need to double check that, but we can read both CSV and

00:17:06.240 --> 00:17:10.359
JSON, and you should be able to just turn on the plist option.

00:17:10.360 --> 00:17:14.159
I will double check, but there's fairly free conversion

00:17:14.160 --> 00:17:18.039
between those three types because hash table is not always

00:17:18.040 --> 00:17:22.039
what you want. And actually, I suspect that slowness that we

00:17:22.040 --> 00:17:24.559
saw in the demo before was because it was allocating hash

00:17:24.560 --> 00:17:29.239
tables for every, like, all of the 50,000 lines. And had it

00:17:29.240 --> 00:17:32.599
been a plist, it would have been faster. Interesting, so

00:17:32.600 --> 00:17:35.399
maybe there's opportunities even if you end up with hash

00:17:35.400 --> 00:17:38.799
lists, but then they're shared strategically and you pay

00:17:38.800 --> 00:17:42.039
the cost of a little extra layer in there that buckets them

00:17:42.040 --> 00:17:46.439
together the way that we might group files by the first four

00:17:46.440 --> 00:17:50.519
characters in the file name once we've got a million files.

NOTE Q: Is the common lisp version ready for 'production' use? Is it complete enough and the API stable enough?

00:17:50.520 --> 00:17:54.479
Anyway, is the Common Lisp version ready for production

00:17:54.480 --> 00:17:59.959
use? Do you want to comment on API stability? I use it all the

00:17:59.960 --> 00:18:04.159
time. I'm writing a game in Common Lisp right now, and I'm

00:18:04.160 --> 00:18:08.559
using transducers everywhere in there, and it doesn't even

00:18:08.560 --> 00:18:11.119
make a dent in the frame rate, and I'm using them

00:18:11.120 --> 00:18:15.359
extensively. Okay, well, I'll just read from chat. Thanks

00:18:15.360 --> 00:18:17.476
so much for the answers.

NOTE Q: Do we need a pre-written \"t-\" version for every already existing reducing function like + or is there a function to construct them from already defined reducer 2-arg functions?

00:18:17.477 --> 00:18:20.439
Do we need a pre-written or t-minus

00:18:20.440 --> 00:18:24.959
version for every already existing reducing function,

00:18:24.960 --> 00:18:30.239
plus, as an example? Or is there a function that constructs,

00:18:30.240 --> 00:18:33.559
in my, I'm going to add the word, auto-visualifies them

00:18:33.560 --> 00:18:37.319
already, auto-defines or something, or just generically

00:18:37.320 --> 00:18:42.239
wraps function calls some way? already defined. This is

00:18:42.240 --> 00:18:49.399
basically fold. Some built-in functions like plus already

00:18:49.400 --> 00:18:52.599
function like reducers. It's a coincidence that they do

00:18:52.600 --> 00:18:56.799
that. But there's an example in the README. Max is one that

00:18:56.800 --> 00:19:00.559
does not act like that. For instance, maybe I could screen

00:19:00.560 --> 00:19:06.479
share later, but if you just type in plus one, If you call plus

00:19:06.480 --> 00:19:10.519
one in Emacs or Common Lisp, you get back one. It actually

00:19:10.520 --> 00:19:15.119
only needs one argument. If you only type plus, it actually

00:19:15.120 --> 00:19:20.839
gives you zero. Plus and multiple satisfy the API of

00:19:20.840 --> 00:19:24.759
reducers. But if you have one that doesn't, like the max

00:19:24.760 --> 00:19:28.759
function, and similarly, just type in plus as a function

00:19:28.760 --> 00:19:32.359
call, just plus with nothing else, and you'll see. No, as a

00:19:32.360 --> 00:19:37.199
function. zero will come out. This basically means it

00:19:37.200 --> 00:19:43.159
satisfies the reducer API. But a function like max does not.

00:19:43.160 --> 00:19:48.399
If you just type in max and then one, it won't work. Pardon me,

00:19:48.400 --> 00:19:54.239
it did. But if you type in max with nothing else, it wouldn't

00:19:54.240 --> 00:19:55.239
work.

00:19:55.240 --> 00:19:58.599
Hence, we have to wrap it in something like fold. I would say

00:19:58.600 --> 00:20:01.919
go look at the fold function. Right, which that I won't do.

00:20:01.920 --> 00:20:04.839
I'm not that well enough prepped. Darn it. Leo would have

00:20:04.840 --> 00:20:08.399
been here, but oh, well, you got me. Yeah, no problem. But

00:20:08.400 --> 00:20:16.879
fold is sort of the ultimate reducer function. Great. So is

00:20:16.880 --> 00:20:26.319
there, where was I? Here we go. We're way past this, right? So

NOTE Q: Is the compelling argument for transducers is that it's a better abstraction?

00:20:26.320 --> 00:20:34.279
is the compiling argument for transducers that it's a

00:20:34.280 --> 00:20:38.879
better abstraction? It seems like there are concerns,

00:20:38.880 --> 00:20:42.399
objections, while problematically valid focused on

00:20:42.400 --> 00:20:45.679
implementation. Can this abstraction allow for advances

00:20:45.680 --> 00:20:50.559
in implementation? Yes, what I've basically done is mostly

00:20:50.560 --> 00:20:55.999
followed the pattern of usage that exists in Clojure and in

00:20:56.000 --> 00:21:01.159
Scheme's SERP 171. In theory, the service level API is the

00:21:01.160 --> 00:21:04.999
same no matter where you're using this, and that's the idea.

00:21:05.000 --> 00:21:08.039
If you learn them in one list, you should be able to use them

00:21:08.040 --> 00:21:12.879
everywhere. Then what it's actually doing under the hood is

00:21:12.880 --> 00:21:18.359
free for us to change around. My implementations are mostly

00:21:18.360 --> 00:21:23.679
based on the scheme with a few alterations here and there.

00:21:23.680 --> 00:21:27.079
And in the Common Lisp case, like adding some Common Lisp

00:21:27.080 --> 00:21:27.959
isms

00:21:27.960 --> 00:21:34.759
to improve usage like UX a little bit. But overall, we are

00:21:34.760 --> 00:21:38.959
free to do whatever we want internally to speed up

00:21:38.960 --> 00:21:42.439
performance. I just haven't done that work. Awesome.

00:21:42.440 --> 00:21:47.239
Awesome. So here's where I have to, where we're getting the

00:21:47.240 --> 00:21:50.079
hook. We've just been pulled off the stream. The viewers

00:21:50.080 --> 00:21:54.079
just saw the crawl by as it sent us over to the other pad where I

00:21:54.080 --> 00:21:57.919
get to jump on and get involved with that now. But I can't

00:21:57.920 --> 00:22:00.359
thank you enough, Colin. Would you like me to stop the

00:22:00.360 --> 00:22:03.799
recording here? Any other comments you'd like to make? Uh,

00:22:03.800 --> 00:22:06.439
yeah, sure. Like, I mean, I'll stick around for any more live

00:22:06.440 --> 00:22:10.639
questions. I'm looking at both IRC and, and, um, uh, big blue

00:22:10.640 --> 00:22:13.239
button here. So if people have more questions, I'll hang

00:22:13.240 --> 00:22:15.959
around for a bit. I'm going to leave the channel open. I see

00:22:15.960 --> 00:22:17.839
you do have a few people in here, so I'm just going to go ahead

00:22:17.840 --> 00:22:20.839
and leave the recording. We can always trim it. Um, trim it

00:22:20.840 --> 00:22:24.279
up. If you, uh, let us know, Hey, the last 10 minutes weren't

00:22:24.280 --> 00:22:26.999
anything, you know, or whatever. No, no pressure, no

00:22:27.000 --> 00:22:29.839
worries, no mistakes. Thank you. Really appreciate you.

00:22:29.840 --> 00:22:31.959
Yep. Thanks a lot.

NOTE Q: Question about how the transducers video was made? Did you use Reveal.js? Do you have a pointer to the html hosted presentation? How did you generate the content for Reveal?

00:22:31.960 --> 00:22:48.399
OK, does anyone else have some questions? I see Mohsen in the

00:22:48.400 --> 00:22:52.839
BigBlueButton chat is asking how I made the video. So the

00:22:52.840 --> 00:22:59.079
presentation itself was done with RevealJS from Org Mode.

00:22:59.080 --> 00:23:03.639
So as you saw, I had a raw Org Mode buffer, which was

00:23:03.640 --> 00:23:09.319
which was the presentation itself, which I then just

00:23:09.320 --> 00:23:11.759
exported with a few certain settings, a few

00:23:11.760 --> 00:23:15.919
customizations. And then for screen recording, I used OBS,

00:23:15.920 --> 00:23:19.719
which worked flawlessly on Arch Linux. I used Sway,

00:23:19.720 --> 00:23:23.159
Wayland, and all of that. So all of that just worked, which

00:23:23.160 --> 00:23:27.999
was very impressive. Where do the HTML host the

00:23:28.000 --> 00:23:51.959
presentation? I don't have that presentation hosted

00:23:51.960 --> 00:23:52.599
anywhere.

00:23:52.600 --> 00:23:59.119
I'll look at the.

00:23:59.120 --> 00:24:00.079
I don't see that.

00:24:00.080 --> 00:24:08.159
Here it is. So we've got the file here as well.

00:24:08.160 --> 00:24:10.999
Looks like that's it for questions, basically.

00:24:11.000 --> 00:24:14.919
Yep, and it looks like everyone's moved on for now. Let's

00:24:14.920 --> 00:24:20.159
see. I mean, it would be so this is answering lounge 81 on IRC.

NOTE Q: From your investigations and tests so far, do you think there would be the necessity of transducers to eventually go down into the C level code for things like using them to solve "infinitely-big" buffer-like interfaces and such?

00:24:20.160 --> 00:24:24.599
Yeah, like, if we really wanted to go that hardcore, maybe

00:24:24.600 --> 00:24:29.439
there's some like C level stuff that we could

00:24:29.440 --> 00:24:36.119
you know, significant demand for such a thing. You know, so

00:24:36.120 --> 00:24:39.239
far there hasn't been such demand, but maybe there will be in

00:24:39.240 --> 00:24:42.519
the future. Yeah, perhaps there's some custom stuff we

00:24:42.520 --> 00:24:43.039
could do.

00:24:43.040 --> 00:24:48.599
And otherwise, magic one.

00:24:48.600 --> 00:25:00.599
Well, it looks like some people are quite happy with this.

00:25:00.600 --> 00:25:14.959
All right. That's about what I've seen. So why don't we end it

00:25:14.960 --> 00:25:19.839
here? I think I can control the recording from my end. If I

00:25:19.840 --> 00:25:23.800
pause it, will that work? All right. Thank you, everyone.