Hopefully the internet goes well. It's a nice Monday
morning here in Tokyo.
Are we connected all right?
Okay, I seem to be struggling still with my audio. One second...
calling. Yeah, you were muted for a moment there. Okay,
there we are. Okay. All right. Sorry about that. I got a mute
out my, my back office chatter. That's kind of distracting
me a little bit. All right. Sorry. I may have lost the plot a
little bit. I think I did. However, find the 1st question.
I got pretty distracted by conversation backstage. Yeah,
no problem. So the first question here, someone's asking,
when they first tried comparing transducers.el, the cl-lib
and Dash bookmark compiled, and they give some detailed
results we're sharing on the stream. Um, they expected
transducers to be slower than CL loop, but faster than CL lib
or dash. However, this isn't the case, any idea why. And so
I'll, I'll come back into their data to show there's they're
showing, um, you know, there's not a lot of detail on the, on
the, on the use case here. We could certainly click through
it, do it.
Oh, I should've waited to zoom until I find my spot here.
There we are.
All right, so there's our example.
Looks like we are doing a simple map and a sum.
Mm-hmm. Yeah, that's right. Yeah, question about
performance. So a case like this, a simple, I just want to rip
through a collection of numbers and sum them all. That's a
case where basically loop is always going to win because
loop is optimized. This is true in both Emacs Lisp and in
Common Lisp. For a case like this where you're not really
doing two nested of chained calls, like you don't have many
sort of what I was compositional steps. If you're just
ripping through a collection of numbers, loop is always
going to win. Transducers kind of shines when you have to do
things that loop can't in terms of expressing yourself. So
there are lots of different transducers that you can chain
together. And in that case, you're kind of prioritizing
developer time and developer happiness because you're
able to yourself more clearly, whereas sometimes those
kind of algorithms can get very hairy if you're just using
loop. Now that sounds like I'm moving the goalposts, and
there's really no excuse for these things not being as
performant as possible. In this specific case, my guess is
that the transducers is slower because it has to do a whole
bunch of like inner function calls in order to actually do
the adding and the collecting. So there's a lot of stuff that
just the raw loop doesn't have to do, which transducers
does. And so in this case, that's why it would be slower.
All right, makes sense.
Um... I cannot comment against Dash. And also a reminder
that transducers both in CL and in Emacs Lisp here doesn't
attempt to do any, you know, fun, you know, inner rewriting
or, you know, what's called an Haskell fusion. Like if you
have two different map steps, like in a row, it's not gonna
see that and somehow fuse them internally. It's a fairly, in
that sense, the implementation is just as is.
to make it you know as raw fast as possible. The idea being
that ergonomics is more important up front. Yeah, that's
kind of a whole fascinating sub-panel, right? My theme this
conference has been, oh, all these different things we
should try to get sub-panels going for and use that. Maybe
fill in the dev track or even have a third track or whatever.
I'm not that concerned about the logistics of squeezing
into the schedule so much. But anyway, interesting, I mean,
to say.
Did we already speak to theoretical texts? No, right? No,
let's continue. Okay, so another question from the group.
Do you know of any theoretical texts on transducers? My
readme, particularly of the Common Lisp implementation,
is the theoretical text on transducers. Rich Hickey has
some YouTube videos which also come close. I mean, he
invented the things. But in terms of having a full
explanation of everything, it's my readme and it's also
the...
The info manual of Guile Scheme, their documentation on
Surfy 171 is what I used to learn transducers and to
re-implement them in other LISPs. So if you just want like a
document explaining them, MyReadMe is actually the
clearest that I've found. Awesome. Okay, next question.
And I'm sorry, you gave a name, you referred to somebody's
videos. Rich Hickey, the inventor of Clojure. Rich Hickey,
thank you. Hope I got the spelling right, and maybe somebody
can catch that and fix it. If not, I'll reach on. Thank you.
Reach on to the next question. Waters (Lazy Series in
Lisp, late 70s) said this should have been done as an
additional compiler feature in compilers, but if not, must
be a macro package. Do you think about that vis your CL,
Fennel, Elisp, porting of transducers? I think that
there's definitely
some Galaxy Brain Lisp author out there is probably smart
enough to turn a bunch of this stuff into macros. I believe
that's how the common Lisp library series works. It sees
that you were calling map or whatever, and it actually knows
that that's a special macro key. in order to be fast. I did not
do that. The implementation as I have it is very simple and
simplicity shouldn't be underestimated.
I love it. What a nice succinct answer. Even I can manage to
type that out as I scroll us to the next question.
So, does t-buffer-read provide a lazy stream
that's line-wise or character-wise or do something else
entirely?
Okay, there are two functions. I showed
t-buffer-read. There's also one called t-file-read,
which does that. You actually have the buffer open,
it's much more clever.
t-buffer-read, I believe, is simpler. As long as you have an
Emacs list, what is called the current buffer active. I'm
fairly sure you're able to just call next-line on it. I don't
believe that I'm doing anything fancy there, looking for
line ends. I believe I'm just grabbing the next line and then
processing that line-wise. Very good.
Can the Elisp library be combined with the stream.el API
or seq in general? I would say that these libraries
are completely orthogonal. You saw that everything
was prefixed by t-.
Basically, transducer is its own zone. However, one thing
that I do in the common lisp, which is theoretically
possible for the Emacs Lisp as well, is kind of like little
shim libraries. So I provide, at least for Common Lisp, for a
number of, you know, popular sort of third-party
collection types, I provide an ability to use them as
sources. Maybe that's what you mean. Like
the built-in containers for Emacs Lisp are already
supported. So, you know, a vector hash table and so on.
make sense so i think what i heard there is yeah go ahead
please sorry in terms of mixing like you know like for
instance you know like seq-map used in transducers
we'll put it that way
i was just gonna say i think it um it it sounds like you're
saying Yeah, probably they are actually. We don't know yet
about any places where they don't play nicely together. So
quite possibly so. We can use sequence and transducers
together, for example. As a source potentially, yeah. It's
very easy because that just uses defgeneric. As long as you
have a new, like if you have a new collection type, as long as
you implement a def method for it somewhere, it'll just
magically work with this library. That's the magic of...
Yeah, as an Emacs user enjoying, you know, sort of the
renaissance of new features it's had, or sorry, Emacs ERC
user for chat. I've seen a lot of awesome stuff get done in the
last couple of years with generic set. JP never was working
on that. And like, that's just making me my eyes pop and go,
wow, that does make a whole lot of things simpler, doesn't
it? I think we're a lot of us running into generics and how
that solves problems in Emacs.
How does one debug a t-comp
expression? Can you talk in terms of single step,
step-by-step, intermediate results of the different
statements you declare? Yes. So in Common Lisp, this is
and sly stickers and things like that. In Emacs Lisp, it's a
little bit, shall we say, more difficult. For step
debugging,
so what comp does is comp internally, it should be a macro,
but currently it's not, although there's work to improve
that. It's doing an internal reduce and it's turning into
one giant kind of composed lambda inside. So I don't know if
step debugging would work there. However, we do have one
function called log, which lets you inspect intermediate
results. So you could technically use that to inject
yourself somewhere into the transduction chain and, you
know, halt or, you know, inspect the current value, et
cetera. So you get a bunch of questions lined up. I think
we're coming up, uh, within our last five minutes, uh,
before some declared, uh, reset time that we have
internally to just roll our closing credits, so to speak.
Um, not that I would want to cut the question and answer
short, but I might have to step away personally. But, um, as
we discussed before, you can just kind of run the QA, however
you want here. Um, or, or take questions offline if you'd
like to answer them off the pad. And I just want to say one more
time. Kitt said it managed later. Thanks again for your talk
for dedicating the time to this live QA. And I think we can see
by the many questions that are here. So I'll try to kind of
flip us through as many of them as I can with our last couple of
minutes, if that sounds good. Alternately, this might be a
good time if you have kind of wrap it up, final thoughts, as
Leo Sopanda saying. By all means, have at. Sure, thanks a
lot. I'd say that if you are still curious, check out the
read-me's because those have a lot of information,
including a full description of the API and everything
that's available.
Otherwise, just give them a shot. Using these things is the
best way to learn them, of course. I use them everywhere,
basically, all across my Emacs list and all across my common
list now. They get a lot of mileage. All right. You're
speaking our language now. As Emacs users, all our ears poke
up when you say, I'm getting a lot of mileage. I'm using it
across everything. Every Emacs user has a story that
harmonizes with that, I think.
So our next question, is
there a path for transducers to enable Elisp processing or
otherwise overly large data sets as if just normal Emacs
buffers, i.e. just pulling one thing at a time. So
essentially stream like under the hood, but buffer like an
interface. I think that makes sense to me. with none of the
usual performance issues, like as if, you know, the history
with long files is what that brings to mind, I guess. Yes, so
as you saw before, the withBufferRead sort of stream
function does have to have the actual buffer in memory, and
then you can go really fast. But there's another one with
file read. Now, again, I haven't tried to optimize that yet.
But in theory, it is able to read right from the underlying
file without having to open it as a buffer first.
Awesome. Ari, the performance issues mentioned, and that
popped up recently in the list and forums, to what extent
does tail call optimization and other mechanisms like
inlining, garbage collection friendliness, and so on,
could these alleviate issues, enable their use at little to
no extra costs? I feel like we're leading the witness here,
but I'm sure you see where we're going. Yeah, no problem. So
in terms of tail optimization, that's already happening
because the internal loop mechanism is using CL labels. And
in Emacs Lisp, CL labels is just a macro that is like
extremely tail recursive. So that's very, very fast. It's
not tail recursive, but it's using like goto. So it's
extremely, extremely fast, like the raw looping of it. So,
okay, well then where does the slowness come from? It's
probably coming from those lambdas and it's probably
coming from, uh, like extra consing, extra allocation
somewhere, which is, um, sort of what you were, what you're
referring to with the GC friendliness. So perhaps there's
some, um, um, yeah, some, like some fusion that I can do to
speed it up. Yeah, that just sounds fascinating endlessly.
Are there options to like read from a CSV, JSON, produce an
alist or plist instead of hash table? Absolutely.
Yes, I need to double check that, but we can read both CSV and
JSON, and you should be able to just turn on the plist option.
I will double check, but there's fairly free conversion
between those three types because hash table is not always
what you want. And actually, I suspect that slowness that we
saw in the demo before was because it was allocating hash
tables for every, like, all of the 50,000 lines. And had it
been a plist, it would have been faster. Interesting, so
maybe there's opportunities even if you end up with hash
lists, but then they're shared strategically and you pay
the cost of a little extra layer in there that buckets them
together the way that we might group files by the first four
characters in the file name once we've got a million files.
Anyway, is the Common Lisp version ready for production
use? Do you want to comment on API stability? I use it all the
time. I'm writing a game in Common Lisp right now, and I'm
using transducers everywhere in there, and it doesn't even
make a dent in the frame rate, and I'm using them
extensively. Okay, well, I'll just read from chat. Thanks
so much for the answers.
Do we need a pre-written or t-minus
version for every already existing reducing function,
plus, as an example? Or is there a function that constructs,
in my, I'm going to add the word, auto-visualifies them
already, auto-defines or something, or just generically
wraps function calls some way? already defined. This is
basically fold. Some built-in functions like plus already
function like reducers. It's a coincidence that they do
that. But there's an example in the README. Max is one that
does not act like that. For instance, maybe I could screen
share later, but if you just type in plus one, If you call plus
one in Emacs or Common Lisp, you get back one. It actually
only needs one argument. If you only type plus, it actually
gives you zero. Plus and multiple satisfy the API of
reducers. But if you have one that doesn't, like the max
function, and similarly, just type in plus as a function
call, just plus with nothing else, and you'll see. No, as a
function. zero will come out. This basically means it
satisfies the reducer API. But a function like max does not.
If you just type in max and then one, it won't work. Pardon me,
it did. But if you type in max with nothing else, it wouldn't
work.
Hence, we have to wrap it in something like fold. I would say
go look at the fold function. Right, which that I won't do.
I'm not that well enough prepped. Darn it. Leo would have
been here, but oh, well, you got me. Yeah, no problem. But
fold is sort of the ultimate reducer function. Great. So is
there, where was I? Here we go. We're way past this, right? So
is the compiling argument for transducers that it's a
better abstraction? It seems like there are concerns,
objections, while problematically valid focused on
implementation. Can this abstraction allow for advances
in implementation? Yes, what I've basically done is mostly
followed the pattern of usage that exists in Clojure and in
Scheme's SERP 171. In theory, the service level API is the
same no matter where you're using this, and that's the idea.
If you learn them in one list, you should be able to use them
everywhere. Then what it's actually doing under the hood is
free for us to change around. My implementations are mostly
based on the scheme with a few alterations here and there.
And in the Common Lisp case, like adding some Common Lisp
isms
to improve usage like UX a little bit. But overall, we are
free to do whatever we want internally to speed up
performance. I just haven't done that work. Awesome.
Awesome. So here's where I have to, where we're getting the
hook. We've just been pulled off the stream. The viewers
just saw the crawl by as it sent us over to the other pad where I
get to jump on and get involved with that now. But I can't
thank you enough, Colin. Would you like me to stop the
recording here? Any other comments you'd like to make? Uh,
yeah, sure. Like, I mean, I'll stick around for any more live
questions. I'm looking at both IRC and, and, um, uh, big blue
button here. So if people have more questions, I'll hang
around for a bit. I'm going to leave the channel open. I see
you do have a few people in here, so I'm just going to go ahead
and leave the recording. We can always trim it. Um, trim it
up. If you, uh, let us know, Hey, the last 10 minutes weren't
anything, you know, or whatever. No, no pressure, no
worries, no mistakes. Thank you. Really appreciate you.
Yep. Thanks a lot.
OK, does anyone else have some questions? I see Mohsen in the
BigBlueButton chat is asking how I made the video. So the
presentation itself was done with RevealJS from Org Mode.
So as you saw, I had a raw Org Mode buffer, which was
which was the presentation itself, which I then just
exported with a few certain settings, a few
customizations. And then for screen recording, I used OBS,
which worked flawlessly on Arch Linux. I used Sway,
Wayland, and all of that. So all of that just worked, which
was very impressive. Where do the HTML host the
presentation? I don't have that presentation hosted
anywhere.
I'll look at the.
I don't see that.
Here it is. So we've got the file here as well.
Looks like that's it for questions, basically.
Yep, and it looks like everyone's moved on for now. Let's
see. I mean, it would be so this is answering lounge 81 on IRC.
Yeah, like, if we really wanted to go that hardcore, maybe
there's some like C level stuff that we could
you know, significant demand for such a thing. You know, so
far there hasn't been such demand, but maybe there will be in
the future. Yeah, perhaps there's some custom stuff we
could do.
And otherwise, magic one.
Well, it looks like some people are quite happy with this.
All right. That's about what I've seen. So why don't we end it
here? I think I can control the recording from my end. If I
pause it, will that work? All right. Thank you, everyone.