[Speaker 0]: Do we have any listeners?
It's you and I. I have a question.
How many tests do you have for hyperbole and
How would you rate the test coverage compared
to other packages? Well,
that's a tricky 1. Shall I spell it out loud
and then maybe type it at the same time?
So, I believe it's around like more than 300
test cases now. But I cannot compare the test
coverage to any other
other package. Maybe I can type that later.
What do you say, Badal?
[Speaker 1]: package. I have no knowledge of any Yeah,
sure, yeah, that's totally fine.
Feel free to just answer them with voice.
[Speaker 0]: Yeah, yeah. There's another question.
1 small suggestion to me,
should means optional,
where shall or must means required.
Not sure if it is too late to make a major
grammar change like that.
Very nice presentation.
So thanks for presentation,
but the package ERT, well,
it's not something that we have come up with.
It's a standard package.
So I believe it has been around for a long
time. So, but please feel free to make
suggestions and maybe you can,
you know, like do a copy or like an alias for
that. If you believe it makes more sense for
your test cases to have that instead.
And then we have another question here.
For your info, you may find this helpful for
running MX test lint both from a command line
and from within MX with a transit menu.
GitHub alpha papa make sure,
yes. It also works on remote CI.
Yeah, thank you, Alpha Papa.
I think I've looked into that,
but we haven't made any use of that.
But maybe you'll inspire me to give it
another look.
[Speaker 2]: Hey guys.
[Speaker 0]: I remember, I recognize that voice.
Hi, Bob. Hey, how are you?
Congratulations, man. Thanks,
Hugh. Thank you. I have another question
here. It is easy to run ad hoc tests inside
an Emacs session given the command line
scripts you need to run to get the batch test
session running? You said it's to run an
ad-hoc test. I'm not sure I understand that
question. Yes, please.
[Speaker 1]: Maybe I can rephrase. Sure.
So I think what I understand is that since
you have to use some of these command lines
scripts to get a batch test session running,
is it easy to run ad hoc tests in an Emacs
session or does that, like in your
experience, has that been difficult?
[Speaker 0]: Well, from the command line,
if you look at the command line,
you'll see that it's only like a few image
functions to call to get that behavior to run
the batch tests. So I think we made some
support function for that in hyperbole.
So it's not, I don't think it's possible out
of the box to do it, but it's not complicated
to do it.
[Speaker 2]: You can define a test anytime,
right? Just like a new function.
So that's ad hoc. You just write your test
and you can run it.
[Speaker 0]: Yeah, yeah, I mean, of course,
but I got the impression it was about running
all your tests like we did with the command
line. Well, so the question is more about how
would you run all your test cases from within
Emacs? And the easy answer to that is
actually you load all your test case files,
and then you run ERT with the T as the test
selector and then it will run all your test
cases.
[Speaker 1]: Right. And I think they have expanded on
their question a little bit as well,
clarifying that. In other words,
can you tweak tests in an Emacs session and
run them right away? Which I believe,
if I understand correctly what Bob was
saying, you can basically define or redefine
functions on the fly and then have them be
run, right?
[Speaker 0]: Yes, yes. You just go into that test case and
you just change it and you run it again.
And either you have to sort of load it or you
can use like the commercial thing I did.
You use hyperbole and just hit meta return on
the test case and it will load it and run the
test case again. So that's of course what you
normally do when you're defining a test or
debug a test case or develop a test case.
Just start with something small,
just make sure maybe you can prepare the test
properly and run it again and again and again
until you're ready with it.
That's a good point. You can definitely do
that and that's part of how I normally
develop the test cases that I mean start with
something small so I can see that I get there
maybe the right input in the buffer that I
want to test on or something and I expand on
that more and more and add more and more more
and more more
[Speaker 2]: tests to it. You might tell them a bit about
how many test cases you have.
I guess you commented on that and like what
happens, you know, with the CICD pipeline,
every time we commit, you know,
across all the versions and what you have set
up there because you know I wish people could
see it. You can go and check on GitHub and
you can see the logs right of any of the
builds and but tell them a bit about that
Mats because I think that's pretty
impressive.
[Speaker 0]: Well, that's part of more the CI,
CD, part of how we developed this using
GitHub and workflows that you get out of the
box from there. So this more than 300 test
cases on our round for I think 5 different
versions of Emacs when we do a pull request
or a commit. So that's a good way to ensure
that it works from version 27.2
up to the latest master version because
there's some changes in Emacs over different
versions that can affect your functions or
your code.
[Speaker 2]: They all run in parallel and so typically in
under 60 seconds I think you've got all of
them run so you've got pretty extensive
testing which does catch interesting bugs
here and there, right?
[Speaker 0]: Yes, of course it does.
I mean, you normally develop with 1 version
and then you think everything is okay.
But then when you're tested with the
different versions, you find out that there
are some changes and there are things you
might not sort of keep track of what's
happening also. So that's a way to get
noticed that the core developers of Emacs
have changed something that you sort of based
your code on. Now I got another question
here. Did you have to change hyperbole code
and design to be more readily testable as you
were increasing your test coverage?
Well, we haven't done that to a lot,
to a big degree, although I believe that that
is an important thing for sort of the future
to do that because some of the hyperbolic
functions are very complicated and long and
that makes testing them rather difficult.
So, at a few places we have sort of broken up
functions in smaller pieces so it'd be easier
to do like unit tests of the different parts
of it. But there's a lot of more work that
has to be done there.
[Speaker 2]: 1 of the nice things is you know the great
environment in Lisp where we're able to do a
lot of interactive bottom-up testing before
we even get to lighting tech pieces.
So it does tend to be more higher level bugs,
I think, that get caught in cross-functional
interaction. We had 1 recently that was an
Emacs version change. It had been a function
that had existed for a long time.
It had an and rest in it,
in its argument list, so it would assemble
the list of arguments from individual
arguments that you would give it,
and they decided in a recent version,
I think with Stefan's input,
to change that to a list and allow the prior
behavior, but it would issue a warning if you
use the prior behavior.
So all of a sudden, the way you were supposed
to do it became semi-invalid.
And so we started getting the warning,
and we've tried to eliminate all those
warnings in recent hyperbole developments.
So we're like, what do we do?
You know, because we wanted to be backward
compatible to where you couldn't use a list.
It required you to use individual arguments.
And now it's sort of requiring you to do
that. And all of that was caused by the
automatic testing on it.
So you said, Max, you were going to tell us
what you learned. So what are the major
things that you learned in doing all of this
work? All of this work?
[Speaker 0]: Well, I tried to cover some of it in the
presentation, but as I was going along,
the presentation became like twice as long as
fitted into the time we had so I had to cut
it out. But I think some of the core things
still is in the presentation.
From a personal perspective,
And this might not be hard to realize,
but forcing yourself to test functions,
test code really forces you to understand the
code a little bit better in a way that sort
of makes it easier than just to read the
code. I don't know how it is for the rest
listening to this, but for me it works so
that if I just read the code then I don't
sort of become as sharp as I should be but if
I try to write the test case for it then I
really need to understand better of all the
edge cases and all the sort of states and etc
that is involved and I think that's That's
what's sort of 1 of the learning things I
wanted to communicate as well that I don't
think I covered in detail in the
presentation. Maybe all this,
but try it. 1 other sort of more from the fun
side is that I really think it's fun to write
the test. So if you haven't tests in your
package, you should start doing that because
it is fun. It might feel like some extra
work, but it really pays off in the long run,
especially if you have it in like a pipeline
and where you can run it regularly when you
do new commits, et cetera.
So, I mean, that's maybe obvious from,
if you look from the commercial side or your
work side to do it like that.
But even for your hobby project,
it can be very sort of pay off really well.
[Speaker 2]: It's worked really well when we're adding new
functionality or we're changing some of the
plumbing in the system.
You know, you go and you do some surgery and
then you run the tests.
And sometimes 6 to 10 tests will fail.
And you find there, you know,
it tends to be they're all interconnected and
it leads you back to the single source.
You fix that and you know it could be an edge
case and off by 1 or Sometimes it's an
assumption about the way something is used
and it's not actually always true.
And so, Matt's just really good at
identifying some of those scenarios and
keeping us honest, I guess I would say.
So I love, I run it as much as I before,
you know, even before I commit something.
So I get to see, you know,
if anything has progressed.
So yeah, I really recommend this process to
people. I haven't seen it done.
I don't think that, I don't know any other
package that has done it to this level.
And it's been working really great for us.
And I think, well, we'll see too when we
release to the general public.
[Speaker 0]: But Bob, also, maybe the test part of
different packages is not the first thing you
look at. So I know there are packages that
have testing, a lot of testing,
but how much, much testing they have or not,
I don't know. It's not what you normally look
into when you look at someone's else code.
You look maybe on the functionality side but
not on how they've done the sort of the
quality side. So there could be other
packages out there that are well equipped.
[Speaker 2]: I hope so. I hope so.
[Speaker 0]: What's the craziest bug you found when
writing these tests? Well,
What springs to my mind just now is that we
were doing some tests or I would do some
tests for when you narrow,
what do you say that? When you,
in outlining, when you sort of compress
things in an outline, so you just,
sorry Bob, maybe you have it,
[Speaker 2]: When you hide text.
[Speaker 0]: What I'm looking for? Yeah,
when you hide. So I was doing some cursor
movement over that. And I always assume that
if you do like a prefix argument to like a
simple cursor movement,
like control F moving 1 character position,
and you would give it the,
and then the prefix, like you want to move
like 2 or 3 positions,
you would do like control U 3 and then
control F and you move 3.
I always assumed that that would be exactly
the same as if you just hit the key control F
3 times, but it's not.
So it's not the bug, it's a feature,
but that was the craziest thing.
I spent the night trying to figure out why
our code was wrong, but It turns out that's
how Emacs behaves. Try it out yourself.
Try to move over the 3 dots at the end of
that and see what happens.
Do it with cursor hitting the key or using a
prefix argument and you see it behaves
differently. That was the craziest thing.
I think there was some other crazy thing or
deep learning also, but I can't come up with
it at the moment. So maybe I can write it in
the Q&A later.
[Speaker 1]: I think we're out of time on the stream,
but people are welcome to join Mats and Bob
here on BigBlueButton to further discuss
this. Thank you both.
[Speaker 0]: Okay, thank you. Thanks,
Makaay. Thank you. I don't know,
Is it only me and Bob here?
So Bob, do you want to say something?
[Speaker 2]: Well, I think it's been a great day.
And I'm glad we did this.
It takes a lot of energy.
I'm just really excited about the progress
that this, and we're actually doing a lot of
QA at work and my professional software work
and looking at you know how we can do more
test driven development and so everybody's
talking about this you know we've got AI over
here that can generate test cases.
But, you know, strangely enough,
with the rapidity of development and web
applications, I think the level of testing
has gone down in recent years compared to
where it used to be, right?
Because the pace has gone up.
And so I think it's starting to turn again
where people are saying,
we can't just release crap into the
Webisphere and we have to better ourselves.
And with all these advanced tool sets that
you have, that you can do CICD testing,
you know, I just, I just see it coming
around, you know, as people develop new
things. So That's kind of exciting to me
because I came from a manufacturing culture
originally where we, our company actually
started a lot of the manufacturing quality
efforts that you saw in Japan and elsewhere
in America for a long time and that was you
know entirely through testing.
We used to just build incredible test cases
because we were combining software with
hardware. And if, you know,
the hardware doesn't work and you ship a
million units, you're,
you're in trouble. So,
that was just something we had to do.
And so it's nice to start to see that curve
come around. And I think,
you know, Matt Vance is very modest,
but I think he's really the 1 that started us
down this path and really made it into a
reality. So everybody else just gets to
benefit from that work.
So thanks.
[Speaker 1]: That's awesome.
[Speaker 0]: Thanks. Okay. Yeah. So if there's nothing
more here, then maybe we should just close
this and I go over to write in the etherpad
the replies we had.
[Speaker 1]: Right, yeah, I think, let's see,
I see 1 other person here,
I believe Ihor just joined us.
Yeah. Yeah, so if you do want to discuss with
Mats and Bob, you're welcome to,
otherwise, yeah, we can close the room now.
[Speaker 3]: Well, I think I missed most of the talk
because I had power outage,
but the part I heard was about the mock
library. And you mentioned that you don't
like CL-let, but instead you use mock.
[Speaker 0]: Yeah, I was more saying that you have to do a
lot more work when you use the CL letdef.
It's for more ambitious and maybe more
complicated cases where you want to really
make a new implementation,
test implementation. If you use the mock,
you get a lot of things out of the box,
verifying that you actually,
like the mock was actually called for
instance, whereas if you do with the CLLatf,
you would have to take correct track of that
yourself. And so, so a lot of more work.
Oh yeah.
[Speaker 3]: I'm saying that most of the time CLLess is
used for simple cases actually.
Because, just for example,
the function always returns the same.
And it tends to be simple lambda that ignores
all the input arguments.
So that's really trivial most of the time but
I actually thought the opposite that mock is
supposed to be used for non-trivial cases.
[Speaker 0]: Sorry, what was the question?
Mock was supposed to be used for non-trivial.
Yeah I mean I don't know how to explain this.
I mean, CLF can be used for non-trivial
definitely. You can define then any behavior
you want. You can write your own function,
but you need to keep track of whether that
function is called or not,
for instance. So you have to make note of
that the function was called so you can fire
sort of an error in case your function wasn't
called because that would be 1 error case.
[Speaker 3]: So you mean the mock fires an error if the
mocked function was actually not called?
[Speaker 0]: Yes, it does. Yes. So if your assumptions,
you sort of document with the mock also your
assumptions how your code is going to be
called. And if those are wrong,
you will get an error.
So you would, so if the implementation would
maybe change, for instance,
and not call the thing you're mocking,
then you will notice that.
But if you see a letdef,
then you will have to keep track of that
yourself. Okay, I see.
I see.
[Speaker 3]: And you know, our mode also uses a lot of
test. In our mode, we have a lot of tests
[Speaker 0]: Ah, okay. Yeah. Yeah. I'm sure I have.
[Speaker 3]: also. We rely on CLLatF for,
we don't use third-party libraries at all.
[Speaker 0]: Oh, you use CLLatF, okay.
Yeah. Yeah. Yeah. At First I found it very
powerful to use that, but then I sort of,
I learned more about how we can use the
mocking library for what I needed.
And I prefer that at the moment.
[Speaker 3]: I see, that is interesting.
Because I had seen it,
but I didn't consider that it's gonna be
useful even in simple cases.
[Speaker 0]: It has its limitations.
So it's like life, how you turn depends.
But maybe I should look more into the org
mode and the test case to learn more about
that. So thanks for pointing that out.
[Speaker 3]: We are trying to cover as much as we can.
It's almost impossible for org.
But yeah, we keep adding more tests.
[Speaker 0]: That's great.
Someone's typing. I don't know.
Any more questions? No?
Okay, then I'll go back and try to document
this in the etherpad. Thank you everybody for
[Speaker 1]: Thank you guys. Great work.
[Speaker 0]: joining. Great. Thank you.
Take care. Bye-bye.
[Speaker 1]: Take care. Bye. Silence.