1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
|
[[!meta title="What I learned by writing test cases for GNU Hyperbole"]]
[[!meta copyright="Copyright © 2023 Mats Lidell"]]
[[!inline pages="internal(2023/info/test-nav)" raw="yes"]]
<!-- Initially generated with emacsconf-publish-talk-page and then left alone for manual editing -->
<!-- You can manually edit this file to update the abstract, add links, etc. --->
# What I learned by writing test cases for GNU Hyperbole
Mats Lidell (he, him, his) - IRC: matsl, @matsl@mastodon.acc.sunet.se, <mailto:matsl@gnu.org>
[[!inline pages="internal(2023/info/test-before)" raw="yes"]]
I'm maintaining GNU Hyperbole. I volunteered for that at a time when FSF was asking for one
since it was unmaintained. I did not have much elisp experience but a passion for the
package. Not much happened.
To my great delight a few years ago the author of Hyperbole Bob Weiner joined the band and
we started together to actively develop Hyperbole again.
One of my focus areas in that work has been to add test cases. We have now gone from no
tests to over 300 ert tests for the package. This talk is about my test case journey. What
I have learned by doing that.
# Discussion
## Questions and answers
- Q:How many tests do you have for Hyperbole and how wouild you rate
the test coverage compared to other packages?
- A:
- With all tests including the interactive we have 354 tests.
Havng said that I must point out that the size of the tests
can be very different. I tend to split tests so they are
logically (in some sense) different. So that if a test fails
it will more likely point you to what the error is. This
makes it become more tests. Codewise you could collect
similar tests to one ert-deftest making the name of the test
point out some group or collection of functions, but I
don't do that!
- I have not studied other packages so I don't know how our
test coverage compares to other packages. In fact I don't
know what code coverage we have. That is another thing to
look into.
- Q: One small suggestion, to me 'should' means optional, whereas
'shall' or 'must' means required. Not sure if it is too late to
make a major grammar change like that :) Very nice presentation. (I
see :))
- A: The assertions come from the ert package so any changes would
have to be suggested to that. I guess you could make your own
version of the assestions using aliases for should et al.
- Q: FYI, you may find this helpful for running Emacs tests/lints,
both from a command line and from within Emacs with a Transient
menu: <https://github.com/alphapapa/makem.sh> It also works on
remote CI.
- A: Thanks for the suggestion. I did have a look at makem.sh but
a long time ago so I don't remember why we did not try to apply
it. I might give it another look now when I have used plain ert
more.
- Q: Is it easy to run ad hoc tests inside of an Emacs session, given
the command line scripts you need to run to get a batch test session
running? In other words, can you tweak tests in an Emacs session and
run them right away?
- A:
- Yes, in principle you just load your tests and run them all
using `ert` and give it the test selector `t`. That runs
all loaded tests.
- If you want to modify a test you can do that. You change it,
evaluate it, and run it again. Just as you change any
function.
- Q: Did you have to change Hyperbole code and design to be more
readily testable as you were increasing your test coverage?
- A:
- Yes, we have done that to a small extent but we should do
more of that. Some Hyperbole functions are large and by that
complicated to test. Splitting them into smaller logical
parts can make testing easier.
- Also moving code into pure functions and avoid side effects
is a good thing. Pure functions are easier to test. Maybe
haveing the side effects separated out into fewer places.
This has not been applied but is something I have been
thinking about. With side effects I here mean things like
adding or modifying text in buffers.
- Q: What's the craziest bug you found when writing these tests?
- A: This is not a bug but I always assumed giving a prefix
argument to a cursor movement would give the same result as
hitting the key the same amount of times. So like C-u 2 C-f
would be the same as hitting the C-f key twise. It is not! When
moving over a hidden area, the three dots '...' at the end of
folded line in org-mode or outline-mode, you get different
behavior. Trying to write a test case for the kotl-mode and its
folded behavior teached me that.
- Q: Why do you prefer el-mock to mocking using cl-letf. (Question
asked in BBB)
- - With cl-letf you need to keep track if the mocked
functionality is being called or not. The el-mock package
does that for you which is what you normally want. Doing
this with cl-letf means the definition becomes longer and
more complicated. Sort of blurs the picture. el-mock is more
to the point.
- BUT since cl-letf does allow you do define a "new"
function it is more powerful and it can be the only option
in cases where el-mock is too limited. So it is good to know
of this possibility with cl-letf when el-mock does not
provide what you need.
[[!inline pages="internal(2023/info/test-after)" raw="yes"]]
[[!inline pages="internal(2023/info/test-nav)" raw="yes"]]
|
