[[!meta title="What I learned by writing test cases for GNU Hyperbole"]] [[!meta copyright="Copyright © 2023 Mats Lidell"]] [[!inline pages="internal(2023/info/test-nav)" raw="yes"]] # What I learned by writing test cases for GNU Hyperbole Mats Lidell (he, him, his) - IRC: matsl, @matsl@mastodon.acc.sunet.se, [[!inline pages="internal(2023/info/test-before)" raw="yes"]] I'm maintaining GNU Hyperbole. I volunteered for that at a time when FSF was asking for one since it was unmaintained. I did not have much elisp experience but a passion for the package. Not much happened. To my great delight a few years ago the author of Hyperbole Bob Weiner joined the band and we started together to actively develop Hyperbole again. One of my focus areas in that work has been to add test cases. We have now gone from no tests to over 300 ert tests for the package. This talk is about my test case journey. What I have learned by doing that. # Discussion ## Questions and answers - Q:How many tests do you have for Hyperbole and how wouild you rate the test coverage compared to other packages? - A:  - With all tests including the interactive we have 354 tests. Havng said that I must point out that the size of the tests can be very different. I tend to split tests so they are logically (in some sense) different. So that if a test fails it will more likely point you to what the error is. This makes it become more tests. Codewise you could collect similar tests to one ert-deftest making the name of the test point out some group or collection of functions, but I don't do that! - I have not studied other packages so I don't know how our test coverage compares to other packages. In fact I don't know what code coverage we have. That is another thing to look into. - Q: One small suggestion, to me 'should' means optional, whereas 'shall' or 'must' means required. Not sure if it is too late to make a major grammar change like that :) Very nice presentation. (I see :)) - A: The assertions come from the ert package so any changes would have to be suggested to that. I guess you could make your own version of the assestions using aliases for should et al. - Q: FYI, you may find this helpful for running Emacs tests/lints, both from a command line and from within Emacs with a Transient menu:   It also works on remote CI. - A: Thanks for the suggestion. I did have a look at makem.sh but a long time ago so I don't remember why we did not try to apply it. I might give it another look now when I have used plain ert more. - Q: Is it easy to run ad hoc tests inside of an Emacs session, given the command line scripts you need to run to get a batch test session running? In other words, can you tweak tests in an Emacs session and run them right away? - A:  - Yes, in principle you just load your tests and run them all using `ert` and give it the test selector `t`. That runs all loaded tests.  - If you want to modify a test you can do that. You change it, evaluate it, and run it again. Just as you change any function. - Q: Did you have to change Hyperbole code and design to be more readily testable as you were increasing your test coverage? - A:  - Yes, we have done that to a small extent but we should do more of that. Some Hyperbole functions are large and by that complicated to test. Splitting them into smaller logical parts can make testing easier.  - Also moving code into pure functions and avoid side effects is a good thing. Pure functions are easier to test. Maybe haveing the side effects separated out into fewer places. This has not been applied but is something I have been thinking about. With side effects I here mean things like adding or modifying text in buffers.  - Q: What's the craziest bug you found when writing these tests? - A: This is not a bug but I always assumed giving a prefix argument to a cursor movement would give the same result as hitting the key the same amount of times. So like C-u 2 C-f would be the same as hitting the C-f key twise. It is not! When moving over a hidden area, the three dots '...' at the end of folded line in org-mode or outline-mode, you get different behavior. Trying to write a test case for the kotl-mode and its folded behavior teached me that. - Q: Why do you prefer el-mock to mocking using cl-letf. (Question asked in BBB) - - With cl-letf you need to keep track if the mocked functionality is being called or not. The el-mock package does that for you which is what you normally want. Doing this with cl-letf means the definition becomes longer and more complicated. Sort of blurs the picture. el-mock is more to the point. - BUT since cl-letf does allow you do define a "new" function it is more powerful and it can be the only option in cases where el-mock is too limited. So it is good to know of this possibility with cl-letf when el-mock does not provide what you need. [[!inline pages="internal(2023/info/test-after)" raw="yes"]] [[!inline pages="internal(2023/info/test-nav)" raw="yes"]]