diff options
Diffstat (limited to '2023/captions/emacsconf-2023-test--what-i-learned-by-writing-test-cases-for-gnu-hyperbole--mats-lidell--main.vtt')
| -rw-r--r-- | 2023/captions/emacsconf-2023-test--what-i-learned-by-writing-test-cases-for-gnu-hyperbole--mats-lidell--main.vtt | 1371 |
1 files changed, 1371 insertions, 0 deletions
diff --git a/2023/captions/emacsconf-2023-test--what-i-learned-by-writing-test-cases-for-gnu-hyperbole--mats-lidell--main.vtt b/2023/captions/emacsconf-2023-test--what-i-learned-by-writing-test-cases-for-gnu-hyperbole--mats-lidell--main.vtt new file mode 100644 index 00000000..d07b8201 --- /dev/null +++ b/2023/captions/emacsconf-2023-test--what-i-learned-by-writing-test-cases-for-gnu-hyperbole--mats-lidell--main.vtt @@ -0,0 +1,1371 @@ +WEBVTT captioned by sachac, checked by sachac + +NOTE Introduction + +00:00:03.120 --> 00:00:07.439 +Hi everyone! I'm Mats Lidell. + +00:00:07.440 --> 00:00:09.879 +I'm going to talk about my journey + +00:00:09.880 --> 00:00:12.480 +writing test cases for GNU Hyperbole + +00:00:12.481 --> 00:00:19.399 +and what I learned on the way. + +00:00:19.400 --> 00:00:24.079 +So, why write tests for GNU Hyperbole? + +00:00:24.080 --> 00:00:25.679 +There is some background. + +00:00:25.680 --> 00:00:27.959 +I'm the co-maintainer of GNU Hyperbole + +00:00:27.960 --> 00:00:33.479 +together with Bob Weiner. Bob is the author of the package. + +00:00:33.480 --> 00:00:34.680 +The package is available through + +00:00:34.681 --> 00:00:38.799 +the Emacs package manager and GNU Elpa + +00:00:38.800 --> 00:00:42.599 +if you would want to try it out. + +00:00:42.600 --> 00:00:46.359 +The package has some age. I think it dates back to + +00:00:46.360 --> 00:00:50.119 +a first release around 1993, which is also + +00:00:50.120 --> 00:00:54.799 +when I got in contact with the package the first time. + +00:00:54.800 --> 00:00:58.239 +I was a user of the package for many years. + +00:00:58.240 --> 00:01:03.119 +Later, I became the maintainer of the package for the FSF. + +00:01:03.120 --> 00:01:04.679 +That was although I did not have + +00:01:04.680 --> 00:01:09.039 +much knowledge of Emacs Lisp, + +00:01:09.040 --> 00:01:12.679 +and I still have a lot to learn. + +00:01:12.680 --> 00:01:15.959 +A few years ago, we started to work actively on the package, + +00:01:15.960 --> 00:01:20.839 +with setting up goals and having meetings. + +00:01:20.840 --> 00:01:24.959 +So my starting point is that I had experience + +00:01:24.960 --> 00:01:27.439 +with test automation from development + +00:01:27.440 --> 00:01:30.599 +in C++, Java and Python + +00:01:30.600 --> 00:01:37.239 +using different x-unit frameworks like cppunit, junit. + +00:01:37.240 --> 00:01:40.039 +That was in my daytime work where + +00:01:40.040 --> 00:01:41.959 +the technique of using pull requests + +00:01:41.960 --> 00:01:46.719 +with changes backed up by tests were the daily routine. + +00:01:46.720 --> 00:01:49.199 +It was really a requirement for a change to go in + +00:01:49.200 --> 00:01:52.159 +to have supporting test cases. + +00:01:52.160 --> 00:01:58.559 +I believe, a quite common setup and requirement these days. + +00:01:58.560 --> 00:02:02.039 +I also had been an Emacs user for many years, + +00:02:02.040 --> 00:02:04.279 +but with focus on being a user. + +00:02:04.280 --> 00:02:09.839 +So as I mentioned, I have limited Emacs Lisp knowledge. + +00:02:09.840 --> 00:02:11.359 +When we decided to start + +00:02:11.360 --> 00:02:13.959 +to work actively on Hyperbole again, + +00:02:13.960 --> 00:02:15.519 +it was natural for me to look into + +00:02:15.520 --> 00:02:18.679 +raising the quality by adding unit tests. + +00:02:18.680 --> 00:02:20.679 +This also goes hand in hand + +00:02:20.680 --> 00:02:25.239 +with running these regularly as part of a build process. + +00:02:25.240 --> 00:02:28.439 +All in all, following the current best practice + +00:02:28.440 --> 00:02:31.359 +of software development. + +00:02:31.360 --> 00:02:36.479 +But since Hyperbole had no tests at all, + +00:02:36.480 --> 00:02:38.719 +it would not be enough just to add tests + +00:02:38.720 --> 00:02:41.799 +for new or changed functionality. + +00:02:41.800 --> 00:02:44.639 +We wanted to add it even broader; ideally, everywhere. + +00:02:44.640 --> 00:02:48.399 +So work started with adding tests here and there + +00:02:48.400 --> 00:02:52.039 +based on our gut feeling where it would be most useful. + +00:02:52.040 --> 00:02:55.799 +This work is still ongoing. + +00:02:55.800 --> 00:02:58.119 +So this is where my journey starts + +00:02:58.120 --> 00:03:00.759 +with much functionality to test, + +00:03:00.760 --> 00:03:03.359 +no knowledge of what testing frameworks existed, + +00:03:03.360 --> 00:03:11.159 +and not really knowing a lot about Emacs Lisp at all. + +NOTE ERT: Emacs Lisp Regression Testing + +00:03:11.160 --> 00:03:13.799 +Luckily there is a package for writing tests in Emacs. + +00:03:13.800 --> 00:03:17.919 +It is called ERT: Emacs Lisp Regression Testing. + +00:03:17.920 --> 00:03:20.959 +It contains both support for defining tests and running them. + +00:03:20.960 --> 00:03:24.639 +Defining a test is done with the macro `ert-deftest`. + +00:03:24.640 --> 00:03:28.919 +In its simplest form, a test has a name, a doc string, and a body. + +00:03:28.920 --> 00:03:31.439 +The doc string is where you typically can give + +00:03:31.440 --> 00:03:33.799 +a detailed description of the test + +00:03:33.800 --> 00:03:35.559 +and has space for more info + +00:03:35.560 --> 00:03:42.279 +than what can be given in the test name. + +00:03:42.280 --> 00:03:45.239 +The body is where all the interesting things happen. + +00:03:45.240 --> 00:03:51.959 +It is here you prepare the test, run it and verify the outcome. + +00:03:51.960 --> 00:03:54.239 +Schematically, it looks like this. + +00:03:54.240 --> 00:04:00.239 +You have the ert-deftest, you have the test name, + +00:04:00.240 --> 00:04:02.799 +and the doc string, and then the body. + +00:04:02.800 --> 00:04:06.559 +It is in the body where everything interesting happens. + +00:04:06.560 --> 00:04:09.759 +The test is prepared, the function of the test is executed, + +00:04:09.760 --> 00:04:13.119 +and the outcome of the test is evaluated. + +00:04:13.120 --> 00:04:14.359 +Did the test succeed or not? + +NOTE Assertions with `should` + +00:04:14.360 --> 00:04:18.479 +The verification of a test is performed with + +00:04:18.480 --> 00:04:21.479 +one or more so-called assertions. + +00:04:21.480 --> 00:04:24.999 +In ERT, they are implemented + +00:04:25.000 --> 00:04:26.599 +with the macro `should` + +00:04:26.600 --> 00:04:33.559 +together with a set of related macros. + +00:04:33.560 --> 00:04:35.519 +`should` takes a form as argument, + +00:04:35.520 --> 00:04:37.839 +and if the form evaluates to nil, + +00:04:37.840 --> 00:04:48.580 +the test has failed. So let's look at an example. + +00:04:48.581 --> 00:04:51.919 +This simple test verifies that the function `+` + +00:04:51.920 --> 00:04:56.919 +can add the numbers 2 and 3 and get the result 5. + +NOTE Running a test case + +00:04:56.920 --> 00:05:01.959 +So now we have defined a test case. How do we run it? + +00:05:01.960 --> 00:05:03.919 +The ERT package has the function (or + +00:05:03.920 --> 00:05:09.519 +rather convenience alias) `ert`. It takes a test selector. + +00:05:09.520 --> 00:05:19.759 +The test name works as a selector for running just one test. + +00:05:19.760 --> 00:05:27.900 +So here we have the example. Let's evaluate it. + +00:05:27.901 --> 00:05:34.519 +We define it and then we run it using ERT. + +00:05:34.520 --> 00:05:42.399 +As you see, we get prompted for a test selector + +00:05:42.400 --> 00:05:46.319 +but we only have one test case defined at the moment. + +00:05:46.320 --> 00:05:55.919 +It's the example 0. So let's hit RET. + +00:05:55.920 --> 00:05:58.959 +As you see here, we get some output + +00:05:58.960 --> 00:06:01.359 +describing what we have just done. + +00:06:01.360 --> 00:06:04.839 +There is one test case it has passed, zero failed, + +00:06:04.840 --> 00:06:07.839 +zero skipped, total 1 of 1 test case + +00:06:07.840 --> 00:06:14.439 +and some time stamps for the execution. + +00:06:14.440 --> 00:06:18.519 +We also see this green mark here indicating one test case + +00:06:18.520 --> 00:06:23.039 +and that it was successful. + +00:06:23.040 --> 00:06:29.659 +For inspecting the test, we can hit the letter `l` + +00:06:29.660 --> 00:06:32.839 +which shows all the `should` forms + +00:06:32.840 --> 00:06:37.779 +that was executed during this test case. + +00:06:37.780 --> 00:06:39.919 +So here we see that we have the `should`, + +00:06:39.920 --> 00:06:47.999 +one `should` executed, and we see the form equals to 2, + +00:06:48.000 --> 00:06:49.799 +and it was 5 equals to 5. + +00:06:49.800 --> 00:06:54.559 +So a good example of a successful test case. + +NOTE Debug a test + +00:06:54.560 --> 00:06:57.919 +So now we've seen how we can run a test case. + +00:06:57.920 --> 00:07:03.799 +Can we debug it? Yes. For debugging a test case, + +00:07:03.800 --> 00:07:07.939 +the `ert-deftest` can be set up using `edebug-defun`, + +00:07:07.940 --> 00:07:10.319 +just as a function or macro is set up + +00:07:10.320 --> 00:07:18.819 +or instrumented for debugging. So let's try that. + +00:07:18.820 --> 00:07:24.119 +So we try `edebug-defun` here. + +00:07:24.120 --> 00:07:28.279 +Now it's instrumented for debugging. + +00:07:28.280 --> 00:07:35.659 +And we run it, `ert`, and we're inside the debugger, + +00:07:35.660 --> 00:07:40.679 +and we can inspect here what's happening. + +00:07:40.680 --> 00:07:46.960 +Step through it and yes it succeeded just as before. + +NOTE Commercial break: Hyperbole + +00:07:50.380 --> 00:07:56.879 +It's time for a commercial break! + +00:07:56.880 --> 00:08:00.079 +Hyperbole itself can help with running tests + +00:08:00.080 --> 00:08:03.639 +and also help with running them in debug mode. + +00:08:03.640 --> 00:08:08.519 +That is because hyperbole identifies the `ert-deftest` + +00:08:08.520 --> 00:08:12.679 +as an implicit button. An implicit button is basically + +00:08:12.680 --> 00:08:13.759 +a string or pattern + +00:08:13.760 --> 00:08:16.799 +that Hyperbole has assigned some meaning to. + +00:08:16.800 --> 00:08:19.959 +For the string `ert-deftest`, it is to run the test case. + +00:08:19.960 --> 00:08:24.559 +You activate the button with the action-key. + +00:08:24.560 --> 00:08:27.079 +The standard binding is the middle mouse button, + +00:08:27.080 --> 00:08:33.040 +or from the keyboard, M-RET. + +00:08:33.041 --> 00:08:34.799 +So let's try that. + +00:08:34.800 --> 00:08:42.219 +We move the cursor here and then we type M-RET. + +00:08:42.220 --> 00:08:47.959 +And boom, the test case was executed. + +00:08:47.960 --> 00:08:54.479 +And to run it in debug mode we type C-u M-RET + +00:08:54.480 --> 00:08:57.719 +to get the assist key, and then we're in the debugger. + +00:08:57.720 --> 00:09:10.479 +So that's pretty useful and convenient. + +NOTE Instrument function on the fly + +00:09:10.480 --> 00:09:13.719 +A related useful feature here is the step-in functionality + +00:09:13.720 --> 00:09:16.399 +bound to the letter i in `debug-mode`. + +00:09:16.400 --> 00:09:18.119 +It allows you to step into a function + +00:09:18.120 --> 00:09:20.479 +and continue debugging from there. + +00:09:20.480 --> 00:09:22.839 +For the cases where your test does not do what you want, + +00:09:22.840 --> 00:09:25.119 +looking at what happens in the function of the test + +00:09:25.120 --> 00:09:37.259 +can be really useful. Let's try that with another example. + +00:09:37.260 --> 00:09:43.359 +So here we have two helper functions, one `f1-add`, + +00:09:43.360 --> 00:09:47.439 +that use the built-in `+` function + +00:09:47.440 --> 00:09:52.239 +and then we have `my-add` that uses that function. + +00:09:52.240 --> 00:09:59.399 +So we're going to test myadd. + +00:09:59.400 --> 00:10:02.919 +And then let's run this. + +00:10:02.920 --> 00:10:05.959 +Let's run this using hyperbole in debug mode + +00:10:05.960 --> 00:10:10.079 +C-u M-RET. We're in the debugger again, + +00:10:10.080 --> 00:10:15.639 +and let's step up front to my function under test + +00:10:15.640 --> 00:10:19.359 +and then press `i` for getting it instrumented + +00:10:19.360 --> 00:10:23.019 +and going into it for debugging. + +00:10:23.020 --> 00:10:25.139 +And here we can expect that it's getting + +00:10:25.140 --> 00:10:26.559 +the arguments 1 and 3, + +00:10:26.560 --> 00:10:30.999 +and it returns the result 4 as expected. + +00:10:31.000 --> 00:10:39.119 +And yes, of course, our test case will then succeed. + +NOTE Mocking + +00:10:39.120 --> 00:10:41.839 +The next tool in our toolbox is mocking. + +00:10:41.840 --> 00:10:46.239 +Mocking is needed when we want to simulate the response + +00:10:46.240 --> 00:10:49.279 +from a function used by the function under test. + +00:10:49.280 --> 00:10:53.139 +That is the implementation of the function. + +00:10:53.140 --> 00:10:56.119 +This could be for various reasons. + +00:10:56.120 --> 00:11:00.879 +One example could be because it would be hard or impossible + +00:11:00.880 --> 00:11:04.199 +in the test setup to get the behavior you want to test for, + +00:11:04.200 --> 00:11:06.279 +like an external error case. + +00:11:06.280 --> 00:11:08.679 +But the mock can also be used to verify + +00:11:08.680 --> 00:11:11.619 +that the function is called with a specific argument. + +00:11:11.620 --> 00:11:14.559 +We can view it as a way to isolate the function on the test + +00:11:14.560 --> 00:11:16.719 +from its dependencies. + +00:11:16.720 --> 00:11:18.959 +So in order to test the function in isolation, + +00:11:18.960 --> 00:11:22.079 +we need to cut out any dependencies to external behavior. + +00:11:22.080 --> 00:11:25.839 +Most obvious would be dependencies to external resources, + +00:11:25.840 --> 00:11:27.639 +such as web pages. As an example: + +00:11:27.640 --> 00:11:30.639 +Hyperbole contains functionality to link you to + +00:11:30.640 --> 00:11:34.239 +social media resources and other resources on the net. + +00:11:34.240 --> 00:11:37.899 +Testing that would require the test system to call out + +00:11:37.900 --> 00:11:39.639 +to the social media resources + +00:11:39.640 --> 00:11:43.539 +and would depend on it being available, etc. + +00:11:43.540 --> 00:11:45.479 +Nothing technically stops a test case + +00:11:45.480 --> 00:11:47.239 +to depend on the external resources, + +00:11:47.240 --> 00:11:51.319 +but would, if nothing else, be flaky or slow. + +00:11:51.320 --> 00:11:53.759 +It could be part of an end-to-end suite + +00:11:53.760 --> 00:11:57.179 +where we want to test that it works all the way. + +00:11:57.180 --> 00:11:59.719 +In this case, we want to look at the isolated case + +00:11:59.720 --> 00:12:04.099 +that can be run with no dependency on external resources. + +00:12:04.100 --> 00:12:06.679 +What you want to do is to replace the function with a mock + +00:12:06.680 --> 00:12:10.339 +that behaves as the real function would do. + +00:12:10.340 --> 00:12:11.639 +The package I have found + +00:12:11.640 --> 00:12:14.319 +and have used for mocking is `el-mock`. + +00:12:14.320 --> 00:12:21.839 +The workhorse in this package is the `with-mock` macro. + +00:12:21.840 --> 00:12:26.519 +It looks like this: `with-mock` followed by a body. + +00:12:26.520 --> 00:12:30.439 +In the execution of the body, stubs and mocks + +00:12:30.440 --> 00:12:32.899 +defined in the body is respected. + +00:12:32.900 --> 00:12:39.199 +Let's look at some examples to make that clearer. + +00:12:39.200 --> 00:12:42.079 +In this case, we have the macro `with-mock`. + +00:12:42.080 --> 00:12:43.959 +It works so that the expression + +00:12:43.960 --> 00:12:48.639 +`stub + => 10` is interpreted + +00:12:48.640 --> 00:12:51.919 +so that the function `+` will be replaced with the stub. + +00:12:51.920 --> 00:12:56.779 +The stub will return 10 regardless how it is called. + +00:12:56.780 --> 00:12:58.119 +Note that the stub function + +00:12:58.120 --> 00:13:00.199 +does not have to be called at this level + +00:13:00.200 --> 00:13:02.799 +but could be called at any level in the call chain. + +00:13:02.800 --> 00:13:07.479 +By knowing how the function under test is implemented + +00:13:07.480 --> 00:13:09.319 +and how the implementation works, + +00:13:09.320 --> 00:13:11.959 +you can find function calls you want to mock + +00:13:11.960 --> 00:13:14.999 +to force certain behavior that you want to test, + +00:13:15.000 --> 00:13:18.999 +or to avoid calls to external resources, slow calls, etc. + +00:13:19.000 --> 00:13:21.959 +Simply isolate the function under test + +00:13:21.960 --> 00:13:26.119 +and simulate its environment. + +00:13:26.120 --> 00:13:28.639 +Mock is a little bit more sophisticated + +00:13:28.640 --> 00:13:30.079 +and depends on the arguments + +00:13:30.080 --> 00:13:31.479 +that the mock function is called with. + +00:13:31.480 --> 00:13:33.847 +Or more precise, it is checked + +00:13:33.848 --> 00:13:35.519 +after the `with-mock` clause + +00:13:35.520 --> 00:13:38.079 +that the arguments match the arguments it was called with + +00:13:38.080 --> 00:13:39.759 +or even if it was called at all. + +00:13:39.760 --> 00:13:41.839 +If it is called with other arguments + +00:13:41.840 --> 00:13:43.719 +there will be an error, + +00:13:43.720 --> 00:13:46.479 +and if it's not called, it is also an error. + +00:13:46.480 --> 00:13:48.359 +So this way, we are sure that the function + +00:13:48.360 --> 00:13:51.319 +we were expected to be called actually was called. + +00:13:51.320 --> 00:13:53.399 +An important piece of the testing. + +00:13:53.400 --> 00:13:56.239 +So we are sure that the mock we have provided + +00:13:56.240 --> 00:14:03.999 +actually is triggered by the test case. + +00:14:04.000 --> 00:14:08.159 +So here we have an example of `with-mock` + +00:14:08.160 --> 00:14:18.879 +where the `f1-add` function is mocked, + +00:14:18.880 --> 00:14:21.999 +so that if it's called with 2 and 3 as arguments, + +00:14:22.000 --> 00:14:24.919 +it will return 10. Then we have a test case + +00:14:24.920 --> 00:14:27.999 +where we try the `my-add` function, + +00:14:28.000 --> 00:14:30.319 +as you might remember, and call that with 2 and 3 + +00:14:30.320 --> 00:14:32.799 +and see that it should also then return 10 + +00:14:32.800 --> 00:14:41.239 +because it's using `f1-add`. + +NOTE cl-letf + +00:14:41.240 --> 00:14:44.559 +Moving over to `cl-letf`. + +00:14:44.560 --> 00:14:47.679 +In rare occasions, the limitations of `el-mock` means + +00:14:47.680 --> 00:14:50.239 +you would want to implement a full-fledged function + +00:14:50.240 --> 00:14:52.979 +to be used under test. + +00:14:52.980 --> 00:14:55.439 +Then the macro `cl-letf` can be useful. + +00:14:55.440 --> 00:14:57.879 +However, you need to handle the case yourself + +00:14:57.880 --> 00:15:00.099 +if the function was not called. + +00:15:00.100 --> 00:15:03.519 +Looking through the test cases where I have used `cl-letf`, + +00:15:03.520 --> 00:15:06.119 +I think most can be implemented using plain mocking. + +00:15:06.120 --> 00:15:11.239 +Cases left is where the args to the mock might be different + +00:15:11.240 --> 00:15:13.739 +due to environment issues. + +00:15:13.740 --> 00:15:24.099 +In that case, a static mock will not work. + +NOTE Hooks + +00:15:24.100 --> 00:15:30.719 +Another trick is that functions that uses hooks. + +00:15:30.720 --> 00:15:35.639 +You can overload or replace the hooks to do the testing. + +00:15:35.640 --> 00:15:40.759 +So you can use the hook function just to do the verification + +00:15:40.760 --> 00:15:43.119 +and not do anything useful in the hook. + +00:15:43.120 --> 00:15:45.079 +Also, here you need to be careful + +00:15:45.080 --> 00:15:55.719 +to make sure the test handler is called and nothing else. + +NOTE Side effects and initial buffer state + +00:15:55.720 --> 00:15:57.679 +So far we have been talking about testing + +00:15:57.680 --> 00:15:59.039 +and what the function returns. + +00:15:59.040 --> 00:16:01.119 +In the best of words, we have a pure function + +00:16:01.120 --> 00:16:02.959 +that only depends on its arguments + +00:16:02.960 --> 00:16:04.939 +and produces no side effects. + +00:16:04.940 --> 00:16:06.899 +Many operations produce side effects + +00:16:06.900 --> 00:16:09.479 +or operate on the contents of buffers + +00:16:09.480 --> 00:16:12.379 +such as writing a message in the message buffer, + +00:16:12.380 --> 00:16:15.659 +change the state of a buffer, move point etc. + +00:16:15.660 --> 00:16:18.859 +Hyperbole is not an exception. Quite the contrary. + +00:16:18.860 --> 00:16:20.839 +Much of the functions creating links + +00:16:20.840 --> 00:16:24.420 +are just about updating buffers. + +00:16:24.421 --> 00:16:28.559 +This poses a special problem for tests. + +00:16:28.560 --> 00:16:29.839 +The test gets longer + +00:16:29.840 --> 00:16:31.919 +since you need to create buffers and files, + +00:16:31.920 --> 00:16:33.279 +initialize the contents. + +00:16:33.280 --> 00:16:35.159 +Verifying the outcome becomes trickier + +00:16:35.160 --> 00:16:39.019 +since you need to make sure you look at the right place. + +00:16:39.020 --> 00:16:41.039 +At the end of the test, you need to clean up, + +00:16:41.040 --> 00:16:43.439 +both for not leaving a lot of garbage + +00:16:43.440 --> 00:16:45.279 +in buffers and files around, + +00:16:45.280 --> 00:16:48.479 +and even worse, not cause later tests + +00:16:48.480 --> 00:16:50.959 +to depend on the leftovers from the other tests. + +00:16:50.960 --> 00:16:53.079 +Here are some functions and variables + +00:16:53.080 --> 00:17:05.099 +I have found useful for this. + +NOTE with-temp-buffer + +00:17:05.100 --> 00:17:09.199 +For creating tests: `with-temp-buffer`: + +00:17:09.200 --> 00:17:11.919 +it provides you a temp buffer that you visit, + +00:17:11.920 --> 00:17:13.719 +and afterwards, there is no need to clean up. + +00:17:13.720 --> 00:17:16.519 +This is the first choice if that is all you need. + +NOTE make-temp-file + +00:17:16.520 --> 00:17:20.519 +`make-temp-file`: If you need a file, + +00:17:20.520 --> 00:17:21.959 +this is the function to use. + +00:17:21.960 --> 00:17:24.279 +It creates a temp file or a directory. + +00:17:24.280 --> 00:17:26.959 +The file can be filled with initial contents. + +00:17:26.960 --> 00:17:31.019 +This needs to be cleaned up after a test. + +00:17:31.020 --> 00:17:33.287 +Moving on to verifying and debugging: + +NOTE buffer-string + +00:17:33.288 --> 00:17:38.247 +`buffer-string`: returns the full contents + +00:17:38.248 --> 00:17:39.499 +of the buffer as a string. + +00:17:39.500 --> 00:17:41.399 +That can sound a bit voluminous, + +00:17:41.400 --> 00:17:46.139 +but since tests are normally small, this often works well. + +00:17:46.140 --> 00:17:48.439 +I have in particular found good use of comparing + +00:17:48.440 --> 00:17:50.399 +the contents of buffers with the empty string. + +00:17:50.400 --> 00:17:53.359 +That would give an error, but as we have seen + +00:17:53.360 --> 00:17:56.079 +with the output produced by the `should` assertion, + +00:17:56.080 --> 00:17:58.079 +this is almost like a print statement + +00:17:58.080 --> 00:18:01.199 +and can be compared with the good old technique + +00:18:01.200 --> 00:18:04.399 +of debugging with print statements. + +00:18:04.400 --> 00:18:06.247 +There might be other ways to do the same + +00:18:06.248 --> 00:18:09.919 +as we saw with debugging. + +NOTE buffer-name + +00:18:09.920 --> 00:18:13.719 +buffer-name: Getting the buffer name is good + +00:18:13.720 --> 00:18:16.239 +to verify what buffer we are looking at. + +00:18:16.240 --> 00:18:18.359 +I often found it useful to check + +00:18:18.360 --> 00:18:21.119 +that my assumptions on what buffer I am acting on + +00:18:21.120 --> 00:18:23.399 +is correct by adding `should` clauses + +00:18:23.400 --> 00:18:25.399 +in the middle of the test execution + +00:18:25.400 --> 00:18:27.399 +or after preparing the test input. + +00:18:27.400 --> 00:18:31.679 +Sometimes Emacs can switch buffers in strange ways, + +00:18:31.680 --> 00:18:34.199 +maybe because the test case is badly written, + +00:18:34.200 --> 00:18:37.239 +and making sure your assumptions are correct + +00:18:37.240 --> 00:18:40.339 +is a good sanity check. + +00:18:40.340 --> 00:18:42.239 +Even the ert package does + +00:18:42.240 --> 00:18:44.879 +some buffer and windows manipulation for its reporting + +00:18:44.880 --> 00:18:47.487 +that I have not fully learned how to master, + +00:18:47.488 --> 00:18:51.979 +so assertion for checking the sanity of the test is good. + +NOTE major-mode + +00:18:51.980 --> 00:18:55.679 +Finally, `major-mode`: Verify the buffer has the proper mode. + +00:18:55.680 --> 00:19:02.679 +Can also be very useful and is a good sanity check. + +NOTE unwind-protect + +00:19:02.680 --> 00:19:06.599 +Finally, cleaning up. `unwind-protect`. + +00:19:06.600 --> 00:19:09.039 +The tool for cleaning up is the `unwind-protect` form + +00:19:09.040 --> 00:19:12.479 +which ensures that the unwind forms + +00:19:12.480 --> 00:19:15.439 +always are executed regardless of the outcome of the body. + +00:19:15.440 --> 00:19:20.419 +So if your test fails, you are sure the cleanup is executed. + +00:19:20.420 --> 00:19:22.759 +Let's look at unwind-protect together with + +00:19:22.760 --> 00:19:30.519 +the temporary file example. Many tests look like this. + +00:19:30.520 --> 00:19:35.279 +You create some resource, you call `unwind-protect`, + +00:19:35.280 --> 00:19:42.759 +you do the test, and then afterwards you do the cleanup. + +00:19:42.760 --> 00:19:46.359 +The cleanup for a file and a buffer is so common, + +00:19:46.360 --> 00:19:50.999 +so I have created a helper for that. + +00:19:51.000 --> 00:19:56.559 +It looks like this. + +00:19:56.560 --> 00:19:59.179 +The trick with the `buffer-modified` flag + +00:19:59.180 --> 00:20:00.719 +is to avoid getting prompted + +00:20:00.720 --> 00:20:03.219 +for killing a buffer that is not saved. + +00:20:03.220 --> 00:20:05.439 +The test buffers are often in the state + +00:20:05.440 --> 00:20:15.099 +where they have not been saved but modified. + +NOTE Input, with-simulated-input + +00:20:15.100 --> 00:20:19.679 +Another problem for tests are input. + +00:20:19.680 --> 00:20:21.559 +In the middle of execution a function + +00:20:21.560 --> 00:20:24.039 +might want to have some interaction with the user. + +00:20:24.040 --> 00:20:26.959 +Testing this poses a problem, not only in that + +00:20:26.960 --> 00:20:31.199 +the input matters, but also as how even to get the test case + +00:20:31.200 --> 00:20:34.079 +to recognize the input!? + +00:20:34.080 --> 00:20:36.039 +Ideally the tests are run in batch mode, + +00:20:36.040 --> 00:20:38.919 +which in some sense means no user interaction. + +00:20:38.920 --> 00:20:42.999 +In batch mode, there is no event loop running. + +00:20:43.000 --> 00:20:47.179 +Fortunately, there is a package `with-simulated-input` + +00:20:47.180 --> 00:20:53.259 +that gets you around these issues. + +00:20:53.260 --> 00:20:55.399 +This is a macro that allows us + +00:20:55.400 --> 00:20:56.999 +to define a set of characters + +00:20:57.000 --> 00:20:59.079 +that will be read by the function under the test, + +00:20:59.080 --> 00:21:02.579 +and all of this works in batch mode. It looks like this. + +00:21:02.580 --> 00:21:04.159 +We have `with-simulated-input`, + +00:21:04.160 --> 00:21:09.839 +and then a string of characters, and then a body. + +00:21:09.840 --> 00:21:11.647 +The form takes a string of keys + +00:21:11.648 --> 00:21:13.119 +and runs the rest of the body, + +00:21:13.120 --> 00:21:15.439 +and if there are input required, + +00:21:15.440 --> 00:21:18.119 +it is picked from the string of keys. + +00:21:18.120 --> 00:21:20.421 +In our example, the `read-string` call + +00:21:20.422 --> 00:21:21.719 +will read up until RET, + +00:21:21.720 --> 00:21:26.119 +and then return the characters read. + +00:21:26.120 --> 00:21:29.639 +As you see in the example, space needs to be provided + +00:21:29.640 --> 00:21:38.459 +by the string SPC, as return by the string RET. + +NOTE Running all tests + +00:21:38.460 --> 00:21:40.799 +So now we have seen ways to create test cases + +00:21:40.800 --> 00:21:43.219 +and even make it possible to run some of them + +00:21:43.220 --> 00:21:44.679 +that has I/O in batch mode. + +00:21:44.680 --> 00:21:47.279 +But the initial goal was to run them all at once. + +00:21:47.280 --> 00:21:48.919 +How do you do that? + +00:21:48.920 --> 00:21:51.759 +Let's go back to the `ert` command. + +00:21:51.760 --> 00:21:53.799 +It prompts for a test selector. + +00:21:53.800 --> 00:21:56.279 +If we give it the selector `t`, + +00:21:56.280 --> 00:21:59.259 +it will run all tests we have currently defined. + +00:21:59.260 --> 00:22:05.779 +Let's try that with the subset of the Hyperbole tests. + +00:22:05.780 --> 00:22:09.559 +Here is the test folder in the Hyperbole directory. + +00:22:09.560 --> 00:22:18.819 +Let's go up here and load all the demo tests. + +00:22:18.820 --> 00:22:21.207 +And then try to run `ert`. + +00:22:21.208 --> 00:22:26.119 +Now we see that we have a bunch of test cases. + +00:22:26.120 --> 00:22:27.919 +We can all run them individually, + +00:22:27.920 --> 00:22:31.719 +but we can run them with `t` instead. + +00:22:31.720 --> 00:22:35.459 +We will run them all at once. + +00:22:35.460 --> 00:22:51.419 +So now, ert is executing all our test cases. + +00:22:51.420 --> 00:22:57.079 +So here we have a nice green display + +00:22:57.080 --> 00:23:03.219 +with all the test cases. + +NOTE Batch mode + +00:23:03.220 --> 00:23:08.159 +So that was fine, but we were still running it manually + +00:23:08.160 --> 00:23:11.980 +by calling ert. How could we run it from the command line? + +00:23:17.180 --> 00:23:21.499 +Ert comes with functions for running it in batch mode. + +00:23:21.500 --> 00:23:25.639 +For Hyperbole, we use `make` for repetitive tasks. + +00:23:25.640 --> 00:23:27.119 +So we have a make target + +00:23:27.120 --> 00:23:29.279 +that uses the ert batch functionality, + +00:23:29.280 --> 00:23:33.259 +and this is the line from the Makefile. + +00:23:33.260 --> 00:23:35.479 +This is a bit detailed, + +00:23:35.480 --> 00:23:37.539 +but you see that we have a part here + +00:23:37.540 --> 00:23:40.779 +where we load the test dependencies. + +00:23:40.780 --> 00:23:43.520 +For getting the packages + +00:23:43.521 --> 00:23:48.459 +such as `el-mock` and `with-simulated-input` etc. loaded. + +00:23:48.460 --> 00:23:53.559 +We also have... I also want to point out here the call to + +00:23:53.560 --> 00:23:58.159 +or the setting of `auto-save-default` to `nil` + +00:23:58.160 --> 00:24:02.439 +to get away with the prompt for excessive backup files + +00:24:02.440 --> 00:24:05.059 +that can pile up after running the tests a few times. + +NOTE Skipping tests + +00:24:05.060 --> 00:24:06.879 +Even with the help of simulated input, + +00:24:06.880 --> 00:24:08.919 +not all tests can be run in batch mode. + +00:24:08.920 --> 00:24:10.559 +They would simply not work there + +00:24:10.560 --> 00:24:12.439 +and have to be run in an interactive Emacs + +00:24:12.440 --> 00:24:14.179 +with the running event loop. + +00:24:14.180 --> 00:24:17.919 +One trick still to be able to use batch mode for automation + +00:24:17.920 --> 00:24:20.319 +is to put the guard at the top of each test case + +00:24:20.320 --> 00:24:22.559 +as the first thing to be executed, + +00:24:22.560 --> 00:24:25.719 +so that it kicks in before anything else and stops Emacs + +00:24:25.720 --> 00:24:27.199 +to try to run the test case. + +00:24:27.200 --> 00:24:35.519 +Now, it looks like this: `(skip-unless (not noninteractive))`. + +00:24:35.520 --> 00:24:38.639 +So when ert sees that the test should be skipped, it skips it + +00:24:38.640 --> 00:24:40.439 +and makes a note of that, + +00:24:40.440 --> 00:24:44.579 +so you will see how many tests that have been skipped. + +00:24:44.580 --> 00:24:47.559 +Too bad. We have a number of test cases defined, + +00:24:47.560 --> 00:24:51.359 +and to run them, we need to run them manually. Well sort of. + +00:24:51.360 --> 00:24:53.807 +Not being able to run all tests easily + +00:24:53.808 --> 00:24:58.419 +is a bit counterproductive + +00:24:58.420 --> 00:25:00.999 +since our goal is to run all tests. + +00:25:01.000 --> 00:25:04.719 +There is however no ert function to run tests in batch mode + +00:25:04.720 --> 00:25:06.779 +with an interactive Emacs. + +00:25:06.780 --> 00:25:08.479 +The closest I have got is either + +00:25:08.480 --> 00:25:10.079 +to start the Emacs from the command line + +00:25:10.080 --> 00:25:12.439 +calling the ert function as we just have seen, + +00:25:12.440 --> 00:25:14.799 +and then killing it manually when done; + +00:25:14.800 --> 00:25:19.599 +or add a function to extract the contents of the ERT buffer + +00:25:19.600 --> 00:25:24.599 +when done and echo it to standard output. + +00:25:24.600 --> 00:25:27.800 +This is how it looks in the Makefile + +00:25:27.801 --> 00:25:31.207 +to get the behavior of cutting and paste, + +00:25:31.208 --> 00:25:34.580 +getting the ERT output into a file + +00:25:34.581 --> 00:25:36.239 +so we can then kill Emacs + +00:25:36.240 --> 00:25:44.799 +and spit out the content of the ERT buffer. + +00:25:44.800 --> 00:25:47.739 +One final word here is that + +00:25:47.740 --> 00:25:54.559 +when you run this in a continuous integration pipeline, + +00:25:54.560 --> 00:25:59.399 +you might not have a TTY for getting Emacs to start, + +00:25:59.400 --> 00:26:03.200 +and that is then another problem + +00:26:03.201 --> 00:26:05.160 +with getting the interactive mode. + +NOTE Conclusion + +00:26:08.460 --> 00:26:11.120 +We have reached the end of the talk. + +00:26:11.121 --> 00:26:14.159 +If you have any new ideas + +00:26:14.160 --> 00:26:16.759 +or have some suggestions for improvements, + +00:26:16.760 --> 00:26:18.239 +feel free to reach out + +00:26:18.240 --> 00:26:21.100 +because I am still on the learning curve of writing, + +00:26:21.101 --> 00:26:25.299 +how to write good test cases. + +00:26:25.300 --> 00:26:27.639 +If you look at the test cases we have in Hyperbole + +00:26:27.640 --> 00:26:29.799 +and you think they might contradict what I am saying here, + +00:26:29.800 --> 00:26:32.579 +it is OK. It is probably right. + +00:26:32.580 --> 00:26:34.599 +I have changed the style as I go + +00:26:34.600 --> 00:26:36.639 +and we have not yet refactored all tests + +00:26:36.640 --> 00:26:38.579 +to benefit from new designs. + +00:26:38.580 --> 00:26:40.599 +That is also the beauty of the test case. + +00:26:40.600 --> 00:26:43.319 +As long as it serves its purpose, it is not terrible + +00:26:43.320 --> 00:26:47.799 +if it is not optimal or not having the best style. + +00:26:47.800 --> 00:26:55.240 +And yes, thanks for listening. Bye. |