diff options
Diffstat (limited to '2023/info/test-after.md')
| -rw-r--r-- | 2023/info/test-after.md | 66 |
1 files changed, 66 insertions, 0 deletions
diff --git a/2023/info/test-after.md b/2023/info/test-after.md index 8be786ed..e4fb08b2 100644 --- a/2023/info/test-after.md +++ b/2023/info/test-after.md @@ -4,6 +4,9 @@ <a name="test-mainVideo-transcript"></a> # Transcript + +Introduction + [[!template new="1" text="""Hi everyone! I'm Mats Lidell.""" start="00:00:03.120" video="mainVideo-test" id="subtitle"]] [[!template text="""I'm going to talk about my journey""" start="00:00:07.440" video="mainVideo-test" id="subtitle"]] [[!template text="""writing test cases for GNU Hyperbole""" start="00:00:09.880" video="mainVideo-test" id="subtitle"]] @@ -57,6 +60,9 @@ [[!template text="""with much functionality to test,""" start="00:02:58.120" video="mainVideo-test" id="subtitle"]] [[!template text="""no knowledge of what testing frameworks existed,""" start="00:03:00.760" video="mainVideo-test" id="subtitle"]] [[!template text="""and not really knowing a lot about Emacs Lisp at all.""" start="00:03:03.360" video="mainVideo-test" id="subtitle"]] + +ERT: Emacs Lisp Regression Testing + [[!template new="1" text="""Luckily there is a package for writing tests in Emacs.""" start="00:03:11.160" video="mainVideo-test" id="subtitle"]] [[!template text="""It is called ERT: Emacs Lisp Regression Testing.""" start="00:03:13.800" video="mainVideo-test" id="subtitle"]] [[!template text="""It contains both support for defining tests and running them.""" start="00:03:17.920" video="mainVideo-test" id="subtitle"]] @@ -75,6 +81,9 @@ [[!template text="""The test is prepared, the function of the test is executed,""" start="00:04:06.560" video="mainVideo-test" id="subtitle"]] [[!template text="""and the outcome of the test is evaluated.""" start="00:04:09.760" video="mainVideo-test" id="subtitle"]] [[!template text="""Did the test succeed or not?""" start="00:04:13.120" video="mainVideo-test" id="subtitle"]] + +Assertions with `should` + [[!template new="1" text="""The verification of a test is performed with""" start="00:04:14.360" video="mainVideo-test" id="subtitle"]] [[!template text="""one or more so-called assertions.""" start="00:04:18.480" video="mainVideo-test" id="subtitle"]] [[!template text="""In ERT, they are implemented""" start="00:04:21.480" video="mainVideo-test" id="subtitle"]] @@ -85,6 +94,9 @@ [[!template text="""the test has failed. So let's look at an example.""" start="00:04:37.840" video="mainVideo-test" id="subtitle"]] [[!template text="""This simple test verifies that the function `+`""" start="00:04:48.581" video="mainVideo-test" id="subtitle"]] [[!template text="""can add the numbers 2 and 3 and get the result 5.""" start="00:04:51.920" video="mainVideo-test" id="subtitle"]] + +Running a test case + [[!template new="1" text="""So now we have defined a test case. How do we run it?""" start="00:04:56.920" video="mainVideo-test" id="subtitle"]] [[!template text="""The ERT package has the function (or""" start="00:05:01.960" video="mainVideo-test" id="subtitle"]] [[!template text="""rather convenience alias) `ert`. It takes a test selector.""" start="00:05:03.920" video="mainVideo-test" id="subtitle"]] @@ -108,6 +120,9 @@ [[!template text="""one `should` executed, and we see the form equals to 2,""" start="00:06:39.920" video="mainVideo-test" id="subtitle"]] [[!template text="""and it was 5 equals to 5.""" start="00:06:48.000" video="mainVideo-test" id="subtitle"]] [[!template text="""So a good example of a successful test case.""" start="00:06:49.800" video="mainVideo-test" id="subtitle"]] + +Debug a test + [[!template new="1" text="""So now we've seen how we can run a test case.""" start="00:06:54.560" video="mainVideo-test" id="subtitle"]] [[!template text="""Can we debug it? Yes. For debugging a test case,""" start="00:06:57.920" video="mainVideo-test" id="subtitle"]] [[!template text="""the `ert-deftest` can be set up using `edebug-defun`,""" start="00:07:03.800" video="mainVideo-test" id="subtitle"]] @@ -118,6 +133,9 @@ [[!template text="""And we run it, `ert`, and we're inside the debugger,""" start="00:07:28.280" video="mainVideo-test" id="subtitle"]] [[!template text="""and we can inspect here what's happening.""" start="00:07:35.660" video="mainVideo-test" id="subtitle"]] [[!template text="""Step through it and yes it succeeded just as before.""" start="00:07:40.680" video="mainVideo-test" id="subtitle"]] + +Commercial break: Hyperbole + [[!template new="1" text="""It's time for a commercial break!""" start="00:07:50.380" video="mainVideo-test" id="subtitle"]] [[!template text="""Hyperbole itself can help with running tests""" start="00:07:56.880" video="mainVideo-test" id="subtitle"]] [[!template text="""and also help with running them in debug mode.""" start="00:08:00.080" video="mainVideo-test" id="subtitle"]] @@ -135,6 +153,9 @@ [[!template text="""And to run it in debug mode we type C-u M-RET""" start="00:08:47.960" video="mainVideo-test" id="subtitle"]] [[!template text="""to get the assist key, and then we're in the debugger.""" start="00:08:54.480" video="mainVideo-test" id="subtitle"]] [[!template text="""So that's pretty useful and convenient.""" start="00:08:57.720" video="mainVideo-test" id="subtitle"]] + +Instrument function on the fly + [[!template new="1" text="""A related useful feature here is the step-in functionality""" start="00:09:10.480" video="mainVideo-test" id="subtitle"]] [[!template text="""bound to the letter i in `debug-mode`.""" start="00:09:13.720" video="mainVideo-test" id="subtitle"]] [[!template text="""It allows you to step into a function""" start="00:09:16.400" video="mainVideo-test" id="subtitle"]] @@ -156,6 +177,9 @@ [[!template text="""the arguments 1 and 3,""" start="00:10:25.140" video="mainVideo-test" id="subtitle"]] [[!template text="""and it returns the result 4 as expected.""" start="00:10:26.560" video="mainVideo-test" id="subtitle"]] [[!template text="""And yes, of course, our test case will then succeed.""" start="00:10:31.000" video="mainVideo-test" id="subtitle"]] + +Mocking + [[!template new="1" text="""The next tool in our toolbox is mocking.""" start="00:10:39.120" video="mainVideo-test" id="subtitle"]] [[!template text="""Mocking is needed when we want to simulate the response""" start="00:10:41.840" video="mainVideo-test" id="subtitle"]] [[!template text="""from a function used by the function under test.""" start="00:10:46.240" video="mainVideo-test" id="subtitle"]] @@ -231,6 +255,9 @@ [[!template text="""as you might remember, and call that with 2 and 3""" start="00:14:28.000" video="mainVideo-test" id="subtitle"]] [[!template text="""and see that it should also then return 10""" start="00:14:30.320" video="mainVideo-test" id="subtitle"]] [[!template text="""because it's using `f1-add`.""" start="00:14:32.800" video="mainVideo-test" id="subtitle"]] + +cl-letf + [[!template new="1" text="""Moving over to `cl-letf`.""" start="00:14:41.240" video="mainVideo-test" id="subtitle"]] [[!template text="""In rare occasions, the limitations of `el-mock` means""" start="00:14:44.560" video="mainVideo-test" id="subtitle"]] [[!template text="""you would want to implement a full-fledged function""" start="00:14:47.680" video="mainVideo-test" id="subtitle"]] @@ -243,12 +270,18 @@ [[!template text="""Cases left is where the args to the mock might be different""" start="00:15:06.120" video="mainVideo-test" id="subtitle"]] [[!template text="""due to environment issues.""" start="00:15:11.240" video="mainVideo-test" id="subtitle"]] [[!template text="""In that case, a static mock will not work.""" start="00:15:13.740" video="mainVideo-test" id="subtitle"]] + +Hooks + [[!template new="1" text="""Another trick is that functions that uses hooks.""" start="00:15:24.100" video="mainVideo-test" id="subtitle"]] [[!template text="""You can overload or replace the hooks to do the testing.""" start="00:15:30.720" video="mainVideo-test" id="subtitle"]] [[!template text="""So you can use the hook function just to do the verification""" start="00:15:35.640" video="mainVideo-test" id="subtitle"]] [[!template text="""and not do anything useful in the hook.""" start="00:15:40.760" video="mainVideo-test" id="subtitle"]] [[!template text="""Also, here you need to be careful""" start="00:15:43.120" video="mainVideo-test" id="subtitle"]] [[!template text="""to make sure the test handler is called and nothing else.""" start="00:15:45.080" video="mainVideo-test" id="subtitle"]] + +Side effects and initial buffer state + [[!template new="1" text="""So far we have been talking about testing""" start="00:15:55.720" video="mainVideo-test" id="subtitle"]] [[!template text="""and what the function returns.""" start="00:15:57.680" video="mainVideo-test" id="subtitle"]] [[!template text="""In the best of words, we have a pure function""" start="00:15:59.040" video="mainVideo-test" id="subtitle"]] @@ -274,16 +307,25 @@ [[!template text="""to depend on the leftovers from the other tests.""" start="00:16:48.480" video="mainVideo-test" id="subtitle"]] [[!template text="""Here are some functions and variables""" start="00:16:50.960" video="mainVideo-test" id="subtitle"]] [[!template text="""I have found useful for this.""" start="00:16:53.080" video="mainVideo-test" id="subtitle"]] + +with-temp-buffer + [[!template new="1" text="""For creating tests: `with-temp-buffer`:""" start="00:17:05.100" video="mainVideo-test" id="subtitle"]] [[!template text="""it provides you a temp buffer that you visit,""" start="00:17:09.200" video="mainVideo-test" id="subtitle"]] [[!template text="""and afterwards, there is no need to clean up.""" start="00:17:11.920" video="mainVideo-test" id="subtitle"]] [[!template text="""This is the first choice if that is all you need.""" start="00:17:13.720" video="mainVideo-test" id="subtitle"]] + +make-temp-file + [[!template new="1" text="""`make-temp-file`: If you need a file,""" start="00:17:16.520" video="mainVideo-test" id="subtitle"]] [[!template text="""this is the function to use.""" start="00:17:20.520" video="mainVideo-test" id="subtitle"]] [[!template text="""It creates a temp file or a directory.""" start="00:17:21.960" video="mainVideo-test" id="subtitle"]] [[!template text="""The file can be filled with initial contents.""" start="00:17:24.280" video="mainVideo-test" id="subtitle"]] [[!template text="""This needs to be cleaned up after a test.""" start="00:17:26.960" video="mainVideo-test" id="subtitle"]] [[!template text="""Moving on to verifying and debugging:""" start="00:17:31.020" video="mainVideo-test" id="subtitle"]] + +buffer-string + [[!template new="1" text="""`buffer-string`: returns the full contents""" start="00:17:33.288" video="mainVideo-test" id="subtitle"]] [[!template text="""of the buffer as a string.""" start="00:17:38.248" video="mainVideo-test" id="subtitle"]] [[!template text="""That can sound a bit voluminous,""" start="00:17:39.500" video="mainVideo-test" id="subtitle"]] @@ -297,6 +339,9 @@ [[!template text="""of debugging with print statements.""" start="00:18:01.200" video="mainVideo-test" id="subtitle"]] [[!template text="""There might be other ways to do the same""" start="00:18:04.400" video="mainVideo-test" id="subtitle"]] [[!template text="""as we saw with debugging.""" start="00:18:06.248" video="mainVideo-test" id="subtitle"]] + +buffer-name + [[!template new="1" text="""buffer-name: Getting the buffer name is good""" start="00:18:09.920" video="mainVideo-test" id="subtitle"]] [[!template text="""to verify what buffer we are looking at.""" start="00:18:13.720" video="mainVideo-test" id="subtitle"]] [[!template text="""I often found it useful to check""" start="00:18:16.240" video="mainVideo-test" id="subtitle"]] @@ -312,8 +357,14 @@ [[!template text="""some buffer and windows manipulation for its reporting""" start="00:18:42.240" video="mainVideo-test" id="subtitle"]] [[!template text="""that I have not fully learned how to master,""" start="00:18:44.880" video="mainVideo-test" id="subtitle"]] [[!template text="""so assertion for checking the sanity of the test is good.""" start="00:18:47.488" video="mainVideo-test" id="subtitle"]] + +major-mode + [[!template new="1" text="""Finally, `major-mode`: Verify the buffer has the proper mode.""" start="00:18:51.980" video="mainVideo-test" id="subtitle"]] [[!template text="""Can also be very useful and is a good sanity check.""" start="00:18:55.680" video="mainVideo-test" id="subtitle"]] + +unwind-protect + [[!template new="1" text="""Finally, cleaning up. `unwind-protect`.""" start="00:19:02.680" video="mainVideo-test" id="subtitle"]] [[!template text="""The tool for cleaning up is the `unwind-protect` form""" start="00:19:06.600" video="mainVideo-test" id="subtitle"]] [[!template text="""which ensures that the unwind forms""" start="00:19:09.040" video="mainVideo-test" id="subtitle"]] @@ -331,6 +382,9 @@ [[!template text="""for killing a buffer that is not saved.""" start="00:20:00.720" video="mainVideo-test" id="subtitle"]] [[!template text="""The test buffers are often in the state""" start="00:20:03.220" video="mainVideo-test" id="subtitle"]] [[!template text="""where they have not been saved but modified.""" start="00:20:05.440" video="mainVideo-test" id="subtitle"]] + +Input, with-simulated-input + [[!template new="1" text="""Another problem for tests are input.""" start="00:20:15.100" video="mainVideo-test" id="subtitle"]] [[!template text="""In the middle of execution a function""" start="00:20:19.680" video="mainVideo-test" id="subtitle"]] [[!template text="""might want to have some interaction with the user.""" start="00:20:21.560" video="mainVideo-test" id="subtitle"]] @@ -357,6 +411,9 @@ [[!template text="""and then return the characters read.""" start="00:21:21.720" video="mainVideo-test" id="subtitle"]] [[!template text="""As you see in the example, space needs to be provided""" start="00:21:26.120" video="mainVideo-test" id="subtitle"]] [[!template text="""by the string SPC, as return by the string RET.""" start="00:21:29.640" video="mainVideo-test" id="subtitle"]] + +Running all tests + [[!template new="1" text="""So now we have seen ways to create test cases""" start="00:21:38.460" video="mainVideo-test" id="subtitle"]] [[!template text="""and even make it possible to run some of them""" start="00:21:40.800" video="mainVideo-test" id="subtitle"]] [[!template text="""that has I/O in batch mode.""" start="00:21:43.220" video="mainVideo-test" id="subtitle"]] @@ -377,6 +434,9 @@ [[!template text="""So now, ert is executing all our test cases.""" start="00:22:35.460" video="mainVideo-test" id="subtitle"]] [[!template text="""So here we have a nice green display""" start="00:22:51.420" video="mainVideo-test" id="subtitle"]] [[!template text="""with all the test cases.""" start="00:22:57.080" video="mainVideo-test" id="subtitle"]] + +Batch mode + [[!template new="1" text="""So that was fine, but we were still running it manually""" start="00:23:03.220" video="mainVideo-test" id="subtitle"]] [[!template text="""by calling ert. How could we run it from the command line?""" start="00:23:08.160" video="mainVideo-test" id="subtitle"]] [[!template text="""Ert comes with functions for running it in batch mode.""" start="00:23:17.180" video="mainVideo-test" id="subtitle"]] @@ -393,6 +453,9 @@ [[!template text="""or the setting of `auto-save-default` to `nil`""" start="00:23:53.560" video="mainVideo-test" id="subtitle"]] [[!template text="""to get away with the prompt for excessive backup files""" start="00:23:58.160" video="mainVideo-test" id="subtitle"]] [[!template text="""that can pile up after running the tests a few times.""" start="00:24:02.440" video="mainVideo-test" id="subtitle"]] + +Skipping tests + [[!template new="1" text="""Even with the help of simulated input,""" start="00:24:05.060" video="mainVideo-test" id="subtitle"]] [[!template text="""not all tests can be run in batch mode.""" start="00:24:06.880" video="mainVideo-test" id="subtitle"]] [[!template text="""They would simply not work there""" start="00:24:08.920" video="mainVideo-test" id="subtitle"]] @@ -430,6 +493,9 @@ [[!template text="""you might not have a TTY for getting Emacs to start,""" start="00:25:54.560" video="mainVideo-test" id="subtitle"]] [[!template text="""and that is then another problem""" start="00:25:59.400" video="mainVideo-test" id="subtitle"]] [[!template text="""with getting the interactive mode.""" start="00:26:03.201" video="mainVideo-test" id="subtitle"]] + +Conclusion + [[!template new="1" text="""We have reached the end of the talk.""" start="00:26:08.460" video="mainVideo-test" id="subtitle"]] [[!template text="""If you have any new ideas""" start="00:26:11.121" video="mainVideo-test" id="subtitle"]] [[!template text="""or have some suggestions for improvements,""" start="00:26:14.160" video="mainVideo-test" id="subtitle"]] |