diff options
Diffstat (limited to '2023/captions/emacsconf-2023-gc--emacsgcstats-does-garbage-collection-actually-slow-down-emacs--ihor-radchenko--main.vtt')
| -rw-r--r-- | 2023/captions/emacsconf-2023-gc--emacsgcstats-does-garbage-collection-actually-slow-down-emacs--ihor-radchenko--main.vtt | 1780 |
1 files changed, 1780 insertions, 0 deletions
diff --git a/2023/captions/emacsconf-2023-gc--emacsgcstats-does-garbage-collection-actually-slow-down-emacs--ihor-radchenko--main.vtt b/2023/captions/emacsconf-2023-gc--emacsgcstats-does-garbage-collection-actually-slow-down-emacs--ihor-radchenko--main.vtt new file mode 100644 index 00000000..d1e2a5f6 --- /dev/null +++ b/2023/captions/emacsconf-2023-gc--emacsgcstats-does-garbage-collection-actually-slow-down-emacs--ihor-radchenko--main.vtt @@ -0,0 +1,1780 @@ +WEBVTT captioned by sachac + +NOTE Introduction + +00:00:00.000 --> 00:00:04.639 +Hello everyone, my name is Ihor Radchenko, + +00:00:04.640 --> 00:00:07.599 +and you may know me from Org Mailing List. + +00:00:07.600 --> 00:00:09.799 +However, today I'm not going to talk about Org Mode. + +00:00:09.800 --> 00:00:11.919 +Today I'm going to talk about + +00:00:11.920 --> 00:00:14.959 +Emacs performance and how it's affected + +00:00:14.960 --> 00:00:19.039 +by its memory management code. + +00:00:19.040 --> 00:00:21.639 +First, I will introduce the basic concepts + +00:00:21.640 --> 00:00:26.439 +of Emacs memory management and what garbage collection is. + +00:00:26.440 --> 00:00:30.559 +Then I will show you user statistics + +00:00:30.560 --> 00:00:34.959 +collected from volunteer users over the last half year + +00:00:34.960 --> 00:00:39.319 +and I will end with some guidelines + +00:00:39.320 --> 00:00:44.719 +on how to tweak Emacs garbage collection customizations + +00:00:44.720 --> 00:00:47.479 +to optimize Emacs performance + +00:00:47.480 --> 00:00:51.079 +and when it's necessary or not to do. + +NOTE About garbage collection in Emacs + +00:00:51.080 --> 00:00:54.519 +Let's begin. What is garbage collection? + +00:00:54.520 --> 00:00:56.519 +To understand what is garbage collection, + +00:00:56.520 --> 00:00:59.039 +we need to realize that anything you do in Emacs + +00:00:59.040 --> 00:01:02.119 +is some kind of command. Any command is most likely + +00:01:02.120 --> 00:01:05.839 +running some Elisp code. Every time you run Elisp code, + +00:01:05.840 --> 00:01:09.239 +you most likely need to locate certain memory in RAM. + +00:01:09.240 --> 00:01:12.879 +Some of this memory is retained for a long time + +00:01:12.880 --> 00:01:15.559 +and some of this memory is transient. + +00:01:15.560 --> 00:01:19.119 +Of course, Emacs has to clear this transient memory + +00:01:19.120 --> 00:01:21.439 +from time to time, to not occupy all the possible RAM + +00:01:21.440 --> 00:01:21.447 +in the computer. In this small example, + +00:01:21.448 --> 00:01:28.639 +we have one global variable + +00:01:28.640 --> 00:01:31.279 +that is assigned a value, + +00:01:31.280 --> 00:01:33.079 +but when assigning the value, + +00:01:33.080 --> 00:01:35.679 +we first allocate a temporary variable + +00:01:35.680 --> 00:01:37.119 +and then a temporary list + +00:01:37.120 --> 00:01:40.079 +and only retain some part of this list + +00:01:40.080 --> 00:01:42.079 +in this global variable. + +00:01:42.080 --> 00:01:44.799 +In terms of memory graph + +00:01:44.800 --> 00:01:50.359 +we can represent this as two variable slots, + +00:01:50.360 --> 00:01:53.159 +one transient, one permanent, + +00:01:53.160 --> 00:01:56.199 +and then a list of three cons cells, + +00:01:56.200 --> 00:02:01.959 +part of which is retained as a global variable + +00:02:01.960 --> 00:02:04.999 +but part of it which is a temporary variable symbol. + +00:02:05.000 --> 00:02:07.679 +The first term of the list is not used + +00:02:07.680 --> 00:02:09.759 +and it might be cleared at some point. + +NOTE Garbage collection in Emacs + +00:02:09.760 --> 00:02:12.239 +So that's what Emacs does. + +00:02:12.240 --> 00:02:15.919 +Every now and then, Emacs goes through all the memory + +00:02:15.920 --> 00:02:19.119 +and identifies which part of the memory are not used + +00:02:19.120 --> 00:02:23.759 +and then clear them so that it can free up the RAM. + +00:02:23.760 --> 00:02:25.919 +This process is called garbage collection + +00:02:25.920 --> 00:02:28.919 +and Emacs uses a very simple and old algorithm + +00:02:28.920 --> 00:02:30.559 +which is called Mark & Sweep. + +00:02:30.560 --> 00:02:33.759 +So doing this mark and sweep process + +00:02:33.760 --> 00:02:34.879 +is basically two stages. + +00:02:34.880 --> 00:02:40.039 +First, Emacs scans all the memory that is allocated + +00:02:40.040 --> 00:02:42.759 +and then identifies which memory is still in use + +00:02:42.760 --> 00:02:45.519 +which is linked to some variables, for example, + +00:02:45.520 --> 00:02:47.599 +and which memory is not used anymore + +00:02:47.600 --> 00:02:49.559 +even though it was allocated in the past. + +00:02:49.560 --> 00:02:52.999 +The second stage [??] whenever a memory is not, + +00:02:53.000 --> 00:02:59.319 +that is not allocated. During the process + +00:02:59.320 --> 00:03:00.759 +Emacs cannot do anything now. + +00:03:00.760 --> 00:03:04.159 +So basically, every time Emacs scans the memory, + +00:03:04.160 --> 00:03:07.199 +it freezes up and doesn't respond to anything, + +00:03:07.200 --> 00:03:10.959 +and if it takes too much time so that users can notice it, + +00:03:10.960 --> 00:03:13.399 +then of course Emacs is not responsive at all, + +00:03:13.400 --> 00:03:19.439 +and if this garbage collection is triggered too frequently, + +00:03:19.440 --> 00:03:22.399 +then it's not just not responsive every now and then. + +00:03:22.400 --> 00:03:24.679 +It's also not responsive all the time, + +00:03:24.680 --> 00:03:26.079 +almost all the time, + +00:03:26.080 --> 00:03:27.679 +so it cannot even normally type or stuff + +00:03:27.680 --> 00:03:32.439 +or do some normal commands. + +00:03:32.440 --> 00:03:36.719 +This mark and sweep algorithm is taking longer + +00:03:36.720 --> 00:03:40.199 +the more memory Emacs uses. So basically, + +00:03:40.200 --> 00:03:44.439 +the more buffers you open, the more packages you load, + +00:03:44.440 --> 00:03:48.319 +the more complex commands you run, the more memory is used, + +00:03:48.320 --> 00:03:52.279 +and basically, the longer Emacs takes + +00:03:52.280 --> 00:03:57.919 +to perform a single garbage collection. + +00:03:57.920 --> 00:04:02.279 +Of course, Emacs being Emacs + +00:04:02.280 --> 00:04:06.039 +this garbage collection can be tweaked. + +00:04:06.040 --> 00:04:08.279 +In particular users can tweak + +00:04:08.280 --> 00:04:10.639 +how frequently Emacs does garbage collection + +00:04:10.640 --> 00:04:13.879 +using two basic variables: `gc-cons-threshold` + +00:04:13.880 --> 00:04:15.519 +and `gc-cons-percentage`. + +00:04:15.520 --> 00:04:21.599 +`gc-cons-threshold` is the raw number of kilobytes + +00:04:21.600 --> 00:04:22.479 +Emacs needs to allocate + +00:04:22.480 --> 00:04:25.959 +before triggering another garbage collection, + +00:04:25.960 --> 00:04:27.799 +and the `gc-cons-percentage` is similar, + +00:04:27.800 --> 00:04:30.399 +but it's defined in terms of fraction + +00:04:30.400 --> 00:04:34.759 +of already-allocated memory. + +00:04:34.760 --> 00:04:38.239 +If you follow various Emacs forums, + +00:04:38.240 --> 00:04:41.959 +you may be familiar with people complaining about + +00:04:41.960 --> 00:04:46.479 +garbage collection. There are many many suggestions + +00:04:46.480 --> 00:04:48.039 +about what to do with it. + +00:04:48.040 --> 00:04:54.079 +Most frequently, you see `gc-cons-threshold` + +00:04:54.080 --> 00:04:56.879 +recommended to be increased, + +00:04:56.880 --> 00:05:01.439 +and a number of pre-packaged Emacs distributions + +00:05:01.440 --> 00:05:04.319 +like Doom Emacs do increase it. + +00:05:04.320 --> 00:05:07.279 +I have seen suggestions which are actually horrible + +00:05:07.280 --> 00:05:10.479 +to disable garbage collection temporarily + +00:05:10.480 --> 00:05:14.359 +or for a long time. + +00:05:14.360 --> 00:05:17.519 +Which is nice... You can see it quite frequently, + +00:05:17.520 --> 00:05:19.399 +which indicates there might be some problem. + +00:05:19.400 --> 00:05:23.959 +However, every time one user poses about this problem, + +00:05:23.960 --> 00:05:26.879 +it's just one data point and it doesn't mean + +00:05:26.880 --> 00:05:28.879 +that everyone actually suffers from it. + +00:05:28.880 --> 00:05:33.719 +It doesn't mean that everyone should do it. + +00:05:33.720 --> 00:05:35.919 +So in order to understand if this garbage collection + +00:05:35.920 --> 00:05:39.959 +is really a problem which is a common problem + +00:05:39.960 --> 00:05:44.919 +we do need some kind of statistics + +00:05:44.920 --> 00:05:46.919 +and only using the actual statistics + +00:05:46.920 --> 00:05:52.759 +we can understand if it should be recommended for everyone + +00:05:52.760 --> 00:05:54.999 +to tweak the defaults or like whether + +00:05:55.000 --> 00:05:57.159 +it should be recommended for certain users + +00:05:57.160 --> 00:05:59.479 +or maybe it should be asked Emacs devs + +00:05:59.480 --> 00:06:01.559 +to do something about the defaults. + +00:06:01.560 --> 00:06:07.959 +And what I did some time ago is exactly this. + +00:06:07.960 --> 00:06:09.959 +I tried to collect the user statistics. + +00:06:09.960 --> 00:06:14.519 +So I wrote a small package on Elp + +00:06:14.520 --> 00:06:18.159 +and some users installed this package + +00:06:18.160 --> 00:06:22.119 +and then reported back these statistics + +00:06:22.120 --> 00:06:24.279 +of the garbage collection for their particular use. + +00:06:24.280 --> 00:06:30.799 +By now we have obtained 129 user submissions + +00:06:30.800 --> 00:06:34.039 +with over 1 million GC records in there. + +00:06:34.040 --> 00:06:38.119 +So like some of these submissions + +00:06:38.120 --> 00:06:43.159 +used default GC settings without any customizations. + +00:06:43.160 --> 00:06:46.039 +Some used increased GC cost threshold + +00:06:46.040 --> 00:06:47.799 +and GC cost percentage. + +00:06:47.800 --> 00:06:53.319 +So using this data we can try to draw + +00:06:53.320 --> 00:06:56.879 +some reliable conclusions on what should be done + +00:06:56.880 --> 00:06:59.919 +and whether should anything be done about garbage collection + +00:06:59.920 --> 00:07:02.639 +on Emacs dev level or at least on user level. + +00:07:02.640 --> 00:07:05.639 +Of course we need to keep in mind + +00:07:05.640 --> 00:07:07.279 +that there's some kind of bias + +00:07:07.280 --> 00:07:08.999 +because it's more likely + +00:07:09.000 --> 00:07:11.719 +that users already have problems with GC + +00:07:11.720 --> 00:07:13.239 +or they think they have problems with GC + +00:07:13.240 --> 00:07:15.919 +will report and submit the data. + +00:07:15.920 --> 00:07:19.999 +But anyway having s statistics is much more useful + +00:07:20.000 --> 00:07:22.079 +than just having anecdotal evidences + +00:07:22.080 --> 00:07:25.519 +from one or other reddit posts. + +00:07:25.520 --> 00:07:28.759 +And just one thing I will do + +00:07:28.760 --> 00:07:30.879 +during the rest of my presentation + +00:07:30.880 --> 00:07:32.839 +is that for all the statistics + +00:07:32.840 --> 00:07:35.679 +I will normalize user data + +00:07:35.680 --> 00:07:37.879 +so that every user contributes equally. + +00:07:37.880 --> 00:07:40.279 +For example if one user submits like + +00:07:40.280 --> 00:07:43.119 +100 hours Emacs uptime statistics + +00:07:43.120 --> 00:07:46.279 +and other users submit one hour Emacs uptime + +00:07:46.280 --> 00:07:52.879 +then I will anyway make it so that they contribute equally. + +00:07:52.880 --> 00:07:56.359 +Let's start from one of the most obvious things + +00:07:56.360 --> 00:07:57.679 +we can look into is + +00:07:57.680 --> 00:08:00.599 +which is the time it takes for garbage collection + +00:08:00.600 --> 00:08:05.879 +to single garbage collection process. + +00:08:05.880 --> 00:08:11.839 +Here you see frequency distribution of GC duration + +00:08:11.840 --> 00:08:14.999 +for all the 129 users we got + +00:08:15.000 --> 00:08:22.279 +and you can see that most of the garbage collections + +00:08:22.280 --> 00:08:26.999 +are done quite quickly in less than 0.1 second + +00:08:27.000 --> 00:08:32.199 +and less than 0.1 second is usually just not noticeable. + +00:08:32.200 --> 00:08:34.519 +So even though there is garbage collection + +00:08:34.520 --> 00:08:39.639 +it will not interrupt the work in Emacs. + +00:08:39.640 --> 00:08:43.279 +However there is a fraction of users + +00:08:43.280 --> 00:08:45.279 +who experience garbage collection + +00:08:45.280 --> 00:08:48.399 +it takes like 0.2, 0.3 or even half a second + +00:08:48.400 --> 00:08:50.399 +which will be quite noticeable. + +00:08:50.400 --> 00:08:55.279 +For the purposes of this study + +00:08:55.280 --> 00:08:59.399 +I will consider that anything that is less than 0.1 second + +00:08:59.400 --> 00:09:02.639 +which is insignificant so like you will not notice it + +00:09:02.640 --> 00:09:04.159 +and it's like obviously + +00:09:04.160 --> 00:09:07.479 +all the Emacs usage will be just normal. + +00:09:07.480 --> 00:09:11.639 +But if it's more than 0.1 or 0.2 seconds + +00:09:11.640 --> 00:09:13.799 +then it will be very noticeable + +00:09:13.800 --> 00:09:16.079 +and you will see that Emacs hang for a little while + +00:09:16.080 --> 00:09:21.319 +or not so little while. In terms of numbers + +00:09:21.320 --> 00:09:26.239 +it's better to plot the statistics not as a distribution + +00:09:26.240 --> 00:09:28.199 +but as a cumulative distribution. + +00:09:28.200 --> 00:09:31.559 +So like at every point of this graph + +00:09:31.560 --> 00:09:37.159 +you'll see like for example here 0.4 seconds + +00:09:37.160 --> 00:09:42.279 +you have this percent of like almost 90% of users + +00:09:42.280 --> 00:09:49.279 +have no more than 0.4 gc duration. + +00:09:49.280 --> 00:09:53.239 +So like we can look here if we take one + +00:09:53.240 --> 00:09:56.879 +gc critical gc duration which is 0.1 second + +00:09:56.880 --> 00:10:00.279 +0.1 second and look at how many users have + +00:10:00.280 --> 00:10:02.439 +it so we have 56% which is like + +00:10:02.440 --> 00:10:09.439 +44% users have less than 0.1 second gc duration + +00:10:09.440 --> 00:10:12.839 +and the rest 56% have more than 0.1 second. + +00:10:12.840 --> 00:10:16.279 +So you can see like more than half of users + +00:10:16.280 --> 00:10:20.559 +actually have noticeable gc delay + +00:10:20.560 --> 00:10:22.999 +so the Emacs freezes for some noticeable time + +00:10:23.000 --> 00:10:27.479 +and a quarter of users actually have very noticeable + +00:10:27.480 --> 00:10:31.799 +so like Emacs freezes such that you see an actual delay + +00:10:31.800 --> 00:10:36.879 +that Emacs actually has + +00:10:36.880 --> 00:10:44.079 +which is quite significant and important point. + +00:10:44.080 --> 00:10:47.719 +But apart from the duration of each individual gc + +00:10:47.720 --> 00:10:49.839 +it is important to see how frequent it is + +00:10:49.840 --> 00:10:52.879 +because even if you do notice a delay + +00:10:52.880 --> 00:10:54.959 +even a few seconds delay + +00:10:54.960 --> 00:10:56.999 +it doesn't matter if it happens once + +00:10:57.000 --> 00:10:59.199 +during the whole Emacs session. + +00:10:59.200 --> 00:11:05.039 +So if you look into frequency distribution again here + +00:11:05.040 --> 00:11:13.639 +I plot time between subsequent garbage collections + +00:11:13.640 --> 00:11:17.959 +versus how frequent it is and we have very clear trend + +00:11:17.960 --> 00:11:21.799 +that most of the garbage collections are quite frequent + +00:11:21.800 --> 00:11:25.159 +like we talk about every few seconds a few tens of seconds. + +00:11:25.160 --> 00:11:30.039 +There's a few outliers which are at very round numbers + +00:11:30.040 --> 00:11:35.839 +like 60 seconds, 120 seconds, 300 seconds. + +00:11:35.840 --> 00:11:37.879 +These are usually timers so like + +00:11:37.880 --> 00:11:40.319 +you have something running on timer + +00:11:40.320 --> 00:11:43.599 +and then it is complex command + +00:11:43.600 --> 00:11:45.079 +and it triggers garbage collection + +00:11:45.080 --> 00:11:48.079 +but it's not the majority. + +00:11:48.080 --> 00:11:51.279 +Again to run the numbers + +00:11:51.280 --> 00:11:53.679 +it's better to look into cumulative distribution + +00:11:53.680 --> 00:11:56.039 +and see that 50% of garbage collections + +00:11:56.040 --> 00:11:58.279 +are basically less than 10 seconds apart. + +00:11:58.280 --> 00:12:02.359 +And we can combine it with previous data + +00:12:02.360 --> 00:12:07.479 +and we look into whatever garbage collection + +00:12:07.480 --> 00:12:09.959 +takes less than 10 seconds from each other + +00:12:09.960 --> 00:12:13.119 +and also takes more than say 0.1 seconds. + +00:12:13.120 --> 00:12:15.319 +So and then we see that + +00:12:15.320 --> 00:12:17.639 +one quarter of all garbage collections + +00:12:17.640 --> 00:12:21.039 +are just noticeable and also frequent + +00:12:21.040 --> 00:12:23.679 +and 9% are not like + +00:12:23.680 --> 00:12:27.199 +more than 0.2% very noticeable and also frequent. + +00:12:27.200 --> 00:12:30.079 +So basically it constitutes Emacs freezing. + +00:12:30.080 --> 00:12:33.559 +So 9% of all the garbage collection Emacs freezing. + +00:12:33.560 --> 00:12:37.319 +Of course if you remember there is a bias + +00:12:37.320 --> 00:12:40.519 +but 9% is quite significant number. + +00:12:40.520 --> 00:12:44.319 +So garbage collection can really slow down things + +00:12:44.320 --> 00:12:48.239 +not for everyone but for significant fraction of users. + +00:12:48.240 --> 00:12:52.159 +Another thing I'd like to look into + +00:12:52.160 --> 00:12:55.399 +is what I call agglomerated GCs. + +00:12:55.400 --> 00:12:57.959 +What I mean by agglomerated is + +00:12:57.960 --> 00:13:00.359 +when you have one garbage collection + +00:13:00.360 --> 00:13:02.999 +and then another garbage immediately after it. + +00:13:03.000 --> 00:13:05.559 +So in terms of numbers I took + +00:13:05.560 --> 00:13:08.719 +every subsequent garbage collection + +00:13:08.720 --> 00:13:10.399 +which is either immediately after + +00:13:10.400 --> 00:13:13.039 +or no more than one second after each. + +00:13:13.040 --> 00:13:16.159 +So from point of view of users is like + +00:13:16.160 --> 00:13:19.999 +multiple garbage collection they add up together + +00:13:20.000 --> 00:13:22.999 +into one giant garbage collection. + +00:13:23.000 --> 00:13:25.839 +And if you look into numbers + +00:13:25.840 --> 00:13:29.559 +of how many agglomerated garbage collections there are + +00:13:29.560 --> 00:13:32.119 +you can see even numbers over 100. + +00:13:32.120 --> 00:13:35.479 +So 100 garbage collection going one after another. + +00:13:35.480 --> 00:13:39.159 +Even if you think about each garbage collection + +00:13:39.160 --> 00:13:42.719 +taking 0.1 second we look into 100 of them + +00:13:42.720 --> 00:13:44.639 +it's total 10 seconds. + +00:13:44.640 --> 00:13:46.839 +It's like Emacs hanging forever + +00:13:46.840 --> 00:13:53.519 +or like a significant number is also 10. + +00:13:53.520 --> 00:13:55.999 +So again this would be very annoying to meet such thing. + +00:13:56.000 --> 00:13:57.879 +How frequently does it happen? + +00:13:57.880 --> 00:14:00.279 +Again we can plot cumulative distribution + +00:14:00.280 --> 00:14:03.879 +and we see that 20 percent like 19 percent + +00:14:03.880 --> 00:14:07.199 +of all the garbage collection are at least two together + +00:14:07.200 --> 00:14:13.679 +and 8 percent like more than 10. So like you think about oh + +00:14:13.680 --> 00:14:15.639 +each garbage collection is not taking much time + +00:14:15.640 --> 00:14:24.479 +but when you have 10 of them yeah that becomes a problem. + +00:14:24.480 --> 00:14:29.919 +Another thing is to answer a question + +00:14:29.920 --> 00:14:32.959 +that some people complain about is that + +00:14:32.960 --> 00:14:35.799 +longer you use Emacs the slower Emacs become. + +00:14:35.800 --> 00:14:43.039 +Of course it may be caused by garbage collection + +00:14:43.040 --> 00:14:48.519 +and I wanted to look into how garbage collection time + +00:14:48.520 --> 00:14:49.679 +and other statistics, + +00:14:49.680 --> 00:14:53.199 +other parameters are evolving over time. + +00:14:53.200 --> 00:14:58.559 +And what I can see here is a cumulative distribution + +00:14:58.560 --> 00:15:03.719 +of GC duration for like first 10 minutes of Emacs uptime + +00:15:03.720 --> 00:15:06.479 +first 100 minutes first 1000 minutes. + +00:15:06.480 --> 00:15:10.199 +And if you look closer then you see + +00:15:10.200 --> 00:15:14.519 +that each individual garbage collection on average + +00:15:14.520 --> 00:15:18.959 +takes longer as you use Emacs longer. + +00:15:18.960 --> 00:15:24.039 +However this longer is not much it's like maybe 10 percent + +00:15:24.040 --> 00:15:29.479 +like basically garbage collection gets like + +00:15:29.480 --> 00:15:34.719 +slow Emacs down more as you use Emacs more but not much. + +00:15:34.720 --> 00:15:38.359 +So basically if you do you see Emacs + +00:15:38.360 --> 00:15:40.639 +being slower and slower over time + +00:15:40.640 --> 00:15:43.159 +it's probably not really garbage collection + +00:15:43.160 --> 00:15:45.839 +because it doesn't change too much. + +00:15:45.840 --> 00:15:48.119 +And if you look into time + +00:15:48.120 --> 00:15:50.839 +between individual garbage collections + +00:15:50.840 --> 00:15:53.719 +and you see that the time actually increases + +00:15:53.720 --> 00:15:56.719 +as you use Emacs longer which makes sense + +00:15:56.720 --> 00:15:58.839 +because initially like first few minutes + +00:15:58.840 --> 00:16:01.479 +you have all kind of packages loading + +00:16:01.480 --> 00:16:04.239 +like all the port loading and then later + +00:16:04.240 --> 00:16:07.239 +everything is loaded and things become more stable. + +00:16:07.240 --> 00:16:12.879 +So the conclusion on this part is that + +00:16:12.880 --> 00:16:16.399 +if Emacs becomes slower in a long session + +00:16:16.400 --> 00:16:18.479 +it's probably not caused by garbage collection. + +00:16:18.480 --> 00:16:23.679 +And one word of warning of course is that + +00:16:23.680 --> 00:16:27.919 +it's all nice and all when I present the statistics + +00:16:27.920 --> 00:16:29.279 +but it's only an average + +00:16:29.280 --> 00:16:34.079 +and if you are an actual user like here is one example + +00:16:34.080 --> 00:16:37.159 +which shows a total garbage collection time + +00:16:37.160 --> 00:16:40.119 +like accumulated together over Emacs uptime + +00:16:40.120 --> 00:16:43.199 +and you see different lines + +00:16:43.200 --> 00:16:45.559 +which correspond to different sessions of one user + +00:16:45.560 --> 00:16:48.679 +and you see they are wildly different + +00:16:48.680 --> 00:16:51.439 +like one time there is almost no garbage collection + +00:16:51.440 --> 00:16:54.679 +another time you see garbage collection + +00:16:54.680 --> 00:16:56.999 +because probably Emacs is used more early + +00:16:57.000 --> 00:16:59.599 +or like different pattern of usage + +00:16:59.600 --> 00:17:03.159 +and even during a single Emacs session + +00:17:03.160 --> 00:17:04.599 +you see a different slope + +00:17:04.600 --> 00:17:06.439 +of this curve which means that + +00:17:06.440 --> 00:17:09.279 +sometimes garbage collection is infrequent + +00:17:09.280 --> 00:17:11.479 +and sometimes it's much more frequent + +00:17:11.480 --> 00:17:14.479 +so it's probably much more noticeable one time + +00:17:14.480 --> 00:17:15.639 +and less noticeable other time. + +00:17:15.640 --> 00:17:19.719 +So if you think about these statistics of course + +00:17:19.720 --> 00:17:23.359 +they only represent an average usage + +00:17:23.360 --> 00:17:26.359 +but sometimes it can get worse sometimes it can get better. + +00:17:26.360 --> 00:17:33.759 +The last parameter I'd like to talk about is + +00:17:33.760 --> 00:17:35.799 +garbage collection during Emacs init. + +00:17:35.800 --> 00:17:40.439 +Basically if you think about what happens during Emacs init + +00:17:40.440 --> 00:17:41.919 +like when Emacs just starting up + +00:17:41.920 --> 00:17:44.479 +then whatever garbage collection + +00:17:44.480 --> 00:17:46.759 +there it's one or it's several times + +00:17:46.760 --> 00:17:51.239 +it all contributes to Emacs taking longer to start. + +00:17:51.240 --> 00:17:56.559 +And again we can look into the statistic + +00:17:56.560 --> 00:18:01.159 +and see what is the total GC duration after Emacs init + +00:18:01.160 --> 00:18:06.159 +and we see that 50% of all the submissions + +00:18:06.160 --> 00:18:10.279 +garbage collection adds up more than one second + +00:18:10.280 --> 00:18:14.919 +to Emacs init time and for 20% of users + +00:18:14.920 --> 00:18:17.079 +it's extra three seconds Emacs start time + +00:18:17.080 --> 00:18:18.479 +which is very significant + +00:18:18.480 --> 00:18:21.479 +especially for people who are used to Vim + +00:18:21.480 --> 00:18:23.919 +which can start in like a fraction of a second + +00:18:23.920 --> 00:18:26.239 +and here it just does garbage collection + +00:18:26.240 --> 00:18:27.439 +because garbage collection is not + +00:18:27.440 --> 00:18:29.239 +everything Emacs does during startup + +00:18:29.240 --> 00:18:31.999 +adds up more to the load. + +00:18:32.000 --> 00:18:36.119 +Okay that's all nice and all + +00:18:36.120 --> 00:18:38.679 +but what can we do about these statistics + +00:18:38.680 --> 00:18:40.159 +can we draw any conclusions + +00:18:40.160 --> 00:18:43.239 +and the answer is of course + +00:18:43.240 --> 00:18:46.079 +like the most important conclusion here is that + +00:18:46.080 --> 00:18:49.439 +yes garbage collection can slow down Emacs + +00:18:49.440 --> 00:18:52.679 +at least for some people and what to do about it + +00:18:52.680 --> 00:18:55.319 +there are two variables which you can tweak + +00:18:55.320 --> 00:18:58.719 +it's because gcconce threshold gcconce percentage + +00:18:58.720 --> 00:19:03.159 +and having the statistics I can at least look a little bit + +00:19:03.160 --> 00:19:08.879 +into what is the effect of increasing these variables + +00:19:08.880 --> 00:19:12.439 +like most people just increase gcconce threshold + +00:19:12.440 --> 00:19:16.959 +and like all the submissions people did increase + +00:19:16.960 --> 00:19:19.919 +and doesn't make much sense to decrease it + +00:19:19.920 --> 00:19:21.079 +like to make things worse + +00:19:21.080 --> 00:19:27.639 +of course for these statistics + +00:19:27.640 --> 00:19:31.559 +the exact values of this increased thresholds + +00:19:31.560 --> 00:19:33.839 +are not always the same + +00:19:33.840 --> 00:19:36.479 +but at least we can look into some trends + +00:19:36.480 --> 00:19:44.759 +so first and obvious thing we can observe + +00:19:44.760 --> 00:19:46.759 +is when we compare + +00:19:46.760 --> 00:19:50.399 +the standard gc settings standard thresholds + +00:19:50.400 --> 00:19:53.999 +and increased thresholds for time between + +00:19:54.000 --> 00:19:57.479 +subsequent gcs and as one may expect + +00:19:57.480 --> 00:19:59.559 +if you increase the threshold + +00:19:59.560 --> 00:20:02.679 +Emacs will do garbage collection less frequently + +00:20:02.680 --> 00:20:05.279 +so the spacing between garbage collection increases + +00:20:05.280 --> 00:20:07.599 +okay the only thing is that + +00:20:07.600 --> 00:20:10.719 +if garbage collection is less frequent + +00:20:10.720 --> 00:20:14.079 +then each individual garbage collection becomes longer + +00:20:14.080 --> 00:20:18.159 +so if you think about increasing + +00:20:18.160 --> 00:20:22.239 +garbage collection thresholds be prepared + +00:20:22.240 --> 00:20:26.519 +that in each individual time Emacs freezes will take longer + +00:20:26.520 --> 00:20:31.599 +this is one caveat when we talk about + +00:20:31.600 --> 00:20:34.079 +this agglomerated gcs which are one after other + +00:20:34.080 --> 00:20:36.759 +like if you increase the threshold sufficiently + +00:20:36.760 --> 00:20:42.319 +then whatever happened that garbage collections + +00:20:42.320 --> 00:20:44.399 +were like done one after other + +00:20:44.400 --> 00:20:47.599 +we can now make it so that they are actually separated + +00:20:47.600 --> 00:20:51.559 +so like you don't see one giant freeze caused by + +00:20:51.560 --> 00:20:52.919 +like 10 gcs in a row + +00:20:52.920 --> 00:20:55.759 +instead you can make it so that they are separated + +00:20:55.760 --> 00:20:59.079 +and in statistics it's very clear + +00:20:59.080 --> 00:21:02.959 +that the number of agglomerated garbage collections + +00:21:02.960 --> 00:21:06.919 +decreases dramatically when you increase the thresholds + +00:21:06.920 --> 00:21:11.759 +it's particularly evident when we look into startup time + +00:21:11.760 --> 00:21:17.279 +if you look at gc duration during Emacs startup + +00:21:17.280 --> 00:21:19.159 +and if we look into what happens + +00:21:19.160 --> 00:21:20.879 +when you increase the thresholds + +00:21:20.880 --> 00:21:23.799 +it's very clear that Emacs startup become faster + +00:21:23.800 --> 00:21:26.279 +when you increase gc thresholds + +00:21:26.280 --> 00:21:33.359 +so that's all for actual user statistics + +00:21:33.360 --> 00:21:35.439 +and now let's try to run into + +00:21:35.440 --> 00:21:38.079 +some like actual recommendations + +00:21:38.080 --> 00:21:42.639 +on what numbers to set and before we start + +00:21:42.640 --> 00:21:44.399 +let me explain a little bit about + +00:21:44.400 --> 00:21:46.479 +the difference between these two variables + +00:21:46.480 --> 00:21:48.879 +which is gc constant threshold and gc constant percentage + +00:21:48.880 --> 00:21:52.359 +so if you think about Emacs memory + +00:21:52.360 --> 00:21:55.239 +like there's a certain memory allocated by Emacs + +00:21:55.240 --> 00:21:58.479 +and then as you run commands and turn using Emacs + +00:21:58.480 --> 00:22:00.079 +there is more memory allocated + +00:22:00.080 --> 00:22:04.639 +and Emacs decides when to do garbage collection + +00:22:04.640 --> 00:22:06.079 +according these two variables + +00:22:06.080 --> 00:22:08.759 +and actually what it does it chooses the larger one + +00:22:08.760 --> 00:22:12.119 +so say you have you are late in Emacs session + +00:22:12.120 --> 00:22:14.039 +you have a lot of Emacs memory allocated + +00:22:14.040 --> 00:22:17.119 +then you have gc constant percentage + +00:22:17.120 --> 00:22:19.919 +which is percent of the already allocated memory + +00:22:19.920 --> 00:22:25.119 +and that percent is probably going to be the largest + +00:22:25.120 --> 00:22:28.319 +because you have more memory + +00:22:28.320 --> 00:22:32.559 +and memory means that percent of it is larger + +00:22:32.560 --> 00:22:36.359 +so like you have a larger number cost + +00:22:36.360 --> 00:22:37.719 +by gc constant percentage + +00:22:37.720 --> 00:22:43.079 +so in this scenario when Emacs session is already running + +00:22:43.080 --> 00:22:45.319 +for a long time and there is a lot of memory allocated + +00:22:45.320 --> 00:22:50.119 +you have gc constant percentage + +00:22:50.120 --> 00:22:52.279 +controlling the garbage collection + +00:22:52.280 --> 00:22:54.999 +while early in Emacs there is not much memory placed + +00:22:55.000 --> 00:22:58.719 +Emacs just starting up then gc constant threshold + +00:22:58.720 --> 00:23:01.639 +is controlling how frequently garbage collection happens + +00:23:01.640 --> 00:23:04.799 +because smaller allocated memory + +00:23:04.800 --> 00:23:06.839 +means its percentage will be a small number + +00:23:06.840 --> 00:23:12.319 +so in terms of default values at least + +00:23:12.320 --> 00:23:14.239 +gc constant threshold is 800 kilobytes + +00:23:14.240 --> 00:23:18.799 +and gc constant percentage is 10 + +00:23:18.800 --> 00:23:24.159 +so gc constant percentage becomes larger than that threshold + +00:23:24.160 --> 00:23:28.919 +when you have more than eight megabytes of allocated memory + +00:23:28.920 --> 00:23:31.039 +by Emacs which is quite early + +00:23:31.040 --> 00:23:34.279 +and it will probably hold just during the startup + +00:23:34.280 --> 00:23:36.799 +and once you start using your maximum + +00:23:36.800 --> 00:23:38.919 +and once you load all the histories + +00:23:38.920 --> 00:23:42.039 +all the kinds of buffers it's probably going to take + +00:23:42.040 --> 00:23:43.959 +more than much more than eight megabytes + +00:23:43.960 --> 00:23:50.639 +so now we understand this + +00:23:50.640 --> 00:23:53.279 +we can draw certain recommendations + +00:23:53.280 --> 00:23:57.279 +about tweaking the gc thresholds + +00:23:57.280 --> 00:24:01.159 +so first of all I need to emphasize + +00:24:01.160 --> 00:24:03.639 +that any time you increase gc threshold + +00:24:03.640 --> 00:24:07.199 +an individual garbage collection time increases + +00:24:07.200 --> 00:24:08.759 +so it's not free at all + +00:24:08.760 --> 00:24:10.999 +if you don't have problems with garbage collection + +00:24:11.000 --> 00:24:13.519 +which is half of the users don't have much problem + +00:24:13.520 --> 00:24:15.079 +you don't need to tweak anything + +00:24:15.080 --> 00:24:19.359 +only when gc is frequent and slow + +00:24:19.360 --> 00:24:23.399 +when Emacs is really really present frequently + +00:24:23.400 --> 00:24:27.119 +you may consider increasing gc thresholds only + +00:24:27.120 --> 00:24:31.479 +and in particular I recommend + +00:24:31.480 --> 00:24:33.279 +increasing gc constant percentage + +00:24:33.280 --> 00:24:36.359 +because that's what mostly controls gc + +00:24:36.360 --> 00:24:40.079 +when Emacs is running for long session + +00:24:40.080 --> 00:24:43.039 +and the numbers are probably like + +00:24:43.040 --> 00:24:46.519 +yeah we can estimate the effect of these numbers + +00:24:46.520 --> 00:24:49.679 +like for example if you have a default value of 0.1 percent + +00:24:49.680 --> 00:24:52.759 +for gc constant percentage 0.1 which is 10 percent + +00:24:52.760 --> 00:24:55.039 +and then increase it twice + +00:24:55.040 --> 00:24:58.639 +obviously you get twice less frequent gcs + +00:24:58.640 --> 00:25:02.559 +but it will come at the cost of extra 10 percent gc time + +00:25:02.560 --> 00:25:05.839 +and if you increase 10 times you can think about + +00:25:05.840 --> 00:25:08.719 +10 less 10 x less frequent gcs + +00:25:08.720 --> 00:25:12.199 +but almost twice longer individual garbage collection time + +00:25:12.200 --> 00:25:16.919 +so probably you want to set the number closer to 0.1 + +00:25:16.920 --> 00:25:23.399 +another part of the users may actually + +00:25:23.400 --> 00:25:28.359 +try to optimize Emacs startup time + +00:25:28.360 --> 00:25:30.759 +which is quite frequent problem + +00:25:30.760 --> 00:25:34.919 +in this case it's probably better to increase gc constant + +00:25:34.920 --> 00:25:38.199 +but not too much so like + +00:25:38.200 --> 00:25:40.239 +first of all it makes sense to check + +00:25:40.240 --> 00:25:43.319 +whether garbage collection is a problem at all + +00:25:43.320 --> 00:25:45.999 +during startup and there are two variables + +00:25:46.000 --> 00:25:50.199 +which can show what is happening this garbage collection + +00:25:50.200 --> 00:25:53.719 +so gc done is a variable that shows + +00:25:53.720 --> 00:25:55.039 +how many garbage collection + +00:25:55.040 --> 00:26:00.159 +like what is the number of garbage collections triggered + +00:26:00.160 --> 00:26:02.599 +like when you check the value + +00:26:02.600 --> 00:26:04.039 +or right after you start Emacs + +00:26:04.040 --> 00:26:04.799 +you will see that + +00:26:04.800 --> 00:26:08.519 +number and gc elapsed variable + +00:26:08.520 --> 00:26:11.599 +which gives you a number of seconds + +00:26:11.600 --> 00:26:14.959 +which Emacs spent in doing garbage collection + +00:26:14.960 --> 00:26:16.879 +so this is probably the most important variable + +00:26:16.880 --> 00:26:20.719 +and if you see it's large then you may consider tweaking it + +00:26:20.720 --> 00:26:26.799 +for the Emacs startup we can estimate some bounds + +00:26:26.800 --> 00:26:30.039 +because in the statistics I never saw anything + +00:26:30.040 --> 00:26:32.439 +that is more than 10 seconds extra + +00:26:32.440 --> 00:26:34.439 +which even 10 seconds is probably like + +00:26:34.440 --> 00:26:39.119 +a really really hard upper bound so + +00:26:39.120 --> 00:26:44.479 +or say if you want to decrease the gc contribution + +00:26:44.480 --> 00:26:47.479 +like order of magnitude or like two orders of magnitudes + +00:26:47.480 --> 00:26:50.879 +let's say like as a really hard top estimate + +00:26:50.880 --> 00:26:55.079 +then it corresponds to 80 megabytes gc constant + +00:26:55.080 --> 00:26:58.959 +and probably much less so like + +00:26:58.960 --> 00:27:00.679 +there's no point setting it + +00:27:00.680 --> 00:27:04.159 +to a few hundred megabytes of course + +00:27:04.160 --> 00:27:08.439 +there's one caveat which is important to keep in + +00:27:08.440 --> 00:27:14.039 +mind though that increasing the gc thresholds + +00:27:14.040 --> 00:27:16.399 +is not just increasing individual gc time + +00:27:16.400 --> 00:27:20.399 +there's also an actual real impact on the RAM usage + +00:27:20.400 --> 00:27:23.839 +so like if you increase gc threshold + +00:27:23.840 --> 00:27:26.879 +it increases the RAM usage of Emacs + +00:27:26.880 --> 00:27:29.639 +and you shouldn't think that like okay + +00:27:29.640 --> 00:27:33.159 +I increased the threshold by like 100 megabytes + +00:27:33.160 --> 00:27:37.119 +then 100 megabytes extra RAM usage doesn't matter + +00:27:37.120 --> 00:27:38.679 +it's not 100 megabytes + +00:27:38.680 --> 00:27:42.319 +because less frequent garbage collection means + +00:27:42.320 --> 00:27:45.639 +it will lead to memory fragmentation + +00:27:45.640 --> 00:27:50.439 +so in practice if you increase the thresholds + +00:27:50.440 --> 00:27:52.799 +to tens or hundreds of megabytes + +00:27:52.800 --> 00:27:55.919 +we are talking about gigabytes extra RAM usage + +00:27:55.920 --> 00:27:59.719 +for me personally when I tried to play with gc thresholds + +00:27:59.720 --> 00:28:02.879 +I have seen Emacs taking two gigabytes like + +00:28:02.880 --> 00:28:05.519 +compared to several times less + +00:28:05.520 --> 00:28:09.039 +when with default settings so it's not free at all + +00:28:09.040 --> 00:28:13.639 +and only like either when you have a lot of free RAM + +00:28:13.640 --> 00:28:16.839 +and you don't care or when your Emacs is really slow + +00:28:16.840 --> 00:28:19.559 +then you may need to consider this + +00:28:19.560 --> 00:28:23.239 +tweaking these defaults so again don't tweak defaults + +00:28:23.240 --> 00:28:24.239 +if you don't really have a problem + +00:28:24.240 --> 00:28:29.839 +and of course this RAM problem is a big big deal + +00:28:29.840 --> 00:28:35.679 +for Emacs devs because from from the point of single user + +00:28:35.680 --> 00:28:38.839 +you have like normal laptop most likely like normal PC + +00:28:38.840 --> 00:28:42.079 +with a lot of RAM you don't care about these things too much + +00:28:42.080 --> 00:28:48.999 +but Emacs in general can run on like all kinds of machines + +00:28:49.000 --> 00:28:51.679 +including low-end machines with very limited RAM + +00:28:51.680 --> 00:28:55.359 +and anytime Emacs developers consider increasing + +00:28:55.360 --> 00:28:57.959 +the defaults for garbage collection + +00:28:57.960 --> 00:29:01.479 +it's like they always have to consider + +00:29:01.480 --> 00:29:02.959 +if you increase them too much + +00:29:02.960 --> 00:29:07.919 +then Emacs may just stop running on certain platforms + +00:29:07.920 --> 00:29:14.439 +so that's a very big consideration in terms + +00:29:14.440 --> 00:29:16.639 +of the global defaults for everyone + +00:29:16.640 --> 00:29:22.199 +although I have to I would say that it might be related + +00:29:22.200 --> 00:29:24.479 +to the safe to increase GCCons threshold + +00:29:24.480 --> 00:29:27.919 +because it mostly affects startup and during startup + +00:29:27.920 --> 00:29:31.279 +it's probably not the peak usage of Emacs + +00:29:31.280 --> 00:29:35.599 +and like as Emacs runs for longer + +00:29:35.600 --> 00:29:38.199 +it's probably where most of RAM will be used later + +00:29:38.200 --> 00:29:44.399 +on the other hand GCCons percentage is much more debating + +00:29:44.400 --> 00:29:46.159 +because it has pros and cons + +00:29:46.160 --> 00:29:47.719 +it will increase the RAM usage + +00:29:47.720 --> 00:29:50.999 +it will increase the individual GC time so + +00:29:51.000 --> 00:29:56.119 +if we consider changing it it's much more tricky + +00:29:56.120 --> 00:29:59.479 +and we have discussing probably measure the impact on users + +00:29:59.480 --> 00:30:05.799 +and a final note on or from the point of view + +00:30:05.800 --> 00:30:07.319 +of Emacs development is + +00:30:07.320 --> 00:30:11.039 +that this simple mark-and-sweep algorithm + +00:30:11.040 --> 00:30:14.119 +is like a very old and not the state-of-the-art algorithm + +00:30:14.120 --> 00:30:17.799 +there are variants of garbage collection + +00:30:17.800 --> 00:30:19.479 +that are like totally non-blocking + +00:30:19.480 --> 00:30:22.479 +so Emacs just doesn't have to freeze + +00:30:22.480 --> 00:30:24.279 +during the garbage collection + +00:30:24.280 --> 00:30:26.839 +or there are variants of garbage collection algorithm + +00:30:26.840 --> 00:30:30.079 +that do not scan all the memory just fraction of it + +00:30:30.080 --> 00:30:33.439 +and scan another fraction less frequently + +00:30:33.440 --> 00:30:36.999 +so there are actually ways just to change + +00:30:37.000 --> 00:30:39.799 +the garbage collection algorithm to make things much faster + +00:30:39.800 --> 00:30:44.199 +of course like just changing the numbers of variables + +00:30:44.200 --> 00:30:47.079 +like the numbers of variable values + +00:30:47.080 --> 00:30:50.079 +is much more tricky and one has to implement it + +00:30:50.080 --> 00:30:52.239 +obviously it would be nice if someone implements it + +00:30:52.240 --> 00:30:55.639 +but so far it's not happening so yeah it would be nice + +00:30:55.640 --> 00:30:59.359 +but maybe not not so quickly + +00:30:59.360 --> 00:31:02.159 +there is more chance to change the defaults here + +00:31:02.160 --> 00:31:07.479 +to conclude let me reiterate the most important points + +00:31:07.480 --> 00:31:11.919 +so from point of view of users you need to understand that + +00:31:11.920 --> 00:31:14.479 +yes garbage collection may be a problem + +00:31:14.480 --> 00:31:16.679 +but not for everyone so like + +00:31:16.680 --> 00:31:21.079 +you should only think about changing the variables + +00:31:21.080 --> 00:31:23.559 +when you really know that garbage collection + +00:31:23.560 --> 00:31:27.479 +is the problem for you so if you have slow Emacs startup + +00:31:27.480 --> 00:31:30.919 +slow Emacs startup and you know that it's caused by + +00:31:30.920 --> 00:31:32.479 +garbage collection like by + +00:31:32.480 --> 00:31:35.999 +you can check the GC elapsed variable + +00:31:36.000 --> 00:31:39.679 +then you may increase GC count threshold + +00:31:39.680 --> 00:31:42.119 +like to few tens of megabytes not more + +00:31:42.120 --> 00:31:44.479 +it doesn't make sense to increase it much more + +00:31:44.480 --> 00:31:48.239 +and if you really have major problems + +00:31:48.240 --> 00:31:49.759 +with Emacs being slaggy + +00:31:49.760 --> 00:31:52.519 +then you can increase GC count percentage + +00:31:52.520 --> 00:31:55.999 +to like 0.2 0.3 maybe + +00:31:56.000 --> 00:31:57.679 +one is probably overkill + +00:31:57.680 --> 00:32:02.759 +but do watch your Emacs ROM usage it may be really impacted + +00:32:02.760 --> 00:32:09.719 +for Emacs developers I'd like to emphasize + +00:32:09.720 --> 00:32:12.439 +that there is a real problem with garbage collection + +00:32:12.440 --> 00:32:17.959 +and nine percent of all the garbage collection + +00:32:17.960 --> 00:32:22.079 +data points we have correspond + +00:32:22.080 --> 00:32:24.959 +to really slow noticeable Emacs precision + +00:32:24.960 --> 00:32:28.039 +and really frequent less than 10 seconds + +00:32:28.040 --> 00:32:32.319 +I'd say that it's really worth + +00:32:32.320 --> 00:32:35.279 +increasing GC count threshold at least during startup + +00:32:35.280 --> 00:32:40.159 +because it really impacts the Emacs startup time + +00:32:40.160 --> 00:32:41.519 +making Emacs startup much faster + +00:32:41.520 --> 00:32:44.799 +ideally we need to reimplement + +00:32:44.800 --> 00:32:48.599 +the garbage collection algorithm of course it's not easy + +00:32:48.600 --> 00:32:50.159 +but it would be really nice + +00:32:50.160 --> 00:32:56.399 +and for GC count percentage defaults it's hard to say + +00:32:56.400 --> 00:33:00.759 +we may consider changing it but it's up to discussion + +00:33:00.760 --> 00:33:03.119 +and we probably need to be conservative here + +00:33:03.120 --> 00:33:06.039 +so we came to the end of my talk + +00:33:06.040 --> 00:33:09.319 +and this presentation + +00:33:09.320 --> 00:33:11.839 +all the data will be available publicly + +00:33:11.840 --> 00:33:17.079 +and you can reproduce all the statistic graphs if you wish + +00:33:17.080 --> 00:33:21.920 +and thank you for attention |