summaryrefslogblamecommitdiffstats
path: root/2022/captions/emacsconf-2022-survey--results-of-the-2022-emacs-survey--timothy--main.vtt
blob: 62844b3d6779c1f8806e3d725af5b28cf740314d (plain) (tree)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
1037
1038
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060
1061
1062
1063
1064
1065
1066
1067
1068
1069
1070
1071
1072
1073
1074
1075
1076
1077
1078
1079
1080
1081
1082
1083
1084
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116
1117
1118
1119
1120
1121
1122
1123
1124
1125
1126
1127
1128
1129
1130
1131
1132
1133
1134
1135
1136
1137
1138
1139
1140
1141
1142
1143
1144
1145
1146
1147
1148
1149
1150
1151
1152
1153
1154
1155
1156
1157
1158
1159
1160
1161
1162
1163
1164
1165
1166
1167
1168
1169
1170
1171
1172
1173
1174
1175
1176
1177
1178
1179
1180
1181
1182
1183
1184
1185
1186
1187
1188
1189
1190
1191
1192
1193
1194
1195
1196
1197
1198
1199
1200
1201
1202
1203
1204
1205
1206
1207
1208
1209
1210
1211
1212
1213
1214
1215
1216
1217
1218
1219
1220
1221
1222
1223
1224
1225
1226
1227
1228
1229
1230
1231
1232
1233
1234
1235
1236
1237
1238
1239
1240
1241
1242
1243
1244
1245
1246
1247
1248
1249
1250
1251
1252
1253
1254
1255
1256
1257
1258
1259
1260
1261
1262
1263
1264
1265
1266
1267
1268
1269
1270
1271
1272
1273
1274
1275
1276
1277
1278
1279
1280
1281
1282
1283
1284
1285
1286
1287
1288
1289
1290
1291
1292
1293
1294
1295
1296
1297
1298
1299
1300
1301
1302
1303
1304
1305
1306
1307
1308
1309
1310
1311
1312
1313
1314
1315
1316
1317
1318
1319
1320
1321
1322
1323
1324
1325
1326
1327
1328
1329
1330
1331
1332
1333
1334
1335
1336
1337
1338
1339
1340
1341
1342
1343
1344
1345
1346
1347
1348
1349
1350
1351
1352
1353
1354
1355
1356
1357
1358
1359
1360
1361
1362
1363
1364
1365
1366
1367
1368
1369
1370
1371
1372
1373
1374
1375
1376
1377
1378
1379
1380
1381
1382
1383
1384
1385
1386
1387
1388
1389
1390
1391
1392
1393
1394
1395
1396
1397
1398
1399
1400
1401
1402
1403
1404
1405
1406
1407
1408
1409
1410
1411
1412
1413
1414
1415
1416
1417
1418
1419
1420
1421
1422
1423
1424
1425
1426
1427
1428
1429
1430
1431
1432
1433
1434
1435
1436
1437
1438
1439
1440
1441
1442
1443
1444
1445
1446
1447
1448
1449
1450
1451
1452
1453
1454
1455
1456
1457
1458
1459
1460
1461
1462
1463
1464
1465
1466
1467
1468
1469
1470
1471
1472
1473
1474
1475
1476
1477
1478
1479
1480
1481
1482
1483
1484
1485
1486
1487
1488
1489
1490
1491
1492
1493
1494
1495
1496
1497
1498
1499
1500
1501
1502
1503
1504
1505
1506
1507
1508
1509
1510
1511
1512
1513
1514
1515
1516
1517
1518
1519
1520
1521
1522
1523
1524
1525
1526
1527
1528
1529
1530
1531
1532
1533
1534
1535
1536
1537
1538
1539
1540
1541
1542
1543
1544
1545
1546
1547
1548
1549
1550
1551
1552
1553
1554
1555
1556
1557
1558
1559
1560
1561
1562
1563
1564
1565
1566
1567
1568
1569
1570
1571
1572
1573
1574
1575
1576
1577
1578
1579
1580
1581
1582
1583
1584
1585
1586
1587
1588
1589
1590
1591
1592
1593
1594
1595
1596
1597
1598
1599
1600
1601
1602
1603
1604
1605
1606
1607
1608
1609
1610
1611
1612
1613
1614













































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































                                                            
WEBVTT captioned by sachac

NOTE Introduction

00:00:00.000 --> 00:00:06.839
Hello everyone and thanks for tuning in. I'm Timothy,

00:00:06.840 --> 00:00:08.559
and in this talk, we'll be going over

00:00:08.560 --> 00:00:11.342
the 2022 Emacs User Survey.

00:00:11.970 --> 00:00:15.078
Since this is the first time we're discussing this,

00:00:15.079 --> 00:00:18.399
we'll be going over the survey itself a bit,

00:00:18.400 --> 00:00:21.199
how it's being put together and run,

00:00:21.200 --> 00:00:24.199
and then we'll have a little taste of the results

00:00:24.200 --> 00:00:26.039
with more analysis to be published in the future.

NOTE The 2020 Emacs User Survey

00:00:26.040 --> 00:00:32.399
To start with though, a bit of background.

00:00:32.400 --> 00:00:36.679
So in 2020, we had an Emacs User Survey

00:00:36.680 --> 00:00:38.839
run by Adrien Brochard.

00:00:38.840 --> 00:00:41.359
Now this is, to the best of my knowledge,

00:00:41.360 --> 00:00:45.559
the first time that a large-scale Emacs User Survey

00:00:45.560 --> 00:00:48.039
has actually been run.

00:00:48.040 --> 00:00:50.439
About 7,000 people responded to the survey,

00:00:50.440 --> 00:00:53.239
so in many respects, it was quite successful.

00:00:53.240 --> 00:00:56.519
And what's significant about this is that

00:00:56.520 --> 00:00:57.679
with this being the first time

00:00:57.680 --> 00:00:59.999
that a large-scale survey has been run,

00:01:00.000 --> 00:01:01.719
it actually provided some insight

00:01:01.720 --> 00:01:06.719
into questions about how the community is using Emacs

00:01:06.720 --> 00:01:09.959
that allow for much better guesses

00:01:09.960 --> 00:01:15.359
than just speculation based on the small number of people

00:01:15.360 --> 00:01:16.919
who respond on the mailing list usually.

00:01:16.920 --> 00:01:24.879
So, why are we doing another survey? Well, to start with,

00:01:24.880 --> 00:01:28.799
in order to get the most value out of an Emacs User Survey,

00:01:28.800 --> 00:01:32.519
it's quite helpful if the information in it is recent.

00:01:32.520 --> 00:01:35.439
Furthermore, we can actually get some more value

00:01:35.440 --> 00:01:38.039
if we can examine trends,

00:01:38.040 --> 00:01:41.199
shifts in the way that people are using Emacs,

00:01:41.200 --> 00:01:42.919
where the pain points lie,

00:01:42.920 --> 00:01:45.479
what people are enjoying the most, etc.

00:01:45.480 --> 00:01:46.520
So in both of these respects,

00:01:46.521 --> 00:01:49.599
it's to our benefit if the survey

00:01:49.600 --> 00:01:51.519
is actually a regular event,

00:01:51.520 --> 00:01:54.359
instead of just something that's run once.

NOTE The design of the survey

00:01:54.360 --> 00:01:57.159
Now, with this in mind,

00:01:57.160 --> 00:02:00.959
we ran the 2022 Emacs User Survey with the plan

00:02:00.960 --> 00:02:05.079
that this will actually become an annual event.

00:02:05.080 --> 00:02:08.999
In the design of the survey, there are a few goals here.

00:02:09.000 --> 00:02:11.520
The main one is of the user community.

00:02:11.521 --> 00:02:14.520
Now, user community is a rather nebulous phrase.

00:02:14.521 --> 00:02:17.520
In this case, what's meant in particular

00:02:17.521 --> 00:02:21.020
is value in questions, for example,

00:02:21.021 --> 00:02:23.839
things like pain points with Emacs,

00:02:23.840 --> 00:02:27.119
which versions people are using,

00:02:27.120 --> 00:02:30.239
which capabilities people are making the most use of,

00:02:30.240 --> 00:02:34.519
which could potentially be helpful to both emacs-devel

00:02:34.520 --> 00:02:36.520
but also our collection of Emacs package maintainers

00:02:36.521 --> 00:02:38.020
and the whole community.

00:02:38.021 --> 00:02:40.799
Actually, I think going beyond just the packages,

00:02:40.800 --> 00:02:46.039
we've also got the people who develop tutorials, guides,

00:02:46.040 --> 00:02:49.279
and all of that sort of surrounding activity,

00:02:49.280 --> 00:02:51.020
which can benefit from a clear understanding

00:02:51.021 --> 00:02:56.020
of how Emacs users use Emacs.

00:02:56.021 --> 00:02:58.519
Separately to that,

00:02:58.520 --> 00:03:01.639
I think as an Emacs user myself,

00:03:01.640 --> 00:03:02.839
that it's rather interesting to see

00:03:02.840 --> 00:03:04.479
how other people are using Emacs

00:03:04.480 --> 00:03:07.079
and what their experience is. So yes, basically,

00:03:07.080 --> 00:03:08.559
you've got utility and interest

00:03:08.560 --> 00:03:10.719
as the two separate driving factors

00:03:10.720 --> 00:03:14.020
as we try to pick questions, which actually can give us

00:03:14.021 --> 00:03:16.520
all of this without taking up too much

00:03:16.521 --> 00:03:18.559
of the respondents time.

NOTE Survey frameworks

00:03:18.560 --> 00:03:24.399
Now, last time in 2020, the Emacs survey that Adrien ran

00:03:24.400 --> 00:03:27.079
used, I think Google Forms, if I recall correctly,

00:03:27.080 --> 00:03:28.799
with an option to send in responses manually.

00:03:28.800 --> 00:03:33.159
This worked, but it's not great,

00:03:33.160 --> 00:03:35.079
particularly given that this is for a survey

00:03:35.080 --> 00:03:37.199
being run in an ardently FOSS community.

00:03:37.200 --> 00:03:38.959
Ideally, we actually want

00:03:38.960 --> 00:03:40.799
to find a survey framework

00:03:40.800 --> 00:03:44.319
that respects the priorities of users, is open source,

00:03:44.320 --> 00:03:46.359
ideally free and open source,

00:03:46.360 --> 00:03:49.999
and is a relatively pleasant experience.

00:03:50.000 --> 00:03:53.079
Unfortunately, looking at available options,

00:03:53.080 --> 00:03:56.879
it seems that one always has to compromise on at least one,

00:03:56.880 --> 00:03:58.020
if not all of those criteria,

00:03:58.021 --> 00:04:01.020
which is quite far from ideal.

NOTE Writing a new survey framework in Julia

00:04:01.021 --> 00:04:04.359
So what's the obvious solution?

00:04:04.360 --> 00:04:06.639
Okay, we should just write a new survey framework.

00:04:06.640 --> 00:04:10.679
Obviously, this is easier said than done.

00:04:10.680 --> 00:04:12.239
But around a year ago,

00:04:12.240 --> 00:04:13.639
I actually started doing exactly this.

00:04:13.640 --> 00:04:17.679
I've used the programming language Julia quite a bit

00:04:17.680 --> 00:04:21.020
on a day to day basis. And there just so happens to be

00:04:21.021 --> 00:04:23.199
a web framework for that called Genie.

00:04:23.200 --> 00:04:24.719
So I thought I'd give it a shot.

00:04:24.720 --> 00:04:26.559
And well, here we are today.

00:04:26.560 --> 00:04:28.479
I ended up putting something together,

00:04:28.480 --> 00:04:34.279
which could take a set of questions written in Julia

00:04:34.280 --> 00:04:35.839
and using a survey library,

00:04:35.840 --> 00:04:38.799
actually pass that into this helpful structure

00:04:38.800 --> 00:04:44.119
and then construct HTML forms based on that,

00:04:44.120 --> 00:04:47.020
and ingest results from the HTML forms,

00:04:47.021 --> 00:04:48.520
and just sort of handle that altogether.

00:04:48.521 --> 00:04:52.439
Now, all of this ends up being fed into an SQLite DB.

00:04:52.440 --> 00:04:55.159
So everything's there, even part responses.

00:04:55.160 --> 00:04:57.599
One of the goals with the actual design of this has been

00:04:57.600 --> 00:05:01.119
to just minimize what's actually done on the client side.

00:05:01.120 --> 00:05:05.559
So that means JavaScript, cookies, the whole lot.

00:05:05.560 --> 00:05:08.759
Basically, as far as this could reasonably be taken,

00:05:08.760 --> 00:05:14.599
we've just got static HTML being shoved to the user,

00:05:14.600 --> 00:05:16.719
or respondent rather. And then we just

00:05:16.720 --> 00:05:18.519
take an HTTP post request back

00:05:18.520 --> 00:05:20.919
and update the results that way.

00:05:20.920 --> 00:05:24.239
Now by doing things like actually paging the survey,

00:05:24.240 --> 00:05:26.559
we can allow for incremental saving of results

00:05:26.560 --> 00:05:30.559
and a few other niceties while essentially preserving

00:05:30.560 --> 00:05:36.319
an experience that doesn't really require any data

00:05:36.320 --> 00:05:37.319
of any particular capabilities, which is sort of

00:05:37.320 --> 00:05:40.199
a nice, clean, minimal experience as far as I'm concerned.

NOTE In practice

00:05:40.200 --> 00:05:45.679
So how does this actually look like in practice?

00:05:45.680 --> 00:05:48.119
Well, one of the nice things about this is

00:05:48.120 --> 00:05:51.479
because the question itself is written in Julia,

00:05:51.480 --> 00:05:54.279
we can get some nice features like custom validators

00:05:54.280 --> 00:05:57.919
and other fancy behavior and directly specify

00:05:57.920 --> 00:06:01.119
how we actually want questions to be registered

00:06:01.120 --> 00:06:04.439
in the database. So here we have, for example,

00:06:04.440 --> 00:06:06.679
two questions we had from this email survey.

00:06:06.680 --> 00:06:09.959
One is a multi-select. Another one is just putting in

00:06:09.960 --> 00:06:14.399
the number of years people have used Emacs for.

00:06:14.400 --> 00:06:16.159
I think this gives a brief overview of the capabilities.

00:06:16.160 --> 00:06:19.599
One of the things I'd like to draw particular attention

00:06:19.600 --> 00:06:20.759
to here is in the multi-select,

00:06:20.760 --> 00:06:22.199
you'll see an array of options,

00:06:22.200 --> 00:06:24.319
the first one of which actually maps for different value

00:06:24.320 --> 00:06:25.879
to be stored for convenience.

00:06:25.880 --> 00:06:29.119
And then the final one is a special one, :other,

00:06:29.120 --> 00:06:30.359
and you can see that's a bit different to the rest

00:06:30.360 --> 00:06:32.599
where it's got that colon function,

00:06:32.600 --> 00:06:33.719
it's a symbol, not a string.

00:06:33.720 --> 00:06:37.639
And this is quite a nice one because the way

00:06:37.640 --> 00:06:39.279
that this framework's been designed,

00:06:39.280 --> 00:06:41.759
when we have an :other value like that,

00:06:41.760 --> 00:06:44.199
instead of it just being a sort of tick box "Other",

00:06:44.200 --> 00:06:47.199
it actually provides the option to write

00:06:47.200 --> 00:06:50.559
your own different response to all of the above.

NOTE Results

00:06:50.560 --> 00:06:55.319
Okay, so at the very end, we've now got

00:06:55.320 --> 00:06:58.519
a completely FOSS survey framework, rather nice.

00:06:58.520 --> 00:07:00.020
So the set of what were these...

00:07:00.021 --> 00:07:01.119
Decent array of input types.

00:07:01.120 --> 00:07:02.639
It would be nice to expand, but at the moment

00:07:02.640 --> 00:07:04.599
I think we could just about describe it as a rich set.

00:07:04.600 --> 00:07:07.159
Zero JavaScript required, but a little bit useful

00:07:07.160 --> 00:07:08.079
for progressive enhancement.

00:07:08.080 --> 00:07:12.759
As demonstrated, we can get some fancy validation going on.

00:07:12.760 --> 00:07:16.679
And then because we've got the results

00:07:16.680 --> 00:07:18.559
tied into this quite nicely,

00:07:18.560 --> 00:07:20.999
we can actually have them available live

00:07:21.000 --> 00:07:22.999
and in quite a number of formats.

00:07:23.000 --> 00:07:25.439
I'm not sure how much you saw in the architecture diagram,

00:07:25.440 --> 00:07:27.079
but we've got all sorts of things here.

00:07:27.080 --> 00:07:29.679
CSV, TSV, plain text, JSON,

00:07:29.680 --> 00:07:32.119
just grab a copy of the SQLite database,

00:07:32.120 --> 00:07:33.319
but only the relevant bits.

00:07:33.320 --> 00:07:35.879
Or something called JLD2,

00:07:35.880 --> 00:07:37.999
which preserves a lot of type information

00:07:38.000 --> 00:07:39.599
and a few other nice things.

NOTE Going forward

00:07:39.600 --> 00:07:43.799
Now, what are we going to do going forward from here?

00:07:43.800 --> 00:07:46.159
Well, there are a few minor issues here.

00:07:46.160 --> 00:07:48.599
For example, there's a memory leak issue which is going on,

00:07:48.600 --> 00:07:51.839
resulting in the service being restarted,

00:07:51.840 --> 00:07:54.519
I think every day or two, while the survey was running.

00:07:54.520 --> 00:07:56.159
I actually have the suspicion

00:07:56.160 --> 00:07:57.639
that that's largely responsible for

00:07:57.640 --> 00:08:01.479
about 1% of respondents, which is about 75 people,

00:08:01.480 --> 00:08:04.399
who described the survey experience as not great.

00:08:04.400 --> 00:08:08.199
Overall though, the feedback has been quite positive.

00:08:08.200 --> 00:08:09.919
There's been some detailed written feedback,

00:08:09.920 --> 00:08:12.799
but just from the quick great/okay/not great options,

00:08:12.800 --> 00:08:14.839
we had about two-thirds of people saying

00:08:14.840 --> 00:08:16.839
that the user experience was great,

00:08:16.840 --> 00:08:19.199
which is really nice to hear the first time being run.

00:08:19.200 --> 00:08:22.839
A few other things would be nice to add, for example,

00:08:22.840 --> 00:08:25.759
in future control flow. By this, I mean

00:08:25.760 --> 00:08:27.879
the option to present different questions

00:08:27.880 --> 00:08:28.999
based on previous answers

00:08:29.000 --> 00:08:31.199
would be quite nice to streamline the experience.

00:08:31.200 --> 00:08:33.519
For example, having a set of questions

00:08:33.520 --> 00:08:37.239
for first-time respondents or people who are involved

00:08:37.240 --> 00:08:42.239
in the packaging side of things

00:08:42.240 --> 00:08:45.079
without actually cluttering the experience

00:08:45.080 --> 00:08:46.039
for everybody else. That'd be quite nice.

00:08:46.040 --> 00:08:48.599
Further to this, all of this,

00:08:48.600 --> 00:08:51.879
I think on top of the standard web interface,

00:08:51.880 --> 00:08:53.599
it'd be quite nice to actually write a server API.

00:08:53.600 --> 00:08:55.520
And the particular reason why I mentioned this

00:08:55.521 --> 00:08:58.020
is because this could potentially allow for

00:08:58.021 --> 00:09:00.359
basically an Emacs survey package.

00:09:00.360 --> 00:09:03.039
I mean, we already use Emacs for so many things,

00:09:03.040 --> 00:09:05.519
might as well fill the survey out from within it as well.

00:09:05.520 --> 00:09:11.159
Okay, so this is how the survey has been conducted.

NOTE Responses

00:09:11.160 --> 00:09:13.679
Now, what are the responses look like?

00:09:13.680 --> 00:09:16.039
Now, at this stage, I was actually hoping

00:09:16.040 --> 00:09:18.919
to get into some somewhat sophisticated analysis

00:09:18.920 --> 00:09:22.599
because there's quite a bit that you can dig out

00:09:22.600 --> 00:09:24.239
of the data responses that we've received.

00:09:24.240 --> 00:09:27.879
However, unfortunately, I've been much more limited on time

00:09:27.880 --> 00:09:30.039
than I'd hoped for, so that's going to have to come later.

00:09:30.040 --> 00:09:33.559
For now, we're just going to take a bit of a peek

00:09:33.560 --> 00:09:35.959
at some of the really basic answers.

00:09:35.960 --> 00:09:38.239
Well, it's not even really analysis.

00:09:38.240 --> 00:09:40.239
Expect to see lots of pie charts, basically.

00:09:40.240 --> 00:09:42.999
But there's still a bit of interest there,

00:09:43.000 --> 00:09:44.359
so we'll go through a bit of that

00:09:44.360 --> 00:09:47.119
and just give a bit of a tease

00:09:47.120 --> 00:09:50.319
as to what might come in the future.

00:09:50.320 --> 00:09:51.919
So to sum up for starters,

00:09:51.920 --> 00:09:55.079
we've had about 6,500 responses.

00:09:55.080 --> 00:09:58.359
It is worth noting that a thousand of those are partials,

00:09:58.360 --> 00:10:02.199
so people who gave up on the survey partway through.

00:10:02.200 --> 00:10:05.399
Given that the 2020 survey had about 7000 responses,

00:10:05.400 --> 00:10:06.999
I'll tell you we're basically on par here.

00:10:07.000 --> 00:10:10.399
This ran over a month and interestingly,

00:10:10.400 --> 00:10:12.239
about half of these respondents

00:10:12.240 --> 00:10:13.799
did not participate in the 2020 survey.

00:10:13.800 --> 00:10:16.199
I think at this point,

00:10:16.200 --> 00:10:17.679
it's not really clear what to make of that.

00:10:17.680 --> 00:10:21.359
There's been a two-year gap between the surveys.

00:10:21.360 --> 00:10:25.159
It's been done, well, it's been done quite differently,

00:10:25.160 --> 00:10:29.639
and yes, there's not enough, really, to say.

00:10:29.640 --> 00:10:31.999
What could be interesting though is actually,

00:10:32.000 --> 00:10:33.839
once this starts running regularly,

00:10:33.840 --> 00:10:36.799
we can see whether there's regular churn

00:10:36.800 --> 00:10:38.520
in the survey respondents,

00:10:38.521 --> 00:10:40.020
or if we have a consistent core

00:10:40.021 --> 00:10:42.020
with people who respond each year,

00:10:42.021 --> 00:10:46.159
and then just people who come by every now and then and go,

00:10:46.160 --> 00:10:47.759
"Oh, why not respond to this year's survey?"

00:10:47.760 --> 00:10:51.479
But we're going to have to wait a bit to actually see

00:10:51.480 --> 00:10:52.759
how people treat the survey.

00:10:52.760 --> 00:10:57.519
Now these responses came from quite a wide range of places

00:10:57.520 --> 00:11:02.519
we've got 115 nations represented here. Collectively,

00:11:02.520 --> 00:11:04.039
these ones have spent about a thousand hours

00:11:04.040 --> 00:11:06.959
giving us information. So I think, if nothing else,

00:11:06.960 --> 00:11:10.479
just from the effort that people have put into

00:11:10.480 --> 00:11:12.879
actually giving us useful data to work with,

00:11:12.880 --> 00:11:13.599
it's worth giving at least a good effort

00:11:13.600 --> 00:11:15.999
to actually trying to extract some value

00:11:16.000 --> 00:11:16.999
out of these responses.

NOTE Geography

00:11:17.000 --> 00:11:20.879
Now, overall we found a lot of responses came from America,

00:11:20.880 --> 00:11:23.199
no surprises there, but as mentioned,

00:11:23.200 --> 00:11:24.020
we've got a good mix around the globe.

00:11:24.021 --> 00:11:29.159
The usual suspects for the rest of the responses,

00:11:29.160 --> 00:11:33.279
a whole bunch in Europe, a whole bunch around Asia,

00:11:33.280 --> 00:11:36.799
a bit in Australasia as well and yes,

00:11:36.800 --> 00:11:38.959
there's nothing particularly surprising here,

00:11:38.960 --> 00:11:41.399
there's a lot of inline expectations.

00:11:41.400 --> 00:11:42.839
What I find a bit more interesting, though,

00:11:42.840 --> 00:11:45.359
is if we actually normalise

00:11:45.360 --> 00:11:48.079
the number of responses from each nation

00:11:48.080 --> 00:11:50.079
by the population of said nations,

00:11:50.080 --> 00:11:54.239
essentially giving a popularity of Emacs

00:11:54.240 --> 00:11:57.359
or at least of Emacs early respondents for each nation,

00:11:57.360 --> 00:12:00.919
we end up finding that Europe, particularly Scandinavia,

00:12:00.920 --> 00:12:02.199
becomes a bit of a hotspot.

00:12:02.200 --> 00:12:04.519
So I'm not sure what's going on

00:12:04.520 --> 00:12:07.319
in Sweden, Finland and Norway,

00:12:07.320 --> 00:12:10.919
but it seems to be particularly popular around there.

00:12:10.920 --> 00:12:14.199
It's also worth noting that we now find

00:12:14.200 --> 00:12:18.319
that the proportion of respondents

00:12:18.320 --> 00:12:21.799
in countries like America, Canada, Australia

00:12:21.800 --> 00:12:24.039
and most of Europe actually becomes

00:12:24.040 --> 00:12:26.399
quite comparable with each other,

00:12:26.400 --> 00:12:30.239
which yes, once again, sort of lines up

00:12:30.240 --> 00:12:32.279
with these responses, expectations from the last slide.

NOTE Gender

00:12:32.280 --> 00:12:36.279
Okay, getting into some of the other

00:12:36.280 --> 00:12:38.599
demographic information.

00:12:38.600 --> 00:12:40.319
The demographic information was new to this survey.

00:12:40.320 --> 00:12:44.479
In the 2020 survey, people were asked what they think

00:12:44.480 --> 00:12:47.199
of being asked about some demographic information

00:12:47.200 --> 00:12:50.199
in a future survey, and the overwhelming response is, "Sure,

00:12:50.200 --> 00:12:52.759
I don't really mind." And so that's what we've done here.

00:12:52.760 --> 00:12:56.279
One of the ones of somewhat interest

00:12:56.280 --> 00:12:59.759
is the age gender breakdown. So we expect Emacs

00:12:59.760 --> 00:13:03.119
to be used predominantly among people in software

00:13:03.120 --> 00:13:05.839
and programming and within the industry,

00:13:05.840 --> 00:13:08.599
I think it's quite widely documented

00:13:08.600 --> 00:13:14.520
to have about a sort of 75-25%, roughly, split

00:13:14.521 --> 00:13:14.759
between male and female.

00:13:14.760 --> 00:13:19.359
Interestingly, in Emacs,

00:13:19.360 --> 00:13:22.879
it's a much more aggressively-biased result.

00:13:22.880 --> 00:13:28.679
So we had about 96% of respondents are male

00:13:28.680 --> 00:13:34.559
with just 4% for the rest. Interestingly, though,

00:13:34.560 --> 00:13:35.359
if we look at the young respondents,

00:13:35.360 --> 00:13:41.719
say for example, under 25, we go from 96% male to 88%.

00:13:41.720 --> 00:13:46.119
So it's fair to say that the young respondents are

00:13:46.120 --> 00:13:49.199
in this respect, a somewhat more diverse group.

00:13:49.200 --> 00:13:52.399
Hopefully, as future surveys go on,

00:13:52.400 --> 00:13:54.399
we'll see this continue not die off

00:13:54.400 --> 00:13:58.719
to the sort of well, at this point,

00:13:58.720 --> 00:14:02.919
it's more like 99% if you look at the older ages.

00:14:02.920 --> 00:14:04.439
But we'll see.

NOTE Occupations

00:14:04.440 --> 00:14:07.919
Occupations was an interesting slide as well.

00:14:07.920 --> 00:14:09.399
Interesting question as well.

00:14:09.400 --> 00:14:11.559
We've got the usual suspects here. I mean,

00:14:11.560 --> 00:14:15.079
it's a text editor, well, Lisp machine

00:14:15.080 --> 00:14:17.639
masquerading as a text editor, mainly used for programming,

00:14:17.640 --> 00:14:20.639
and so we expect lots of software development

00:14:20.640 --> 00:14:23.519
and that sort of thing. But that's only about

00:14:23.520 --> 00:14:25.399
just over half of the responses.

00:14:25.400 --> 00:14:28.679
We've got a huge chunk from academia,

00:14:28.680 --> 00:14:29.999
and then really just an odd bag

00:14:30.000 --> 00:14:30.879
of all sorts of other things,

00:14:30.880 --> 00:14:33.079
including things which you wouldn't really associate

00:14:33.080 --> 00:14:35.359
with programming and software at all.

00:14:35.360 --> 00:14:39.639
Things like creative writing, publishing, legal, yes.

00:14:39.640 --> 00:14:41.719
And then you've got this chunk of Other,

00:14:41.720 --> 00:14:43.239
which is I think here is

00:14:43.240 --> 00:14:46.679
the fourth most popular option here.

00:14:46.680 --> 00:14:49.399
And what we have here is about 500 different responses

00:14:49.400 --> 00:14:51.839
from a huge range of activities.

00:14:51.840 --> 00:14:54.359
It's really quite interesting to read things like

00:14:54.360 --> 00:14:56.919
I think, things like "naval officer",

00:14:56.920 --> 00:15:01.319
and just... All sorts of surprising occupations for Emacs.

00:15:01.320 --> 00:15:04.799
And I think this is a particular area

00:15:04.800 --> 00:15:10.199
because I imagine compared to other code editors,

00:15:10.200 --> 00:15:13.879
sort of your VS Code, remember like

00:15:13.880 --> 00:15:18.959
that Emacs may have a particularly diverse set

00:15:18.960 --> 00:15:23.599
of industry occupations represented in its users.

00:15:23.600 --> 00:15:28.359
Now, if you look at where the response actually came from,

00:15:28.360 --> 00:15:31.039
we've got the usual suspects up top,

00:15:31.040 --> 00:15:33.959
Hacker News and r/emacs.

00:15:33.960 --> 00:15:40.119
But then we actually get a much more graduated breakdown

00:15:40.120 --> 00:15:43.679
than in the 2020 survey.

00:15:43.680 --> 00:15:46.279
We do think familiar results here like IRC, Telegram,

00:15:46.280 --> 00:15:48.639
Emacs China, and Twitter.

00:15:48.640 --> 00:15:50.839
But now you've got a few new entries,

00:15:50.840 --> 00:15:53.519
things like the Fediverse, Discourse, Matrix,

00:15:53.520 --> 00:15:56.119
which didn't pop up previously.

00:15:56.120 --> 00:15:59.079
So I think this is yes, quite a nice sign in terms of

00:15:59.080 --> 00:16:02.520
actually hitting a wide range

00:16:02.521 --> 00:16:05.999
of pockets of Emacs users across different platforms,

00:16:06.000 --> 00:16:10.319
which bodes well for the potential representiveness

00:16:10.320 --> 00:16:11.319
of this survey.

NOTE Free and open source software

00:16:11.320 --> 00:16:15.119
Unsurprisingly, if we're talking about Emacs

00:16:15.120 --> 00:16:17.919
and particularly people who are quite engaged in it,

00:16:17.920 --> 00:16:19.679
which are the respondents to this survey,

00:16:19.680 --> 00:16:25.359
we find that we also get quite a high degree of care

00:16:25.360 --> 00:16:27.479
for free and open source software.

00:16:27.480 --> 00:16:30.519
So if you have a look here,

00:16:30.520 --> 00:16:35.279
only about a quarter of users

00:16:35.280 --> 00:16:39.799
didn't express a strong preference towards FOSS software.

00:16:39.800 --> 00:16:43.759
In fact, we had over a quarter saying that

00:16:43.760 --> 00:16:49.239
they would accept significant or even any compromise

00:16:49.240 --> 00:16:52.199
to use a FOSS user software

00:16:52.200 --> 00:16:55.759
over a proprietary alternative,

00:16:55.760 --> 00:16:59.679
which given the nature of Emacs,

00:16:59.680 --> 00:17:00.639
not terribly surprising,

00:17:00.640 --> 00:17:02.439
but a strong showing nonetheless.

NOTE Emacs versions

00:17:02.440 --> 00:17:05.599
Now, let's start getting to things

00:17:05.600 --> 00:17:07.719
which are actually useful for

00:17:07.720 --> 00:17:11.479
potential Emacs development and packaging.

00:17:11.480 --> 00:17:13.599
If you're thinking about supporting Emacs versions,

00:17:13.600 --> 00:17:16.599
it looks like you can do fantastically well

00:17:16.600 --> 00:17:20.639
in terms of hitting most users if you support Emacs 27+.

00:17:20.640 --> 00:17:23.159
That hits about 96% of respondents.

00:17:23.160 --> 00:17:26.199
Interestingly though, you can actually make an argument

00:17:26.200 --> 00:17:27.119
for being even more aggressive.

00:17:27.120 --> 00:17:30.319
I mean, if you have a look at Emacs 28+,

00:17:30.320 --> 00:17:32.359
that's still over three quarters of respondents.

00:17:32.360 --> 00:17:35.799
We've got, at this point, a quarter

00:17:35.800 --> 00:17:37.279
using the unreleased HEAD version,

00:17:37.280 --> 00:17:40.159
even though it's getting close to release.

00:17:40.160 --> 00:17:43.039
Obviously here, as stated, we're hitting

00:17:43.040 --> 00:17:44.599
a sort of more engaged with the community

00:17:44.600 --> 00:17:47.799
subset of Emacs users, but still,

00:17:47.800 --> 00:17:49.879
I think it's interesting to see that

00:17:49.880 --> 00:17:52.639
with Emacs's increasingly frequent update schedule,

00:17:52.640 --> 00:17:54.999
that users are actually picking up those updates

00:17:55.000 --> 00:17:56.359
quite promptly as they roll out.

NOTE Languages

00:17:56.360 --> 00:18:02.079
Continuing on with how people actually use Emacs: languages.

00:18:02.080 --> 00:18:05.199
We've got the usual suspects here: lots of Python,

00:18:05.200 --> 00:18:08.959
quite a bit of JavaScript and C, lots of shell.

00:18:08.960 --> 00:18:11.879
What I find quite interesting though is

00:18:11.880 --> 00:18:12.799
if we actually bring in

00:18:12.800 --> 00:18:16.719
the 2020 Stack Overflow language usage survey data,

00:18:16.720 --> 00:18:19.239
and that maps quite well

00:18:19.240 --> 00:18:20.079
to the array of language options we provided here.

00:18:20.080 --> 00:18:21.199
They had a general Lisp option,

00:18:21.200 --> 00:18:23.919
which I've folded into Common Lisp

00:18:23.920 --> 00:18:26.919
since they listed Clojure separately.

00:18:26.920 --> 00:18:29.679
I think that seems like a fairly safe bet.

00:18:29.680 --> 00:18:31.919
But other than that, the only languages that we missed

00:18:31.920 --> 00:18:35.839
are Scheme and Elisp.

00:18:35.840 --> 00:18:37.879
What we can do is we can look at

00:18:37.880 --> 00:18:41.199
the relative popularity of different languages

00:18:41.200 --> 00:18:44.519
from our Emacs user survey compared to Stack Overflows.

00:18:44.520 --> 00:18:48.319
What do we find? Well, Clojure and Common Lisp

00:18:48.320 --> 00:18:51.639
far above the rest, I imagine in no small part due to

00:18:51.640 --> 00:18:54.959
the fantastic SLIME and Cider packages.

00:18:54.960 --> 00:18:59.559
Following that, we see Haskell being particularly prominent,

00:18:59.560 --> 00:19:00.639
and then a collection of other languages,

00:19:00.640 --> 00:19:06.199
your Erlang, Elixir, Julia, Perl and the rest.

00:19:06.200 --> 00:19:10.959
And then lastly, if we have a look at the ones

00:19:10.960 --> 00:19:13.439
which have significantly diminished popularity

00:19:13.440 --> 00:19:17.719
compared to Stack Overflow, we end up with, I think,

00:19:17.720 --> 00:19:20.159
what I could probably cast as more enterprising languages.

00:19:20.160 --> 00:19:25.799
Things like C#, Java, Typescript and the like.

NOTE Prose

00:19:25.800 --> 00:19:31.559
So, that's interesting. Now, earlier

00:19:31.560 --> 00:19:33.239
when we were looking at the split of Emacs users,

00:19:33.240 --> 00:19:37.239
we found that we actually had a fair few

00:19:37.240 --> 00:19:42.199
in more creative areas, like writing and publishing.

00:19:42.200 --> 00:19:44.479
So if looking at prose, we'd expect a decent chunk

00:19:44.480 --> 00:19:47.039
to be using Emacs for prose, but it's actually more

00:19:47.040 --> 00:19:48.719
than just a little bit, it's a little slice.

00:19:48.720 --> 00:19:50.599
We've got a whopping about a third of users

00:19:50.600 --> 00:19:54.719
saying they frequently use Emacs for writing prose.

00:19:54.720 --> 00:19:55.999
I'd imagine that the availability

00:19:56.000 --> 00:19:57.799
of things like Org mode and AUCTeX

00:19:57.800 --> 00:20:03.399
probably help like this.

NOTE Packages

00:20:03.400 --> 00:20:05.119
Moving on to other packages, or more packages,

00:20:05.120 --> 00:20:08.879
we've actually got a very similar split here

00:20:08.880 --> 00:20:13.199
to the 2020 survey. Org has seen a bit of a growth

00:20:13.200 --> 00:20:16.039
in popularity. We've got some new arrivals here as well.

00:20:16.040 --> 00:20:18.479
For example, Vertico has popped onto the scene

00:20:18.480 --> 00:20:21.279
and overtaken Ivy here, along with

00:20:21.280 --> 00:20:24.519
a few other new packages like Consult.

00:20:24.520 --> 00:20:27.599
Other than that, quite comparable.

00:20:27.600 --> 00:20:29.999
What's rather interesting, though, I find here is that

00:20:30.000 --> 00:20:33.719
when you have people who listed a small number of packages,

00:20:33.720 --> 00:20:39.439
they actually predominantly listed packages

00:20:39.440 --> 00:20:41.319
other than the most common set.

00:20:41.320 --> 00:20:43.959
So if we have a lot of people who only listed one package,

00:20:43.960 --> 00:20:48.959
basically two-thirds of that,

00:20:48.960 --> 00:20:51.479
or actually three-quarters of those responses

00:20:51.480 --> 00:20:53.879
were saying other packages,

00:20:53.880 --> 00:20:56.279
despite the fact that overall packages

00:20:56.280 --> 00:20:58.599
other than the highlighted selection here

00:20:58.600 --> 00:21:01.399
only constitute a quarter of responses.

00:21:01.400 --> 00:21:04.919
So there might be something a bit more to look at there.

NOTE Documentation

00:21:04.920 --> 00:21:07.799
Now when people are using packages,

00:21:07.800 --> 00:21:11.039
we also asked what types of documentation

00:21:11.040 --> 00:21:14.399
people would like to see more of on package READMEs.

00:21:14.400 --> 00:21:17.159
Basically we've got a big mix here.

00:21:17.160 --> 00:21:20.079
It seems like generally people are interested in

00:21:20.080 --> 00:21:23.839
seeing more in various forms, whether it be tutorials,

00:21:23.840 --> 00:21:29.479
overviews, screenshots, comparisons, or clips and videos.

00:21:29.480 --> 00:21:32.919
So full READMEs with a lot of context

00:21:32.920 --> 00:21:38.439
seem to be quite desirable from this.

NOTE Moving forward

00:21:38.440 --> 00:21:42.359
Now moving forward, what are we going to do?

00:21:42.360 --> 00:21:45.039
So 800 people gave some detailed feedback on the survey.

00:21:45.040 --> 00:21:47.759
That's quite nice. I'm going to be taking a good read

00:21:47.760 --> 00:21:50.799
of all of those responses and use that

00:21:50.800 --> 00:21:55.639
to improve the process and also the set of questions.

00:21:55.640 --> 00:22:00.759
Now all of you can also give some feedback on the questions,

00:22:00.760 --> 00:22:02.679
both that you found most useful in this survey,

00:22:02.680 --> 00:22:04.799
ones that you think might not add much value,

00:22:04.800 --> 00:22:07.039
and/or new questions

00:22:07.040 --> 00:22:08.359
that you think might be a good addition.

00:22:08.360 --> 00:22:11.119
Once I've done a bit more analysis,

00:22:11.120 --> 00:22:13.119
particularly the more sophisticated analysis

00:22:13.120 --> 00:22:17.159
which I'm planning, which will probably come out actually

00:22:17.160 --> 00:22:18.719
maybe in the first quarter of next year,

00:22:18.720 --> 00:22:22.919
we can see which questions there seem to have provided

00:22:22.920 --> 00:22:25.039
the most interesting or surprising results

00:22:25.040 --> 00:22:26.559
and those are probably worth keeping.

00:22:26.560 --> 00:22:31.959
Lastly, once we actually have an API

00:22:31.960 --> 00:22:33.279
and potentially even an Emacs package,

00:22:33.280 --> 00:22:36.159
we could automate a large number of the questions,

00:22:36.160 --> 00:22:38.999
things like Emacs version, set of packages used,

00:22:39.000 --> 00:22:41.039
and that could just streamline the experience

00:22:41.040 --> 00:22:42.279
of actually filling out the survey,

00:22:42.280 --> 00:22:44.199
make it a bit more frictionless.

NOTE Time

00:22:44.200 --> 00:22:47.319
Now talking of the question of questions,

00:22:47.320 --> 00:22:49.319
a quick survey is a good survey.

00:22:49.320 --> 00:22:52.959
If we're asking people to dedicate their time

00:22:52.960 --> 00:22:56.279
to fill out this, it's good to try to get as much value

00:22:56.280 --> 00:22:59.759
without asking them to donate much of their time.

00:22:59.760 --> 00:23:02.399
How has the survey done in this respect?

00:23:02.400 --> 00:23:04.119
I'm actually very happy with how it's done.

00:23:04.120 --> 00:23:06.639
We get a few comments from the feedback saying

00:23:06.640 --> 00:23:07.759
that it was a bit of a long side,

00:23:07.760 --> 00:23:10.759
but the median time was about 12 minutes,

00:23:10.760 --> 00:23:13.759
which doesn't seem too bad, and most commonly

00:23:13.760 --> 00:23:16.399
we saw people completing it in about 8 minutes.

00:23:16.400 --> 00:23:18.879
For a once-per-year survey,

00:23:18.880 --> 00:23:20.519
I think this seems fairly reasonable.

00:23:20.520 --> 00:23:24.279
Getting closer to a 5-10 minute range would be nice,

00:23:24.280 --> 00:23:26.199
but this isn't far off.

NOTE How long the survey is open for

00:23:26.200 --> 00:23:30.879
Lastly, we're also going to be considering

00:23:30.880 --> 00:23:32.719
how long the survey is open for.

00:23:32.720 --> 00:23:36.719
So from the initial opening date,

00:23:36.720 --> 00:23:38.479
what we have here is a plot of

00:23:38.480 --> 00:23:41.919
the page which people ended up on

00:23:41.920 --> 00:23:43.399
and when they started the survey.

00:23:43.400 --> 00:23:46.759
So what we can see is a huge spike in the first few days.

00:23:46.760 --> 00:23:50.239
I've just realised that this plot

00:23:50.240 --> 00:23:53.399
is actually labelled incorrectly.

00:23:53.400 --> 00:23:55.679
Please disregard the minutes to complete the survey.

00:23:55.680 --> 00:23:58.839
This should be days after survey opening

00:23:58.840 --> 00:24:01.519
that a response is actually submitted.

00:24:01.520 --> 00:24:05.399
And what we have here is a big spike

00:24:05.400 --> 00:24:08.679
in popularity in the first week basically,

00:24:08.680 --> 00:24:10.599
and then it trickles down

00:24:10.600 --> 00:24:11.959
to a fairly consistent level after that.

00:24:11.960 --> 00:24:15.839
I'm about to publish a last call for survey responses,

00:24:15.840 --> 00:24:18.279
so I'll see if any final bump happens,

00:24:18.280 --> 00:24:20.039
but this indicates that we can probably just

00:24:20.040 --> 00:24:23.079
have the survey open for a week or two

00:24:23.080 --> 00:24:25.199
and that should be sufficient.

NOTE Plan going forward

00:24:25.200 --> 00:24:30.839
Alright, so what's the general plan going forwards?

00:24:30.840 --> 00:24:35.639
Well, as stated earlier, the idea is to run this annually

00:24:35.640 --> 00:24:38.399
and then consistently improve the questions,

00:24:38.400 --> 00:24:41.039
the experience, and the analysis that's done.

00:24:41.040 --> 00:24:43.559
This year has been the hardest by far

00:24:43.560 --> 00:24:45.839
because a lot had to be set up from scratch.

00:24:45.840 --> 00:24:50.159
The hope is that moving on from here,

00:24:50.160 --> 00:24:51.799
a lot of it can be reused.

00:24:51.800 --> 00:24:54.039
For example, with my comments about

00:24:54.040 --> 00:24:56.439
more sophisticated analysis being down the line,

00:24:56.440 --> 00:24:58.439
once that's all worked out,

00:24:58.440 --> 00:25:00.719
as long as nothing changes too drastically,

00:25:00.720 --> 00:25:03.559
we should be able to reuse a lot of that work

00:25:03.560 --> 00:25:05.759
quite easily in future years.

00:25:05.760 --> 00:25:08.599
Alright, that's it for now.

00:25:08.600 --> 00:25:11.879
Hopefully, you've found this an interesting peek

00:25:11.880 --> 00:25:13.359
into how the survey is operated

00:25:13.360 --> 00:25:15.319
and some of the initial results,

00:25:15.320 --> 00:25:18.919
and hopefully, I'll see you around next year

00:25:18.920 --> 00:25:36.960
for the 2023 survey. Thanks for listening.