Hello everyone and thanks for tuning in. I'm Timothy,
and in this talk, we'll be going over
the 2022 Emacs User Survey.
Since this is the first time we're discussing this,
we'll be going over the survey itself a bit,
how it's being put together and run,
and then we'll have a little taste of the results
with more analysis to be published in the future.
To start with though, a bit of background.
So in 2020, we had an Emacs User Survey
run by Adrien Brochard.
Now this is, to the best of my knowledge,
the first time that a large-scale Emacs User Survey
has actually been run.
About 7,000 people responded to the survey,
so in many respects, it was quite successful.
And what's significant about this is that
with this being the first time
that a large-scale survey has been run,
it actually provided some insight
into questions about how the community is using Emacs
that allow for much better guesses
than just speculation based on the small number of people
who respond on the mailing list usually.
So, why are we doing another survey? Well, to start with,
in order to get the most value out of an Emacs User Survey,
it's quite helpful if the information in it is recent.
Furthermore, we can actually get some more value
if we can examine trends,
shifts in the way that people are using Emacs,
where the pain points lie,
what people are enjoying the most, etc.
So in both of these respects,
it's to our benefit if the survey
is actually a regular event,
instead of just something that's run once.
Now, with this in mind,
we ran the 2022 Emacs User Survey with the plan
that this will actually become an annual event.
In the design of the survey, there are a few goals here.
The main one is of the user community.
Now, user community is a rather nebulous phrase.
In this case, what's meant in particular
is value in questions, for example,
things like pain points with Emacs,
which versions people are using,
which capabilities people are making the most use of,
which could potentially be helpful to both emacs-devel
but also our collection of Emacs package maintainers
and the whole community.
Actually, I think going beyond just the packages,
we've also got the people who develop tutorials, guides,
and all of that sort of surrounding activity,
which can benefit from a clear understanding
of how Emacs users use Emacs.
Separately to that,
I think as an Emacs user myself,
that it's rather interesting to see
how other people are using Emacs
and what their experience is. So yes, basically,
you've got utility and interest
as the two separate driving factors
as we try to pick questions, which actually can give us
all of this without taking up too much
of the respondents time.
Now, last time in 2020, the Emacs survey that Adrien ran
used, I think Google Forms, if I recall correctly,
with an option to send in responses manually.
This worked, but it's not great,
particularly given that this is for a survey
being run in an ardently FOSS community.
Ideally, we actually want
to find a survey framework
that respects the priorities of users, is open source,
ideally free and open source,
and is a relatively pleasant experience.
Unfortunately, looking at available options,
it seems that one always has to compromise on at least one,
if not all of those criteria,
which is quite far from ideal.
So what's the obvious solution?
Okay, we should just write a new survey framework.
Obviously, this is easier said than done.
But around a year ago,
I actually started doing exactly this.
I've used the programming language Julia quite a bit
on a day to day basis. And there just so happens to be
a web framework for that called Genie.
So I thought I'd give it a shot.
And well, here we are today.
I ended up putting something together,
which could take a set of questions written in Julia
and using a survey library,
actually pass that into this helpful structure
and then construct HTML forms based on that,
and ingest results from the HTML forms,
and just sort of handle that altogether.
Now, all of this ends up being fed into an SQLite DB.
So everything's there, even part responses.
One of the goals with the actual design of this has been
to just minimize what's actually done on the client side.
So that means JavaScript, cookies, the whole lot.
Basically, as far as this could reasonably be taken,
we've just got static HTML being shoved to the user,
or respondent rather. And then we just
take an HTTP post request back
and update the results that way.
Now by doing things like actually paging the survey,
we can allow for incremental saving of results
and a few other niceties while essentially preserving
an experience that doesn't really require any data
of any particular capabilities, which is sort of
a nice, clean, minimal experience as far as I'm concerned.
So how does this actually look like in practice?
Well, one of the nice things about this is
because the question itself is written in Julia,
we can get some nice features like custom validators
and other fancy behavior and directly specify
how we actually want questions to be registered
in the database. So here we have, for example,
two questions we had from this email survey.
One is a multi-select. Another one is just putting in
the number of years people have used Emacs for.
I think this gives a brief overview of the capabilities.
One of the things I'd like to draw particular attention
to here is in the multi-select,
you'll see an array of options,
the first one of which actually maps for different value
to be stored for convenience.
And then the final one is a special one, :other,
and you can see that's a bit different to the rest
where it's got that colon function,
it's a symbol, not a string.
And this is quite a nice one because the way
that this framework's been designed,
when we have an :other value like that,
instead of it just being a sort of tick box "Other",
it actually provides the option to write
your own different response to all of the above.
Okay, so at the very end, we've now got
a completely FOSS survey framework, rather nice.
So the set of what were these...
Decent array of input types.
It would be nice to expand, but at the moment
I think we could just about describe it as a rich set.
Zero JavaScript required, but a little bit useful
for progressive enhancement.
As demonstrated, we can get some fancy validation going on.
And then because we've got the results
tied into this quite nicely,
we can actually have them available live
and in quite a number of formats.
I'm not sure how much you saw in the architecture diagram,
but we've got all sorts of things here.
CSV, TSV, plain text, JSON,
just grab a copy of the SQLite database,
but only the relevant bits.
Or something called JLD2,
which preserves a lot of type information
and a few other nice things.
Now, what are we going to do going forward from here?
Well, there are a few minor issues here.
For example, there's a memory leak issue which is going on,
resulting in the service being restarted,
I think every day or two, while the survey was running.
I actually have the suspicion
that that's largely responsible for
about 1% of respondents, which is about 75 people,
who described the survey experience as not great.
Overall though, the feedback has been quite positive.
There's been some detailed written feedback,
but just from the quick great/okay/not great options,
we had about two-thirds of people saying
that the user experience was great,
which is really nice to hear the first time being run.
A few other things would be nice to add, for example,
in future control flow. By this, I mean
the option to present different questions
based on previous answers
would be quite nice to streamline the experience.
For example, having a set of questions
for first-time respondents or people who are involved
in the packaging side of things
without actually cluttering the experience
for everybody else. That'd be quite nice.
Further to this, all of this,
I think on top of the standard web interface,
it'd be quite nice to actually write a server API.
And the particular reason why I mentioned this
is because this could potentially allow for
basically an Emacs survey package.
I mean, we already use Emacs for so many things,
might as well fill the survey out from within it as well.
Okay, so this is how the survey has been conducted.
Now, what are the responses look like?
Now, at this stage, I was actually hoping
to get into some somewhat sophisticated analysis
because there's quite a bit that you can dig out
of the data responses that we've received.
However, unfortunately, I've been much more limited on time
than I'd hoped for, so that's going to have to come later.
For now, we're just going to take a bit of a peek
at some of the really basic answers.
Well, it's not even really analysis.
Expect to see lots of pie charts, basically.
But there's still a bit of interest there,
so we'll go through a bit of that
and just give a bit of a tease
as to what might come in the future.
So to sum up for starters,
we've had about 6,500 responses.
It is worth noting that a thousand of those are partials,
so people who gave up on the survey partway through.
Given that the 2020 survey had about 7000 responses,
I'll tell you we're basically on par here.
This ran over a month and interestingly,
about half of these respondents
did not participate in the 2020 survey.
I think at this point,
it's not really clear what to make of that.
There's been a two-year gap between the surveys.
It's been done, well, it's been done quite differently,
and yes, there's not enough, really, to say.
What could be interesting though is actually,
once this starts running regularly,
we can see whether there's regular churn
in the survey respondents,
or if we have a consistent core
with people who respond each year,
and then just people who come by every now and then and go,
"Oh, why not respond to this year's survey?"
But we're going to have to wait a bit to actually see
how people treat the survey.
Now these responses came from quite a wide range of places
we've got 115 nations represented here. Collectively,
these ones have spent about a thousand hours
giving us information. So I think, if nothing else,
just from the effort that people have put into
actually giving us useful data to work with,
it's worth giving at least a good effort
to actually trying to extract some value
out of these responses.
Now, overall we found a lot of responses came from America,
no surprises there, but as mentioned,
we've got a good mix around the globe.
The usual suspects for the rest of the responses,
a whole bunch in Europe, a whole bunch around Asia,
a bit in Australasia as well and yes,
there's nothing particularly surprising here,
there's a lot of inline expectations.
What I find a bit more interesting, though,
is if we actually normalise
the number of responses from each nation
by the population of said nations,
essentially giving a popularity of Emacs
or at least of Emacs early respondents for each nation,
we end up finding that Europe, particularly Scandinavia,
becomes a bit of a hotspot.
So I'm not sure what's going on
in Sweden, Finland and Norway,
but it seems to be particularly popular around there.
It's also worth noting that we now find
that the proportion of respondents
in countries like America, Canada, Australia
and most of Europe actually becomes
quite comparable with each other,
which yes, once again, sort of lines up
with these responses, expectations from the last slide.
Okay, getting into some of the other
demographic information.
The demographic information was new to this survey.
In the 2020 survey, people were asked what they think
of being asked about some demographic information
in a future survey, and the overwhelming response is, "Sure,
I don't really mind." And so that's what we've done here.
One of the ones of somewhat interest
is the age gender breakdown. So we expect Emacs
to be used predominantly among people in software
and programming and within the industry,
I think it's quite widely documented
to have about a sort of 75-25%, roughly, split
between male and female.
Interestingly, in Emacs,
it's a much more aggressively-biased result.
So we had about 96% of respondents are male
with just 4% for the rest. Interestingly, though,
if we look at the young respondents,
say for example, under 25, we go from 96% male to 88%.
So it's fair to say that the young respondents are
in this respect, a somewhat more diverse group.
Hopefully, as future surveys go on,
we'll see this continue not die off
to the sort of well, at this point,
it's more like 99% if you look at the older ages.
But we'll see.
Occupations was an interesting slide as well.
Interesting question as well.
We've got the usual suspects here. I mean,
it's a text editor, well, Lisp machine
masquerading as a text editor, mainly used for programming,
and so we expect lots of software development
and that sort of thing. But that's only about
just over half of the responses.
We've got a huge chunk from academia,
and then really just an odd bag
of all sorts of other things,
including things which you wouldn't really associate
with programming and software at all.
Things like creative writing, publishing, legal, yes.
And then you've got this chunk of Other,
which is I think here is
the fourth most popular option here.
And what we have here is about 500 different responses
from a huge range of activities.
It's really quite interesting to read things like
I think, things like "naval officer",
and just... All sorts of surprising occupations for Emacs.
And I think this is a particular area
because I imagine compared to other code editors,
sort of your VS Code, remember like
that Emacs may have a particularly diverse set
of industry occupations represented in its users.
Now, if you look at where the response actually came from,
we've got the usual suspects up top,
Hacker News and r/emacs.
But then we actually get a much more graduated breakdown
than in the 2020 survey.
We do think familiar results here like IRC, Telegram,
Emacs China, and Twitter.
But now you've got a few new entries,
things like the Fediverse, Discourse, Matrix,
which didn't pop up previously.
So I think this is yes, quite a nice sign in terms of
actually hitting a wide range
of pockets of Emacs users across different platforms,
which bodes well for the potential representiveness
of this survey.
Unsurprisingly, if we're talking about Emacs
and particularly people who are quite engaged in it,
which are the respondents to this survey,
we find that we also get quite a high degree of care
for free and open source software.
So if you have a look here,
only about a quarter of users
didn't express a strong preference towards FOSS software.
In fact, we had over a quarter saying that
they would accept significant or even any compromise
to use a FOSS user software
over a proprietary alternative,
which given the nature of Emacs,
not terribly surprising,
but a strong showing nonetheless.
Now, let's start getting to things
which are actually useful for
potential Emacs development and packaging.
If you're thinking about supporting Emacs versions,
it looks like you can do fantastically well
in terms of hitting most users if you support Emacs 27+.
That hits about 96% of respondents.
Interestingly though, you can actually make an argument
for being even more aggressive.
I mean, if you have a look at Emacs 28+,
that's still over three quarters of respondents.
We've got, at this point, a quarter
using the unreleased HEAD version,
even though it's getting close to release.
Obviously here, as stated, we're hitting
a sort of more engaged with the community
subset of Emacs users, but still,
I think it's interesting to see that
with Emacs's increasingly frequent update schedule,
that users are actually picking up those updates
quite promptly as they roll out.
Continuing on with how people actually use Emacs: languages.
We've got the usual suspects here: lots of Python,
quite a bit of JavaScript and C, lots of shell.
What I find quite interesting though is
if we actually bring in
the 2020 Stack Overflow language usage survey data,
and that maps quite well
to the array of language options we provided here.
They had a general Lisp option,
which I've folded into Common Lisp
since they listed Clojure separately.
I think that seems like a fairly safe bet.
But other than that, the only languages that we missed
are Scheme and Elisp.
What we can do is we can look at
the relative popularity of different languages
from our Emacs user survey compared to Stack Overflows.
What do we find? Well, Clojure and Common Lisp
far above the rest, I imagine in no small part due to
the fantastic SLIME and Cider packages.
Following that, we see Haskell being particularly prominent,
and then a collection of other languages,
your Erlang, Elixir, Julia, Perl and the rest.
And then lastly, if we have a look at the ones
which have significantly diminished popularity
compared to Stack Overflow, we end up with, I think,
what I could probably cast as more enterprising languages.
Things like C#, Java, Typescript and the like.
So, that's interesting. Now, earlier
when we were looking at the split of Emacs users,
we found that we actually had a fair few
in more creative areas, like writing and publishing.
So if looking at prose, we'd expect a decent chunk
to be using Emacs for prose, but it's actually more
than just a little bit, it's a little slice.
We've got a whopping about a third of users
saying they frequently use Emacs for writing prose.
I'd imagine that the availability
of things like Org mode and AUCTeX
probably help like this.
Moving on to other packages, or more packages,
we've actually got a very similar split here
to the 2020 survey. Org has seen a bit of a growth
in popularity. We've got some new arrivals here as well.
For example, Vertico has popped onto the scene
and overtaken Ivy here, along with
a few other new packages like Consult.
Other than that, quite comparable.
What's rather interesting, though, I find here is that
when you have people who listed a small number of packages,
they actually predominantly listed packages
other than the most common set.
So if we have a lot of people who only listed one package,
basically two-thirds of that,
or actually three-quarters of those responses
were saying other packages,
despite the fact that overall packages
other than the highlighted selection here
only constitute a quarter of responses.
So there might be something a bit more to look at there.
Now when people are using packages,
we also asked what types of documentation
people would like to see more of on package READMEs.
Basically we've got a big mix here.
It seems like generally people are interested in
seeing more in various forms, whether it be tutorials,
overviews, screenshots, comparisons, or clips and videos.
So full READMEs with a lot of context
seem to be quite desirable from this.
Now moving forward, what are we going to do?
So 800 people gave some detailed feedback on the survey.
That's quite nice. I'm going to be taking a good read
of all of those responses and use that
to improve the process and also the set of questions.
Now all of you can also give some feedback on the questions,
both that you found most useful in this survey,
ones that you think might not add much value,
and/or new questions
that you think might be a good addition.
Once I've done a bit more analysis,
particularly the more sophisticated analysis
which I'm planning, which will probably come out actually
maybe in the first quarter of next year,
we can see which questions there seem to have provided
the most interesting or surprising results
and those are probably worth keeping.
Lastly, once we actually have an API
and potentially even an Emacs package,
we could automate a large number of the questions,
things like Emacs version, set of packages used,
and that could just streamline the experience
of actually filling out the survey,
make it a bit more frictionless.
Now talking of the question of questions,
a quick survey is a good survey.
If we're asking people to dedicate their time
to fill out this, it's good to try to get as much value
without asking them to donate much of their time.
How has the survey done in this respect?
I'm actually very happy with how it's done.
We get a few comments from the feedback saying
that it was a bit of a long side,
but the median time was about 12 minutes,
which doesn't seem too bad, and most commonly
we saw people completing it in about 8 minutes.
For a once-per-year survey,
I think this seems fairly reasonable.
Getting closer to a 5-10 minute range would be nice,
but this isn't far off.
Lastly, we're also going to be considering
how long the survey is open for.
So from the initial opening date,
what we have here is a plot of
the page which people ended up on
and when they started the survey.
So what we can see is a huge spike in the first few days.
I've just realised that this plot
is actually labelled incorrectly.
Please disregard the minutes to complete the survey.
This should be days after survey opening
that a response is actually submitted.
And what we have here is a big spike
in popularity in the first week basically,
and then it trickles down
to a fairly consistent level after that.
I'm about to publish a last call for survey responses,
so I'll see if any final bump happens,
but this indicates that we can probably just
have the survey open for a week or two
and that should be sufficient.
Alright, so what's the general plan going forwards?
Well, as stated earlier, the idea is to run this annually
and then consistently improve the questions,
the experience, and the analysis that's done.
This year has been the hardest by far
because a lot had to be set up from scratch.
The hope is that moving on from here,
a lot of it can be reused.
For example, with my comments about
more sophisticated analysis being down the line,
once that's all worked out,
as long as nothing changes too drastically,
we should be able to reuse a lot of that work
quite easily in future years.
Alright, that's it for now.
Hopefully, you've found this an interesting peek
into how the survey is operated
and some of the initial results,
and hopefully, I'll see you around next year
for the 2023 survey. Thanks for listening.