path: root/2023/captions/emacsconf-2023-web--emacs-saves-the-web-maybe--yuchen-pei--main.vtt

WEBVTT captioned by ken, checked by sachac

NOTE Overview

00:00:00.000 --> 00:00:03.199
Hello, I am Yuchen, and I will be talking about

00:00:03.200 --> 00:00:06.839
how Emacs may be used to save user freedom on the web.

00:00:06.840 --> 00:00:09.679
I will begin by describing the background issues,

00:00:09.680 --> 00:00:12.359
followed by solutions outside of Emacs.

00:00:12.360 --> 00:00:14.879
Then I will move into the main business of describing

00:00:14.880 --> 00:00:17.799
several ways to address the issues using Emacs,

00:00:17.800 --> 00:00:20.599
including free clients in Emacs, web browsers,

00:00:20.600 --> 00:00:23.399
also known as universal clients in Emacs,

00:00:23.400 --> 00:00:27.119
approaches using Emacs web server and Emacs web framework,

00:00:27.120 --> 00:00:29.319
which allows one to write an Emacs package

00:00:29.320 --> 00:00:30.759
and get a web app for free,

00:00:30.760 --> 00:00:35.679
as well as using Emacs as a Firefox extension.

NOTE Background problems

00:00:35.680 --> 00:00:37.159
OK, let's now move on to

00:00:37.160 --> 00:00:39.559
the background issues for this topic.

00:00:39.560 --> 00:00:42.639
Many of you probably already know what is free software.

00:00:42.640 --> 00:00:45.480
It is software that respects four user freedoms,

00:00:45.481 --> 00:00:48.999
including freedom 0, which is the freedom to use,

00:00:49.000 --> 00:00:52.179
freedom 1 is the freedom to study and modify a program,

00:00:52.279 --> 00:00:53.988
freedom 2 is the freedom to

00:00:54.488 --> 00:00:57.239
distribute exact copies of a program,

00:00:57.240 --> 00:01:01.679
and freedom 3 is the freedom to distribute modified copies.

00:01:01.680 --> 00:01:04.039
Different environments have different norms

00:01:04.040 --> 00:01:06.819
with regards to user freedom.

00:01:06.820 --> 00:01:11.239
For example, GNU/Linux distributions

00:01:11.240 --> 00:01:13.439
default to free software,

00:01:13.440 --> 00:01:15.519
even though the official kernel Linux

00:01:15.520 --> 00:01:18.419
contains non-free code, like non-free firmware.

00:01:18.420 --> 00:01:23.059
What I mean is, people generally expect free software

00:01:23.060 --> 00:01:25.759
in these environments.

00:01:25.760 --> 00:01:27.359
There's plenty of free software

00:01:27.360 --> 00:01:29.039
built on other free software,

00:01:29.040 --> 00:01:31.219
so generally people can accomplish tasks

00:01:31.220 --> 00:01:33.599
using free software only.

00:01:33.600 --> 00:01:37.279
Emacs, by comparison, is even better.

00:01:37.280 --> 00:01:41.219
It has freedom built-in, as it is highly customizable

00:01:41.220 --> 00:01:44.679
with self-documenting configurations.

00:01:44.680 --> 00:01:49.599
When a Lisp form is evaluated by the user in Emacs,

00:01:49.600 --> 00:01:53.159
the change is instantly reflected in the environment.

00:01:53.160 --> 00:01:56.719
Thus, it converts users to hackers effortlessly.

00:01:56.720 --> 00:01:58.439
From writing setq statements,

00:01:58.440 --> 00:02:00.639
which is similar to configurations

00:02:00.640 --> 00:02:01.959
in the majority of other programs,

00:02:01.960 --> 00:02:03.399
to writing functions,

00:02:03.400 --> 00:02:05.439
which are building blocks of Elisp features,

00:02:05.440 --> 00:02:08.139
to writing features and publishing packages,

00:02:08.140 --> 00:02:09.799
it is a natural progression.

00:02:10.099 --> 00:02:15.039
In this sense, Emacs perhaps has

00:02:15.040 --> 00:02:18.839
the most gentle learning curve for hackers.

00:02:18.840 --> 00:02:21.099
On the other hand, the default license

00:02:21.100 --> 00:02:22.599
in the Emacs community

00:02:22.600 --> 00:02:26.100
is GNU General Public License version 3 or later,

00:02:26.200 --> 00:02:29.039
which is the best free software license

00:02:29.040 --> 00:02:32.299
apart from the Affero license.

00:02:32.300 --> 00:02:35.019
Now let's move on to web browsers,

00:02:35.020 --> 00:02:39.239
which by contrast does not default to freedom.

00:02:39.240 --> 00:02:42.199
For one thing, free software JavaScript projects

00:02:42.200 --> 00:02:45.779
default to Expat license,

00:02:45.780 --> 00:02:49.399
which is also commonly known as the MIT license,

00:02:49.400 --> 00:02:53.279
which is a lax permissive license that could be exploited

00:02:53.280 --> 00:02:55.919
as developers could write non-free derivatives

00:02:55.920 --> 00:02:59.679
and subjugate user freedom.

00:02:59.680 --> 00:03:03.159
This also contributes to the JavaScript trap.

00:03:03.160 --> 00:03:06.719
Most popular web browsers nowadays simply download and run

00:03:06.720 --> 00:03:10.819
any JavaScript code requested by the web page.

00:03:10.820 --> 00:03:15.319
Generally speaking, there are two camps on this issue.

00:03:15.320 --> 00:03:19.039
One side would say JavaScript is simply part of life,

00:03:19.040 --> 00:03:22.039
and an integral part of the so-called modern web.

00:03:22.040 --> 00:03:25.299
Just accept it, and there is no point in fighting it.

00:03:25.300 --> 00:03:28.288
Indeed, it can be frustrating when greeted by

00:03:28.388 --> 00:03:31.799
"This page requires JavaScript and cookies to continue,"

00:03:31.800 --> 00:03:34.719
or even a blank page when opening a web page

00:03:34.720 --> 00:03:38.439
while disabling JavaScript.

00:03:38.440 --> 00:03:42.159
The other camp takes a more principled position

00:03:42.160 --> 00:03:44.839
and says JavaScript is unnecessary.

00:03:44.840 --> 00:03:47.279
I mean, people use the web mainly for

00:03:47.280 --> 00:03:48.519
database-like operations

00:03:48.520 --> 00:03:51.679
to interact with data stored on other people's computers,

00:03:51.680 --> 00:03:55.359
like querying, creating, updating, deleting.

00:03:55.360 --> 00:03:58.959
I mean, 99% of the things happen in getting data,

00:03:58.960 --> 00:04:01.239
including reading news, watching videos,

00:04:01.240 --> 00:04:03.339
downloading images, etc.,

00:04:03.340 --> 00:04:06.079
and posting data, including publishing

00:04:06.080 --> 00:04:10.479
this sort of materials, publishing news comments, videos.

00:04:10.480 --> 00:04:12.399
Why does this need any programs

00:04:12.400 --> 00:04:16.199
to do funny computations, right?

00:04:16.200 --> 00:04:18.980
Modern web browsers are also a pain to use.

00:04:19.080 --> 00:04:20.980
They are the opposite to Emacs

00:04:21.080 --> 00:04:26.759
in terms of customization capabilities.

00:04:26.760 --> 00:04:29.359
Such problems on the client side

00:04:29.360 --> 00:04:31.919
is the main focus of this talk.

00:04:31.920 --> 00:04:34.319
On the server side, the issue is known as SaaSS,

00:04:34.320 --> 00:04:38.460
service as a software substitute.

00:04:38.760 --> 00:04:42.420
It is about doing computing for users

00:04:42.421 --> 00:04:44.540
on other people's computers,

00:04:44.541 --> 00:04:48.439
which the user has no visibility, let alone control.

00:04:48.440 --> 00:04:51.940
Examples include translation or photo editing

00:04:51.941 --> 00:04:55.359
in so-called web applications.

00:04:55.360 --> 00:04:59.919
Another example would be web applications

00:04:59.920 --> 00:05:02.159
make recommendations based on user data

00:05:02.160 --> 00:05:05.959
and suggest what the users read or watch next.

00:05:05.960 --> 00:05:09.959
On the one hand, SaaSS is an intractable problem

00:05:09.960 --> 00:05:11.799
because free software is all about user freedom

00:05:11.800 --> 00:05:13.759
on one's own computer,

00:05:13.760 --> 00:05:16.079
not someone else's computer.

00:05:16.080 --> 00:05:18.780
On the other hand, this is also a lesser problem

00:05:18.880 --> 00:05:21.599
because it has trivial solutions,

00:05:21.600 --> 00:05:25.839
which is self-hosting and keeping computations local.

00:05:25.840 --> 00:05:28.679
Wouldn't it be nice to use a photo editing web application,

00:05:28.680 --> 00:05:31.939
but without the web?

NOTE Solutions outside of Emacs

00:05:31.940 --> 00:05:36.400
Right, now let's move on to solutions outside of Emacs

00:05:36.401 --> 00:05:39.039
that tackle these problems.

00:05:39.040 --> 00:05:42.959
There are generally two ways to fix this issue.

00:05:42.960 --> 00:05:45.399
One is blocking non-free JavaScript,

00:05:45.400 --> 00:05:48.979
and the other is substituting with free programs.

00:05:48.980 --> 00:05:50.439
Let's start with blocking.

00:05:50.440 --> 00:05:54.859
LibreJS, for example, is a Firefox extension

00:05:54.860 --> 00:05:56.919
blocking non-free, non-trivial JavaScript.

00:05:56.920 --> 00:05:59.820
It works by intercepting, filtering

00:05:59.821 --> 00:06:01.759
all requests for JavaScript,

00:06:01.760 --> 00:06:05.599
recognizing the ones that are trivial or free,

00:06:05.600 --> 00:06:10.999
and blocking the execution of the others.

00:06:11.000 --> 00:06:13.879
As an experiment, I logged the LibreJS output

00:06:13.880 --> 00:06:15.559
for about two weeks,

00:06:15.560 --> 00:06:19.739
and during which, of all the web pages I loaded,

00:06:19.740 --> 00:06:25.000
23 domains have at least some LibreJS-compliant scripts.

00:06:25.001 --> 00:06:28.679
That is not much, though I did use other means

00:06:28.680 --> 00:06:30.819
to reduce the scenarios

00:06:30.920 --> 00:06:35.399
where I need to load web pages with JavaScript in Firefox,

00:06:35.400 --> 00:06:40.719
like using a text browser like Lynx.

00:06:40.720 --> 00:06:44.239
Then there's also NoScript, which is like LibreJS,

00:06:44.240 --> 00:06:49.499
but it blocks all scripts, whether free or non-free,
trivial or non-trivial.

00:06:49.500 --> 00:06:54.359
So the problem with blocking is that

00:06:54.360 --> 00:06:57.559
blocking with certain scripts and accepting others,

00:06:57.560 --> 00:07:00.579
there are like... I can think of two problems.

00:07:00.679 --> 00:07:02.779
One is that it does not help with Freedom 1,

00:07:02.879 --> 00:07:07.959
which is the freedom to allow users to modify a program

00:07:07.960 --> 00:07:13.079
and use it in place of the original program.

00:07:13.080 --> 00:07:15.839
And also it does not help

00:07:15.840 --> 00:07:18.859
when the non-free JavaScript is mandatory

00:07:18.860 --> 00:07:20.719
for the functioning of the web page.

00:07:20.720 --> 00:07:22.839
For example, some pages are blank

00:07:22.840 --> 00:07:27.079
when non-free JavaScript is not executed.

00:07:27.080 --> 00:07:35.180
So now let's move on to the substitution, the other method.

00:07:36.280 --> 00:07:38.919
Let's start with userscript.

00:07:38.920 --> 00:07:41.760
It is a script, it is a user-specified JavaScript

00:07:41.761 --> 00:07:43.039
injected to a web page.

00:07:43.040 --> 00:07:48.480
A typical example of userscript tool is GreaseMonkey.

00:07:48.481 --> 00:07:53.159
Another idea is a proxy that replaces scripts in place,

00:07:53.160 --> 00:07:55.819
that is, sending user-specified scripts

00:07:55.919 --> 00:08:00.899
as a response to requests for such scripts.

00:08:00.900 --> 00:08:04.759
So one example would be Haketilo, however you pronounce it.

00:08:04.760 --> 00:08:09.619
It's a tool that's built on top of mitmproxy.

00:08:09.620 --> 00:08:11.719
It is supposed to do this.

00:08:11.720 --> 00:08:14.599
I haven't used GreaseMonkey nor Haketilo

00:08:14.600 --> 00:08:16.599
for these purposes yet,

00:08:16.600 --> 00:08:20.779
so I can't say much about these options.

00:08:20.780 --> 00:08:24.359
So then there are also free clients

00:08:24.360 --> 00:08:26.479
which replace the whole frontend,

00:08:26.480 --> 00:08:30.660
instead of a script requested by web pages

00:08:30.661 --> 00:08:32.499
from the official web clients.

00:08:32.500 --> 00:08:37.359
People often refer to them as alternative frontend.

00:08:37.360 --> 00:08:39.359
YouTube is perhaps the best example

00:08:39.360 --> 00:08:41.279
as there are so many free clients,

00:08:41.280 --> 00:08:43.621
including Invidious for the web,

00:08:43.622 --> 00:08:46.239
youtube-dl and yt-dlp on the command line,

00:08:46.240 --> 00:08:50.279
MPV and VLC as GUI desktop, LibreTube

00:08:50.280 --> 00:08:53.259
and NewPipe for Android and so on.

00:08:53.260 --> 00:08:56.759
Youtube-dl and yt-dlp are especially versatile

00:08:56.760 --> 00:08:59.459
as they work with many video and audio sites

00:08:59.460 --> 00:09:02.520
with extractors written in Python,

00:09:02.620 --> 00:09:06.299
so people can add extractors like extensions.

00:09:06.300 --> 00:09:09.421
A similar tool would be woob,

00:09:09.422 --> 00:09:12.739
short for web outside of the browsers.

00:09:12.740 --> 00:09:16.820
It is a command-line and GUI program

00:09:16.920 --> 00:09:23.199
that interacts with many web services, even banks.

00:09:23.200 --> 00:09:25.839
And there are browser extensions

00:09:25.840 --> 00:09:28.859
that automatically redirect to these clients.

00:09:28.860 --> 00:09:31.639
For example, Redirector and Libredirect

00:09:31.640 --> 00:09:35.199
redirect to the free web clients.

00:09:35.200 --> 00:09:39.699
One could use OpenWith, another extension,

00:09:39.700 --> 00:09:42.159
to redirect to free non-web clients,

00:09:42.160 --> 00:09:46.380
for example by opening YouTube links with MPV.

NOTE Emacs solutions

00:09:46.480 --> 00:09:50.999
Now let us move to Emacs-based solutions.

00:09:51.000 --> 00:09:54.599
They are based on the same ideas but using Emacs.

NOTE Free clients in Emacs

00:09:54.600 --> 00:09:57.479
First, free clients in Emacs.

00:09:57.480 --> 00:10:00.639
Basically alternative frontends written in Elisp.

00:10:00.640 --> 00:10:03.359
There are several advantages.

00:10:03.360 --> 00:10:06.199
For example, integration with other Emacs tools,

00:10:06.200 --> 00:10:09.559
good for archiving, making use of Emacs libraries,

00:10:09.560 --> 00:10:12.488
extensibility, thanks to Emacs' own

00:10:12.489 --> 00:10:14.900
extensibility and customizability.

00:10:15.000 --> 00:10:18.619
Examples include mastodon.el for mastodon,

00:10:18.620 --> 00:10:22.679
or mastorg for viewing and archiving toots with org,

00:10:22.680 --> 00:10:28.899
sx for Stack Exchange, buildbot.el for buildbot, etc.

00:10:28.900 --> 00:10:31.900
Here's an example of mastorg displaying

00:10:31.901 --> 00:10:34.420
the hierarchy of a toot in org.

00:10:34.520 --> 00:10:39.820
Just wait. Right.

00:10:39.920 --> 00:10:43.900
So this is the toot itself, this is a first reply,

00:10:44.000 --> 00:10:48.479
this is a reply to the reply, and so on.

00:10:48.480 --> 00:10:53.079
And here is an example of

00:10:53.080 --> 00:11:05.719
opening a Stack Exchange link using sx.

00:11:05.720 --> 00:11:07.020
Let's check out the tag.

00:11:11.120 --> 00:11:28.399
So we can browse the Stack Exchange Emacs site
with ease.

00:11:28.400 --> 00:11:31.079
The idea is quite simple.

00:11:31.080 --> 00:11:35.620
Just use APIs to get data and display it in Emacs,

00:11:35.720 --> 00:11:40.819
or just to scrape, like requesting HTML and processing it.

00:11:40.820 --> 00:11:42.079
An example of scraping is hnreader,

00:11:44.180 --> 00:11:47.199
which scrapes Hacker News web pages

00:11:47.299 --> 00:11:49.779
and renders them in Org buffers.

00:11:49.780 --> 00:11:52.379
Here's how hnreader fetches

00:11:52.380 --> 00:11:56.319
and displays the Hacker News front page.

00:11:58.520 --> 00:12:03.999
And one could go into the comments,

00:12:04.000 --> 00:12:09.159
which shows a similar hierarchy to mastorg's output.

00:12:14.360 --> 00:12:19.000
And of course, there are limitations for this method,

00:12:19.001 --> 00:12:22.539
which is not limited to Emacs.

00:12:22.540 --> 00:12:24.521
There are basically limitations

00:12:24.522 --> 00:12:28.419
to any ad hoc bespoke clients,

00:12:28.420 --> 00:12:31.519
which is catch-up games with remote server,

00:12:31.520 --> 00:12:34.559
which may change the API interface endpoints

00:12:34.560 --> 00:12:37.539
or even structure of the responses.

00:12:37.540 --> 00:12:43.020
This brings us to web browsers in Emacs.

NOTE Web browsers in Emacs

00:12:43.021 --> 00:12:45.159
Web browsers are universal clients

00:12:45.160 --> 00:12:47.199
because all sites support browsers.

00:12:47.200 --> 00:12:48.919
So in a world of no JavaScript,

00:12:48.920 --> 00:12:52.739
there will be no need to write bespoke clients.

00:12:52.740 --> 00:12:53.479
In such a world,

00:12:53.480 --> 00:12:56.739
instead of using JavaScript code to fetch JSON,

00:12:56.740 --> 00:13:00.119
web developers make server do the heavy lifting

00:13:00.120 --> 00:13:02.859
and just send the complete HTML over.

00:13:02.860 --> 00:13:05.479
Okay, back to reality.

00:13:05.480 --> 00:13:07.659
EWW, the default Emacs browser,

00:13:07.660 --> 00:13:11.379
is what people refer to as a text browser,

00:13:11.380 --> 00:13:16.899
even though it is not text only and it supports images too.

00:13:16.900 --> 00:13:20.679
It is a good solid browser that supports forms, etc.

00:13:20.680 --> 00:13:24.079
The downside is that it does not support CSS,

00:13:24.080 --> 00:13:28.159
so the formatting could be a bit ugly sometimes.

00:13:28.160 --> 00:13:30.119
There are some other browsers in Emacs too,

00:13:30.120 --> 00:13:34.279
like emacs-w3m, which is backed by w3m,

00:13:34.280 --> 00:13:36.439
and Luwak, which is backed by Lynx.

00:13:36.440 --> 00:13:39.099
Sorry for the naming, by the way.

00:13:39.100 --> 00:13:41.519
They often consist of a backend

00:13:41.520 --> 00:13:44.879
that fetches URL and parses HTML.

00:13:44.880 --> 00:13:47.199
For example, the built-in URL package

00:13:47.200 --> 00:13:50.599
and the libxml2 binding in Emacs are decent enough.

00:13:50.600 --> 00:13:53.188
And the frontend that renders the HTML,

00:13:53.189 --> 00:13:56.599
like shr or lynx, etc.

00:13:56.699 --> 00:14:04.739
There is also an xwidget-webkit,

00:14:04.740 --> 00:14:07.759
but this browser executes JavaScript,

00:14:07.760 --> 00:14:10.539
so it does not really help in this case.

00:14:10.540 --> 00:14:14.239
Browser extensions on Emacs are effortless,

00:14:14.240 --> 00:14:17.459
as they can be written as Emacs packages.

00:14:17.460 --> 00:14:19.279
For example, one could easily write

00:14:19.280 --> 00:14:21.959
Elisp scripts with similar functionalities

00:14:21.960 --> 00:14:24.921
to libredirect and openwith

00:14:24.922 --> 00:14:29.881
to redirect links, to rewrite URLs,

00:14:30.181 --> 00:14:36.860
or to open, say, a YouTube URL with MPV,

00:14:37.061 --> 00:14:39.700
but with even more flexibility.

00:14:39.800 --> 00:14:41.779
For example, here's how one could

00:14:41.780 --> 00:14:44.839
transform a Zoom link to a dial-in number

00:14:44.840 --> 00:14:47.479
so that it is easier to join a Zoom meeting

00:14:47.480 --> 00:14:50.359
without running non-free JavaScript.

00:14:50.360 --> 00:14:53.039
This might still be bad for privacy,

00:14:53.040 --> 00:14:55.999
but at least it's good for freedom.

00:14:58.699 --> 00:15:00.279
As mentioned before,

00:15:00.379 --> 00:15:03.919
one shortcoming of these Emacs-based browsers,

00:15:03.920 --> 00:15:08.079
Emacs web browsers, is no support for CSS,

00:15:08.080 --> 00:15:11.319
so the formatting could leave a lot to be desired.

00:15:11.320 --> 00:15:12.959
Maybe someone would write

00:15:12.960 --> 00:15:17.159
an Emacs browser package backed by wkhtmltopdf,

00:15:17.160 --> 00:15:20.639
which, when opening a URL,

00:15:20.640 --> 00:15:26.380
it calls wkhtmltopdf to convert the web page to PDF

00:15:26.480 --> 00:15:29.540
and opens in, say, pdf-view-mode of the pdf-tools,

00:15:29.640 --> 00:15:31.039
thus containing formatting,

00:15:31.040 --> 00:15:33.999
and all the URL clicks resolve to the same actions.

00:15:34.000 --> 00:15:42.399
Also, wkhtmltopdf contains a flag that disables JavaScript.

00:15:43.300 --> 00:15:45.239
Another idea would be to use Firefox

00:15:45.240 --> 00:15:49.679
as a processor to fetch URLs.

00:15:50.280 --> 00:15:54.559
Maybe it can be used to pass back the HTML

00:15:54.560 --> 00:15:56.519
after executing free JavaScript,

00:15:56.520 --> 00:16:01.439
say, if Firefox has LibreJS installed.

00:16:01.440 --> 00:16:04.940
This requires Firefox to send back the DOM,

00:16:05.040 --> 00:16:08.039
which could be achieved using native messaging.

00:16:08.040 --> 00:16:09.719
More on that later.

00:16:09.720 --> 00:16:14.239
Alternatively, one could also write a Firefox extension

00:16:14.240 --> 00:16:17.639
that sends the DOM in an existing tab back to Emacs.

00:16:17.640 --> 00:16:20.079
But thinking more about it,

00:16:20.080 --> 00:16:22.959
I don't think this is actually a useful idea,

00:16:23.059 --> 00:16:27.039
because most of the sites that work under LibreJS

00:16:27.139 --> 00:16:34.419
also are useful when all JavaScript is blocked.

00:16:34.420 --> 00:16:37.039
So, this means these sites are viewable

00:16:37.040 --> 00:16:42.159
under EWW, Luwak, etc.

00:16:42.160 --> 00:16:43.639
And another issue is that

00:16:43.640 --> 00:16:46.559
this could also make running non-free JavaScript easier,

00:16:46.560 --> 00:16:52.379
which is harmful to user freedom.

NOTE emacs-web-server - overview

00:16:52.380 --> 00:16:54.239
OK, let's move on to the idea

00:16:54.240 --> 00:16:55.679
of running Emacs as a web server,

00:16:55.680 --> 00:16:58.559
so that Emacs client packages are web apps

00:16:58.560 --> 00:17:00.319
serving as alternative frontends.

00:17:00.320 --> 00:17:02.239
Why would we want to do this?

00:17:02.240 --> 00:17:06.079
Well, as much as one wants to be always in Emacs,

00:17:06.080 --> 00:17:08.339
it is not always feasible.

00:17:08.340 --> 00:17:10.719
For example, one may be on the go

00:17:10.720 --> 00:17:12.519
and needs to look up something on the phone.

00:17:12.520 --> 00:17:14.879
On the other hand, Emacs client packages

00:17:14.880 --> 00:17:16.159
are just alternative frontends

00:17:16.160 --> 00:17:18.119
but written in Elisp and run in Emacs.

00:17:18.120 --> 00:17:20.759
With the help of emacs-web-server package,

00:17:20.760 --> 00:17:23.579
we can access Emacs packages on the web.

00:17:23.580 --> 00:17:26.439
emacs-web-server package is not something new,

00:17:26.440 --> 00:17:30.379
but seems to be underused in the community somehow.

NOTE emacs-web-server - hello emacs!

00:17:30.380 --> 00:17:33.359
OK, let's start with a simple example called hello-emacs.

00:17:33.360 --> 00:17:35.239
It is pretty straightforward.

00:17:35.240 --> 00:17:38.639
Just require the web server feature

00:17:38.640 --> 00:17:40.999
and run ws-start to start a server process

00:17:41.000 --> 00:17:43.359
and send the string "hello emacs"

00:17:43.360 --> 00:17:45.539
to the process regardless of the request.

00:17:45.540 --> 00:17:48.479
As you can see, it is going to be available

00:17:48.480 --> 00:17:51.219
at port 9000 of localhost.

00:17:51.319 --> 00:17:52.999
Let's try it out.

00:17:53.000 --> 00:18:01.839
We need to first evaluate this code block.

00:18:01.840 --> 00:18:03.939
And it works.

00:18:03.940 --> 00:18:10.839
To stop a server, just run ws-stop on the web server object.

00:18:10.840 --> 00:18:14.959
Let's evaluate.

00:18:14.960 --> 00:18:17.579
Yep, it stopped.

NOTE emacs-web-server - yolo

00:18:17.580 --> 00:18:19.999
OK, now let's move on to something funny

00:18:20.000 --> 00:18:22.219
that you should never run on the public web.

00:18:22.220 --> 00:18:23.919
I call it yolo.el.

00:18:23.920 --> 00:18:25.359
It uses htmlize

00:18:25.360 --> 00:18:28.159
to make any Emacs buffer available on the web.

00:18:28.160 --> 00:18:28.999
Let's try it out.

00:18:29.000 --> 00:18:32.999
Just require the thing and start the server by yolo-start.

00:18:33.000 --> 00:18:38.119
And it's available at port 9999.

00:18:38.120 --> 00:18:41.599
By default, the root domain shows the splash screen

00:18:41.600 --> 00:18:42.919
which needs to be available.

00:18:42.920 --> 00:18:46.719
Running display-splash-screen ensures that,

00:18:47.219 --> 00:18:48.839
but here I've already run it.

00:18:48.939 --> 00:18:54.359
So let's have a look.

00:18:54.560 --> 00:18:56.639
And here we have the splash screen.

00:18:56.640 --> 00:19:00.239
Emacs tutorial and such.

00:19:00.240 --> 00:19:03.279
Unfortunately, none of these links work,

00:19:05.480 --> 00:19:08.000
which is something we will revisit later.

00:19:10.000 --> 00:19:15.381
So, to show an arbitrary buffer,

00:19:15.481 --> 00:19:19.981
just use the buffer name as a path.

00:19:20.081 --> 00:19:24.761
For example, the slide has the buffer named web.org,

00:19:24.861 --> 00:19:28.080
so we can display it.

00:19:34.581 --> 00:19:36.540
Let's try something fancier,

00:19:36.941 --> 00:19:40.000
like the man page of ffmpeg.

00:19:40.880 --> 00:19:44.719
So this is the man page of ffmpeg.

00:19:45.120 --> 00:19:48.420
And the buffer name is a bit more complicated.

00:19:48.520 --> 00:19:51.639
I have the URL available here.

00:19:59.140 --> 00:20:05.979
It's missing a star.

00:20:05.980 --> 00:20:10.659
It's pretty neat if you ask me.

00:20:12.560 --> 00:20:14.879
And, yeah, what else?

00:20:14.880 --> 00:20:22.699
Well, we can also browse EWW in Firefox.

00:20:22.700 --> 00:20:30.599
For example, let's check out gnu.org,

00:20:30.600 --> 00:20:33.679
and note that the buffer name is EWW with stars.

00:20:39.080 --> 00:20:41.879
So, ah, it works.

00:20:41.979 --> 00:20:50.899
And it has all the graphics even.

00:20:50.900 --> 00:20:55.639
Now, how about we do it the other way around?

00:20:55.640 --> 00:21:10.779
So we load the current slide web.org using this funny thing.

00:21:10.780 --> 00:21:12.239
And it works.

00:21:14.040 --> 00:21:19.939
Not as nice as the Org buffer, though.

00:21:19.940 --> 00:21:27.439
Right, and now that gives me some funny idea.

00:21:27.440 --> 00:21:31.359
So I'm a firm believer that memes are meant to be enjoyed

00:21:31.360 --> 00:21:33.199
in silence rather than read out loud.

00:21:33.200 --> 00:21:38.759
So I will jump straight to trying this idea,

00:21:38.760 --> 00:21:48.959
which is loading the EWW buffer URL with EWW itself.

00:21:49.860 --> 00:21:53.839
Loading, loading, loading.

00:21:53.840 --> 00:21:56.199
Spoiler alert, it never loads.

00:21:59.100 --> 00:22:03.120
So that concludes the demo.

00:22:03.220 --> 00:22:06.439
And so we can stop the server, web server, with `yolo-stop`.

00:22:06.440 --> 00:22:13.399
So one could extend yolo to serve arbitrary Emacs commands,

00:22:13.400 --> 00:22:15.439
making it even more dangerous.

00:22:15.440 --> 00:22:26.019
That is, for example, `localhost:9000/m-x/magit-status`

00:22:26.119 --> 00:22:27.720
would run `magit-status`

00:22:27.820 --> 00:22:33.499
and show the magit-status buffer in the web browser.

00:22:34.500 --> 00:22:43.119
Or localhost:9000/m-x/eww/

00:22:43.120 --> 00:22:46.759
any arbitrary URL to browse arbitrary URL

00:22:46.760 --> 00:22:50.819
with EWW inside of Firefox.

00:22:50.820 --> 00:22:53.879
It can serve as a way to block all JavaScript,

00:22:53.880 --> 00:22:56.799
because EWW does not support JavaScript.

00:22:56.800 --> 00:23:00.079
And enforce preferred colorscheme in Firefox,

00:23:00.080 --> 00:23:02.839
since htmlize, as you have noticed,

00:23:02.840 --> 00:23:06.439
faithfully reflects the theme used in Emacs.

NOTE emacs-web-server - emacs web framework

00:23:07.940 --> 00:23:10.239
Okay, so we know that yolo is unsafe

00:23:10.339 --> 00:23:11.440
and needs to be refined.

00:23:11.540 --> 00:23:13.439
In fact, we don't necessarily want

00:23:13.440 --> 00:23:15.599
to run Emacs on a web browser.

00:23:15.600 --> 00:23:17.279
After all, a modern web browser is

00:23:17.280 --> 00:23:19.079
something one has to fight all the time

00:23:19.080 --> 00:23:21.600
and should be avoided whenever possible.

00:23:21.601 --> 00:23:24.479
We want to instead be able to access things

00:23:24.480 --> 00:23:26.459
when forced to be in a web browser,

00:23:26.460 --> 00:23:28.359
in which case only the motivations

00:23:28.360 --> 00:23:31.299
of an alternative frontend apply.

00:23:31.300 --> 00:23:35.360
Moreover, the ideal situation is an Emacs web framework,

00:23:35.460 --> 00:23:36.799
a tool that automatically

00:23:36.800 --> 00:23:39.199
transforms Emacs packages to web apps,

00:23:39.200 --> 00:23:41.799
so that one does not need to write extra code

00:23:41.800 --> 00:23:45.559
to get a web app that does the same thing as the package.

00:23:45.560 --> 00:23:49.099
We also need all links in the web pages to work.

00:23:49.100 --> 00:23:52.399
As noted before, the links on the yolo Emacs splash screen

00:23:52.400 --> 00:23:53.839
do not work.

00:23:53.840 --> 00:23:58.199
So here's a proof-of-concept example. It's called bom.el.

00:23:58.200 --> 00:24:00.119
It gets some weather forecast data

00:24:00.120 --> 00:24:03.079
from the Australian Bureau of Meteorology

00:24:03.080 --> 00:24:05.559
and displays it in an org buffer.

00:24:05.560 --> 00:24:09.279
So let's try it out. One could do `M-x bom`,

00:24:09.280 --> 00:24:15.219
which shows an org buffer with links to each state.

00:24:15.220 --> 00:24:17.199
So based in Melbourne, naturally,

00:24:17.200 --> 00:24:21.839
I would like to find out the weather of Victoria.

00:24:21.840 --> 00:24:27.839
And yes, to execute this command. Wait, wait, wait. Right.

00:24:27.840 --> 00:24:33.459
And we are at a buffer that shows

00:24:33.460 --> 00:24:36.119
the weather forecast of the whole of Victoria

00:24:36.120 --> 00:24:39.379
in the hierarchy. Note that this back button

00:24:39.479 --> 00:24:46.639
takes you to the previous page.

00:24:46.640 --> 00:24:47.919
So here are the regions of Victoria.

00:24:47.920 --> 00:24:53.799
I think Melbourne is in Central.

00:24:53.800 --> 00:24:54.719
And yeah, it shows

00:24:54.720 --> 00:24:57.259
the seven-day weather forecast of Melbourne.

00:24:57.260 --> 00:25:00.359
You can also reach this page by running,

00:25:00.360 --> 00:25:08.199
let's see, directly `M-x bom-state`.

00:25:08.600 --> 00:25:09.759
Vic.

00:25:13.960 --> 00:25:18.399
OK. So this works.

00:25:18.400 --> 00:25:21.280
And this is bom as an Emacs package.

00:25:21.380 --> 00:25:23.980
Now let's check out bom as a web app

00:25:23.981 --> 00:25:28.039
transformed by Emacs web framework.

00:25:28.040 --> 00:25:30.319
So start the web server with bom-start.

00:25:33.020 --> 00:25:39.559
And let's try it out. It's at 9000 again.

00:25:39.560 --> 00:25:42.359
Oops. Invalid path. Oh, that's because

00:25:42.360 --> 00:25:46.119
it makes exactly one command to one path.

00:25:46.120 --> 00:25:49.300
So remember that we used the bom command

00:25:49.301 --> 00:25:50.719
to show the landing page.

00:25:50.720 --> 00:25:54.340
So here we need the bom in the path as well.

00:25:54.440 --> 00:26:00.679
And it shows the same landing page, except in HTML.

00:26:00.680 --> 00:26:07.259
Let's check out Victoria weather forecast as before.

00:26:07.260 --> 00:26:12.279
And it shows an HTML converted from the org buffer

00:26:12.280 --> 00:26:17.559
using ox export HTML, whatever.

00:26:17.560 --> 00:26:20.259
And you can see even the back button is here.

00:26:20.359 --> 00:26:26.219
That takes you to /bom.

00:26:26.220 --> 00:26:29.139
So let's have a look at Melbourne. Here it is.

00:26:29.140 --> 00:26:31.379
Hooray, it works.

00:26:31.380 --> 00:26:33.860
So, yeah, as usual,

00:26:33.960 --> 00:26:40.559
you can stop the web server with `M-x bom-stop`.

00:26:40.560 --> 00:26:43.660
Right. And alternatively,

00:26:43.760 --> 00:26:48.499
it can also be deployed directly in terminal

00:26:48.500 --> 00:26:56.099
in a dedicated Emacs daemon.

00:26:56.100 --> 00:26:58.279
So you can see that there's a one-one correspondence

00:26:58.280 --> 00:27:03.099
between the Emacs package interface and the web interface.

00:27:03.100 --> 00:27:06.039
And that implies some restrictions to the Emacs package

00:27:06.040 --> 00:27:09.159
for the Emacs web framework to be able to do its job. Right.

00:27:09.160 --> 00:27:13.439
For example, the package needs to have an Org interface

00:27:13.440 --> 00:27:15.519
and the links that trigger other commands

00:27:15.520 --> 00:27:17.799
need to be in Elisp links

00:27:17.800 --> 00:27:20.759
so that the Emacs web framework

00:27:20.760 --> 00:27:24.799
can translate it to web server URL path.

00:27:24.800 --> 00:27:28.919
Note that Emacs web server framework is not a real package.

00:27:28.920 --> 00:27:33.339
I wrote some functions in bom.el serving the purpose,

00:27:33.340 --> 00:27:35.719
and they should be separated out eventually

00:27:35.720 --> 00:27:37.759
without much trouble.

00:27:37.760 --> 00:27:39.999
One could get weather forecast

00:27:40.000 --> 00:27:42.219
without running JavaScript anyway,

00:27:42.220 --> 00:27:45.199
which makes bom.el less important

00:27:45.200 --> 00:27:48.319
as an alternative web client.

00:27:48.320 --> 00:27:50.519
Though it does provide, dare I say,

00:27:50.520 --> 00:27:52.380
a clean and minimal interface

00:27:52.480 --> 00:27:55.719
compared to common weather forecast web pages.

00:27:55.720 --> 00:27:58.639
Other more relevant use cases could be Mastodon,

00:27:58.640 --> 00:28:01.319
whose official web client requires JavaScript

00:28:01.320 --> 00:28:03.479
to display a post.

00:28:03.480 --> 00:28:08.559
The mastorg package that shows an Org hierarchy of toots

00:28:08.560 --> 00:28:12.079
rooted as a given toot could be a low-hanging fruit.

00:28:12.179 --> 00:28:15.199
The limitation of Org interface requirements

00:28:15.200 --> 00:28:17.879
can also be relaxed in further work,

00:28:17.880 --> 00:28:21.639
if one could extend Emacs web framework

00:28:21.640 --> 00:28:24.199
to translate back and forth between Emacs widgets,

00:28:24.200 --> 00:28:28.639
say, including buttons and web page widgets,

00:28:28.640 --> 00:28:30.599
including links.

00:28:30.600 --> 00:28:32.599
Another more far-fetched idea would be

00:28:32.600 --> 00:28:35.799
to translate to other types of interfaces,

00:28:35.800 --> 00:28:42.120
like GNU/Linux or Android GUI.

00:28:44.020 --> 00:28:47.479
How about animations? Say, M-x butterfly,

00:28:47.480 --> 00:28:53.999
or even web games from Emacs games?

00:28:54.000 --> 00:29:00.099
Possibilities are unlimited in this, as always, in Emacs.

00:29:00.100 --> 00:29:03.159
I also noticed some limitations

00:29:03.160 --> 00:29:07.439
when trying to actually host bom.el on the public web.

00:29:07.440 --> 00:29:12.939
Given the limited access to the Emacs server,

00:29:13.540 --> 00:29:16.719
I was comfortable enough to give bom.el a go

00:29:16.720 --> 00:29:18.799
to serve it on the public web.

00:29:18.800 --> 00:29:20.559
However, I immediately stopped

00:29:20.560 --> 00:29:22.879
after noticing how slow it is.

00:29:22.880 --> 00:29:24.719
It can take more than 30 seconds

00:29:24.720 --> 00:29:27.839
to load a page of weather forecast for a state.

00:29:27.840 --> 00:29:30.999
I am also not sure how many simultaneous connections

00:29:31.000 --> 00:29:32.379
it can handle.

00:29:32.380 --> 00:29:36.439
In any case, I think the package emacs-web-server

00:29:36.440 --> 00:29:40.419
could do with some performance enhancement.

NOTE Firefox with emacs for extensions

00:29:40.420 --> 00:29:43.999
Right. Because of the time constraints,

00:29:44.000 --> 00:29:45.759
I will briefly touch one final idea,

00:29:45.760 --> 00:29:50.320
which is to use Emacs as a Firefox browser extension.

00:29:50.420 --> 00:29:52.800
We already have org-protocol,

00:29:52.900 --> 00:29:54.439
which allows Firefox to communicate

00:29:54.440 --> 00:29:55.919
with a running Emacs server

00:29:55.920 --> 00:29:59.779
by sending an org-protocol URL to the latter.

00:29:59.780 --> 00:30:03.159
It can be used not just for capturing or storing links,

00:30:03.160 --> 00:30:10.119
but to execute arbitrary code on any component of the URL.

00:30:10.120 --> 00:30:11.679
However, it is fire and forget,

00:30:11.680 --> 00:30:16.479
and Emacs cannot tell Firefox what to do.

00:30:16.480 --> 00:30:17.919
There may be a length restriction, too.

00:30:17.920 --> 00:30:20.399
For example, Firefox may not be able to send back

00:30:20.400 --> 00:30:22.419
the whole DOM.

00:30:22.420 --> 00:30:26.219
This claim needs to be verified, though.

00:30:26.220 --> 00:30:30.019
Native messaging is one solution to this problem.

00:30:30.020 --> 00:30:31.639
It is a two-way communication channel

00:30:31.640 --> 00:30:35.319
between a Firefox web extension and a local system process

00:30:35.320 --> 00:30:37.839
started by the web extension.

00:30:37.840 --> 00:30:40.399
The process could be an Emacs server,

00:30:40.400 --> 00:30:42.399
which would make Emacs effectively

00:30:42.400 --> 00:30:48.679
a Firefox web browser extension.

00:30:48.680 --> 00:30:51.999
In this case, Elisp would be the main extension language,

00:30:52.000 --> 00:30:53.619
rather than JavaScript.

00:30:53.620 --> 00:30:56.159
However, JavaScript is still needed at the Firefox end

00:30:56.160 --> 00:30:59.220
of the communication channel.

00:30:59.320 --> 00:31:01.159
As a simple example of this idea,

00:31:01.160 --> 00:31:04.439
Firefox could ask Emacs to redirect a URL

00:31:04.440 --> 00:31:08.319
by removing tracking and using alternative frontend, etc.

00:31:08.320 --> 00:31:12.479
However, I was not able to implement this

00:31:12.480 --> 00:31:14.279
due to some tricky business

00:31:14.280 --> 00:31:15.639
with enforcing synchronicity

00:31:15.640 --> 00:31:17.119
that allows the web extension

00:31:17.120 --> 00:31:20.199
to wait for responses from Emacs.

00:31:20.200 --> 00:31:25.359
Some further work, I suppose.

NOTE Thank you

00:31:25.360 --> 00:31:28.154
That concludes my talk.

00:31:28.254 --> 00:31:31.440
Thank you for your attention.