path: root/2021/captions/emacsconf-2021-dashboard--productivity-dashboards-with-emacs-and-kindle--mehmet-tekman--main.vtt

WEBVTT

00:00.000 --> 00:01.567
Hi everyone! I'm Mehmet Tekman,

00:01.567 --> 00:02.483
and I'm here to talk to you

00:02.483 --> 00:03.700
about using Amazon Kindles

00:03.700 --> 00:05.040
as a productivity dashboard for

00:05.040 --> 00:07.359
your various projects.

00:07.359 --> 00:09.519
In a nutshell, you describe your machines,

00:09.519 --> 00:11.317
your commands, and your schedules

00:11.317 --> 00:13.120
in an Org-Mode file,

00:13.120 --> 00:14.933
and then you just initialize

00:14.933 --> 00:16.960
your Kindle devices.

00:16.960 --> 00:18.367
These devices are asleep

00:18.367 --> 00:19.117
most of the time,

00:19.117 --> 00:20.720
but they wake up at scheduled times

00:20.720 --> 00:22.033
to retrieve content

00:22.033 --> 00:24.800
from the centralized server.

00:24.800 --> 00:27.599
Content can be Org mode and Emacs-based,

00:27.599 --> 00:29.500
or it can be from Web content,

00:29.500 --> 00:42.840
or it can just be static images and WAV.

00:42.840 --> 00:45.600
If you, like me, struggle to

00:45.600 --> 00:46.833
keep your life under tabs,

00:46.833 --> 00:48.300
or find it very hard to separate

00:48.300 --> 00:49.417
your work life from your home life,

00:49.417 --> 00:51.283
then you, like me, likely need

00:51.283 --> 00:52.917
some kind of passive background service

00:52.917 --> 00:54.083
that reminds you of where you are

00:54.083 --> 00:55.267
and what you are supposed to be doing.

00:55.267 --> 00:56.960
Even if it's just a sign saying,

00:56.960 --> 00:58.640
"You're at home! Relax!"

00:58.640 --> 01:00.400
An Amazon Kindle is perfect for this.

01:00.400 --> 01:01.717
In a nutshell, it's a cheap

01:01.717 --> 01:03.117
black and white e-ink device

01:03.117 --> 01:03.800
that can go for weeks

01:03.800 --> 01:05.033
without needing a single charge.

01:05.033 --> 01:06.767
Every year, Amazon brings out

01:06.767 --> 01:07.983
an incrementally better model,

01:07.983 --> 01:09.333
which makes the old devices obsolete,

01:09.333 --> 01:11.067
and you can find these older models

01:11.067 --> 01:13.360
for 5 euros on second-hand websites.

01:13.360 --> 01:15.360
Plus it runs Linux, has WiFi networking,

01:15.360 --> 01:16.987
and has a dedicated forum of hackers

01:16.987 --> 01:19.200
for getting the most out of the device.

01:19.200 --> 01:20.366
Some drawbacks of this is that

01:20.366 --> 01:22.799
the device often comes with unwanted bloat:

01:22.799 --> 01:24.050
over-the-air updates,

01:24.050 --> 01:25.833
it phones home to Amazon regularly,

01:25.833 --> 01:27.033
it has a secret microphone

01:27.033 --> 01:27.983
embedded in the device,

01:27.983 --> 01:29.433
and it has a bunch of creepy

01:29.433 --> 01:30.633
seemingly interdependent

01:30.633 --> 01:31.439
background processes,

01:31.439 --> 01:34.083
where killing one kind of kills the others

01:34.083 --> 01:36.560
risking that you will break the device.

01:36.560 --> 01:37.799
But this is where the community

01:37.799 --> 01:38.883
really shines through,

01:38.883 --> 01:40.483
since the friendly (and not-so-friendly)

01:40.483 --> 01:41.583
users (and developers)

01:41.583 --> 01:43.200
from the MobileRead forums have pretty much

01:43.200 --> 01:44.560
scraped out a good portion of the

01:44.560 --> 01:46.960
harmful Amazon scripts from the device.

01:46.960 --> 01:48.550
Some of the devices even use

01:48.550 --> 01:49.483
Awesome Window Manager,

01:49.483 --> 01:50.800
meaning you can really play around

01:50.800 --> 01:51.620
with the existing system

01:51.620 --> 01:52.633
without having to create

01:52.633 --> 01:54.233
your own X11 server.

01:54.233 --> 01:55.377
This then empowers users

01:55.377 --> 01:57.600
to display whatever they want on the device.

01:57.600 --> 01:59.040
One project that really got this going

01:59.040 --> 02:00.560
was the Kindle-Dash dashboard from

02:00.560 --> 02:02.320
Pascal Widdershoven, who really refined a

02:02.320 --> 02:03.483
lot of the internal scripts

02:03.483 --> 02:05.439
to stabilize the device.

02:05.439 --> 02:06.600
However, the project then

02:06.600 --> 02:07.650
puts the onus on the device

02:07.650 --> 02:08.560
to retrieve the data from

02:08.560 --> 02:09.950
somewhere else over the internet,

02:09.950 --> 02:10.753
and so you still need to 

02:10.753 --> 02:11.440
generate the content

02:11.440 --> 02:13.200
and place it on the web somewhere.

02:13.200 --> 02:14.640
Plus you need to do this and manage it

02:14.640 --> 02:17.360
for every Kindle device that you have.

02:17.360 --> 02:18.500
Kindle-Sync, however,

02:18.500 --> 02:19.867
is an entirely different beast,

02:19.867 --> 02:21.800
albeit one that builds off of the works

02:21.800 --> 02:23.440
of the aforementioned projects.

02:23.440 --> 02:24.800
It assumes that instead of just having

02:24.800 --> 02:26.050
one Kindle device around

02:26.050 --> 02:27.133
that you wish to re-purpose

02:27.133 --> 02:28.080
for productivity purposes,

02:28.080 --> 02:28.983
that you actually have

02:28.983 --> 02:30.117
multiple Kindle devices

02:30.117 --> 02:30.794
that you want to manage

02:30.794 --> 02:32.720
and configure in tandem.

02:32.720 --> 02:33.633
Everything is managed

02:33.633 --> 02:35.667
from a dedicated server (or a raspberry pi)

02:35.667 --> 02:37.440
which distributes jobs to multiple Kindles,

02:37.440 --> 02:39.519
running on different update timers.

02:39.519 --> 02:40.786
These timers are all managed

02:40.786 --> 02:41.486
from the server,

02:41.486 --> 02:43.017
and all the Kindle device has to do is:

02:43.017 --> 02:45.200
to wake up, power on the WiFi,

02:45.200 --> 02:47.280
receive some media, display the media, and

02:47.280 --> 02:49.680
receive a barebones RTC sleep request.

02:49.680 --> 02:51.040
Then it sleeps for the requested time,

02:51.040 --> 02:52.800
consuming no power, whilst displaying the

02:52.800 --> 02:55.200
desired media. That is maybe 10 seconds

02:55.200 --> 02:57.599
of awake time between each request.

02:57.599 --> 02:58.933
Cron does not actually run

02:58.933 --> 02:59.933
on the Kindle device itself,

02:59.933 --> 03:01.600
simply because it does not reliably work.

03:01.600 --> 03:04.050
All of this is handled by the server.

03:04.050 --> 03:05.599
With the server-client model,

03:05.599 --> 03:08.000
it also tries to restrict Amazon access.

03:08.000 --> 03:09.517
SSH keys are shared

03:09.517 --> 03:11.217
only from the client to the server,

03:11.217 --> 03:12.517
but not from the server to the client,

03:12.517 --> 03:13.920
so the Kindle cannot connect

03:13.920 --> 03:16.319
to the Raspberry Pi without a password.

03:16.319 --> 03:18.033
IPtables rules are also set

03:18.033 --> 03:19.483
so that the Kindle cannot phone home

03:19.483 --> 03:20.667
to Amazon, and the connections

03:20.667 --> 03:23.200
are restricted to just the LAN.

03:23.200 --> 03:24.820
So I got very curious at one point

03:24.820 --> 03:26.133
and decided to see how long

03:26.133 --> 03:27.599
a Kindle could last on a single charge

03:27.599 --> 03:28.560
in such an arrangement,

03:28.560 --> 03:30.640
so that every 15 minutes for 18 hours,

03:30.640 --> 03:31.599
I tested the device

03:31.599 --> 03:32.959
by sending a media item

03:32.959 --> 03:35.200
and recording the battery level.

03:35.200 --> 03:36.159
The Kindle doesn't seem to

03:36.159 --> 03:36.959
report the battery level

03:36.959 --> 03:37.760
very continuously,

03:37.760 --> 03:39.040
but at discrete percentages,

03:39.040 --> 03:39.840
so that you could end up with

03:39.840 --> 03:42.159
a graph that looks like this.

03:42.159 --> 03:43.617
Assuming you have half the charge,

03:43.617 --> 03:45.200
and use it once every hour -

03:45.200 --> 03:48.319
it will drop by 10% battery in 76 hours,

03:48.319 --> 03:49.760
which is roughly three days.

03:49.760 --> 03:50.640
It's hard to extrapolate

03:50.640 --> 03:52.400
with only three good summarized data points,

03:52.400 --> 03:53.519
of which the number of requests

03:53.519 --> 03:54.879
per battery level appear to diminish

03:54.879 --> 03:56.640
as shown in the table below,

03:56.640 --> 03:58.560
but the final result yields 76 requests

03:58.560 --> 04:00.799
with an average loss of 0.5% battery life

04:00.799 --> 04:02.273
per request. Which is not bad!

04:02.273 --> 04:04.400
Assuming you do a request every 2 hours

04:04.400 --> 04:06.000
from 8am to 8pm,

04:06.000 --> 04:07.040
and let it sleep at night,

04:07.040 --> 04:09.040
then that's approximately 6 requests a day,

04:09.040 --> 04:10.400
which could easily last a device

04:10.400 --> 04:11.280
for a month.

04:11.280 --> 04:12.586
The ksync script does

04:12.586 --> 04:13.200
essentially everything:

04:13.200 --> 04:14.799
from generating and fetching the media,

04:14.799 --> 04:16.720
to initializing all Kindle devices,

04:16.720 --> 04:18.160
generating the server cronjobs,

04:18.160 --> 04:19.199
log report summaries,

04:19.199 --> 04:20.400
editing the config tables,

04:20.400 --> 04:21.199
and much more.

04:21.199 --> 04:22.880
The media operations are comparatively

04:22.880 --> 04:23.680
much more complex

04:23.680 --> 04:25.280
and encompass a few media use cases

04:25.280 --> 04:26.400
such as fetching the weather

04:26.400 --> 04:28.560
(though only from Open Weather Maps)

04:28.560 --> 04:30.000
and retrieving Google Calendar views

04:30.000 --> 04:32.000
by week, month, agenda, and four day view.

04:32.000 --> 04:33.199
You can retrieve Org-Mode data

04:33.199 --> 04:34.639
from an Emacs instance on the server,

04:34.639 --> 04:35.360
which in my case

04:35.360 --> 04:36.720
I produce views for an agenda

04:36.720 --> 04:39.360
or a sparse tree of my main projects file.

04:39.360 --> 04:41.120
Finally we have gallery and wavfile,

04:41.120 --> 04:42.240
which are static resources

04:42.240 --> 04:44.000
which will never change once generated.

04:44.000 --> 04:45.199
The idea is that you feed it

04:45.199 --> 04:46.400
text and an image location,

04:46.400 --> 04:47.040
and it generates

04:47.040 --> 04:48.720
a Kindle-compatible image

04:48.720 --> 04:51.280
using imagemagick as a backend for it.

04:51.280 --> 04:52.240
In the case of the wavfile,

04:52.240 --> 04:54.160
it uses espeak on the backend.

04:54.160 --> 04:55.280
The below is summarized

04:55.280 --> 04:56.317
from the help-me text

04:56.317 --> 04:57.199
in the main ksync file,

04:57.199 --> 04:58.160
but essentially, you need to

04:58.160 --> 04:59.919
define your config in the CSV files,

04:59.919 --> 05:01.440
which we talk about in the next section;

05:01.440 --> 05:03.120
initialize all your Kindle devices,

05:03.120 --> 05:04.720
i.e. copy over SSH keys,

05:04.720 --> 05:06.160
kill all the unnecessary services,

05:06.160 --> 05:07.840
and prime them for media collection;

05:07.840 --> 05:08.720
and ensure that you have

05:08.720 --> 05:10.080
all your static media generated

05:10.080 --> 05:11.440
and fetchable; and finally

05:11.440 --> 05:12.720
you then refresh the scheduling

05:12.720 --> 05:14.240
on the server.

05:14.240 --> 05:15.759
Okay, so this is all good and well,

05:15.759 --> 05:17.039
and we now know what the server does

05:17.039 --> 05:18.400
and how to probe and inspect it -

05:18.400 --> 05:19.759
but how does the server generate

05:19.759 --> 05:21.120
much of the content?

05:21.120 --> 05:22.080
So a lot of the content

05:22.080 --> 05:23.360
will be dynamically generated,

05:23.360 --> 05:24.639
meaning it cannot be cached

05:24.639 --> 05:26.720
and is likely to change from hour to hour.

05:26.720 --> 05:28.400
The media content that is generated here

05:28.400 --> 05:29.759
are mostly PNG images

05:29.759 --> 05:30.567
and have a timestamp

05:30.567 --> 05:32.320
embedded in their filenames.

05:32.320 --> 05:33.520
The Emacs-specific content

05:33.520 --> 05:34.560
consists of a few views,

05:34.560 --> 05:36.000
namely the org-gcal views,

05:36.000 --> 05:37.600
org-agenda, and org-calories --

05:37.600 --> 05:39.520
essentially anything that Emacs can display

05:39.520 --> 05:42.000
and that you want to capture into an image.

05:42.000 --> 05:43.360
Emacs can't (as far as I know)

05:43.360 --> 05:44.639
render graphics in a headless way,

05:44.639 --> 05:45.600
so what we do instead

05:45.600 --> 05:48.240
is run Emacs in a dummy minimal X11 session

05:48.240 --> 05:50.080
via "xvrb-run."

05:50.080 --> 05:51.680
From inside, you can take screenshots

05:51.680 --> 05:52.233
as you would in 

05:52.233 --> 05:53.440
a normal desktop environment,

05:53.440 --> 05:54.400
but with the benefit that

05:54.400 --> 05:56.479
you don't actually need to invoke a desktop

05:56.479 --> 05:58.560
or interfere with an existing one.

05:58.560 --> 05:59.840
The minimal elisp shown here

05:59.840 --> 06:00.720
is all that is required

06:00.720 --> 06:02.400
to output your desired image from Emacs

06:02.400 --> 06:04.479
and configure it for the Kindle environment.

06:04.479 --> 06:05.360
On the web side of things,

06:05.360 --> 06:06.400
we don't really need to invoke

06:06.400 --> 06:07.520
a dummy X11 session

06:07.520 --> 06:09.120
because Chromium can run headless

06:09.120 --> 06:09.919
and can be controlled

06:09.919 --> 06:11.600
by the node library "puppeteer"

06:11.600 --> 06:13.039
to render dynamic content,

06:13.039 --> 06:14.560
focus on regions of the webpage,

06:14.560 --> 06:16.080
and take snapshots.

06:16.080 --> 06:17.600
The static content comprises

06:17.600 --> 06:19.600
of two types: images and audio.

06:19.600 --> 06:21.520
The content is accessed by a key,

06:21.520 --> 06:22.560
in this case Batman,

06:22.560 --> 06:23.600
and the content information

06:23.600 --> 06:25.199
is given by the "--extra" parameter

06:25.199 --> 06:26.960
which describes either or both

06:26.960 --> 06:30.880
an image and text.

06:30.880 --> 06:32.248
Okay, so now we have content,

06:32.248 --> 06:33.600
how do we schedule this content

06:33.600 --> 06:34.960
to appear on our desired machines

06:34.960 --> 06:36.400
at desired times?

06:36.400 --> 06:37.759
Everything is run via cron.

06:37.759 --> 06:38.720
So previously we saw that

06:38.720 --> 06:40.880
we only needed the tables MACHINES.csv,

06:40.880 --> 06:43.440
COMMANDS.csv, and multiple TIME_*.csv tables

06:43.440 --> 06:44.880
for the shell script to work.

06:44.880 --> 06:46.479
But Org-Mode does this far easier,

06:46.479 --> 06:47.919
since you can just have everything

06:47.919 --> 06:49.039
in the same file,

06:49.039 --> 06:50.720
and with the helper minor-mode,

06:50.720 --> 06:51.360
manage everything

06:51.360 --> 06:53.120
from a single Org-Mode document.

06:53.120 --> 06:55.120
Here I have 4 kindles and their shortnames.

06:55.120 --> 06:56.160
Yes, I even have a Kindle

06:56.160 --> 06:57.520
hanging outside my door.

06:57.520 --> 06:58.960
I have 11 defined commands

06:58.960 --> 07:00.800
which represent the views I want to see,

07:00.800 --> 07:02.319
and there are 4 timetables I use,

07:02.319 --> 07:02.800
but you can have

07:02.800 --> 07:04.319
everything on one, if you like.

07:04.319 --> 07:05.360
Rows are machine names,

07:05.360 --> 07:06.800
and columns are corresponding hours

07:06.800 --> 07:07.840
at which they run.

07:07.840 --> 07:09.440
Trust me, it's easier to configure

07:09.440 --> 07:10.960
repeating tasks just by repeating them

07:10.960 --> 07:12.720
multiple times, because at least this way,

07:12.720 --> 07:13.680
it's human readable,

07:13.680 --> 07:14.880
and the script which converts these

07:14.880 --> 07:15.759
to a cronjob

07:15.759 --> 07:18.800
collapses the repeating tasks by itself.

07:18.800 --> 07:20.560
The ksync script can be called

07:20.560 --> 07:23.120
from within the config.org file

07:23.120 --> 07:24.683
using this convenient 

07:24.683 --> 07:26.960
use-package declaration.

07:26.960 --> 07:28.319
All that one needs to do

07:28.319 --> 07:30.560
is to configure the ENVIRONMENT_VARIABLES

07:30.560 --> 07:32.880
by setting them in this table

07:32.880 --> 07:34.479
where you set the repo name,

07:34.479 --> 07:36.160
the config directory,

07:36.160 --> 07:37.599
where the media shall go,

07:37.599 --> 07:38.960
and the server IP,

07:38.960 --> 07:39.919
although this can be

07:39.919 --> 07:41.360
automatically detected.

07:41.360 --> 07:42.240
The package allows you

07:42.240 --> 07:43.440
to export your tables

07:43.440 --> 07:46.720
by running C-c C-c on them,

07:46.720 --> 07:49.199
and allows you to update all the jobs

07:49.199 --> 07:52.319
related to each of your clients.

07:52.319 --> 07:53.759
You can also initialize clients

07:53.759 --> 07:55.120
using this package --

07:55.120 --> 07:56.479
for either all of them

07:56.479 --> 07:58.479
or individual clients --

07:58.479 --> 07:59.599
and the package comes with

07:59.599 --> 08:01.120
some convenience functions

08:01.120 --> 08:02.720
to do this automatically

08:02.720 --> 08:06.720
for all tables in the buffer.

08:06.720 --> 08:08.319
With this, I want to say a big thank you

08:08.319 --> 08:09.840
to Takaaki Ishikawa

08:09.840 --> 08:11.520
for his fantastic "org-tree-slide"

08:11.520 --> 08:12.879
presentation package.

08:12.879 --> 08:14.136
To Pascal Widdershoven

08:14.136 --> 08:15.803
and David Hamp-Gonsalves,

08:15.803 --> 08:16.633
for their fantastic 

08:16.633 --> 08:17.840
kindle-dash repositories,

08:17.840 --> 08:19.903
for which some of my internal Kindle scripts

08:19.903 --> 08:20.720
are derived from.

08:20.720 --> 08:22.160
Also a big thanks to the friendly

08:22.160 --> 08:23.520
and not-so-friendly users and hackers

08:23.520 --> 08:24.960
in the MobileRead forums.

08:24.960 --> 08:25.919
And finally, a big thanks

08:25.919 --> 08:26.960
to the Emacs community

08:26.960 --> 08:28.270
and the conference organizers.

08:28.270 --> 08:31.120
Thank you! [captions by Mehmet]