Hello, I'm Joseph Turner.
This talk is about hyperdrive.el,
peer-to-peer file system in Emacs.
Hyperdrive is a JavaScript library
developed by the Holepunch team
for sharing files on a peer-to-peer network.
It's released under the Apache 2.0 license.
Here's how it works.
To participate as a peer, you run a node,
a lightweight local server that allows you
to connect with other nodes on the network.
You can create a hyperdrive or multiple hyperdrives,
and you can author files within them.
Each hyperdrive is automatically assigned
a globally unique link
that starts with `hyper://`.
When you share that link with someone,
they have access to your hyperdrive.
Anyone who has that link can load the hyperdrive
from the network and view its content.
When you load a Hyperdrive file from the network,
your node caches that data locally
and automatically begins seeding it
back to the network,
making it available for others to download from you.
Hyperdrives are single writer.
This means that when you create a new drive,
you are the only one who can make changes to it.
Others can view it
and can seed it back to the network,
but you're the only one who can modify it.
Hyperdrives are offline first.
This means that when you load data from the network,
it's stored locally on your machine
for you to view later,
even when you're disconnected from other peers.
You can also create new drives
and modify your drives when you're offline,
and then share those changes
once you connect with peers later.
Hyperdrives are local first.
This means that when you are connected with
other peers on a local area network,
even if none of the peers involved
are connected to the broader Internet,
you can still share files.
Hyperdrives are sparsely replicated.
This means that you can download individual files
from a hyperdrive without having to download
the whole thing.
This saves on disk space and also allows you
to quickly load just the files
that you're interested in.
Hyperdrives are mutable.
You can add files, change files,
remove files from a hyperdrive.
And when peers load your drive
using the very same link,
they will be able to load
the latest changes that you've published.
Hyperdrives are versioned.
This means that when you make changes to a file,
the previous versions of those files are not lost.
Peers can load the old versions of a file
that was changed or deleted, for example,
simply by specifying the version number
of the hyperdrive when the file still existed
or existed in a previous state.
I'm presenting this talk on behalf of USHIN.
USHIN is a tiny nonprofit whose mission is to
promote personal, community and global health
through free and open universal shared information
for everybody.
USHIN was founded in the early 90s
by Paula Maas, Steve Nash and others
with the goal of creating
a distributed health information network
that would allow people to find, share, compare
and deliberate health information
from a variety of sources.
Since then, the scope of the project has broadened
to include all kinds of information.
And in recent years, we've been focusing on
building with peer-to-peer software.
About three years ago,
we started working with Mauve Signweaver,
who has since then been our steadfast
peer-to-peer explorer and guide.
This year, we started working on
this hyperdrive.el Emacs package,
and Adam has been the powerhouse
behind the Emacs Lisp development.
About a month ago,
Jonas Bernoulli started joining with us,
and he has been offering his expertise
in the realm of user interface design
using his Transient library.
And Protesilaos Stavrou has been not only valuable
in terms of user design and feedback,
but he created a wonderful
basic introduction to Hyperdrive. Take a look.
[Prot]: Hello everyone!
My name is Protesilaos, also known as Prot.
In this video, I want to show you
the basics of hyperdrive.el.
This is a package for Emacs that lets us connect to
the Hyperdrive peer-to-peer network.
We can browse existing drives on the network,
meaning existing file systems,
or create and maintain our own hyperdrive,
to which we can add, remove or edit files.
We will see this together.
What I have here in front of me is a basic
use-package declaration for hyperdrive.el.
All I am doing is binding `hyperdrive-menu`
to a key and also activating
the menu bar entry of hyperdrive.
Let me show you very quickly, `hyperdrive-menu`.
And this is the sort of interface that it brings up.
We will take a closer look at it.
As for the menu bar,
we have `hyperdrive-menu` over here,
and we will take a look at this as well.
Suffice to say that
you can get the same functionality
with the menu bar or with `hyperdrive-menu`,
as well as by calling the commands directly
with `M-x` or their respective key bindings.
I won't cover everything in that regard,
but please bear this fact in mind.
Let's start then with what I have here
in this other tab, which is a set of files.
I have prepared in my local file system - a hyperdrive.
This contains a set of files
that I ultimately want to share
on the peer-to-peer network,
meaning that I want this to eventually be
reflected in my own hyperdrive.
What I did to get started is
I invoked `hyperdrive-menu`.
The very first thing you need to do
to get started with Hyperdrive,
either to browse or to create,
is to start the gateway,
meaning to be able to connect to
the peer-to-peer network.
You see here in `hyperdrive-menu` that there is
an indicator next to the "Gateway" heading
telling me that the gateway is on.
I started it with `G` and then `s`.
Once you start it,
you can type `h` to visit an existing drive,
`N` to create a drive,
or `L` to open a link to an existing hyperdrive.
You may get this link via email, for example.
In this case, let me be over here
and let me just kill this buffer.
What I want in this case is to select a hyperdrive.
You see here I have my own hyperdrive
and also the hyperdrive of USHIN,
the developers of hyperdrive.el.
What I want to do in this case,
you see there are various options available.
I want to type `v` to view a file.
And in this case,
I will just type the forward slash,
which means to view the root directory
of this hyperdrive. And I am here.
It is an empty drive apart from
my credentials over here.
There is nothing more really to see.
But what I want to do is
to start adding files to this drive.
Let me use the menu bar for this.
I will hover over to the menu bar,
and I will find the "Hyperdrive" submenu.
Then I will go find where the drives are.
And you see that there are writable drives,
meaning drives that I manage,
and read-only drives, meaning drives of other users.
In this case, USHIN is read-only, of course,
and Protesilaos is writable.
So what I want to do is upload a file to Protesilaos.
And it prompts me in the Minibuffer for a file.
I will just add the README.
It is asking me, "Where do you want to add it?"
And in square brackets, the default is to add it
to the root directory of the hyperdrive.
So I will just type `RET` to select the default.
And there it is. It is already in the drive.
Let me add the COPYING file as well,
using the same method.
I will go find my drive. "Upload File."
And let's add the COPYING.
And I also want to add it to the root directory.
So already I have two files.
This is looking good.
Now I am opening the file inside of hyperdrive.
This is not the same file
as I have in my local directory.
What I can do here, for example, is I can...
Let me open another file here, the COPYING.
I can create an Org link.
Notice that these are Org files.
So I will use the standard `org-store-link` command.
You can see the key binding I invoked
and the command it calls
on the top right corner of my screen.
So what I did is
I copied a link to this hyperdrive file.
And in this other file over here,
I want to say "Free/libre."
So I am editing my hyperdrive terms:
"Check the," and I will paste the link here,
"for how to use my files."
If we check what is here,
you will see that this is a link
inside of the hyperdrive.
And it's pointing specifically
to that heading over there,
which has this unique identifier,
as you can see over here.
So this is wonderful.
I want to rename it to "Check the COPYING
for how to use my files."
I will save this.
And now what I want to do is,
I want to invoke `hyperdrive-menu`.
And you will see now the menu, unlike earlier,
has more commands, more stuff we can do with it.
For example, I can type `w` to copy a URL.
And by typing `w`, you will notice...
Let me go to the `*scratch*` buffer to paste this in.
You will notice what the URL is.
It is my hyperdrive, my unique identifier,
and then forward slash, meaning the root directory,
and then README.org.
That is the file I was editing.
Let's do `hyperdrive-menu` again.
Let's remove the `*scratch*` buffer.
And let's do `hyperdrive-menu`.
And you will notice that there are options
to download the file, for example.
If you are reading somebody else's file,
you can download it to your own file system.
Let me do that. "Download."
And it is asking me, "Where do you want to save this?"
For now, I will save it in the `/tmp/`, like this.
Let's call it test.org.
Okay, `/tmp/test.org`.
Let me go and visit `test.org`.
And there it is.
It downloaded it just like that.
This is how you can, for example,
download the pictures and videos
that I will eventually share on my hyperdrive.
Let's invoke `hyperdrive-menu` again.
And let's go up to the parent, you see,
with the caret (`^`) sign.
This will take me to the parent directory,
in this case, the root directory of my hyperdrive.
Let me do it a bit differently.
The same idea, a bit differently.
For those of you who are familiar with
Dired and the `dired-jump` command,
Dired is the standard file manager of Emacs.
And `dired-jump` is a command that lets you jump
from the current file
to the directory that contains that file.
So you see, I am here.
The `dired-jump` command, by default,
is bound to Ctrl-x, Ctrl-j (`C-x C-j`).
So if I do `C-x C-j`, in this case,
it invokes a command.
You can see the name of it, `hyperdrive-up`,
which is functionally equivalent to `dired-jump`.
It does the same thing, meaning that it took me
to the parent directory of this file.
I think this is very helpful.
This listing over here, in general,
tries to mimic or to reuse
the knowledge you already have of Dired.
For example, if you type `o`,
it will open the file at point in the other window,
same as in Dired.
Whereas if you type `RET`,
it would open it in the current window.
Again, same as what you will do in Dired.
Let's see over here.
You have options to jump with `j`,
which is using Minibuffer completion to go to a file.
Right now, I only have two files,
but the idea is the same.
It's, again, what you would do in Dired
if you type `j` with the default key bindings, though,
not with Evil mode or something else.
Let's see again what we have over here.
You can create a bookmark, and this will work,
but no need to show you everything.
The idea is that you create a bookmark
the way you create any Emacs bookmark,
to a file, to a directory.
It doesn't matter.
And then you can jump to it,
the way bookmarks in Emacs always work.
What I want to show you now
a little bit is the history.
History in Hyperdrive
has to do with the drive itself.
Meaning that individual files
do not have their own history,
but the drive as such has a history.
Whenever you add a file, you remove a file,
or you edit a file, you are incrementing
the versioning of the hyperdrive by one.
So each action corresponds to one unit of history.
If you add a file, remove a file, and edit a file,
this means that you are up three versions.
So whatever your version number is, plus three.
I am on version 24* over here. [* latest, not version 24]
Let me go to this file now,
and let me do `V h`
to see a history of it.
You will notice that between versions 23 and 24,
this file was constant.
But in version 25, we have a change.
When you are in this buffer over here,
you can type the equals sign (`=`),
which is a key binding that will bring up the diff.
So the set of changes between
the previous version and the current version.
And you see here, between versions 23 and 25,
I have this addition.
I think this is wonderful because now
you can always go and check
what is the state of this file.
What is this person up to?
What have they been changing?
And with Hyperdrive, you can also visit…
Let me go to the parent here.
You can always visit a previous history.
So, for example, I will go to the previous history,
and let me see this file again
in its previous history.
I am looking at the file,
but notice that the file now is not editable
because this is in the past.
I cannot rewrite history.
I can only go to the present and then modify it
and then create a new history, a new version.
But this helps me see the state of the file
at that version of the hyperdrive.
So this is the basic idea of it, folks.
What I want to do then is
continue with my process here.
Let me actually do it like this so that you can see.
Continue with the process of
publishing all those files
that I have on my hyperdrive.
For example, this is a picture of a flower
that I have taken.
I think it's very nice.
And this is a video of an eagle
that was flying above me.
And I will share this on the Hyperdrive network.
From the network, by the way,
you can also stream video as well.
It is described in the hyperdrive.el manual,
but I cannot show you everything right now.
I think you get the idea.
The gist is, you have a file system
that you can share with the world
using peer-to-peer technology.
And for me, this is a powerful tool.
This is a valuable proposition
because I can share these large files I have,
these pictures or videos,
which I cannot post on my website
due to bandwidth considerations.
And this way, I can still share with the world
something that I consider interesting.
That's all for today, folks.
Thank you very much for your attention.
Remember that hyperdrive.el is still in development
and things may change.
But the fundamentals are in place
and will remain constant.
That's all for today. Take care. Goodbye.
[Joseph]: Thank you, Prot.
Another fundamental feature of hyperdrive.el
is drive creation.
The first step, as always,
is to make sure that the gateway is running.
So I'll open up `hyperdrive-menu` with `C-c h`.
By the way, my key presses,
as well as the commands that they run,
can be seen at the top right of my screen.
Down here, I see that the gateway is off.
So I'll start it with `G s`.
Now, it takes a few moments
for the gateway to spin up.
So to refresh the menu,
I will close it and open it again.
And now we see that the gateway is on.
I'll press `N` to create a new drive.
Now it's prompting me for a new hyperdrive seed.
A seed is a string of characters
that will be used to generate,
in tandem with my secret master key,
a new public key that will globally,
uniquely identify this drive.
So I'll type in "emacsconf".
And after a moment,
we see the newly created, empty drive.
So I'll open up `hyperdrive-menu` once more.
And I'll press `h` to open the sub-menu that shows
more information about this hyperdrive,
as well as commands related to this drive.
In blue here, we see the seed that I just entered
as well as the public key that it generated.
We also see that the petname
is also set to emacsconf.
The petname is different from the seed.
The petname is my personal, local identifier
for this drive.
I can change it whenever I want.
And while it's not a secret,
it's not displayed to other users.
So I will leave it for now.
"emacsconf" is fine.
But when I go to share this drive,
I'll want to display something more memorable
than this long public key.
And that's what the nickname is for.
I'll change that by pressing `n`,
and I'll type in "Emacs Conference".
Now, when other peers load this hyperdrive
by its URL, which I can copy by pressing `w`,
they will see the nickname
in addition to the public key.
So if hyperdrive.el is like a phone book,
the public keys are akin to phone numbers,
the nickname is like the name that your contacts
give you when they introduce themselves,
and the petname is the name
that you actually write down in your phone book.
I'll show you what nicknames and petnames look like
for drives that are not writable to me.
I'll press `C-g` to close this submenu,
and now I'm back at the main menu.
I'll press `C-u h` to choose a hyperdrive,
and I'll look at Prot's hyperdrive here.
And here we see that the nickname
of Prot's hyperdrive is "Protesilaos".
Now, it's grayed out,
which means that I can't change it.
I can't change it because it's not my hyperdrive.
But I can change the petname if I want it
to show up under a different name.
So I'll press `p`,
and I'll type in "Prot", and hit Enter.
Now I'll open his hyperdrive by pressing `f`.
And I'll pick a path, I'll just hit `RET`
to open the root directory.
And now, when Prot's hyperdrive shows up,
at the top of the screen,
I see that it's identified with the petname "Prot."
Now I'll show off `hyperdrive-mirror`.
`hyperdrive-mirror` is like `hyperdrive-upload-file`,
except that it allows you to upload
an entire directory full of files recursively.
For this example, I will upload
the contents of the emacsconf-mirror directory.
First step is to open the menu. I'll press H,
and then I will choose the emacsconf drive.
In the bottom here, we see the Mirror group.
The first option that I can change
is the source directory.
The source directory is the directory on my local
machine from which files will be uploaded.
By default, the source directory
is set to the current directory of the main buffer.
This is good for now, so I'll leave it.
The target directory is the directory
in the hyperdrive where the files will end up.
By default, it's the root directory,
but for this example,
I'll put these files in a subdirectory called notes.
The filter is the rule that allows you
to programmatically determine which files
in the local directory will be uploaded
into the hyperdrive, and which ones won't.
By default, all files are mirrored, but in this case,
let's say that I want to upload only the Org files,
these first three, and I want to exclude
the markdown file, solar-oven-notes.md.
So I'll press `m f`, and I will choose
the regular expression string option.
If I wanted to, I could choose a named function
or a lambda, but I won't demo that here.
I'll type in `org$`, which will match against
the files that end with "org."
The filter is here, and I'll leave confirmation on.
The confirmation step just allows me to review
the list of files that are going to be uploaded
into the drive before it happens.
So I'll press `m m` to mirror them, and I see here
that these three files are going to be uploaded.
Looks good. I'll press `C-c C-c` to confirm the mirror.
Now it says three files have been uploaded,
and here they are in the drive.
Good, so now I will modify the
fermented-overnight-oats file in the hyperdrive.
So I've loaded it, and I'll add here:
"or other grains - cook them in advance if you want to,"
and I'll save it.
Now this file, fermented-overnight-oats.org,
has been modified on the hyperdrive
more recently than on the file system.
The file system file has not been modified.
So I'll go back to my local directory,
and I'll modify a different file.
In this case, I'll add another hoedown
to the fiddle-tunes.org file.
"Tom and Jerry." That's a good hoedown.
And now I'll mirror again.
So I'll open the menu, and I'll press `h`,
and then I'll open up the emacsconf demo drive.
And now I've also decided that
I want to include the solar-oven-notes.md file.
So I'll remove the filter
so that it's no longer excluded.
I'll leave the rest of the settings the same,
and I'll press `m m` again.
And now we see that the
`*hyperdrive-mirror*` buffer looks different.
So, there are two main groups.
These are the files that are going to be uploaded,
and these are the files that are ignored.
They're not going to be uploaded.
The first subgroup is the files that are new locally.
So the solar-oven-notes.md file is new on my machine,
and it doesn't exist in the hyperdrive.
So the mirror command is going to take that file
and add it to the hyperdrive.
This group contains the files that are newer locally.
So the fiddle-tunes.org file
has been modified on my local machine,
but it hasn't been modified on the hyperdrive.
So `hyperdrive-mirror` is going to take the
updated version and put it on the hyperdrive.
Now, the first group that's going to be ignored
are the files that are older locally.
So the fermented oats file has been modified
on the hyperdrive more recently
than on my local file system.
So `hyperdrive-mirror` isn't going to overwrite
the version of the file in my hyperdrive
with the older local version.
And finally, the emacsconf-preparation.org file
hasn't been modified on either the hyperdrive
or my local file system,
and the timestamp is identical.
So `hyperdrive-mirror` is going to
ignore this file as well.
So the only two files that are going
to be uploaded now are the solar-oven-notes.md file
and the fiddle-tunes.org file.
I'll confirm that with `C-c C-c`.
And now in my hyperdrive
we see that there are four files.
The solar-oven-notes.md file has been uploaded,
and if I open the fiddle-tunes.org file,
we see that it now contains the line "Tom and Jerry,"
which means that it was updated based on
the change to the file on my local file system.
This `hyperdrive-mirror` command is the command
that we use to periodically update
the USHIN hyperdrive with
the contents of the USHIN website.
There's going to be a link to the USHIN hyperdrive
as well as the website at the end of the video.
Now I'll go into a little more detail
about the Hyperdrive history buffer
by showing off the README file in Prot's hyperdrive.
I'll press `C-c h` to open the menu,
`C-u h` to be prompted for a drive.
I'll select Prot's drive,
then I'll press `f` to jump to a file
inside of Prot's drive,
and then `RET` to go to the root directory.
From here, I'll press `j` to jump to
an item in his directory.
I'll press `RET` on the README to load it.
And then finally, I'll open up
`C-c h` to look at the menu.
Here, I see that I'm looking at
the latest version of Prot's README.org file.
I also see that the previous version
of README.org is unknown.
It's unknown because hyperdrives
are sparsely replicated.
That means that when my node loaded this
README.org file, it didn't bother
to load anything else.
It didn't load the previous history
or any other file in his drive.
But now that I want to check out whether
there is a previous version, I'll press `V p`.
And when it loads, I see in the mode line down here
that I'm now looking at version 25 of this drive.
That means that I'm looking at the README.org file
at version 25 of this drive.
I'll open the menu again, and I see
that the same version number is displayed here.
The previous version, before version 25,
is also unknown because we haven't bothered
to load anything before version 25.
I also see that the next version, after version 25,
is the latest version.
So I'll open that up by pressing `V n`.
And now we are back where we started,
at the latest version.
But now we see that the previous version,
the version before the latest version, is version 25.
Because now that our node has loaded the previous version,
it can display that information to us.
From here, I'll open up the history buffer.
I'll press `V h`.
And I can see that there are two known existent
ranges in Prot's README.org history.
This means that the latest version of README.org
was modified at version 39.
And that Prot made four changes
to other files in his hyperdrive
besides the README.org file since then.
Before that, the time that Prot modified
the README.org file was at version 25.
And then he made 13 other changes to other files
inside of this drive.
Before that, we don't know.
We haven't loaded the history.
But since we're curious,
I'll hit `RET` on the unknown line.
And now we see that in Prot's hyperdrive,
the README.org file didn't exist
for the first 22 revisions of his drive.
Then Prot created it at version 23.
Then again, at 25, made a change.
And then made another change at 39.
For good measure, I'll show you the diffs.
The first diff just contains the entire file
because the file didn't exist before version 23.
Then, at version 25, Prot added a link
to the COPYING.org file.
And then, at 39,
Prot changed the link to his own hyperdrive
to be a relative link.
Now, I'll stream a video from the USHIN hyperdrive
that shows off a prototype interface we created
for exploring networks of sources of information.
I'll open up `hyperdrive-menu`.
Press `C-u h` to be prompted for a drive.
I'll select the USHIN drive.
Press `f` to jump to a file in it.
And then I'll jump to the media directory.
Once it loads, I'll press `RET` on the video
that I want to stream.
[Voice from the video]: This is a demonstration
of the subjective trust interface
that the USHIN team built for the u4u.io web app.
[Joseph]: There it is.
Streaming a video from Hyperdrive.
Here's how hyperdrive.el works under the hood.
It uses plz [Please],
the HTTP library that Adam Porter wrote,
to send requests via curl to hyper-gateway.
hyper-gateway is a program that Mauve Signweaver wrote,
which runs a hyperdrive node under the hood.
It also runs a local HTTP server,
which accepts requests to control the node.
For example, if hyperdrive.el wants to show a file
from someone's hyperdrive,
it sends the appropriate link via curl
as a GET request to hyper-gateway.
hyper-gateway then fetches the data from the network
and returns it via curl,
via plz, back to hyperdrive.el.
If hyper-gateway already has a locally cached copy,
it doesn't bother checking the network.
It just sends it straight away.
Likewise, if hyperdrive.el wants to add a file
to a hyperdrive, it sends a PUT request.
hyper-gateway is not installed as part of hyperdrive.el.
It needs to be installed
as a separate, executable program.
We have plans to switch from using hyper-gateway
to another program that Mauve Signweaver
is working on, called hyper-sdk-rpc.
hyper-sdk-rpc will give us more fine-grained control
over the underlying Hyperdrive node,
which will open up some new features.
For example, we'll be able to rename files atomically,
diff directories between versions,
and list the peers that we're currently connected to.
Here are some of the Emacs libraries
that hyperdrive.el depends on.
Hyperdrive.el uses plz to send HTTP requests
to hyper-gateway. Check it out.
It's on GNU ELPA.
ewoc.el is a built-in library that's documented
in the Emacs Lisp manual under the rather cryptic
heading "Abstract Display Functions."
Really, what it does is it allows you
to map a data model to some display.
What we do with it in hyperdrive.el is,
in the directory view, we map directory entry items,
files and directories, to display items.
And ewoc.el makes it easy to update the display
whenever the underlying data changes.
Hyperdrive.el uses persist.el to store
Hyperdrive metadata about known hyperdrives
as well as version history between sessions,
so that the data is cached
when you log out and log back in.
If you're going to be developing a program in Emacs
for Emacs 29 or later,
I recommend looking at multisession.el,
because it's built-in, and it has some features
that persist.el doesn't have.
Thank you for listening to this talk.
I hope you get a chance to try out hyperdrive.el.
Here is a link to the hyperdrive.el manual
in the USHIN hyperdrive.
When you go to paste this link in,
you'll need to combine it back into one line.
This link is available in the hyperdrive.el manual,
which is available from within a browser,
at this link on the USHIN website.
We also have a public conference room that you can join.
It's an XMPP multi-user chat
that the Sopranica team graciously makes available
for us to use.
There's also a Matrix bridge,
if that's more your cup of tea.
Thank you to the EmacsConf organizers.
It's been a pleasure to participate,
and I look forward to hearing the rest of the talks.
Have a good day.