1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
|
[[!meta title="hyperdrive.el: Peer-to-peer filesystem in Emacs"]]
[[!meta copyright="Copyright © 2023 Joseph Turner and Protesilaos Stavrou"]]
[[!inline pages="internal(2023/info/hyperdrive-nav)" raw="yes"]]
<!-- Initially generated with emacsconf-publish-talk-page and then left alone for manual editing -->
<!-- You can manually edit this file to update the abstract, add links, etc. --->
# hyperdrive.el: Peer-to-peer filesystem in Emacs
Joseph Turner and Protesilaos Stavrou, <https://ushin.org> xmpp:discuss@conference.ushin.org (XMPP MUC for USHIN discussion), <mailto:joseph@ushin.org>
[[!inline pages="internal(2023/info/hyperdrive-before)" raw="yes"]]
This talk explores [hyperdrive.el](https://ushin.org/hyperdrive/hyperdrive-manual.html), an Emacs interface to [hyperdrive](https://docs.holepunch.to/building-blocks/hyperdrive), a
peer-to-peer shared filesystem.
Peer-to-peer networks give you the freedom to choose your sources of
information and to communicate directly with them. Emacs is a
[freedom-respecting](https://www.gnu.org/philosophy/free-software-even-more-important.html) text editor/operating system that puts you in the
driver's seat. By bringing these two worlds together, hyperdrive.el
aims to inspire deliberation about digital communication freedom.
Hyperdrives are peer-to-peer shared folders, well-suited for data that
changes over time, like personal blogs. It complements Bittorrent and
IPFS, which work best with unchanging data, like journal archives.
Here's a scenario: Alice creates a new hyperdrive and adds some files.
Her computer returns a public key URL that uniquely identifies the
hyperdrive. Alice shares that URL with Bob, who can then download
Alice's files directly from her computer. No third-party servers are
required to route the connection.
Data is distributed among peers; once Bob has loaded Alice's files,
Carol can get them from Bob (or anyone else who has a copy) even when
Alice is offline. Drives are mutable; when Alice adds/removes/changes
files in the drive, Bob can refresh her drive on his machine to get
the latest changes. Drives are versioned; anyone with the URL can
"check out" prior versions of Alice's drive to see what her files used
to look like.
Currently supported features in hyperdrive.el include:
- [directly edit hyperdrive files](https://ushin.org/hyperdrive/hyperdrive-manual.html#Write-to-a-hyperdrive)
- [dired-like directory view](https://ushin.org/hyperdrive/hyperdrive-manual.html#Directory-view)
- [org-mode link support](https://ushin.org/hyperdrive/hyperdrive-manual.html#Org-mode-links)
- [version history navigation/diffing](https://ushin.org/hyperdrive/hyperdrive-manual.html#View-the-hyperdrive-version-history)
- [built-in bookmark.el integration](https://ushin.org/hyperdrive/hyperdrive-manual.html#Bookmark-a-hyperdrive)
- [local directory mirroring](https://ushin.org/hyperdrive/hyperdrive-manual.html#Mirror-a-whole-directory)
- [audio/video streaming](https://ushin.org/hyperdrive/hyperdrive-manual.html#Stream-audio-and-video)
Planned features include:
- peer discovery (swarming)
- diffing directories between versions
Check out [the manual](https://ushin.org/hyperdrive/hyperdrive-manual.html#Installation) for installation instructions!
You're welcome to join our public XMPP chat room!
- xmpp:discuss@conference.ushin.org ([Join anonymously from your browser](https://anonymous.cheogram.com/discuss@conference.ushin.org))
- #_bifrost_discuss_conference.ushin.org:aria-net.org (Matrix bridge)
Bugs can be submitted to the [ushin issue tracker](https://todo.sr.ht/~ushin/ushin). Patches, comments or
questions can be submitted to the [ushin public inbox](https://lists.sr.ht/~ushin/ushin).
About the speaker:
I'm Joseph Turner. I enjoy fiddle, Aikido, peer-to-peer networks,
Emacs, and swimming in cold water. I work with [USHIN](https://ushin.org/), a tiny
educational US nonprofit whose mission is to promote personal,
community, and global health through free and open universal shared
information for everybody. This year, we're focusing on the
hyperdrive.el project, with the goal of bringing Emacs and
peer-to-peer together.
I am Protesilaos. Friends call me "Prot" and you are welcome to do the
same. I have been an Emacs user for ~4 years. I use Emacs full-time for
practically every aspect of my computing. I am the author and maintainer
of several packages for Emacs and am enthusiastic about its potential
for user freedom.
# Discussion
## Questions and answers
- Q: It's not clear how hyperdrive (not hyperdrive.el) works. Do I
need to install something on my computer to use it? Can I use it
from my phone?
- A: Like the emacs transmission client connects with the
transmission-daemon, hyperdrive.el connects with the
hyper-gateway daemon.
- Accessing hyperdrives on mobile:
<https://github.com/AgregoreWeb/agregore-mobile>
- Q: What lessons have you learned while developing hyperdrive.el?
- A: Great support and guidance from talented folks like Adam,
Mauve, Jonas, Prot!!
- Q: I use multiple computers and my partner also would like acess to
my notes. so 2 questions. How well would this work with using this
to editing my zettelkasten hyperdrive using multiple computers
- A: Hyperdrives are single-writer, so you'd be better off
linking between drives. In the future, we plan to add support
for <https://github.com/nobiot/org-transclusion> to
hyperdrive.el.
- Q2: How well would it work if my and my partner worked on the
same hyperdrive zettelkasten
- A: If you linked between drives, it could work quite well!
- Q: What would be a good way of getting Hyperdrives if you don't
want to install NPM and hava a binary. Could you compile it with
deno or the "rust or zig or go?" cli alternative tool? I would
prefer to download a single binary.
- A: Jonas has been using hyper-gateway installed with a `guix
shell` command. Thank you, Jonas!!!
- Quick gist:
<https://gist.github.com/tarsius/509e9c65c9df1bc243d77cd968d60daa>
- Q: <https://github.com/datrs/hypercore> rust hyperdrive?
- A: I'm not familiar with this rust port yet.
- Q: If you had your druthers, what would make your work on
hyperdrive.el easier?
- A: User feedback!! Please try it out :)
- Q: Have you tried putting a git repo in hyperdrive? Does it work
well?
- A: If you use a bare repository, take care to gc and create pack
files before mirroring to the drive, and only publish periodic
updates, then that might be okay. (And make sure not to repack
old pack files.)
- Q: Is data transferred between nodes in the clear or encrypted?
- A: Encrypted in transit.
- Q: Is there a searchable catalogue of hyperdrives?
- A: Not yet, but we have plans for a distributed "trust"
network that could be used with hyperdrive:
<https://git.sr.ht/~ushin/trust.el>
- Q: Any plans for FUSE or posix semantics?
- A: Not yet. There was
<https://github.com/andrewosh/hyperdrive-fuse> , but it's not
maintained currently.
- Q: Any plans for a TRAMP interface?
- A: Good idea! A TRAMP interface may make it easier to offer
live hyperdrive filename completions.
- Q: How does this comprare to syncthing?
- A: Syncthing is useful for sharing files among a small group of
trusted peers, like an F2F network
(<<https://en.wikipedia.org/wiki/Friend-to-friend>>).
Hyperdrives are useful for publicly sharing a set of files which
you can updated going forward and which others can link to.
- Q: If you edit a file on the hyperdrive, then edit the same file on
the local mirror. How is the conflict handled when you sync the
mirror again?
- A: If I understand correctly, you're asking about what happens
when you write to the same hyperdrive from multiple machines.
The short answer is, "Please don't do that." However, it
appears that the Holepunch team is making progress on
<https://docs.holepunch.to/building-blocks/autobase> for
"autohmatically rebasing" hyperdrive history, effectively
allowing for multi-writer hyperdrives.
- Q: wouldn't user be able to collaborate asynchronously by viewing diffs on a serially "shared" file, in other words, a user would copy another peer's text file, edit and upload their changes, share the link to their updated file so that then others can see diffs and in that way co-create an evolving file?
- A: You could try this. Another idea that we have in the works is integration with <https://github.com/nobiot/org-transclusion>
## Notes
- damn, I never even knew about dired-jump (C-x C-j). Main
differences with (C-x d) from the file are one fewer keystroke and
having the point on the file you came from.
- Btw, hyperdrive looks like another one of those things that would be amazing if I collaborated with anyone using emacs
- Also incidentally hyperdrive is amazing
- I installed hyperdrive.el and tried to run it and immediately got: transient-setup: Symbol’s function definition is void: transient-prefix-object
- Need to upgrade transient.el
- Gah, you are right. I had an obsolete version just hanging out causing pain.
- Okay, I got hyperdrive.el working and it was super easy. hyper://fwsn55wnznts5mpkee16j89ja38nfz6zne4wijzap1z9ka4jsxio/Dird_a333f1_4884540.jpg
- I think hyperdrive is a pretty easy way to share files. Easier than Dropbox. But maybe for delivery? I'm not sure.
- A: What do you mean by "delivery"?
- I was thinking of asset delivery. I write professionally, so I have things like PDFs and other documents to deliver. Sometimes I have big files to deliver to clients. Using something like hyperdrive would be cooler than Dropbox. ... Hmmmm, is there a hyperdrive web gateway?
- A: A good non-Emacs option is <https://github.com/AgregoreWeb/agregore-browser> Agregore is Chromium with built-in support for more protocols besides HTTP, like Hyperdrive, Bittorrent, IPFS, Gemini, and Gun. <https://github.com/RangerMauve/hyper-gateway> , the program that hyperdrive.el connects to as a client is, I think, exactly what you asking about. I don't know of a public HTTP gateway, but you could easily run hyper-gateway on a VPS.
- That would be fun. Then I could control the domain name so it would look cool and professional and like I was extra high-tech.
- A: hyper-gateway lets you serve multiple drives at various subdomains. For example, you could serve your primary drive at <https://reverik.tld>, and then folks could access, e.g., the USHIN drive at <https://aaj45d88g4eenu76rpmwzjiabsof1w8u6fufq6oogyhjk1ubygxy.reverik.tld>
- Great work, everyone. Thank you.
- Prot's presentations are so clear. Perfectly model pedagogy.
- Just like his code: his .el buffers SQUEAK as you C-v through them.
- Prot is indeed a model of paying attention to detail.
- I have learned so much from his videos, and from his code.
- Prot switching from vim/etc to Emacs seems to correspond to an inflexion point in the current Emacs renaissance does it not? Coincidence, or... I wonder what way the causal arrow goes :)
- I think the current "Emacs renaissance" probably dates back to around the time that package.el was developed and then when MELPA came online. momentum seems to have been building since then
- We must also consider Org-mode's inception as a key moment in recent Emacs history
- My Emacs environment would be much poorer without Prot's contributions, that's for sure.
- My hypothesis is that Diogenes himself made his way to Cyprus and spent the past immortal couple thousand years as a mountain hermit waiting to return to mortal life when the Emacs community needed him. Humor aside, there are a few key personalities---many of them here this weekend---who have been disproportionately catalytic to the community recently. Prot is certainly one. Thanks to all of you!
[[!inline pages="internal(2023/info/hyperdrive-after)" raw="yes"]]
[[!inline pages="internal(2023/info/hyperdrive-nav)" raw="yes"]]
|
