Leo Vivier: [About the previous presentation:] At the end... We are right on time, so I'm
sorry if you have a lot of questions before.
You had so many questions and I
couldn't answer all of them
I'm really happy about it but I'm also
really sad that I don't have enough time to do so.
I'm going to try to do a better job this
time of leaving you a little more time
for the questions. So, just before,
a little addendum because I did screw up
in the previous presentation...
You remember I tried to rename the file
and it didn't work?
well it turns out I had two files named "baz," so
my software works great, thank you very much.
[Org Roam Technical Presentation]: All right. So now what I'm going to do
during this presentation
is that I'm going to... Oops, I didn't start
my timer. Just give me
a little second, and let's subtract
one minute. Okay. Good.
So, what I'm going to do right now... It's a
little different from the previous
talk I gave you, and different even
from what Noorah gave you.
There's scaling the mountain as
far as difficulty is concerned.
On this one, I will be telling you about the
technical aspects of org-roam,
because I've been telling you
about the general philosophy
of the notes and the general philosophy
of organization,
but right now, I really want to get into
the nitty gritty about org-roam.
So if we go in the git repository,
this at the very core is org-roam.
For some of you who have no experience whatsoever
developing stuff or programming or
anything along those lines,
this is how all the development around the world
is working. You have a repository, a
git repository, where you have all the
files, all the libraries you're using,
all the programs, all the commands.
Everything is inside your files.
And in a way, this is the org-roam project.
You can see that we have many files.
We have org-roam-buffer, -capture, -compat,
-completion, -dailies, etc. etc.
So, before we dive a little deeper, I just
want to give you a lay of the land, so to speak,
to know where we're heading.
Org-roam is built on top of Org Mode,
and Org Mode gives us plenty of tools
to play around with the files.
I'm moving the glass.
I'm starting to move my hands a little
bit. You know, when I get excited about
something, I move my hand,
and then that stuff happens. So
in org-roam, we have Org Mode and
Org Roam gives us plenty of tools which
are incredibly useful
for writing stuff. So you know we already
have the links,
we already have the hierarchy which is
given by having trees within trees within trees. We have
quote blocks. We have babel blocks.
We have so much stuff. We have an arsenal of
tools that have been developed
for the last 15 years.
When you think about it, org-roam just
wants to create backlinks. It sounds
something very simple, but the problem is
that we need to play nicely with all of those
intricate pieces. The fact is, it takes quite a
lot of expertise to be able to do so
because if... Right now we are in the brain of org-roam,
but if I show you the brain of Org Mode...
So this is the brain of Org Mode.
It looks very simple like this because I
haven't entered the lisp/ folder.
I'm just going to enter it. I'm going
to zoom out a little bit. Don't worry if you
don't see everything.
but I just want you to get the
sheer feel of magnitude
that is Org Mode. So right now, we are
in a very small size... What I'm going to do,
I'm going to skip
one page, one, two,
three, we have... Let's just check how many
lines we have.
Okay. Let me just revert to a fairly
readable sight.
At the bottom, you can see that we have...
Oh it's not showing because it's a
little small. Okay, I'm just going to
resize the window a little bit.
It's not showing up. Give me a second. I
can't see how many lines I have. Okay. So
let's do it the better way.
I'm going to go back at the beginning of
the buffer and we're going to count
how many lines we have. So the bottom, in
the mini buffer (and the mini buffer is this area),
we have 377 lines,
which means 377
libraries within Org Mode. And mind you,
that's not counting all the modules that
we have on the side, which
come on top of Org Mode. Now when you
try to think
about something so elemental as links,
you have to think about how to play well
with every single one of these modules.
Now, obviously not the 370.
Sometimes, you know one module it's not going to do anything.
I'm not sure ob-calc could be doing
anything with it.
But it's something that we have to keep
in mind.
So, really early on, when we started
developing org-roam with Jethro Kuan, my
co-maintainer, you know we had this idea that
we wanted to develop something that was optimized,
something that would scale very
nicely, whether or not you had...
Something that would work as
fast if you had 10 files,
or if you had 100 files, or if you had
10,000 files and maybe more. So the
problem when you do this--
and I'm doing some callbacks to the talk
I gave you earlier today about
a few big files versus many--
I got confused--few big files versus many
small files,
the problem with this is that we need to
think about optimization from the get go.
So one of the decisions we took when
we got started
with org-roam is that if I go in my
test repository--so that's the one in
which we were right before--
we have a file which is called org-roam.db.
Now if I open it, it's not... It's a
little garbage because it's a binary.
But what we have
is a database with which we communicate
via... Sorry, it's an SQL database.
And what this allows us to do
is we store all the information we need
inside this SQL database which allows us
to speed up a lot of the operations that
are necessary for the functioning of org-roam.
So, for instance, if I go back to the
index file that I had before...
Let's just go back to "foo," actually.
This way you'll see a little more on the side.
So you see that on the side we have
two links. I'm not going to click on them;
otherwise, I'm going to open them.
But we have two links.
Now there are many implementations of
the Zettelkasten method inside Emacs
and with Org Mode.
But what we've decided to do
is that every time you have a link--so if
we go to the index again here at point we have
the link "foo." Every time we create a link,
we update our database
to say, okay, so we have a link in the file
"index" which is leading to the file
"foo.org" and it is situated
under the heading "A heading."
If you check the side buffer, you see
that all this information
which I just highlighted to you
is present right here.
Oh, sorry I forgot this. Thank you.
So let's see. Log. Okay.
I'm going to split like this.
I'm going to go back there.
The problem is that I can't show my
keystrokes at the same time as
I'm showing the side buffer, so I'll
keep it right now for your own
discretion. Anyway, getting back to the talk.
So the thing is we have this
SQL database. The goal is to keep it optimized.
Now, why is it better optimized than just
using default Org Mode?
So in my talk about many big files
versus a few--I keep getting.-- you got
what I was saying; I'm not going to
repeat it--
By the way, it is 10 to 10:00.
I'm starting really to be tired now.
Moving on to...
What did I want to show you? So it was
almost... Yes, org-elements.
So what I'm going to do... I'm going to...
I believe it's org-element-parse-buffer.
So I was telling you about org-elements before.
And the main command--sorry, the main
function that is used by org-element
is -parse-buffer. What it does (and you can
see the docstring) is that it
recursively parsed
the buffer and returned structure,
structure being all the information that
we have in this buffer.
So just to show you a little more, we're
going to move into a scratch buffer,
and what we're going to do is that we're
going to write this command
org-element-parse-buffer, and we're going to check the
output of this command.
Sorry, not this one. We're going to go
in the index. So in the index file, you have
a title, you have a heading, you have a
link, etc. etc.
So what I'm going to do, I'm going to
evaluate this text.
Now at the bottom in the mini buffer,
you see an AST, an abstract--
obviously don't remember what the S
stands for... Semantic?--
Huh. Interesting. Anyway. A representation
of the data
in a way that is exploitable by a
machine. Now what I'm going to do--
syntax, thank you--so what I'm going to do,
I'm going to paste it inside the buffer
in a way that is humanly readable.
You can see that we have plenty of
information. We have a section which
starts at the char 1, which ends at the character 45.
We have the content... so "Emacs Scratch"--
oh actually, no, never mind, I did something wrong, I ran
it in the wrong buffer.
So actually what I'm going to do, we're
going to run this command
with the selected window.
Next window. Okay. That's a bit of live
Elisp writing for you right now.
Now if I evaluate this
and paste the content of the buffer,
it is doing its bidding.
So now what we have...
We have a section. We have the keyword
TITLE which you see right here. You have the :value.
If we scroll down a little bit, we have a
heading which is right here. We have the contents,
which should be... Yes, the content is not
listed exactly here, but you have a
paragraph, which is this,
and then you have a link, etc. etc.
It is all parentheses if you're not used to Elisp.
Like, right now, I've selected only the
content of the parenthesis link.
I can move like this etc. etc.
I'm not... It's not an Elisp
lesson that I'm doing right now, but basically,
if we were to use the default tooling of
org-roam--Org Mode, sorry, I keep getting too confused, sorry for that--
would be extremely slow to do what
we're doing. Some people
who are doing some implementations of the
Zettelkasten method inside Emacs have
opted for this method, but the problem is that
we think that it scales poorly.
Now some other people have decided to not do with a database,
and what they do is that they use a tool
which is called ripgrep.
You might know grep, which is a tool that
allows you to search
a file, the content of a file, for a line.
So for instance, if we open vterm here,
let's see... I've opened the term.
I am in this repository.
What I'm going to do is that I'm going to
load the content of the file.
How am I going to do this? I need to move to bash.
Let's do grep
for the line... Which links did we
have... grep foo
inside the file. Is it three? I can't
remember. Okay. Let's do this.
Am I working? No.
Let's go for four? Why, is it eight?
Oh, you know what, I'm just
going to copy the name.
There we go.
Problem with live presentation, always.
You know what, I'm struggling, so I'm
going to drop this point. Anyway,
so grep is a simple tool that allows you
to search the content of a file, but
ripgrep is a solution that is written
in Rust and which is supposed to be--
well, not supposed--which is far more capable.
Now I'd like to talk to you about the future
of org-roam. Right now, I've told you about
the general concept, which is about using
this SQL database, and about
playing nicely with Org Mode.
We think that there's something great
that we can do about org-roam.
Now I've been talking with a lot of
people who are behind Org Mode and you know,
they've told us, "Do you think that
org-roam could have something to bring to
Org Mode? Let's say,
backlinks? Is there something that we
could be doing to
import backlinks into Org Mode?"
We thought about it with Jethro and the problem is
we've always tried to have an
experimental ground,
a very isolated portion of your system
where we could track backlinks and
that's why we use
a slipbox directory, so that we only
track backlinks in one specific place.
But now, because there seems to be so
much interest about the method and we
have so much backing
on Github--we have like
2,600 stars, which is mind-boggling to us---
because we have so much success...
We have plenty of ideas about the future.
One of the key parts of development being the writing
of an external parser for org-roam.
So I've been telling you about org-element. org-element runs
inside Emacs. But what if
we wrote a background process
that could read a file, an Org Mode file,
extract the same type of data that you
see on your screen right now,
so that we could use to update a database
that we could use to compute the
links, so that we could use it
to show org-roam server all the
connections between your nodes?
Now there is a path of improvement here
that is extremely important to us.
But you know, that's the technical aspect.
I'm out of time. I'm just going to
take one more minute to finish on this point.
We believe that org-roam has the potential to be a
think tank,
in a way, for Org Mode and the way we
think about
note-taking in general. I've stressed a
great deal in my first presentation--
sorry, the one I did before Noorah--that
org-roam is really great as a way to
think organically about knowledge.
Honestly, we want to put the
theory into practice with org-roam.
We are holding something which has the
potential to be
a great factor of innovation for the
future, whether it be Org Mode
or even for software in general. You know,
the way to think about...
build nodes of knowledge in a way,
and the way to represent all those ids
with the graph...
the way to basically have a note-taking system that
corresponds to the research that
corresponds to the way you think.
I believe we are really excited about this and if you
want to keep track of the development of org-roam
on my YouTube channel (which is already
linked a little earlier
inside the pad),
I do have a Youtube channel where I try
to present novelties
or the new stuff inside org-roam.
I'll also be recording videos about the
technical aspects, about the direction
that we're taking with org-roam.
If you want to talk with us we are
always available either on IRC channel #org-roam
(I believe there's a dash between org and roam) but also
on the Discourse. I'll be putting all
the links inside the conversation.
And that's me done. So, thank you for
listening. Now I'll be taking
three minutes of questions so as to be
right on time.
(Amin: Many thanks for your awesome talk, Leo.)
Leo: Thank you.
I'm just refreshing the page, and I'm
going to scroll down to my
talk if I can find the right section.
Let me just scroll a little bit.
Reproducible Emacs. No, I think it's
slower...
We have so many questions, so at the
same time I'm pissed because I can't
find it,
but I'm really, really impressed by the
number of questions that we had. (Amin: Oh yeah.
Yours is about I think about line 600
or so.)
Leo: Yes, got it splendid.
So, the questions. "So why not run a
background Emacs for parsing instead of
implementing a new parser?"
I believe we've had this question.
I was giving a similar talk
earlier this week--and this week,
I'm not French, this week, sorry--and
someone asked me this question.
The thing is running a background Emacs
process... You know, it sounds great,
but it's also very limited because all
the problems we have
about concurrency, about threads in Emacs...
Well, yes, we can forward all our calls to
background Emacs
just like when you export a file
with... Sorry,
Amin, could you mute microphone when
you're not speaking, it's a little hard for
me to concentrate?
That's fine. Don't worry.
Where was I? I'm sorry. The
question. Yes.
So, basically, forwarding all the
questions--sorry, all our queries to a
background Emacs.
That is what org export is doing. Like,
you have the ability to
asynchronously export LaTeX documents,
ODT documents from
Org Mode. It uses a very minimal
version of Emacs to do that. But the
problem is that we think that it's not
going to scale as well as a true
genuine background process. Since we
have been talking a lot
as far as the Org Mode development is
concerned about,
writing a proper parser, writing a proper
documentation for the parsing of Org Mode files and
writing a proper document standard
that says, okay, this is how the Org Mode
format works, you know, to
basically have a way to not fall into
the traps of Markdown which has many
many standards...
We need to think about this and we
believe that org-roam has
the ability to think about these questions.
As a person, I'm also really interested
about this.
I can take the questions, Amin, so don't
worry about feeding them to me. So how
often does the
DB index get updated in order to contain
changes within the Org files?
So we have two ways: either we
update as soon as you save a file,
or we have a timer which is an idle
timer, which waits... Okay
the user has not inputted
anything in the last
five seconds, so it's time to queue a
database--parsing--
a rebuild of the data, not an
incrementation
of the database, I should say.
"Did you ever think of..." I believe I
have one more minute and then
I'll hand it to the other folks.
"Do you ever think of opening up or
designing the SQL DB as a general Org
speed up tool outside of org-roam so that
other libraries that do execute
complex queries are able to use it?"
Well, a lot of people have been working on this
I believe alphapapa has been
thinking quite a lot about this.
org-ql is... The ql stands for query language.
I can't remember now what the backend is
for org-ql, but the idea is
relatively the same, you know. It's about
finding ways to optimize the way we
store the data about an Org Mode file
and how we retrieve it,
and SQL for us seems to seem to be a
good idea. Now, obviously,
maybe we could do something about
Org Mode, but the problem is, I think, a
background process
is not necessarily in
the core mentality of Org Mode. But it's
definitely something that we could suggest
when we are a little more mature, because, well,
org-roam was started last February and so
it's a fairly young project in a way.
I see plenty more questions, but
I'm out of time, folks, so I'm not sure.
The other speaker is probably ready.
So what I'll do is I'll probably try to
answer your questions when I get the
time inside the pad,
but feel free to ping me on IRC
or on the different channels we have
for org-roam, and
I'll answer them with as much
energy as I can gather.
All right, thank you so much.
(Amin: Thank you again very much, Leo.)
Leo: And that was me done for today. So you'll see me at the end, but I'm
officially done and I am free of
thoughts. I can focus on
sleeping, probably. (Amin: Awesome)
Leo: See you guys later. Bye.