summaryrefslogtreecommitdiffstats
path: root/2020/info/18.md
blob: c467dcb1378a29231c312bbd37780ca7b200418c (plain) (blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
# Org-roam: Technical Presentation
Leo Vivier

Org-roam is a Roam replica built on top of the all-powerful Org-mode.

Org-roam is a solution for effortless non-hierarchical note-taking
with Org-mode.  With Org-roam, notes flow naturally, making
note-taking fun and easy.  Org-roam should also work as a
plug-and-play solution for anyone already using Org-mode for their
personal wiki.

Org-roam aims to implement the core features of Roam, leveraging the
mature ecosystem around Org-mode where possible.  Eventually, we hope
to further introduce features enabled by the Emacs ecosystem.

The purpose of the talk is to present some technical aspects of
Org-roam.  From the very beginning, we wanted Org-roam to scale with
your notes, and this meant that we had to keep a close eye on our
performances.  As we iterated, optimisation remained a top-priority,
leading us to constantly peek under Org-mode's hood.  Not only has
this made us better developers, but it has also uncovered paths of
optimisation for Org-mode itself.

The talk is targeted at software engineers willing to peek under
Org-mode's hood.  A rudimentary understanding of Elisp will be
required.

Points to be covered

-   SQL database via emacsql
-   Elisp libraries
-   Parsing of Org-mode files
-   org-elements.e
-   Parsing with a background-process
-   Ensuring consistency via hooks

<!-- from the pad --->

-   Actual start and end time (EST): Start 2020-11-28T15.39.41; Q&A  2020-11-28T15.56.29; End 2020-11-28T16.01.03


# Questions


## Why not to run a background Emacs for parsing instead of implement a new parser?


### Running a background Emacs progress sounds great, but is still limited. Forwarding all queries to a background Emacs (like org-mode's exporter does) is only feasible with a (??? zaeph can probably fix the answer)


## How often does the DB index get updated in order to contain changes within Org files?


### Either on save, or on idle-timer.


## Did you ever think of opening up (or designing) the SQL DB as a general Org speedup-tool outside of org-roam so that other libraries that do execute complex queries are able to re-use the summarized data?


### FYI, see John Kitchin's work, he uses a SQLite database to index his Org files. <https://kitchingroup.cheme.cmu.edu/blog/2017/01/03/Find-stuff-in-org-mode-anywhere/>

-   John's DB approach is great. However, we should not end up using several DB-index in parallel. ;-)+1


## Obviously with the 'global backlinks' agenda, it would be interesting to combine with the eev stuff from before :-) ( <https://github.com/edrx/eev> )


## about the external program, you could just talk to the PANDOC guys (or Firn [Parses org-files into data structures with Orgize <https://github.com/PoiScript/orgize> ], Logseq [OCaml & Angstrom, for the document parser <https://github.com/mldoc/mldoc> ]), they're very helpful and have already a good org-mode parser


## Is it feasible to have this process of parsing org-roam following the LSP protocol? that would allow to be editor agnostic, and it would save the work to define the communication protocol and any other technical details.


# Notes


## "org-roam just wants to create backlinks"


## org-mode has many many files (377 lines in dired&#x2026; including .elc files)


## If you want to create an index of all the org files using the native format, it would be very slow. So org-roam uses a sqlite database


## ripgrep (written in Rust) is more capable than grep; used by some Zettelkasten


## "Is there something we could do to import backlinks into org mode?"


## "We've always tried to have an experimental ground where we can track backlinks"