1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
|
# One Big-ass Org File or multiple tiny ones? Finally, the End of the debate!
Leo Vivier
[[!template id=vid src="https://mirror.csclub.uwaterloo.ca/emacsconf/2020/emacsconf-2020--12-one-big-ass-org-file-or-multiple-tiny-ones-finally-the-end-of-the-debate--leo-vivier.webm"]]
[Download compressed .webm video (22.3M)](https://mirror.csclub.uwaterloo.ca/emacsconf/2020/smaller/emacsconf-2020--12-one-big-ass-org-file-or-multiple-tiny-ones-finally-the-end-of-the-debate--leo-vivier--vp9-q56-video-original-audio.webm)
Many discussions have been had over the years on the debate between
using few big files versus many small files. However, more often than
not, those discussions devolve in a collection of anecdotes with
barely any science to them.
Once and for all (or, at least until org-element.el get overhauled), I
would like to settle the debate by explaining why the way we parse
Org-mode files becomes slower as our files grow in size or numbers,
and how that affects their browsing and the building of custom-agenda
views.
I feel qualified to talk about this topic for two reasons:
- I went through the trouble of optimising my agenda-views by
implementing clever regex-based skips, so I know the ceiling that
can be reached with the current tech.
- My work on Org-roam has led me to consider the use of an external
parser for Org-mode files, and whilst we are only at the prototyping
stage, we know what is at stake.
I intend the talk to be fairly light-hearted and humorous, which is the
only way we can do true justice to the topic.
<!-- from the pad --->
- Actual start and end time (EST): Start 2020-11-28T13.43.24; Q&A
2020-11-28T13.51; End: 2020-11-28T14.00.07
# Questions
## What's better: one big file or many small ones? :>
For knowledge management: many files (see also org-roam).
Otherwise: one big file to have everything (todos, projects, notes,
etc…) in one single place.
- Possible walk around by some hacks?
## Do you switch between British and French accents?
## What's the Emacs icon
Browser extension for org-protocol (anyone got the link / name?) is
this <https://addons.mozilla.org/en-US/firefox/addon/org-capture/> or
this
<https://addons.mozilla.org/en-US/firefox/addon/edit-with-emacs1/>
## How do you feel about archive files in org mode, how can that work in?
## Could you post links?
## How big are your org files?
Main file: 38000 lines for all GTD-tasks and he does archive.
Karl does use archiving although Karl does use Org tasks even in
knowledge management and those don't get archived most of the time.
## Does it not consume more resources and time to load multiple files than a large file of the same contents?
Dealing with hiding contents is computationally expensive.
- I doubt it is correct. Emacs display engine is quite effective
dealing with invisible text. Moving cursor around is affected, but I
never heard (and never experienced) issues with scrolling on large
(2Mb) org files.
- Actually, Org currently uses overlays to hide text, and the
overhead of the overlays does eventually add up. There's a
working branch that uses text-properties instead, and it may be
merged to Org someday.
- It is on the way ;) I need more feedback (see help request in
<https://updates.orgmode.org/>).
- If I ever have time to even get my Org upgraded to the latest
version, maybe I can think about trying to test that ;)
- Would it help to share the branch on GitHub?
- It would probably make it easier to use and more visible,
so…maybe? :)
- Noted (or rather captured) (using org-mode right? :)
Indeed.
- Karl: whenever I had severe performance issues and somebody was
nice and helped to analyze the issue, "overlays" were the root
cause in probably 90% of the cases. However, an average user
(including me) does not know if a specific feature is implemented
using overlays or not. My Org life is basically try and error ;-)
- alphapapa: FYI, if you use org-indent-mode (or whatever the name
is of the mode that uses overlays to indent contents), you could
disable that to reduce the number of overlays in a
buffer.
- Karl: thanks a bunch. However, some features are delivering
important features to me so that I do have to accept the
performance overhead to a certain level. That's a difficult
trade-off I do have to make from time to time ;-)
## Doesn't using many small org file clutter up your buffer list when generating agenda etc?
Personally, I limit org agenda to just a few files while keeping notes
in many more.
# Notes
- Speaker's emacs.d: <https://github.com/zaeph/.emacs.d>
- Mentioned: <https://karl-voit.at/2020/05/03/current-org-files/> ->
Karl's big Org files.
- org-element.el: <https://orgmode.org/worg/dev/org-element-api.html>.
- single-threaded lisp function that parses the whole file.
- "the problem is to let org-element to make sense of the item (?)
…".
|
