summaryrefslogtreecommitdiffstats
path: root/2020/info/12.md
blob: 429330d819ff478069e186ea9597cd6103994b64 (plain) (blame)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
# One Big-ass Org File or multiple tiny ones?  Finally, the End of the debate!
Leo Vivier

[[!template id=vid src="https://mirror.csclub.uwaterloo.ca/emacsconf/2020/emacsconf-2020--12-one-big-ass-org-file-or-multiple-tiny-ones-finally-the-end-of-the-debate--leo-vivier.webm"]]  
[Download .webm video, 720p, 226MB](https://mirror.csclub.uwaterloo.ca/emacsconf/2020/emacsconf-2020--12-one-big-ass-org-file-or-multiple-tiny-ones-finally-the-end-of-the-debate--leo-vivier.webm)

Many discussions have been had over the years on the debate between
using few big files versus many small files.  However, more often than
not, those discussions devolve in a collection of anecdotes with
barely any science to them.

Once and for all (or, at least until org-element.el get overhauled), I
would like to settle the debate by explaining why the way we parse
Org-mode files becomes slower as our files grow in size or numbers,
and how that affects their browsing and the building of custom-agenda
views.

I feel qualified to talk about this topic for two reasons:

-   I went through the trouble of optimising my agenda-views by
    implementing clever regex-based skips, so I know the ceiling that
    can be reached with the current tech.
-   My work on Org-roam has led me to consider the use of an external
    parser for Org-mode files, and whilst we are only at the prototyping
    stage, we know what is at stake.

I intend the talk to be fairly light-hearted and humorous, which is the
only way we can do true justice to the topic.

<!-- from the pad --->

-   Actual start and end time (EST): Start 2020-11-28T13.43.24; Q&A 2020-11-28T13.51; End: 2020-11-28T14.00.07


# Questions


## What's better: one big file or many small ones? :>

For knowledge management: many files (see also org-roam)

Otherwise: one big file to have everything (todos, projects, notes, etc&#x2026;) in one single place.

-   possible walk around by some hacks?


## Do you switch between British and French accents?


## What's the Emacs icon

Browser extension for org-protocol (anyone got the link / name?) is this <https://addons.mozilla.org/en-US/firefox/addon/org-capture/> or this <https://addons.mozilla.org/en-US/firefox/addon/edit-with-emacs1/>


## How do you feel about archive files in org mode, how can that work in?


## Could you post links?


## How big are your org files?

main file: 38000 lines for all GTD-tasks and he does archive

Karl does use archiving although Karl does use Org tasks even in knowledge management and those don't get archived most of the time.


## Does it not consume more resources and time to load multiple files than a large file of the same contents?

Dealing with hiding contents is computationally expensive.

-   I doubt it is correct. Emacs display engine is quite effective dealing with invisible text. Moving cursor around is affected, but I never heard (and never experienced) issues with scrolling on large (2Mb) org files.

    -   Actually, Org currently uses overlays to hide text, and the overhead of the overlays does eventually add up.  There's a working branch that uses text-properties instead, and it may be merged to Org someday.
    
        -   It is on the way ;) I need more feedback (see help request in <https://updates.orgmode.org/>)
        
            -   If I ever have time to even get my Org upgraded to the latest version, maybe I can think about trying to test that ;)
            
                -   Would it help to share the branch on github?
                
                    -   It would probably make it easier to use and more visible, so&#x2026;maybe?  :)
                    
                        -   Noted (or rather captured) (using org-mode right? :) Indeed
    
    -   Karl: whenever I had severe performance issues and somebody was nice and helped to analyze the issue, "overlays" were the root cause in probably 90% of the cases. However, an average user (including me) does not know if a specific feature is implemented using overlays or not. My Org life is basically  try and error ;-)
    
        -   FYI, if you use org-indent-mode (or whatever the name is of the mode that uses overlays to indent contents), you could disable that to reduce the number of overlays in a buffer. &#x2013;alphapapa
        
            -   Karl: thanks a bunch. However, some features are delivering important features to me so that I do have to accept the performance overhead to a certain level. That's a difficult trade-off I do have to make from time to time ;-)


## Doesn't using many small org file clutter up your buffer list when generating agenda etc?

Personally, I limit org agend to just a few files while keeping notes in many more.


# Notes


- Speaker's emacs.d: <https://github.com/zaeph/.emacs.d>
- Mentioned: <https://karl-voit.at/2020/05/03/current-org-files/> -> Karl's big Org files
- org-element.el: <https://orgmode.org/worg/dev/org-element-api.html>
  - single-threaded lisp function that parses the whole file
- "the problem is to let org-element to make sense of the item (?) &#x2026; "