# One Big-ass Org File or multiple tiny ones?  Finally, the End of the debate!
Leo Vivier

Many discussions have been had over the years on the debate between
using few big files versus many small files.  However, more often than
not, those discussions devolve in a collection of anecdotes with
barely any science to them.

Once and for all (or, at least until org-element.el get overhauled), I
would like to settle the debate by explaining why the way we parse
Org-mode files becomes slower as our files grow in size or numbers,
and how that affects their browsing and the building of custom-agenda
views.

I feel qualified to talk about this topic for two reasons:

-   I went through the trouble of optimising my agenda-views by
    implementing clever regex-based skips, so I know the ceiling that
    can be reached with the current tech.
-   My work on Org-roam has led me to consider the use of an external
    parser for Org-mode files, and whilst we are only at the prototyping
    stage, we know what is at stake.

I intend the talk to be fairly light-hearted and humorous, which is the
only way we can do true justice to the topic.

<!-- from the pad --->

-   Actual start and end time (EST): Start 2020-11-28T13.43.24; Q&A 2020-11-28T13.51; End: 2020-11-28T14.00.07


# Questions


## What's better: one big file or many small ones? :>


### For knowledge management: many files (see also org-roam)


### Otherwise: one big file to have everything (todos, projects, notes, etc&#x2026;) in one single place.

-   possible walk around by some hacks?


## Do you switch between British and French accents?


## What's the Emacs icon


### Browser extension for org-protocol (anyone got the link / name?) is this <https://addons.mozilla.org/en-US/firefox/addon/org-capture/> or this <https://addons.mozilla.org/en-US/firefox/addon/edit-with-emacs1/>


## How do you feel about archive files in org mode, how can that work in?


## Could you post links?


## How big are your org files?


### main file: 38000 lines for all GTD-tasks and he does archive


### Karl does use archiving although Karl does use Org tasks even in knowledge management and those don't get archived most of the time.


## Does it not consume more resources and time to load multiple files than a large file of the same contents?


### Dealing with hiding contents is computationally expensive.

-   I doubt it is correct. Emacs display engine is quite effective dealing with invisible text. Moving cursor around is affected, but I never heard (and never experienced) issues with scrolling on large (2Mb) org files.

    -   Actually, Org currently uses overlays to hide text, and the overhead of the overlays does eventually add up.  There's a working branch that uses text-properties instead, and it may be merged to Org someday.
    
        -   It is on the way ;) I need more feedback (see help request in <https://updates.orgmode.org/>)
        
            -   If I ever have time to even get my Org upgraded to the latest version, maybe I can think about trying to test that ;)
            
                -   Would it help to share the branch on github?
                
                    -   It would probably make it easier to use and more visible, so&#x2026;maybe?  :)
                    
                        -   Noted (or rather captured) (using org-mode right? :) Indeed
    
    -   Karl: whenever I had severe performance issues and somebody was nice and helped to analyze the issue, "overlays" were the root cause in probably 90% of the cases. However, an average user (including me) does not know if a specific feature is implemented using overlays or not. My Org life is basically  try and error ;-)
    
        -   FYI, if you use org-indent-mode (or whatever the name is of the mode that uses overlays to indent contents), you could disable that to reduce the number of overlays in a buffer. &#x2013;alphapapa
        
            -   Karl: thanks a bunch. However, some features are delivering important features to me so that I do have to accept the performance overhead to a certain level. That's a difficult trade-off I do have to make from time to time ;-)


## Doesn't using many small org file clutter up your buffer list when generating agenda etc?


### Personally, I limit org agend to just a few files while keeping notes in many more.


# Notes


## Speaker's emacs.d: <https://github.com/zaeph/.emacs.d>


## Mentioned: <https://karl-voit.at/2020/05/03/current-org-files/> -> Karl's big Org files


## org-element.el: <https://orgmode.org/worg/dev/org-element-api.html>


### single-threaded lisp function that parses the whole file


## "the problem is to let org-element to make sense of the item (?) &#x2026; "