# One Big-ass Org File or multiple tiny ones?  Finally, the End of the debate!
Leo Vivier

[[!template id=vid src="https://mirror.csclub.uwaterloo.ca/emacsconf/2020/emacsconf-2020--12-one-big-ass-org-file-or-multiple-tiny-ones-finally-the-end-of-the-debate--leo-vivier.webm" subtitles="/2020/subtitles/emacsconf-2020--12-one-big-ass-org-file-or-multiple-tiny-ones-finally-the-end-of-the-debate--leo-vivier.vtt"]]  
[Download compressed .webm video (22.3M)](https://mirror.csclub.uwaterloo.ca/emacsconf/2020/smaller/emacsconf-2020--12-one-big-ass-org-file-or-multiple-tiny-ones-finally-the-end-of-the-debate--leo-vivier--vp9-q56-video-original-audio.webm)

Many discussions have been had over the years on the debate between
using few big files versus many small files.  However, more often than
not, those discussions devolve in a collection of anecdotes with
barely any science to them.

Once and for all (or, at least until org-element.el get overhauled), I
would like to settle the debate by explaining why the way we parse
Org-mode files becomes slower as our files grow in size or numbers,
and how that affects their browsing and the building of custom-agenda
views.

I feel qualified to talk about this topic for two reasons:

-   I went through the trouble of optimising my agenda-views by
    implementing clever regex-based skips, so I know the ceiling that
    can be reached with the current tech.
-   My work on Org-roam has led me to consider the use of an external
    parser for Org-mode files, and whilst we are only at the prototyping
    stage, we know what is at stake.

I intend the talk to be fairly light-hearted and humorous, which is the
only way we can do true justice to the topic.

<!-- from the pad --->

- Actual start and end time (EST): Start 2020-11-28T13.43.24; Q&A
  2020-11-28T13.51; End: 2020-11-28T14.00.07

# Questions

## What's better: one big file or many small ones? :>
For knowledge management: many files (see also org-roam).

Otherwise: one big file to have everything (todos, projects, notes,
etc&#x2026;) in one single place.

- Possible walk around by some hacks?

## Do you switch between British and French accents?

## What's the Emacs icon
Browser extension for org-protocol (anyone got the link / name?) is
this <https://addons.mozilla.org/en-US/firefox/addon/org-capture/> or
this
<https://addons.mozilla.org/en-US/firefox/addon/edit-with-emacs1/>

## How do you feel about archive files in org mode, how can that work in?

## Could you post links?

## How big are your org files?
Main file: 38000 lines for all GTD-tasks and he does archive.

Karl does use archiving although Karl does use Org tasks even in
knowledge management and those don't get archived most of the time.

## Does it not consume more resources and time to load multiple files than a large file of the same contents?
Dealing with hiding contents is computationally expensive.

- I doubt it is correct. Emacs display engine is quite effective
  dealing with invisible text. Moving cursor around is affected, but I
  never heard (and never experienced) issues with scrolling on large
  (2Mb) org files.
  - Actually, Org currently uses overlays to hide text, and the
    overhead of the overlays does eventually add up.  There's a
    working branch that uses text-properties instead, and it may be
    merged to Org someday.
    - It is on the way ;) I need more feedback (see help request in
      <https://updates.orgmode.org/>).
      - If I ever have time to even get my Org upgraded to the latest
        version, maybe I can think about trying to test that ;)
        - Would it help to share the branch on GitHub?
          - It would probably make it easier to use and more visible,
            so&#x2026;maybe?  :)
            - Noted (or rather captured) (using org-mode right? :)
              Indeed.
  - Karl: whenever I had severe performance issues and somebody was
    nice and helped to analyze the issue, "overlays" were the root
    cause in probably 90% of the cases. However, an average user
    (including me) does not know if a specific feature is implemented
    using overlays or not. My Org life is basically try and error ;-)
    - alphapapa: FYI, if you use org-indent-mode (or whatever the name
      is of the mode that uses overlays to indent contents), you could
      disable that to reduce the number of overlays in a
      buffer.
      - Karl: thanks a bunch. However, some features are delivering
        important features to me so that I do have to accept the
        performance overhead to a certain level. That's a difficult
        trade-off I do have to make from time to time ;-)

## Doesn't using many small org file clutter up your buffer list when generating agenda etc?
Personally, I limit org agenda to just a few files while keeping notes
in many more.

# Notes
- Speaker's emacs.d: <https://github.com/zaeph/.emacs.d>
- Mentioned: <https://karl-voit.at/2020/05/03/current-org-files/> ->
  Karl's big Org files.
- org-element.el: <https://orgmode.org/worg/dev/org-element-api.html>.
  - single-threaded lisp function that parses the whole file.
- "the problem is to let org-element to make sense of the item (?)
  &#x2026;".