# One Big-ass Org File or multiple tiny ones? Finally, the End of the debate! Leo Vivier [[!template id=vid src="https://mirror.csclub.uwaterloo.ca/emacsconf/2020/emacsconf-2020--12-one-big-ass-org-file-or-multiple-tiny-ones-finally-the-end-of-the-debate--leo-vivier.webm" subtitles="/2020/subtitles/emacsconf-2020--12-one-big-ass-org-file-or-multiple-tiny-ones-finally-the-end-of-the-debate--leo-vivier.vtt"]] [Download compressed .webm video (22.3M)](https://mirror.csclub.uwaterloo.ca/emacsconf/2020/smaller/emacsconf-2020--12-one-big-ass-org-file-or-multiple-tiny-ones-finally-the-end-of-the-debate--leo-vivier--vp9-q56-video-original-audio.webm) Many discussions have been had over the years on the debate between using few big files versus many small files. However, more often than not, those discussions devolve in a collection of anecdotes with barely any science to them. Once and for all (or, at least until org-element.el get overhauled), I would like to settle the debate by explaining why the way we parse Org-mode files becomes slower as our files grow in size or numbers, and how that affects their browsing and the building of custom-agenda views. I feel qualified to talk about this topic for two reasons: - I went through the trouble of optimising my agenda-views by implementing clever regex-based skips, so I know the ceiling that can be reached with the current tech. - My work on Org-roam has led me to consider the use of an external parser for Org-mode files, and whilst we are only at the prototyping stage, we know what is at stake. I intend the talk to be fairly light-hearted and humorous, which is the only way we can do true justice to the topic. - Actual start and end time (EST): Start 2020-11-28T13.43.24; Q&A 2020-11-28T13.51; End: 2020-11-28T14.00.07 # Questions ## What's better: one big file or many small ones? :> For knowledge management: many files (see also org-roam). Otherwise: one big file to have everything (todos, projects, notes, etc…) in one single place. - Possible walk around by some hacks? ## Do you switch between British and French accents? ## What's the Emacs icon in the firefox address bar? Browser extension for org-protocol made by vifon: ## How do you feel about archive files in org mode, how can that work in? ## Could you post links? ## How big are your org files? Main file: 38000 lines for all GTD-tasks and he does archive. Karl does use archiving although Karl does use Org tasks even in knowledge management and those don't get archived most of the time. ## Does it not consume more resources and time to load multiple files than a large file of the same contents? Dealing with hiding contents is computationally expensive. - I doubt it is correct. Emacs display engine is quite effective dealing with invisible text. Moving cursor around is affected, but I never heard (and never experienced) issues with scrolling on large (2Mb) org files. - Actually, Org currently uses overlays to hide text, and the overhead of the overlays does eventually add up. There's a working branch that uses text-properties instead, and it may be merged to Org someday. - It is on the way ;) I need more feedback (see help request in ). - If I ever have time to even get my Org upgraded to the latest version, maybe I can think about trying to test that ;) - Would it help to share the branch on GitHub? - It would probably make it easier to use and more visible, so…maybe? :) - Noted (or rather captured) (using org-mode right? :) Indeed. - Karl: whenever I had severe performance issues and somebody was nice and helped to analyze the issue, "overlays" were the root cause in probably 90% of the cases. However, an average user (including me) does not know if a specific feature is implemented using overlays or not. My Org life is basically try and error ;-) - alphapapa: FYI, if you use org-indent-mode (or whatever the name is of the mode that uses overlays to indent contents), you could disable that to reduce the number of overlays in a buffer. - Karl: thanks a bunch. However, some features are delivering important features to me so that I do have to accept the performance overhead to a certain level. That's a difficult trade-off I do have to make from time to time ;-) ## Doesn't using many small org file clutter up your buffer list when generating agenda etc? Personally, I limit org agenda to just a few files while keeping notes in many more. # Notes - Speaker's emacs.d: - Mentioned: -> Karl's big Org files. - org-element.el: . - single-threaded lisp function that parses the whole file. - "the problem is to let org-element to make sense of the item (?) …".