Hello. If you're listening to this talk,
then you should be at least a bit interested
in Org mode, which is fantastic
because there have been quite a few
interesting developments over the past year or so.
I'm Timothy, as you may have gathered from the last talk,
and I'm also quite involved with the Org project,
so I'd like to go through a few of those developments
over the past year or so and give you a few hints as well
as to what potentially lies around the corner with Org mode.
The starters, slightly on the more boring side
but rather significant change to the project,
occurred with the housekeeping or organisation.
The codebase for the Org project has actually shifted over
from a self-hosted Gogs instance over to Savannah,
which means it's now living right alongside
the Emacs codebase.
This has been accompanied by the creation
of a whole bunch of Org-related repos under
Bastien's (Org's maintainer) personal sourceHut account.
We've got the source of the website, the Org wiki Worg,
as well as Org contrib.
Another recent addition to this list of Org-related repos
is the new Org mode tests--continuous integration.
Now, this is rather important, because while we do recommend
that all contributors actually run make tests
before submitting patches to the Org project,
this doesn't always happen.
It can also actually be a bit harder than you expect
to run the tests because there are a lot
of trans-dependencies you get with Org;
for instance, with all of the various Babel libraries
which actually require other packages
or programming language to be installed on the system.
Having a single self-contained test system
to actually make sure that Org can be regularly
and thoroughly tested should be a great help for actually
ensuring the quality of the contributions.
The funding structure for Org has also undergone a bit
of a shift. Historically, we've just directed everybody
who's interested in financially supporting the Org project
to the maintainer Bastien's personal GitHub sponsors
and LibrePay accounts. Now, early this year,
Bastion has created the Librepay Org mode team account,
which means that you can actually now
support the Org project as opposed to
the person leading the Org project.
Currently, this just distributes donations between Bastien,
Ihor, and myself. However, the idea is that
as the active contributors for the Org project
come and go over time, the list of people on this team
can be changed as seems sensible. The hope here is that
it will simplify both how easy it is
to actually financially support the Org project
as well as how easily people contributing
to the Org project can be supported.
If you're interested in supporting the Org project,
there's never been a better time than now
to have a look at this
and let anybody who might also might be interested know.
Hopefully, this leads to a healthier funding structure
that will scale better into the long term
and thus better support the work that happens with Org.
Now, the project itself has of course also seen quite a bit
of development over the past year.
We've had about 800 comments from 80 contributors.
Within these comments, there's been a lot of polishing
quality-of-life improvements,
and also quite a few new features.
Now, I haven't got nearly enough time
to go through this exhaustively,
so we're just going to go through a quick highlight reel.
There's a collection of export improvements
from things which affect all export backends,
like including remote content
and adding new things like DOI links
and support for encrypted Org files,
as well as a whole lot of export-backend-specific changes.
For example, quite a few backends--
I've mentioned the LaTeX one here,
but also others such as Texinfo--have now got
rich support for various types of attributes and objects.
The HTML backend has had a few things boosted up and well,
if you want to see the full list,
just take a look at the release notes.
We've also seen a similar collection of improvements
with the Babel backends. Once again, this is scattered--
or well, it can be split into two sets of changes.
There's some which affect all of Babel, essentially.
For instance, the new syntax of parsing
the raw content of code blocks, or the changes with Noweb.
For example, :noweb-prefix is a new option that can be used.
And then there's also a collection
of backend-specific changes. So ASCII graphics with PlantUML
or enhanced return capabilities with the ob-python library.
And then of course, as before, a whole collection
of more changes which you can find in the release notes.
Last but by no means least,
there have been quite a few changes within the rest of Org.
So this is, once again, far too many things to list,
but it's things like improved refiling,
capture templates, image preview sizing,
clocktable settings, agenda tweaks,
and well, a whole lot more.
Yes, basically, the essence of what's here
is a lot of little changes
which just address particular use cases in ways
that I don't think anybody's going to be seeing
the impact of all of them,
but I think most people should at least
find one or two things
which actually improve their own usage.
Now these are the sort of assorted
relatively minor improvements,
but there are also some major ones.
And one in particular, citations.
So I think this has been, at this point,
over a decade in the making,
but Org finally has first-class support for citations.
And I have to say, it is marvellous.
You'd hope so, after the labour. I think it is.
It can be said that it's actually worth the wait.
I think out of the various options you've got now,
(for example, the way that Pandoc
and Markdown otherwise[??] do it)
Org has a fantastically succinct and flexible
syntax for citations, which scales really well for all sorts
of different use cases.
Additionally, on the backend side of things,
we've now got a generalised way for handling citations
which has been quite helpful for the--I think
I could say rather rapid development
of multiple citation backends for Org.
And I think it's just fantastic, really, seeing
how quickly Org has gone
from having no support for citations
at the start of this year
to what can be described as
a wonderfully rich and flexible support
with, well, multiple backends for citations.
I think that's something that we can really be proud of.
And it's been a fantastic contribution
for everybody involved in this process.
Okay, so we've had features.
There have also been a whole lot of
quality of life improvements.
Once again, many more than I can reasonably mention here.
So I'm just going to flick it through
a few of them. A few big ones though,
Ihor is responsible for three lovely developments with Org,
one of which is Org fold. So this is a generalisation
of the way that content is folded in Org.
And I think quite a few of you will actually underestimate
how much can be folded in Org.
It's not just a matter of headlines.
It's headlines, code blocks, lists, environments,
all sorts of things can actually be folded in Org.
And the introduction of Org fold is important
for two reasons. One is that it has allowed for
text-property-based folding,
which in Emacs versions less than 29
has a huge difference in performance,
which is particularly apparent with larger files.
The second significant thing about this is that
it now actually provides a more general way
to actually describe changes to the folding structure.
So before there was direct modification of
messing with overlays scattered around the Org code base.
Now we have a much more well organised system
where we use Org fold to say what is and isn't folded
and to manage the state of all of that,
which is, I think, just from a sort of design,
sort of project design approach, a much better system.
We've also got the Org element cache by Ihor.
This is actually something which was discussed
quite a while ago, but has somewhat stalled
due to the difficulty of cache invalidation.
Ihor has sunk a tremendous amount of effort into this
and has improved it to the point
where we've now actually been able to
enable this by default. So what this basically does is
it records lots of information
about the structure of the Org document
and allows for, well, with the appropriate modifications
that Ihor has also made throughout the Org element library
to use this information to speed up various operations
based on the Org document syntax tree.
And so this has been quite--
the improvements have been scattered all over the place,
but for a good example for libraries
or anybody who's wanting to quickly map over Org elements
is `org-element-cache-map', which now provides
a much, much faster way to map over
all of the Org elements in a document.
This also ties into the third major feature from Ihor
which I'd like to mention, which is Org persist.
So this provides a method of persisting values
across Emacs sessions, basically saving them
to a file somewhere and loading them.
Now this works for Elisp values and it also works for files,
which we made use of with the improved capabilities
for remote files and exports.
This has also been used with the `org-element-cache' data.
So now, if you've got a massive Org file
and you open it once, that data can be saved to
with the Org element cache to Org persist,
and the next time you load this file
in another Emacs session,
we can just start with the cached data
instead of having to construct everything from scratch,
which is quite nice. Once again, a change which
much like the other ones,
we will see more of an impact in larger files,
but a very welcome one everywhere.
Now with remote files, there's also been the beginnings
of a bit of an effort with Org
to improve the approach we have to safety.
So in this case previously,
Org would unconditionally download
all the remotes of files that's all referenced.
And now, it's actually going to maintain a list of
sort of safe resources and prompt you
when it's surprised by something,
to work out whether it should
just download this one resource,
mark the whole domain as safe, and a few other options.
We're also going to probably see
a similar approach extend to, for instance,
bits of Babel execution in the future.
Okay bug fixes. It will be remiss of me not to mention that
along with all of the features
and quality of life improvements,
there has been a huge pile of bug fixes.
I think the best way to actually get a look at this
would be to look at the release notes
or maybe even the actual commit log,
but you could also just take my word and say that
there have been a whole load of them
over the past year. So just yes, the code base, I think,
is just gradually getting into better and better shape.
Asynchronous session evaluation is I think possibly
the final quality-of-life improvement
I want to mention. This came early in the year,
just with ob-python, and it's been delayed
because in order to actually make it work,
they've required some fundamental changes to the way
that ob-comint works.
Now that's been implemented,
we've since seen support extended to ob-R,
and hopefully, we'll see more languages join this list
in the not-too-distant future.
Now I guess one bonus which I tacked on just for fun is
it's now more convenient than ever
to actually specify the permissions for tangled files.
Previously you had to give a list expression
which should be evaluated.
Now you can give it directly in octal form
instead of being a list expression
that produces an integer representation
of the octal permissions.
Or you can use ls style: rwx and dashes.
Or even just chmod. I want to be able to execute this
as a user, which will basically modify
the default permission to add that capability.
Alrighty. So that's the Org project itself,
but there's also a whole ecosystem.
So what have we got here?
Well a whole bunch of Zettelkasten
or personal-knowledge-base-type projects.
One of which is logseq, so that's an online open source
Zettelkasten which supports both Markdown
and also Org mode as a first-class format.
Then of course we have Org Roam, which provides
a Zettelkasten built directly on top of Org within Emacs.
Both of these are seen considerably interesting
over the past year.
Moving on to visuals, minad has produced
another lovely minad package in the form of org-modern
which just spruces up the visuals of all documents
and seems to have been quite well received recently
since its release.
Building on top of the citations from earlier,
Andras Simonyi has produced
the wonderful citeproc-org library,
which, if you're not familiar,
allows the capabilities of the citation style language
which has now become something
which is quite widely supported
to be used for Org citation exports.
This means that you've got access to I think at this point
is it thousands or tens of thousands
of different bibliography and citation formats
which is obviously a huge boon to org citations.
Lastly, just to be slightly critical,
I'm actually going to mention the Neovim Org mode project,
because I think this really shows the interest
in Org as the format, beyond just Emacs.
I think I haven't gone into it much here,
but there's been quite a lot of development
with external tools making use of the Org format.
Clearly, we've done quite a few things right,
and so I think it's interesting to see the interest
that exists outside of Emacs,
even without all the lovely tooling we've built,
just out of appreciation of the formatting, its potential.
Speaking of the format, though,
we've also seen three new parsers on the scene this year.
We've got one in Julia, Haskell
and another one for Tree sitter.
The last one, I think, is currently the least capable,
but also potentially the most interesting
in terms of what possibilities it allows for.
Okay. So that's a quick speed run through
some of the developments over the past year.
What's coming next? So there's been
quite a lot of work done with the Org syntax document.
In fact, I've completely written it,
and we've now taken it up to spec
to actually accurately describe the way
that the Org format is, as of Org 9.6.
Now, I think this is quite an important document
for the growth in parsing tools
to help actually ensure
that the way that external tools process Org
is actually in sync with the way that Org mode does.
I think it's worth doing everything we can, really,
to avoid the sort of implementation drift
that we've seen with Markdown.
I don't want anything near that.
This is also quite good for the Org format itself because,
in the process of going through this sort of effort,
it brings attention to irregularities in the syntax
which you might want to resolve, and as well as
helping the robustness of org mode itself.
So ultimately, this is to everybody's benefit, really.
It's also my personal hope that this might actually
get to the point where we consider submitting this
as a text format to the Internet Engineering Task Force
as a new text format. So that would be quite nice.
The Org project itself has a layer on top
of the mailing list called Woof developed by Bastien,
and that's about to have another major release.
So what this is going to do is improve the ease of which
we can actually monitor what's going on
with the mailing list. So this is when people have patches,
bug reports, or other types of things
raised on the mailing list.
It's a nice way to collect the status of those
and put them all in one place.
So improvements to this improve the ease of which
the Org mode project can be managed,
which is always quite nice to see.
There's also been--jumping back to the export
which is mentioned right at the start of this presentation--
we've got the introduction of engraved faces
to LaTeX export. Now what's quite interesting about this
is that it's actually used as Emacs' native font lock
and allows for processing that in a generalized way
to different output formats. So at the moment,
this is just integrated with ox-latex,
but it contains the functionality needed for HTML and ASCII,
and it could also be extended to other formats like ODT.
So we could potentially have full syntax highlighting
based on Emacs exported to, well,
really all of the Org mode backends,
except, I suppose, the plain text ones like Markdown.
And I think from both the capabilities perspective--
because I think, really, font lock in Emacs
from Emacs major modes
tends to blow basically everything else vaguely used
out of the water, whether it be listings, minted
or other efforts--and also from a consistency point of view,
this could be quite a nice development.
Alrighty. Now this talk is "This Year in Org,"
and I think you all may have guessed
this is very much tied into my work with This Month in Org
which started, I think, a bit over a year ago.
And so, as you're all avid readers,
I'm sure you've noticed
that there haven't been as many posts as of late.
Now this isn't because my interest in This Month in Org
has been diminishing. Simply, it's the consequence
of an evaporation of my free time.
However, This Month in Org is still going to stick around.
The only change really is that the title is going to be--
probably continue to be
a bit more aspirational than descriptive
in the near future. We'll see how this goes.
Well, thanks for listening to this overview
of the state of Org at the moment,
and hopefully, I'll see you next year.