[Lukas]: Welcome to our presentation,
Collaborative Data Processing
and Documenting using org-babel.
My name is Lukas Bossert, and I'm
from the RWTH Aachen University
in the city of Aachen, Germany.
[Jonathan]: And my name is Jonathan Hartmann.
I'm also from the IT Center here at RWTH Aachen.
[Lukas]: Great.
And we will show you today how you
can use Org Mode for data processing.
So you see a little workflow what we are going to do.
First, we will give you a slight introduction to Org Mode.
Then we will dive into the part of data preparing.
First, you're going to query the data using the language SPARQL.
Then we're going to clean it using a different language.
And in the main part of our presentation,
we're going to do the data processing, first aggregating
using Python, later on counting items using Org,
and even visualizing it using R. At the end,
we're going to show you how to preserve
the data and the document and its documentation,
first doing in plain exporting, then adding some metadata,
and showing you two different ways, first a manual export,
and also then a batch-processed export.
All right.
Let's dive in to that.
Jonathan, can you give us an introduction about Org Mode?
[Jonathan]: Of course.
So in case anyone isn't familiar with it,
Org Mode, in the words of Carsten Dominik,
is back to the future for plain text.
So this is just a module available for Emacs,
plain-text base.
It's been around since 2003, which
makes it about 20 years old.
And it's extensible and fully customizable.
And especially, it's very convenient, very good
for scientific text production and organization.
So for example, you can do project management, agenda,
diary, journaling, personal knowledge management,
presentation.
Even this is written in Org Mode.
It's an Org Mode presentation.
You can do single source publishing,
which we will do later on, and also
literate programming, which is the core of our talk.
OK.
[Lukas]: So let me stop this presentation here.
So what you see here is the plain text underneath it.
So this is Org Mode.
And Jonathan, since we kind of already
did the introduction together, should we
also do the working part together?
[Jonathan]: Of course.
So you see on the screen there on the right,
that's my screen in Emacs.
And Lukas, why don't you host a session using CRDT,
and I'll connect to your buffer.
[Lukas]: OK. Great.
I do that.
So what I do, I'm using Doom Emacs.
And I can use the `SPC` and then the `l`
for the live share/collab part.
I can use the `s` for share current buffer.
So when I do this, I'm getting asked for some settings.
I'm going with the default settings here.
So default port, no password, and my display name.
And now Emacs is connecting.
And once it's connected, which just takes a couple of seconds,
I can get the URL.
So I'm going back to this menu and using `y`
for copying the URL of the current session.
And this is the URL I'm going to send over to you, Jonathan,
to pick that up.
[Jonathan]: Right.
OK.
And now on my screen, I'm going to do a `SPC l c` for connect.
And I'm going to paste the URL
that Lukas just sent me in here.
Default port, no password.
And we're connecting now.
So this takes a second just to get us synced up.
So we can work on the same document at the same time.
We can follow each other's cursors around.
We can have multiple buffers open and work on them
at the same time.
And so here you see that we are both in the same document.
You can see my cursor popping around.
And you can see we're both editing the same item.
Great.
[Lukas]: So we also see who else is currently in our buffer
with the user overview.
So let me just delete that window.
And that's going to work in our main one.
So we said first part is about data retrieval.
So we should give it a headline.
We said prepare stage.
So what are we going to do first, Jonathan?
[Jonathan]: So what we're going to do,
what this whole document is based upon,
is we're going to pull data from Wikidata using a SPARQL query.
The data we're going to pull is related to the NFDIs,
which here in Germany is the National Forschungsdaten
Infrastructure, which is a sort of collection of universities
that work together on various research projects.
And this is emblematic of the kind of data
that we would be interested in working with here.
So I'm going to paste a--forgive the pre-written code--
I'm going to paste some text in here.
[Lukas]: And while you are talking, I just
keep on documenting what we do
so we can split the work.
[Jonathan]: In here, after a minor technical upset,
is the raw dataset cell.
And it's going to use SPARQL,
which is how we have the syntax highlighting
in our code here.
It's going to go to the URL endpoint
query.wikidata.org/sparql ,
and it's going to return the data as a text CSV,
and it's going to cache that data
so that we don't constantly hammer the API every time
we run this notebook.
So I'm going to run that there.
You can see down at the bottom of my screen,
we're contacting the host query.wikidata.org .
[Lukas]: And there's the result.
[Jonathan]: Yeah, except I think that for our purposes here,
we're just going to limit this to 50 results.
[Lukas]: Oh, yeah.
[Jonathan]: Just so it's a little easier for us to manage.
I'm going to run that again.
There we go.
That looks a little better.
[Lukas]: I think that's fine.
50 items is fine.
So what do we see here, Jonathan?
[Jonathan]: Right.
So the first thing we see when we look at this
is a couple of Q codes at the top,
which are an artifact of Wikidata.
So these are pages which don't have
the label for whichever institution they happen to be.
For our purposes here, we're just going to exclude them.
We could just go on Wikidata and edit them ourselves.
But for now, it's a little more interesting
if we go and remove them.
So I'm going to create a new cell.
Lukas, if you don't mind starting one for data cleaning.
[Lukas]: Oh, yeah.
Good point.
Yeah, data cleaning.
OK.
How do you want to do that, Jonathan?
[Jonathan]: I'm going to use a shell command.
So let's see.
There we go.
And so you can see, here is another cell,
that the cell is now using a shell,
and that we have this thing `:var input=raw-dataset`,
which is the name of the cell above
where we got our data from Wikidata.
This is going to run just a simple shell command.
It's going to take the input and then run `sed` on it
and exclude any records which have a Q
followed by one or more digits afterwards.
That should remove those from our data set.
So I'm going to run that.
That seems to have done the trick.
[Lukas]: Great, yeah.
That's really good.
We got rid of all the Q items.
Very good.
So we just have two-column table: institutions
and consortia.
Very nice.
So let's come to our main part, doing some processing.
Let me give you a headline here, process the data.
What do you want to do first?
[Jonathan]: This is not a very complicated data set,
but let's just do some simple counts first.
I'm going to start with Python,
and we're just going to do some aggregation with Python.
Again, I've got some pre-written code here.
You can see that we've started a cell using Python.
The variable `clean_df` now is equal to `clean-dataset`.
So we're going to take that data
that we retrieved from the SPARQL query,
we're going to run it through the cleaning cell,
and then we're going to import it into this cell.
This is just going to do some simple Python aggregation.
We're going to import `pandas`,
which is the Python data science library,
create a data frame out of our input,
and then aggregate it, grouping on `wLabel`,
and getting a count from that and returning it.
So if we execute that cell...
[Lukas]: Nice, we get institutions and a count.
But what about not ordering it by the alphabet,
but more like ordering by counts?
[Jonathan]: Sure.
So let's do this... `sort_values()`, I think, as the Python.
How does that look?
[Lukas]: Better, but I would like to
have the highest number first
and then ascending.
Well, not ascending, descending.
[Jonathan]: Right, so we can do `ascending=False`.
[Lukas]: This is perfect, I'd say.
[Jonathan]: Great.
[Lukas]: Very good.
OK, that's nice.
We get a good overview here.
But can we also do something else,
like counting how many institutions are
involved in one consortium?
And also using this later on in the text?
[Jonathan]: Sure, so I'm going to put a new...
If you give me another heading down here
for institutions per consortium...
And here we're going to use awk code just to spice things up
and add yet another language in here.
So you can see this is awk.
We're using standard in instead of defining a variable.
But the really interesting thing about this cell
is that we have this `:var consortium="NFDI4Memory"`.
And what this code is doing is
it's counting any time it sees
that particular consortium name
and keeping track of that.
So if we execute this,
Lukas, why don't you execute this one?
[Lukas]: OK, I'm going to enter it.
And I get a result, NFDI4Memory,
because this is our default value for this variable.
And we get the count.
So it's five institutions are involved
in the NFDI4memory consortium.
Great, but the very nice thing, what I think,
is here that we can use this code snippet within our text.
So, blended in seamlessly.
Let me give you an example.
I'm writing out the text.
Now we know how many institutions are in...
Give me an example.
I would like to know how many institutions are
involved in NFDI4Objects, which is a consortium.
So I'm writing `call_` and using
the name of this snippet here, of this cell,
which is `inst-count(`,
and writing my value, `NFDI4Objects`.
As soon as I evaluate this using `C-c C-c`,
I get the result back here.
I can do this even for more.
Or in writing, `call_inst-count`, go with `NFDI4Earth`,
which is another consortium.
`C-c C-c`, it's three institutions.
This can be used throughout your text,
and as soon as the data set changes from in the beginning,
maybe different results requiring Wikidata,
this also will be updated once it's exported.
Very nice, Jonathan.
But I think we did a lot of analysis
on text and counting things.
Can we also do something more visual?
Show me something.
[Jonathan]: Sure.
So what we can do with this, because we just
have two columns here that are sort of related,
we can build a little network plot out of it.
So let's make a network visualization.
We're going to use the `igraph` library from R
and just plot the edges that we see here.
There we go.
There's my little heading and space.
Here is our code.
Again, just to be fancy and keep using
different languages in here, we set a variable called
`NFDI_edges` equal to `clean-dataset`.
So this, again, is sort of cascading
through the original data
that we pulled from the Wikidata endpoint,
cleaning that data, and now it's being inserted
into this cell as well.
But you see the difference here.
Instead of exporting a table, what we're saying
is that there will be a graphics file,
and it will be called network-plot.png.
All right.
And so Lukas, why don't you execute this one?
[Lukas]: There you go.
I can click `C-c C-c`
and I get a nice plot of the network below our cell.
So this is very nice indeed.
So I think it's about time to wrap it up and to export
and to preserve the data and the documentation
that we have in our very last step, calling preserve.
So I would like to do it in two steps.
First, maybe manually exporting it,
but then also doing it in a batch process.
Giving you some insights how to do that manual export.
For example, you can do a LaTeX export.
Let me write down the key combination to do that here.
So you press `SPC m e l o`.
Let me show you how this is done.
So I'm pressing `SPC`.
I'm pressing `m`, which is my local leader.
I'm pressing `e`, which is now the `org-export-dispatch`.
And now I have different options I can choose from.
I want to do a LaTeX export because I want to get in PDF.
So I'm pressing `l`.
Now I've got different options available.
So I'm pressing `o` for a PDF file and open that.
Let's see now the code.
Now this is exporting document.
And what we have here is PDF,
which contains our workflow in the beginning,
our bullet points we have here,
and also the code snippet
that we use for querying the data.
And we have the result below that.
So this is our table with all the data sets.
But as you can see, this is running out of the page.
So this is not very nice using the default settings.
But everything is in this PDF.
I guess we can now show you a way
how to improve this result.
[Jonathan]: Right.
So we have, of course, a version of this
that we prepared ahead of time,
which is more or less identical to the one we just made,
but it has a little more text, a little more explanation,
a little more documentation along with the code.
You can see we have some metadata up at the top,
the title, the authors, a bibliography,
and most importantly, the `custom-export.setup` file,
which lists specifically the sort of LaTeX commands
that we're using and the HTML styles that we're going to use.
And then down at the bottom of this file,
we have our automatic batch process.
Here is one more language we're including in here.
So this is Lisp.
And you can see here we are exporting to HTML, ASCII,
and PDF.
The nice thing about this is that this is a document.
It's a sort of document that we have a couple of
that we can have running automatically and building.
It will export a HTML, an ASCII file, and a PDF file
every time it's run based off of
the most recent data available on Wikidata.
So it's self-documenting.
We have, of course, our data retrieval steps,
our data cleaning steps, our data preparation steps,
and our preservation steps all listed at the same time.
And then you can see over on the right,
there's an example of the HTML file that we get out of this.
We also get a very nicely formatted PDF file,
which doesn't have that little issue
with the overflow of the table.
It's very nicely put together.
And we even have an ASCII file.
And I should also point out very quickly,
while you have this one up, Lukas, after the awk code,
you can see the text for the number of consortia,
or the number of institutions per consortia
is actually printed inline.
[Lukas]: Yeah, you're very right.
So this is what we had as code,
and now this is nicely integrated into our text.
So we got the consortium and number of institutions.
You can't tell a difference between code and text.
[Jonathan]: And those are automatically updated.
So if another institution joins NFDI4Earth,
then the next time this runs, we update the text right here.
It's nothing we have to worry about.
We just pull it directly out of Wikidata.
[Lukas]: And for the sake of completeness,
this is the ASCII file.
That's in the export format.
It contains also everything, code and data.
Yeah, so this is what we wanted to show you,
how to do some data processing,
some collaborative work,
documenting using org-babel.
Thanks for listening.
[Jonathan]: Thank you all, have a good day.