1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
|
[[!sidebar content=""]]
[[!meta title="How to incorporate handwritten notes into Emacs Orgmode"]]
[[!meta copyright="Copyright © 2022 Bala Ramadurai"]]
[[!inline pages="internal(2022/info/handwritten-nav)" raw="yes"]]
<!-- Initially generated with emacsconf-generate-talk-page and then left alone for manual editing -->
<!-- You can manually edit this file to update the abstract, add links, etc. --->
[[!img /i/2022-handwritten-title.png alt="title" size="600x"]]
# How to incorporate handwritten notes into Emacs Orgmode
Bala Ramadurai (his/him, <mailto:bala@balaramadurai.net>)
[[!inline pages="internal(2022/info/handwritten-before)" raw="yes"]]
[[!img /i/2022-handwritten-abstract.png alt="title" style="max-width: 100%; height: auto"]]
Research suggests that note taking is most effective when done by hand. Yes, handwritten notes. Certainly, typewritten or typed out notes are more efficient, however notes written by hand are retained by the human brain much longer.
Integrating handwritten notes into a computer leaned productivity workflow is tricky. Also, in terms of hardware, if all you have is a smartphone, then we need to deal with the situation a bit differently.
This talk will introduce to you a simple system to integrate handwritten notes into your org-mode (Emacs) based productivity workflow.
- Gist of the problem of dealing with handwritten notes
- Advantages of handwritten notes
- Emacs org mode workflow
- Option 1: Hardware for handwriting not available
- Option 2: Hardware for handwriting available
# Discussion
## Notes
- Tesseract is a free and open source tool to do OCR (may or may not
convert handwriting)
- <https://github.com/tesseract-ocr/tesseract>
- Karl Voit: I'm not practicing handwritten OCR at the moment. From
what I've read, offline handwriting OCR (offline OCR = not
recognizing while writing but form a (scanned) document) is only
implemented in a good way in highly proprietary services such as
Microsoft OneNote and Evernote and Apple Notes. Since I don't use
those services on purpose
(<https://karl-voit.at/2018/04/21/end-of-OneNote/>), I don't have
personal experience. I'm still hoping that there will be a FOSS
solution one fine day.
- I'm (rarely) taking notes on my BOOX Note Air e-ink tablet
(which I love). It has decent on-device offline-OCR but it's
really tedious to use it. So usually, I keep the notes as PDF
file and don't care to OCR it. Not great but at the moment I
can live with it.
- After all: I'm much faster when typing on my keyboard
(<https://karl-voit.at/2021/03/21/advantage2-plans/>) - so if
there is no graphical information to be captured, I'm faster
typing in Org mode than writing by hand.
- Do you have a blog article about that? I own a BOOX Air myself but I can't think of a viable workflow using it with handwriting + OCR on-device.
- <https://capture2text.sourceforge.net/> \-- another potential OCR
solution that is FOSS, but supported only on Windows OS
- mathpix seems to support OCR now as well:
- <https://mathpix.com/blog/mathpix-text-ocr>
- <https://mathpix.com/handwriting-recognition>
- Leo Vivier owns an e-ink tablet: Boox Max Lumi (almost A4-size,
bigger brother of Boox Note Air)
- Digital pens and digitizing platforms help to get your notes from
analogue into digital form:
- <https://www.wired.com/gallery/best-smart-pens/>
- Also, livescript uses special pens and tablets (erasable and
correctable), transcribes your handwriting
- [LiveScribe](https://www.amazon.com/Livescribe-Single-Subject-Notebook-4-pack/dp/B001AALJ1I/ref=asc\_df\_B001AALJ1I/?tag=hyprod-20&linkCode=df0&hvadid=167151358503&hvpos=&hvnetw=g&hvrand=14286771041209503377&hvpone=&hvptwo=&hvqmt=&hvdev=c&hvdvcmdl=&hvlocint=&hvlocphy=9027973&hvtargid=pla-359920653567&psc=1)
- I always love the way Bala records his talks :D
- i read every word
- Apple notes scanning is working very well these days.
- OneNote is super locked-in these days.
- Thank you Prof!
- Lol @ just type out the damn note XD. It's a good point, but I have over a year of handwritten notes that I haven't bothered typing out ;-)
- I have had some success with tesseract and my handwritten notes
- has anyone training such a OCR system to make it more accurate for your handwriting or additional symbols.
- shh I've been using Google Cloud Vision to get the text out of my sketches =)
- wow what a refined video. really solid editing. this is enjoyable to watch
- would it be better to use OCR or to read the notes and use speech-to-text?
- Does it work well with free software, klavul
- hmh, there seems to be plugin to integrate orgmode with xournalpp ( https://gitlab.com/vherrmann/org-xournalpp )
- I use capture2text , made a small wrapper
- Would it be possible to make a libre software which could detect handwritten notes as does the Boogie Board and run Emacs on it?
- other: people with short-term requirements and who don't care are using great OCR with cloud-based lock-in services. Therefore, there isn't much incentive to create FOSS.
## Questions and answers
- Q: How do you link the notes together so that you could search
through them in the future?
- A: I'm still in the process of trying to link these topics. Once you convert these notes from handwritten text --or if you have a handwriting device and you can convert it into text-- import it into org-roam and link it the way that you'd normally do it. I started a week ago linking these texts. That would be my response-- which leads into the next question...
- Q: Is it necessary to OCR your handwriting or change the way you
write to use your workflow?
- A: Yes, this is a necessary step. You have to conduct optical character recognition. Because otherwise, indexing and linking becomes a problem later on. If it's just for you and reading your own text and you browse it and read it when you want, then that's different. You don't have to worry about the OCR-- just import the jpeg or png or whatever and put it under the org-mode or org-roam headline and you should be good to go.
However, if you want to search and link then OCR becomes necessary. I'm unfamiliar with anything inside the emacs ecosystem that does OCR. I use tools like OneNote and I think Google Keep does that. I use an external tool to convert if you want linking and indexing. For me sometimes screenshots and handwritten notes have to go together. The key is to have those together.
- Q: What about Searching notes, notes to text while being offline
- A: Yes, there are no two ways about it. Only proprietary software like Google Keep and Microsoft, perhaps Dropbox does it. I would say it's absolutely mandatory for you to convert this... unless the emacs org mode community puts their heads together. If you want me to help by reviewing a package I am keen on doing that and trying it out.
OneNote can search with OCR, as far as Bala is aware.
- Q: Those articles on notetaking seem interesting. Could we get a link
for them?
- A: I will definitely drop a link in the etherpad after the talk to make it available. A good friend in France shared this with me. He had a whole bunch of articles that supported my thesis. (an email was sent with a follow-up request on 10 December)
- Q: Have you looked at taking handwritten notes on a tablet like
Xournal++
- A: [Not yet] The only tools I've used have been OneNote and Google keep. For now, the dropbox method for me is to write the notes, then take a photo with a mobile phone camera, upload it to dropbox-- why dropbox-- because it allows jpeg to be uploaded. I find pdf very cumbersome. I then import it into OneNote which converts it into digitized text. Then I take and move it from the inbox into the org mode system. That's how my workflow is. I have started to experiment with other devices and workflows, but not enough to give you intelligent advice.
Q: Have you tried out the "Remarkable" device and figured out how to link the files back into org mode constructively yet?
- A: Yes, I've heard of that device as well as Book and Amazon has come out with the writing Kindle. These devices do exist, but I'm not sure if they convert to text and put it in a format you can import into org mode. This would be nice, if you can integrate it into emacs. I use dropbox to do all the good stuff. I can certainly do that.
- Q: Something to think about is handwritten and org transcribed notes
de-duplication for searching, do you want one or the other, both?
- A: Thank you for asking this. Transcription has become very important not only transcribing written but voice notes. In spite of so many tools out there, I find it extremely cumbersome to transcribe voice notes. It would be nice if we had voice, and handwritten transcription helps. Actually that's a great idea. I would be very interested in voice and handwriting. I've seen this in software in OneNote and transcription packages. This is really important and a great idea. I volunteer to test this out. Right now I find it very difficult to transcribe, so if this can do it for us that would be amazing.
- side note: org-remark could be useful for this
<https://github.com/nobiot/org-remark>
- Q: How fancy have your imports of handwritten notes been for org mode?
(Bullets, TODO , org tables etc...)
- A: Bala uses a star on paper when writing tasks and makes
them into org tasks during manual transcription. He also uses dates for this. I know then that I need to keep track of those things in a task. It would be really cool to OCR org constructs from paper. TODO items etc. Doing away with all of the intermediate steps would be nice. A writing org-mode as if it was a human language instead of something that is restricted to a computer.
- Just to glorify handwriting. I decided that I would hand write what I was going to talk about. The handwritten notes turned out much better than my plain vanilla talk. I even wrote some jokes this way. Handwriting helps thinking and speaking with an audience. Computers have led to handwriting being less important, but Bala thinks
they're very efficient and important :)
- Q: How often do you instead use keyboard entry to type in and
summarize your notes? Would you consider that a suitable approach
for yourself, e.g. at the end of the day?
- A: When in a hurry, Bala types notes directly into Org Mode.
- He has not yet established a habit of daily journaling but
would like to do so in the future. I have a shortcut for org-roam dailies that shows up in blue text on the calendar. Daily writing would be helpful.
- Handwriting can be easier depending on language/script.
Perhaps handwriting is best for non-Roman script. My mother tongue has a script that is not English.
- Q: Do you have any ideas about mindmapping and incorporating those notes into your process? I've used nebo and other tools, but wish I had a better way of extracting key words.
- A: I have used mindmapping which is hierarchical, in fact I use something called function maps. Just like org-roam and interacting relationships. All of this I've been drawing on my laptop screen. All of this helps me. I don't digitally convert it. It helps me and is a lot easier. When I need to add to it I can do so. I find that doing it on a device and retaining it without conversion is best. Converting does not help me at all. YOu can select it, with a lasso select. Those things can be done. I like it to be that way. Then I link it into org mode.
- Q: Bala, sorry to join late to the party, you might have answer. Do you train your own OCR?
[[!inline pages="internal(2022/info/handwritten-after)" raw="yes"]]
[[!inline pages="internal(2022/info/handwritten-nav)" raw="yes"]]
[[!taglink CategoryOrgMode]]
|
