Greetings. Salaam. This is Mohsen Banan.
I am an Iranian software and internet engineer.
I converted to Emacs in 1986.
It was Emacs version 17 then.
By around 1988, when Emacs version 18
was well in place,
I started living inside of Emacs.
My primary digital environment
has been Emacs ever since.
It has been a good life.
I'm a native Farsi speaker and writer.
I'm not a linguist,
and I do not specialize in
multilingualization, internationalization,
and localization.
My favorite programming language is Lisp,
and I am a bit of an Emacs developer.
This presentation is about use of
Perso-Arabic Scripts with Emacs.
It's an overview presentation.
I won't be digging deep
in many of the mentioned topics.
My goal is to make you aware
of what can be done with Emacs today,
and the potentials that Emacs presents
for Perso-Arabic writers.
The main topics that i'll cover are:
- a brief introduction to Perso-Arabic scripts
- two existing Emacs Persian input methods
- the challenges involved with
making Emacs applications bidirectional-aware
- the ultimate goal of creating
a complete digital environment
for Perso-Arabic writers
I'll also be including various pointers.
So first, let's make sure
that what I'm presenting
is of interest to you.
if you are a Perso-Arabic writer
and if you use Emacs,
you're definitely my intended audience.
If you're an Emacs developer
who wishes to make her Emacs apps
multilingual and bidi-aware,
you're also my intended audience.
For the purposes of this presentation,
in this slide, I'm categorizing scripts
based on directionality and shaping.
Latin letters are not shaped.
Generally speaking,
the shape of a Latin letter
is independent of its position in a word.
Perso-Arabic letters are subject to shaping.
For example the letter mim--
sounding similar to M--
takes three shapes depending on whether
it is in the beginning of a word,
in the middle of a word,
or at the end of a word.
I'll be showing more of how shaping works
in an Emacs session screencast later.
Shaping has ramifications
for Emacs application developers.
For example, if you are combining
initial letters to create a label,
those letters can be shaped together--
which is not what you want.
In such cases, you would need to
explicitly keep them separate.
Latin-based scripts are always
left-to-right.
Perso-Arabic scripts are right-to-left
with letters, but numbers are left-to-right.
So, Perso-Arabic scripts
are bi-directional (BIDI).
Hebrew is also bi-directional,
but Hebrew is not shaped.
More recently, it has become very common
to mix Perso-Arabic and Latin text.
This can become very confusing
if paragraph directionality
is not properly observed.
I'll be providing some examples
as screencasts.
The Emacs display engine
now fully and well supports
both shaping and BIDI.
Since 2012, starting with Emacs version 24,
we can say that Emacs
is a truly multilingual-capable environment.
Like everything else,
multilingual support for Emacs
was added gradually.
Unicode support was added early on.
The framework for input methods evolved
in the 1990s. But it was not till version 24
in 2012 that the display engine
could fully support BIDI.
Hats off to Eli Zaretskii
for his work on Emacs BIDI.
Once full BIDI support was in place
in 2012, I went ahead and added
two Persian input methods to Emacs 24.
So now Emacs fully supports
Perso-Arabic scripts.
By Perso-Arabic script, we're referring to
the Arabic writing system
with various extensions
used by a large number of languages.
Perso-Arabic is the second most
widely used writing system in the world
by the number of countries.
It is the third by the number of users
after the Latin and Chinese scripts.
So, by well supporting Perso-Arabic,
Emacs's potential user base
can be greatly enhanced.
Before focusing on the Persian input methods,
let me quickly summarize
Emacs's input methods model.
Input methods allow you to enter characters
that are not supported by your keyboard.
With Quail maps, we can map ASCII key strings
to multilingual characters.
So we can input any text
from an ASCII keyboard.
You select an input method
with C-x RET C-\ .
We'll try that in a screencast shortly.
Since version 24, Emacs comes loaded with
two Persian input methods:
farsi-isiri-9147 is the standard
traditional Iranian keyboard.
farsi-transliterate-banan is an intuitive
transliteration keyboard for Farsi
which requires near-zero training for use.
I'll be mostly focused on
farsi-transliterate-banan
in this presentation.
So let's try this out.
In this gif-cast, we're going to select
a Persian input method
and write a few simple sentences.
With no training and no documentation,
any Farsi writer familiar with Emacs
can write these, as farsi-transliterate-banan
input method is intuitive.
I'll be using keycast
to show you keys as they are used.
Let me first describe as to what we have
on the screen.
There are three windows in one frame.
Keycast will show commands and keys
on the mode line.
The leftmost window is showing
logs of keycast.
Transformed individual unshaped letters
will appear here.
The middle window is running a tail -f
on the dribble file piped to fold -w1.
This lets you see the raw ASCII characters
as I type them.
The right window is the empty buffer
on the ex.fa file.
Anything that I described here
can be done with virgin Emacs distribution
with nothing added,
but I'm using Blee
(By* Libre-Halaal Emacs Environment)
to show things. You don't need to have blee
for writing the equivalent of the text
in this gif-cast.
First, I'm going to select
the farsi-transliterate-banan.
I'm entering C-x RET C-\ .
Notice the mode -ine
and the prompt at mini-buffer.
With completion, I'm going to select
farsi-transliterate-banan.
Notice that farsi-isiri-9147
was also provided as a choice.
Also notice that the letter 'b' appears
in the left of the mode line of ex.fa.
This indicates which input method
has been selected.
Also notice that cursor is on
the top left corner of ex.fa.
Next, I'm going to enter the 's' character.
Notice the cursor moved to the right,
and unshaped ’seen’ appeared
in the ex.fa buffer, on the mode-line,
and in the keycast log buffer.
Next, I'm going to enter 'l'.
Notice ل in the mode-line,
and notice how س was subjected to shaping.
Next I'm going to write the letter
to write the following:
سلام –حال شما چه طوره؟ –با ایمکس همه کار میشه کرد
"Hello, how are you?"
You can do everything with Emacs.
Generally, same-sounding Latin characters
are used. As usual, vowels are ignored
unless called for.
Notice that in order to get ح,
i repeated 'h' twice.
ش
is the obvious 'sh'.
چ
is the obvious 'ch'.
t, that's the ط is upper case T.
ت
is lowercase t.
That's it. We managed to write in Farsi
with a QWERTY keyboard, intuitively.
Next, we are going to switch back to
globish and write "back to globish".
Notice that the globish sentence
started from the left side.
This is due to proper detection
of paragraph directionality by Emacs.
For the most part, Emacs is self-documenting.
Here we are pointing you to some
relevant self-contained Emacs resources.
The BIDI documentation applies to
all BIDI scripts,
not just Perso-Arabic scripts.
Referring to the code
can also be useful for some.
Here are some pointers.
The Quail translation code
for Persian input methods:
the persian.el file has full details
of the mapping and some documentation.
Next, we'll show the keyboard layouts
as a gif-cast. You can get
relevant documentation for any input method
with the describe-input-method command.
So, let's try that for farsi-transliterate-banan.
We are back in the ex.fa buffer
as one window.
We don't need the keycast logging
and the dribble windows any more.
With the C-\, I reactivate
the farsi input method.
Notice that keycast is still active
on the mode line.
Next, with the C-h C-\ ,
I get the input methods documentation.
I then delete other windows
and keep the help buffer visible.
Notice that beh
in this input methods identifier.
Here is the URL for full documentation
on the web. The keyboard layout itself
is a one-to-one mapping, but towards
making transliteration intuitive.
Multiple keys are sometimes mapped to
the same letter.
For example, both 'i' and 'y' produce yeh.
The usual two letter transliterations
ending with 'h' -- zh, ch, sh, and kh
are provided. The ampersand prefix
is used to support often invisible
BIDI markings. In addition to this
internal documentation,
full documentation is also available.
Complete documentation
for Persian input methods
is available as PLPC-120036.
Next, we'll take a quick look at this
on the web.
You can click on links
in the reveal web-based form
of this presentation.
So let's visit PLPC-120036.
This document fully describes
Persian input methods.
In addition to HTML,
you can also obtain it in PDF.
Let's do that. Of particular interest
in this document are various tables
that enumerate lists of letters
with their association
to both Persian input methods.
Let's take a look at a few of these.
Table 3, mapping of isiri-6219
(the Farsi character set)
to Emacs version input methods
could be of interest to you,
as well as table 8
for BIDI-related control markups,
and table 9, for vowels and other signs.
Having covered input methods,
let's turn our attention
to ramifications of BIDI and Perso-Arabic
on various Emacs applications.
Since 2012, I have been using Persian text
in various Emacs applications.
In short, my experience has been
that most Emacs apps are usable,
but they all have glitches
that could at a minimum
annoy Perso-Arabic users.
In this slide, I'm presenting a summary.
The glitches with Gnus
are not all that significant for me.
BBDB glitches can easily be fixed.
For Calendar, I have customized my own setup
to support Persian and Islamic dates
in Perso-Arabic. Perhaps they should be
merged upstream, instead of
dealing with apps one at a time.
I think it's more reasonable
to consider them collectively.
The glitches that I mentioned
in the previous slide
have two routes:
some are BIDI-specific
and some are Perso-Arabic-specific.
In this slide,
I have classified them as such
and have made some general suggestions,
but all of these at best amount to
tactical approaches.
I think a more strategic approach
is called for.
The right way to address BIDI-awareness
and other awarenesses
is to build them in frameworks
that Emacs apps can then use.
So, I'm proposing that we first create ENML,
the Emacs Native Markup Language,
as a Lisp-ish (perhaps even not fully secure)
super-set of HTML5.
With that in place, we can then build on
the two decades of experience
that have produced various
web application development frameworks
by mimicking one of them.
I don't have any running code
for any of these, but discussing strategy
need not always be futile.
Emacs has immense potentials,
but those potentials cannot be realized
unless we integrate Emacs in the totality
of a specific complete digital ecosystem.
Over the past two decades, I've been
building the contours
of The Libre-Halaal By* (ByStar)
digital ecosystem.
Emacs can then be fully integrated into
ByStar. It's through such integration
that full conviviality of Emacs
can be experienced.
Blee, the ByStar Libre-Halaal
Emacs environment is Emacs
plus a whole lot of Emacs apps
integrated with Debian,
With ByStar services and with BISOS,
the ByStar Internet Services OS.
Perhaps this could be
the topic of a presentation
for the 2022 Emacs Conference.
[captions by Mohsen Banan]