Need for Indian language content in Unicode

Indian readership on the Internet is growing rapidly 42% yoy as per IRS Q32011 (pdf) numbers, while the press growth is  measly3.1%yoy. Many newspapers particularly in Telugu (Andhrajyothy, Sakshi,Vaartha etc) who have started with non Unicode encodings for their online versions have switched over to Unicode in the recent past. Even the district editions are available on the Internet. Unfortunately, the archival is only available for few weeks or few months at most. Even this content is mostly PDF, which can't be searched by search engines, as PDF is mainly an output mechanism for Indian languages which have complex text scripts. These are the major major setbacks to Indian languages, as Search engines, encyclopaedic sites like Wikipedia rely on digital content which is maintained forever to provide links to the information.

For Indian languages, the rendering is a complex process as illustrated by this Telugu example (Ya+E matra->Yi). As pdf only retains the final image, it is not amenable for search (Illustration Via WIkimedia Commons, Author:Hatukanezumi)

Few  Indian language newspapers and Government Planning magazine (Yojana in other than Hindi version) and state government magazines (Sample Page of Andhra Pradesh Magazine in pdf) and websites(notably information and public relations website) are still following legacy fonts and pdf files, as they have not even migrated to Unicode for their content. As Information plays increasingly dominant role in our day to day life, the traditional digital content producers who  tweak their systems to archive their valuable content in Unicode without any time limit will stand to benefit themselves and their readers much more than others.

Indic input standards development for smart touch phones

Though, the Indic language support in basic phones existed from 2005, their usage is close to nil, due to lack of awareness, difficulties of entering Indic text and printing of Hindi characters on keypads meant for  non -Hindi areas.  For  Indic use in SMS, there were  no  standards for  encoding Indic text with GSM/CDMA technology.

IEEE-Standards Association has setup a  India Special Interest Group  with the objective of  spearheading the initiative to improve the contribution from Indian engineers in Standards development. As part of its software and IT sub group, a meeting was organised at Centre for Internet and Society(CIS), Bangalore  to  understand  the status of Indic on mobiles and brainstorm next steps on 24 Feb 2011.

Dr Nadeem from CEWIT, Vijay from Nokia, Dr Mahesh and Naveen J from Brahmi computing,Sunil Abraham and Nirmita Narasimhan, Sri Chandra and myself as volunteers of IEEE-SA attended the meeting. Sri Chandra briefed the participants about the IEEE-SA initiative.

Dr Nadeem presented work done as part of the broadband consortium during the last year towards input, storage, rendering of indic text. The scope included only the basic phones with 12 key keypad and simple input methods without reliance on word dictionaries. The team came up with an intutive input method in the usual alphabetic order of script and switching in of additional layers to speed up data entry to access frequently occurring letters. He also gave away hardcopies of the recommendation to the attendees. He said that he plans to share the recommendation in a larger forum and encourage the vendors to adopt the standard.

After extensive brainstorming on indic input methods status,  it is agreed that touch phones will become the key for extensive use of Indic languages. As touch phones become affordable at price points of less than 5000 Rupees, developing a input standard for them can help in speeding up deployment. The touch phone also does not have limitation on number of keys, so even simple methods can be easy to use. The firmware changes for phones already with public can also be done easily. Sunil from CIS offered to help  promote from policy angle by working with Government apart from hosting future events at CIS.

IEEE-SA as an established leader in standards development, can help the industry through a time tested standard development process, which can protect IP of participants and also help in faster development of standard. The team felt that we are at right point in time to work on the standard and help enable mobile services for Indians in their own languages.

Indian language interface on basic cell phones from Samsung

Continuing from my previous blog post on using Nokia cell phones in Indian languages I will share how Samsung phones work with Indian languages. I understand Samsung uses CDAC technology for its phones. A photo of Samsung Phone SGH-B100 introduced in 2006 is shown here. I use Hindi alphabet to explain the interface, but advise the other indic language readers to transliterate this post into their language using sites like http://girgit.chitthajagat.in/ to see the approximate mapping of alphabets in their language.

First let us take a detailed look at key mapping for Samsung SGH-B100 focussing on most common language alphabet. I tried looking for a manual on the net, but could find only English and Arabic ones. It is bad that Samsung or CDAC did not even provide a manual in Hindi.

Samsung basic cellphones and their indian language keymapping

Key	Language Keypad mapping
1	क ख ग घ ङ 1
2	च छ ज झ ञ 2
3	ट ठ ड ढ ण 3
4	त थ द ध न4
5	प फ ब भ म 5
6	य र ल व श 6
7	ष स ह क्ष त्र 7
8	अ आ इ ई उ with shift * key ऊ ऎ ए ऐ ऒ ओ औ 8
9	ा ि ी ु ू 9
*	Shift key:Halanth form for consonants or additional choices for vowel matras like ॆेैॊोौृ
0	ँ ंः ऋ ॐ 0
#	Space

Typing in the Indian language
Just like English, multiple key strokes are needed to input a letter from the alphabet. As the indian language alphabet is larger than English, 5 letters are mapped to a numeral key. Shift key(*) is used for halanth or vowel matra variants. Though entering indian language text may be tedious, people will realise the benefits in day to day usage. Now let me explain inputing text with the example of writing my first name Arjun (अर्जुन). First change the phone language to your language from the Phone settings menu. Then from the menu select contacts and then add contact using soft buttons. Then you will see the input mode with your language on the bottom row of the display. In case you are not getting your language text, please experiment changing the input mode by using the soft key on the right.

• The first letter is vowel and अ, select key 8.shows the soft menu with 1-5 mapped as अ -उ, select 1
• The second letter is a conjuct of र , ज and उ so press 6, * (shows halanth forms) , 2 (to select halanth form which is required for  connecting the consonants),2 then 3 and 9 then 4
• The third letter is consonant न, so press 4 then 5 . Then you will see अर्जुन.

Share your experiences of using the cell phone in native language. Please note that Hindi is not my mother tongue, so in case of any inadvertent mistakes in this post, please correct through a comment or let me know so that I can fix it.

Samsung's approach seems to reduce the keystrokes for entering text. In the above example, we needed 11 keystrokes versus the need of 16 keystrokes for Nokia.

The use of Indian languages is not popular due to lack of standards in input methods and other telecom technology issues. While an initiative is in progress to standardize use of 7 bit encoding for indian languages as part of 3GPP by CEWiT (presentation on indian language workshop dowloadable from CEWIT website), real benefits will be seen when the best keyboard layout is standardized and adopted by the Vendors. Vendors should get together to discuss the best keyboard layout and agree to offer it for free or on a very nominal fee and non discriminative basis for the benefits of technology to reach the society. Help from professional institutes like IEEE-SA can be taken, to leverage the experience of developing standards acceptable by all stakeholders.

Indian language interfaces on basic cellphones from Nokia

Recently, I was returning from a trip to my native place by Train. My train was late by 45 minutes and I sat down on a platform bench. A middle aged woman was on the bench beside me. She said to me her relative's name and asked my help to place the call, while passing her mobile to me. I took the phone and searched for the contact, confirmed the name with her and then pressed the dial button and gave back the phone to her. I also noticed that the woman is a literate person, as she had in her hand a house warming invitation written in the form of Telugu Poems. As the indic language interfaces have appeared in the phones from Nokia, Samsung from 2005, I reflected for a while on the issues faced by people like her. I also thought about my other relatives, who have phones, but do not feel comfortable to use the phone for making calls or storing contacts. Though they have digital cell phones, they will key in the number as if they were using the Plain old telephone service of past. Many elders wait for their son/daughter to call them and never take the initiative to call on their own. I have seen people with good knowledge of English also fail to use the phone's features like SMS.

When I bought my first personal cell phone in 2005, I saw the Hindi letters printed alongside the English and Numerals on the keypad. I experimented briefly with Hindi interface for some time and switched back to English, as my mother tongue is Telugu and the same was not available on the phone. In my recent trip, I saw a Nokia phone 1650 with Telugu as language option. It is unfortunate that the keypad has Hindi lettering and not Telugu lettering even though the phone was sold in Andhra Pradesh, where Telugu is local language.

While trying to make a presentation for a conference on use of local languages for information sharing, I tried to search the net for information related to indic languages and cell phones on phone company sites, CDAC, TDIL and telecom service provider sites . What I got after extensive search was only  a Hindi user manual of Nokia 1108. I could not get a single snapshot of Nokia/Samsung phones with their Hindi language keypad even in their press releases. Some bloggers uploaded images of some sample phones with Hindi keypad, when they were blogging about some of the phones general features. As those images are copyrighted, I set out to make my own photographs of such phones. I also wanted to write an article of the Indian language interfaces so that other people can get benefit from my experience. So here is a brief note on Nokia and its approach to Indian language interfaces. I use Hindi alphabet to explain the interface, but advise the other indic language readers to transliterate this post into their language using sites like http://girgit.chitthajagat.in/ to see the approximate mapping of alphabets in their language.

First let us take a detailed look at key mapping for Nokia 1108 focussing on most common language alphabet. Please see the Hindi user manual cited above for additional letter mappings.

Nokia basic cellphones and their indian language keymapping

Key	Language Keypad mapping
1	ँ ंः ् 1
2	अ आ इ ई उ ऊ ऋ 2
3	ऎ ए ऐ ऒ ओ औ 3
4	क ख ग घ ङ 4
5	च छ ज झ ञ 5
6	ट ठ ड ढ ण 6
7	त थ द ध न 7
8	प फ ब भ म 8
9	य र ल व श ष स ह 9
*	Halanth form
0	Space
#	Switch input language mode and assist method

Typing in the Indian language
Just like English, multiple key strokes are needed to input a letter from the alphabet. As the indian language alphabet is larger than English, 5 to 6 letters are mapped to a numeral key. Though entering indian language text may be tedious, people will realise the benefits in day to day usage. Now let me explain inputing text with the example of writing my first name Arjun (अर्जुन). First change the phone language to your language. Then from the menu select contacts and then add contact using soft buttons. Then you will see the input mode with your language on the top row of the display. In case you are not getting your language text, please experiment changing the input mode by pressing '#' key.

• The first letter is vowel and अ, select key 2 once.
• The second letter is a conjuct of र , ज and उ so press 9 twice, * one time (to form halanth which is required for  connecting the consonants),5 three times and then 2 four times
• The third letter is consonant न, so press 7 five times. Then you will see अर्जुन.

Share your experiences of using the cell phone in native language. Please note that Hindi is not my mother tongue, so in case of any inadvertent mistakes in this post, please correct through a comment or let me know so that I can fix it.

I will write another post on the Samsung approach of indic language interface. Till then enjoy indic computing if you or your friend owns a Nokia phone supporting an indic language.

Gyanpad(Knowledge Kiosk)

Support for  Indian languages  on computers  gained momentum  around 2005 with support of Unicode.   Five   years  have passed  and I continue to hear the same issues about  fonts, input methods for typing in  native languages. Usage of computers through native languages has not gone beyond  a small minority of people  who are  passionate  about their  native tongue.


 In India, computer sales  are growing  year on year  with negligible impact of  the global recession  and mobile  phone density has grown rapidy beyond  expectations.  Net connectivity  through dial up /ADSL modems/Wireless  is  available throughout the country.  Broadband net   connectivity is not growing rapidly primarily due to  high cost, though  people complain of lower speeds.  Low speed connectivity can still  be very helpful, if put to  use for accessing   knowledge  sites  like Wikipedia.  To  clear the input method hurdles,  I present  an application called Gyanpad (Sanskrit word for Knowledge and English  synonym of  Slate), which in its first avatar allows you to select an Indian language  and  search  Wikipedia and sister sites  in a user friendly  way with just on screen keyboards.  So you can   call   all your  family /friends  and introduce them  to native  language use on computers without them  being scared  by   the  big 101 key physical board with just English letters o n it  .

Gyanpad(Knowledge Kiosk)-Hindi InscriptSearch

Gyanpad supports 15 Indian languages. Here is  a mini user guide for Hindi. You can try this out even if you do not know Hindi, and explain easily to your family and friends. Click   Gyanpad application  (here) .  From now on, just use your  mouse. Select Hindi-Inscript  and click on  'Next' .  Search page loads with Hindi Inscript  "On Screen Keyboard".   You  can see Hindi letter shapes on the screen.  You can adjust  the keyboard  size if you are using  Firefox with  'CTRL'  and '+' or  '-' key s.  (Physical keyboard  needed just for this step,  if you have not configured it earlier).  Suppose you want to type  the word  Hindi  (हिंदी) , move to the  keyboard button showing (ह) and click on it . Continue  the same with    the buttons  showing (ि), (ं)  (द) (ी) and then click on the 'Search' button.  In a second, you will see the search results from wikimedia  sites courtesy Google.  Browse through the same and click on the page you want to see.  The new page will open  and you can navigate using hyperlinks.  If you want to  search for   something else, use back buttons of the browser  to reach the search page.     So you will come to a situation, where you can't see the  Hindi letter on the keyboard. Don't panic ,  just click on the button  with 'Up  Arrow ' (shift)  and you can  see  different set of characters on the buttons. When you have to  form  complex   letter  with  two or more consonants and then a vowel, make use of the combining symbol  on the first panel  called Halanth/Virama (्) after you type each consonant. Always end with  the  appropriate vowel symbol on the first pane.

May be you want  to teach the young ones, who are more comfortable with English than Hindi.  No problem  select Hindi-Phonetic  from the first page and  you will see   Hindi Phonetic Keyboard. The initial  pane has English  lowercase letters. By  clicking  on letters correspond to  English spelling of Hindi words,  you can enter  the word you want to search.  For our  example Hindi  (हिंदी), just   locate    ('h' )('i' ) ('n') ('d') ('I') and  then click  .  You can   use upper case letters by 'Up Arrow'( Shift key)  to get  different   sounds    d (द)  D(ड)  and long vowels.  With 'CTRL+ALT' button, you can  change the keyboard pane  to show   Hindi letters, which you can enter, in case you are not able to get the appropriate Hindi letters  using English keys.

I have built this tiny application  using the Google resources . Google  provides the  'On screen keyboard' (Virtual Keyboard)  option from its home page, when you opt to use the customized site for your language. However, this application  will be useful  in Cybercafes/ Exhibitions/Training Centers/Computer Laboratories, where it is desirable not to customize the settings to avoid incovenience to other users.

Google  is yet  to release Inscript keyboards for other Indian Languages.  So manage with Phonetic Keyboards.   Try  it out and   share your feedback by commenting on this post.    It is my  strong desire  to see  that  information from   Wikipedia  /Government  /Not for Profit institutional  sites  is  available free to every Indian  using  low cost  netbooks/custom computing devices with just touch screen/mouse interface.   These devices  can be made available at every public place,  be it   corner   retail store/bus stop/mall /hospital/...

Did  not understand some   of the words above,  just go straight to  Gyanpad application 

and search in  English/ native language.

When will we have Inscript Mobile?

CDAC has released Inscript 5.1 proposal (requires user details form submission at the site) arrived after discussions with Microsoft IBM and Redhat. The new proposal enhances the Inscript to latest Unicode standard 5.1, by providing backward compatibility with mappings for for ZWJ, ZWNJ and an extended layer to encode the new character codes. The toggle key for basic to extended layers is not defined. Another proposal is to make the mapping language specific rather than script. For the Telugu language, my feedback is that Indo-Arabic numerals need to be retained as default rather than Telugu numerals.

However smart mobiles will become commonplace than PCs in 21st century, this standard is inadequate as most mobiles have keypads with about 35 keys for typing. As the computers are powerful, what is needed is a clever way to accommodate the most practically used letters of the language on this keyboard. As most users are bilingual, phonetic approaches make it convenient for new users to easily switch between the languages. Phonetic based schemes with dictionary support for word hinting from Microsoft and Google are now available for PC. Alternative statistical based input method approaches are becoming available with mobile phones targeting basic 12 keys models. Tirumalakrishna Desikachary has defined one Phonetic keyboard (released along with a font called Pothana) for Telugu language 10 years back. In this short and long forms of vowel are assigned to the same key. Vowel keys are treated as independent vowels if they appear first in the word and as dependent vowels (or matras of the preceding consonant) if they appear after the first letter of the word. This avoids wasting of keys for representing matra forms of vowels. This still uses 2 keys beyond the basic alphabet to accommodate 4 Telugu letters. Out of that only one (letter au) is in popular use and that when it is interchanged with chandrabindu, all the letters that are in popular use are accommodated on the English alphabet keys. Similar arrangement can be worked out for Hindi and other languages based on the initial proposals of Desikachary.
Compared to other phonetic methods like Itrans and RTS, Pothana keeps the mapping simple one to one except for the overloading of vowel keys based on the context. It is desirable to have standard for Phonetic input on mobiles to enable the IT revolution to benefit the masses.

Firefox Indic language usage is not growing rapidly

Firefox is providing insight into download and use of its language versions from Dec-2009. It uses a feature called blocklist, where in the Firefox browser contacts Mozilla servers for updates on the malware websitesm once a day. Through this transaction, Mozilla is able to track the usage of the locale of the browser.

I got access to few Indic language report s (Hindi, Kannada, Malayalam, Marathi, ,Tamil, Telugu) for the past 3 months and tried to analyze the same. In the chart below, the downloads for the week are shown in the chart. Hindi tops the list, among the languages I have analyzed, an expected statistic in view of the larger population.

However the actual usage for the week ending 2010-03-14 is shown below. There is not much change in the last few months in these numbers. Hindi again tops the list, but its proportion of users from outside India is very much low compared to all other languages . Malayalam has large percentage of users outside India relative to to other languages. Hindi users are about 0.1% of the total users from India for Firefox 3.6 Version. Other language percentages are much below. Though the download numbers every week are roughly the same, the usage has not improved much in the last three months. This means that users are curious to find out the language version looks like, but many are not really using it.

It would be interesting to speculate about the reasons. Many users could be using computers provided by their employer. They may not be comfortable with the localized version. Or they might have experienced font display issues, if their system is not configured for rendering the language. Any other guesses and suggestions on how we can convert the downloaders into actual users?
More detailed analysis for Telugu is available in Telugu blogpost entry.