Need for Indian language content in Unicode

Indian readership on the Internet is growing rapidly 42% yoy as per IRS Q32011 (pdf) numbers, while the press growth is  measly3.1%yoy. Many newspapers particularly in Telugu (Andhrajyothy, Sakshi,Vaartha etc) who have started with non Unicode encodings for their online versions have switched over to Unicode in the recent past. Even the district editions are available on the Internet. Unfortunately, the archival is only available for few weeks or few months at most. Even this content is mostly PDF, which can't be searched by search engines, as PDF is mainly an output mechanism for Indian languages which have complex text scripts. These are the major major setbacks to Indian languages, as Search engines, encyclopaedic sites like Wikipedia rely on digital content which is maintained forever to provide links to the information.

For Indian languages, the rendering is a complex process as illustrated by this Telugu example (Ya+E matra->Yi). As pdf only retains the final image, it is not amenable for search (Illustration Via WIkimedia Commons, Author:Hatukanezumi)

Few  Indian language newspapers and Government Planning magazine (Yojana in other than Hindi version) and state government magazines (Sample Page of Andhra Pradesh Magazine in pdf) and websites(notably information and public relations website) are still following legacy fonts and pdf files, as they have not even migrated to Unicode for their content. As Information plays increasingly dominant role in our day to day life, the traditional digital content producers who  tweak their systems to archive their valuable content in Unicode without any time limit will stand to benefit themselves and their readers much more than others.

Indian language interface on basic cell phones from Samsung

Continuing from my previous blog post on using Nokia cell phones in Indian languages I will share how Samsung phones work with Indian languages. I understand Samsung uses CDAC technology for its phones. A photo of Samsung Phone SGH-B100 introduced in 2006 is shown here. I use Hindi alphabet to explain the interface, but advise the other indic language readers to transliterate this post into their language using sites like http://girgit.chitthajagat.in/ to see the approximate mapping of alphabets in their language.

First let us take a detailed look at key mapping for Samsung SGH-B100 focussing on most common language alphabet. I tried looking for a manual on the net, but could find only English and Arabic ones. It is bad that Samsung or CDAC did not even provide a manual in Hindi.

Samsung basic cellphones and their indian language keymapping

Key	Language Keypad mapping
1	क ख ग घ ङ 1
2	च छ ज झ ञ 2
3	ट ठ ड ढ ण 3
4	त थ द ध न4
5	प फ ब भ म 5
6	य र ल व श 6
7	ष स ह क्ष त्र 7
8	अ आ इ ई उ with shift * key ऊ ऎ ए ऐ ऒ ओ औ 8
9	ा ि ी ु ू 9
*	Shift key:Halanth form for consonants or additional choices for vowel matras like ॆेैॊोौृ
0	ँ ंः ऋ ॐ 0
#	Space

Typing in the Indian language
Just like English, multiple key strokes are needed to input a letter from the alphabet. As the indian language alphabet is larger than English, 5 letters are mapped to a numeral key. Shift key(*) is used for halanth or vowel matra variants. Though entering indian language text may be tedious, people will realise the benefits in day to day usage. Now let me explain inputing text with the example of writing my first name Arjun (अर्जुन). First change the phone language to your language from the Phone settings menu. Then from the menu select contacts and then add contact using soft buttons. Then you will see the input mode with your language on the bottom row of the display. In case you are not getting your language text, please experiment changing the input mode by using the soft key on the right.

• The first letter is vowel and अ, select key 8.shows the soft menu with 1-5 mapped as अ -उ, select 1
• The second letter is a conjuct of र , ज and उ so press 6, * (shows halanth forms) , 2 (to select halanth form which is required for  connecting the consonants),2 then 3 and 9 then 4
• The third letter is consonant न, so press 4 then 5 . Then you will see अर्जुन.

Share your experiences of using the cell phone in native language. Please note that Hindi is not my mother tongue, so in case of any inadvertent mistakes in this post, please correct through a comment or let me know so that I can fix it.

Samsung's approach seems to reduce the keystrokes for entering text. In the above example, we needed 11 keystrokes versus the need of 16 keystrokes for Nokia.

The use of Indian languages is not popular due to lack of standards in input methods and other telecom technology issues. While an initiative is in progress to standardize use of 7 bit encoding for indian languages as part of 3GPP by CEWiT (presentation on indian language workshop dowloadable from CEWIT website), real benefits will be seen when the best keyboard layout is standardized and adopted by the Vendors. Vendors should get together to discuss the best keyboard layout and agree to offer it for free or on a very nominal fee and non discriminative basis for the benefits of technology to reach the society. Help from professional institutes like IEEE-SA can be taken, to leverage the experience of developing standards acceptable by all stakeholders.

Indian language interfaces on basic cellphones from Nokia

Recently, I was returning from a trip to my native place by Train. My train was late by 45 minutes and I sat down on a platform bench. A middle aged woman was on the bench beside me. She said to me her relative's name and asked my help to place the call, while passing her mobile to me. I took the phone and searched for the contact, confirmed the name with her and then pressed the dial button and gave back the phone to her. I also noticed that the woman is a literate person, as she had in her hand a house warming invitation written in the form of Telugu Poems. As the indic language interfaces have appeared in the phones from Nokia, Samsung from 2005, I reflected for a while on the issues faced by people like her. I also thought about my other relatives, who have phones, but do not feel comfortable to use the phone for making calls or storing contacts. Though they have digital cell phones, they will key in the number as if they were using the Plain old telephone service of past. Many elders wait for their son/daughter to call them and never take the initiative to call on their own. I have seen people with good knowledge of English also fail to use the phone's features like SMS.

When I bought my first personal cell phone in 2005, I saw the Hindi letters printed alongside the English and Numerals on the keypad. I experimented briefly with Hindi interface for some time and switched back to English, as my mother tongue is Telugu and the same was not available on the phone. In my recent trip, I saw a Nokia phone 1650 with Telugu as language option. It is unfortunate that the keypad has Hindi lettering and not Telugu lettering even though the phone was sold in Andhra Pradesh, where Telugu is local language.

While trying to make a presentation for a conference on use of local languages for information sharing, I tried to search the net for information related to indic languages and cell phones on phone company sites, CDAC, TDIL and telecom service provider sites . What I got after extensive search was only  a Hindi user manual of Nokia 1108. I could not get a single snapshot of Nokia/Samsung phones with their Hindi language keypad even in their press releases. Some bloggers uploaded images of some sample phones with Hindi keypad, when they were blogging about some of the phones general features. As those images are copyrighted, I set out to make my own photographs of such phones. I also wanted to write an article of the Indian language interfaces so that other people can get benefit from my experience. So here is a brief note on Nokia and its approach to Indian language interfaces. I use Hindi alphabet to explain the interface, but advise the other indic language readers to transliterate this post into their language using sites like http://girgit.chitthajagat.in/ to see the approximate mapping of alphabets in their language.

First let us take a detailed look at key mapping for Nokia 1108 focussing on most common language alphabet. Please see the Hindi user manual cited above for additional letter mappings.

Nokia basic cellphones and their indian language keymapping

Key	Language Keypad mapping
1	ँ ंः ् 1
2	अ आ इ ई उ ऊ ऋ 2
3	ऎ ए ऐ ऒ ओ औ 3
4	क ख ग घ ङ 4
5	च छ ज झ ञ 5
6	ट ठ ड ढ ण 6
7	त थ द ध न 7
8	प फ ब भ म 8
9	य र ल व श ष स ह 9
*	Halanth form
0	Space
#	Switch input language mode and assist method

Typing in the Indian language
Just like English, multiple key strokes are needed to input a letter from the alphabet. As the indian language alphabet is larger than English, 5 to 6 letters are mapped to a numeral key. Though entering indian language text may be tedious, people will realise the benefits in day to day usage. Now let me explain inputing text with the example of writing my first name Arjun (अर्जुन). First change the phone language to your language. Then from the menu select contacts and then add contact using soft buttons. Then you will see the input mode with your language on the top row of the display. In case you are not getting your language text, please experiment changing the input mode by pressing '#' key.

• The first letter is vowel and अ, select key 2 once.
• The second letter is a conjuct of र , ज and उ so press 9 twice, * one time (to form halanth which is required for  connecting the consonants),5 three times and then 2 four times
• The third letter is consonant न, so press 7 five times. Then you will see अर्जुन.

Share your experiences of using the cell phone in native language. Please note that Hindi is not my mother tongue, so in case of any inadvertent mistakes in this post, please correct through a comment or let me know so that I can fix it.

I will write another post on the Samsung approach of indic language interface. Till then enjoy indic computing if you or your friend owns a Nokia phone supporting an indic language.

Gyanpad(Knowledge Kiosk)

Support for  Indian languages  on computers  gained momentum  around 2005 with support of Unicode.   Five   years  have passed  and I continue to hear the same issues about  fonts, input methods for typing in  native languages. Usage of computers through native languages has not gone beyond  a small minority of people  who are  passionate  about their  native tongue.


 In India, computer sales  are growing  year on year  with negligible impact of  the global recession  and mobile  phone density has grown rapidy beyond  expectations.  Net connectivity  through dial up /ADSL modems/Wireless  is  available throughout the country.  Broadband net   connectivity is not growing rapidly primarily due to  high cost, though  people complain of lower speeds.  Low speed connectivity can still  be very helpful, if put to  use for accessing   knowledge  sites  like Wikipedia.  To  clear the input method hurdles,  I present  an application called Gyanpad (Sanskrit word for Knowledge and English  synonym of  Slate), which in its first avatar allows you to select an Indian language  and  search  Wikipedia and sister sites  in a user friendly  way with just on screen keyboards.  So you can   call   all your  family /friends  and introduce them  to native  language use on computers without them  being scared  by   the  big 101 key physical board with just English letters o n it  .

Gyanpad(Knowledge Kiosk)-Hindi InscriptSearch

Gyanpad supports 15 Indian languages. Here is  a mini user guide for Hindi. You can try this out even if you do not know Hindi, and explain easily to your family and friends. Click   Gyanpad application  (here) .  From now on, just use your  mouse. Select Hindi-Inscript  and click on  'Next' .  Search page loads with Hindi Inscript  "On Screen Keyboard".   You  can see Hindi letter shapes on the screen.  You can adjust  the keyboard  size if you are using  Firefox with  'CTRL'  and '+' or  '-' key s.  (Physical keyboard  needed just for this step,  if you have not configured it earlier).  Suppose you want to type  the word  Hindi  (हिंदी) , move to the  keyboard button showing (ह) and click on it . Continue  the same with    the buttons  showing (ि), (ं)  (द) (ी) and then click on the 'Search' button.  In a second, you will see the search results from wikimedia  sites courtesy Google.  Browse through the same and click on the page you want to see.  The new page will open  and you can navigate using hyperlinks.  If you want to  search for   something else, use back buttons of the browser  to reach the search page.     So you will come to a situation, where you can't see the  Hindi letter on the keyboard. Don't panic ,  just click on the button  with 'Up  Arrow ' (shift)  and you can  see  different set of characters on the buttons. When you have to  form  complex   letter  with  two or more consonants and then a vowel, make use of the combining symbol  on the first panel  called Halanth/Virama (्) after you type each consonant. Always end with  the  appropriate vowel symbol on the first pane.

May be you want  to teach the young ones, who are more comfortable with English than Hindi.  No problem  select Hindi-Phonetic  from the first page and  you will see   Hindi Phonetic Keyboard. The initial  pane has English  lowercase letters. By  clicking  on letters correspond to  English spelling of Hindi words,  you can enter  the word you want to search.  For our  example Hindi  (हिंदी), just   locate    ('h' )('i' ) ('n') ('d') ('I') and  then click  .  You can   use upper case letters by 'Up Arrow'( Shift key)  to get  different   sounds    d (द)  D(ड)  and long vowels.  With 'CTRL+ALT' button, you can  change the keyboard pane  to show   Hindi letters, which you can enter, in case you are not able to get the appropriate Hindi letters  using English keys.

I have built this tiny application  using the Google resources . Google  provides the  'On screen keyboard' (Virtual Keyboard)  option from its home page, when you opt to use the customized site for your language. However, this application  will be useful  in Cybercafes/ Exhibitions/Training Centers/Computer Laboratories, where it is desirable not to customize the settings to avoid incovenience to other users.

Google  is yet  to release Inscript keyboards for other Indian Languages.  So manage with Phonetic Keyboards.   Try  it out and   share your feedback by commenting on this post.    It is my  strong desire  to see  that  information from   Wikipedia  /Government  /Not for Profit institutional  sites  is  available free to every Indian  using  low cost  netbooks/custom computing devices with just touch screen/mouse interface.   These devices  can be made available at every public place,  be it   corner   retail store/bus stop/mall /hospital/...

Did  not understand some   of the words above,  just go straight to  Gyanpad application 

and search in  English/ native language.

Indian Wikipedias-Grading based on Traffic Analysis

Indian language wikipedia community has so far been looking at more of input statistics, like # of pages, page depth to measure their growth. Using absolute numbers based on content contribution is not a right metric to compare the languages. Outcome based measures like the # of unique visitors per Million speakers is a better measure. Instead of looking at ranking based on value, comparison between groups of languages based on grades can show better insights. I present here the same for Indian languages with more than 1 Million primary and secondary speakers. I have excludeded Sanskrit, Bishnupriya Manipuri (score of >10000) . I have dropped English (>10000) as it is a worldwide language and is not useful for comparison. I have included Chinese(zh) as that language has similar problems as our with regard to use on computers.

It shows Malayalam tops the list and is on par with Chinese (approximately 150). Tamil. Telugu, Marathi are in the second group(>50), Kannada, Urdu,Hindi, Gujarati, Bengali(>25) are in third group. Assuming that these language communities have similar issues with regard to Wikipedia, this trend is expected to continue for the next decade. Only surprise element may be, if Hindi beats the prediction and emerges as the top, because of it being treated as a national language and incorporating it in the curriculum. These statistics I hope will make the Indian Wikipedian communities think about what needs to be done to move to the top grade. Another interesting analysis on how soon these will grow would be, to look at the number of active Wikipedians and what is being done to prmote wikipedia in these communities. I will plan that for a later post. Meawhile I look forward to your feedback.
For those, who are interested in the inputs for this analysis, I have taken the daily statistics on 2009-09-01 and the language population numbers at wiki for generating this chart.

Firefox indic localization session at Barcamp7

Barcampbangalore7, is scheduled for 13,14 September 2008. This is going to be my third barcamp. As is the norm in technology, by the third attempt, any product becomes successful or looses the race. So is any initiative. I thought I should start real contributions to the Barcamp.

I have been working on Firefox Telugu localization for over 9 months and went through a lot of learnings about contributing to the open source world, be it setting up wiki pages for project, creating email-lists or tracking fellow contributors and chasing the coordinators to increase the prirority for our initiative. The Telugu version of Firefox was to have released with Firefox 2.13, but is now likely to be available with firefox 3.0.2 beta . As several other Indian languages are also planned for release, I proposed a session in the upcoming barcamp on this topic.

I propose to cover the following

• Brief history of Firefox, Features of firefox,
• Localization tips
• Other initiatives to strengthen Indic localization.

I welcome other contributors to join me for this session.