On 2/22/11, Gautam John <gautam(a)prathambooks.org> wrote: There are Four Components 1. Input Methods ( GOI approved Inscript layout, Various Popular Layouts , Translitraton Keyboards, Phonetic Keyboards) 2. Encoding ( unicode) 3. Font (Opentype Fonts ie. supporting Unicode) 4. Rendering Engines (this does the shaping of Complex Glyphs using the Open type font table in Fonts . eg. Pango in Gnome, Harfbuzz in KDE, ICU in Openoffice & java based programmes , Uniscribe in Windows etc ) ASCII is not like Unicode. It only understands latin, not any other language. All over India, legacy, non-standard local language "technologies" (ugly hacks) have gained deep roots. Local newspaper websites as well as publishing houses seem to use their own non-standard fonts. This means that documents and web sites get tied to fonts. These fonts may or may not be freely available, and in some extreme cases, may be no longer available at all. If you lose the font, you lose the content as well. Ka in Hindi may be mapped in the position of A in some font , in the position of H in some other font as per the convenience of font developer It is Font dependent. There is a need of Preparing Conversion maps for each Ascii font to convert data encoded in them to unicode. Swathanthra Malayalam Computing's Payyan's (http://wiki.smc.org.in/Payyans ) is a tool developed for converting ASCII to Unicode easily for any Indic Language by building a Font map for each needed font . This tool helped Malayalam Wiktionary to convert many copyright expired books in non standard encodings to Unicode Popular Firefox extension named Padma uses similar encoding conversion tables to display ASCII news websites in Unicode Each Language Communities can answer this question well. In Malayalam we have both screen and print fonts, including one Ornamental font . I dont think this will happen. There is a long history of lobbying for thiswith CDAC from 2001 Onwards and nothing happened. CDAC made enough money by selling ASCII fonts(and still makes) and They cant even think about giving them away with a FOSS License . And during frequent terms they eat more government money for making yet another CD to ship with their FOSS project forks (such ad Bhaathiya OO , IndiFox etc )+ These fonts. In the same way most of the TDIL funding to CDAC for Indic Language technology research does not make output at all or not getting released, even after TDIL's policy decision to release them under a foss license. Searching and sorting algorithms for Indic languages are in development and are not bug free. Indic support is not yet available in most of the search solutions (including FOSS solutions like Lucene or Solr) because of the complex word formation characteristics. Most of the existing applications tries exact string-matching algorithms on Indic content yielding only 20% of results. Indic search algorithms should use language and grammar aware algorithms -- "[It is not] possible to distinguish between 'numerical' and 'nonnumerical' algorithms, as if numbers were somehow different from other kinds of precise information." - Donald Knuth

On 2/23/11, Gautam John <gautam(a)prathambooks.org> wrote: If your language have some errors in Complex Glyph formation, it is a rendering engine issue. You can find more here http://en.wikipedia.org/wiki/Wikipedia:Enabling_complex_text_support_for_In… Rendering Engines like Pango evolved through more than 10 years of patching & correction by language communities. It work Pretty well in most of the indic languages. Harfbuzz(http://www.freedesktop.org/wiki/Software/HarfBuzz) is relatively new player in the field by taking code from Pango QT & ICU . Harfuzz-ng is used in new Firefox 4 as its default Rendering engine .Uniscribe engine in Windows based systems started supporting Indic fonts from Windows XP SP2 onwards. Let me give an example for why Rendering engine is important. Now For latin script wiki's there is PDF download option & Pediapress to print them directly But Such Options are not available for Non Latin wikis Character Rendering is a the block here. Pedia press's library fails to render non latin content , because the library they use is not making use of rendering engines. If a teacher went to internet cafe for reading a wikipedia entry in indian language , she must ensure following things before reading/printing articles 1. ensure the Operating system have Indic support 2. Ensure It have a font to display content correctly 3. Browser renders well Then only she can read it/ print it in human readable form. If there is PDF export facility with server side rendering , it was so easy for her to to take it /print it for students. Sometime back Santhosh Posted his project Pypdflib for testing in this list. It is a library for rendering PDF from Indic language wiki pages . It uses functionality of pango for generating PDF In short Rendering is a major roadblock in reaching wikipedia to masses. The projects like santhosh's effort are very important to fill this gap. Yes. payyan's wikipage have an Howto for creating fontmaps This is the only viable option as of now. Most of the languages have around 10-20 popular fonts . Creating Mapping tables for them is anyway a big task . But if each language communities are contributing, it is not a big task. And Padma project has done mapping of many news website fonts already through the contributions of many people. There is no other free alternative . BTW Document Conversion is a big business and many corporates are working on this area to provide solutions for companies & governments In my opinion, Efforts on this will be waste of time & money .I dont believe in miracles with CDAC. CDACMumbai have a history of GPL Licensing one font series as a part of their indix project , Raghu Series, by Late. Prof. R.K.Joshi, Famous Calligrapher and Researcher in Type faces. http://en.wikipedia.org/wiki/R_K_Joshi Rebranding his Jana Series fonts to raghu series & GPLing them was his long term effort from inside CDAC. But its font tables need to be corrected to make them usable . We did this work for malayalam and Raghu-Malayalam is currently maintained by SMC. Anyway it is an exceptional case Yes. And Problems & instability in unicode encoding also affects this Sometime back GerardM 's post is shared in this list http://ultimategerardm.blogspot.com/2010/12/malayalam-enigma.html Also read these break thoughts on unicode by Indic language communities http://www.j4v4m4n.in/2009/11/07/unicode-or-malayalam/ Anivar -- "[It is not] possible to distinguish between 'numerical' and 'nonnumerical' algorithms, as if numbers were somehow different from other kinds of precise information." - Donald Knuth

On Thu, Feb 24, 2011 at 8:51 AM, Anivar Aravind &lt;anivar.aravind(a)gmail.com&gt; wrote: The 'GPL' that these fonts had was the 'General Public License' wasn't it ? And not the GNU General Public License. I may be mistaken though etc. I've been, in the past, known to berate and sigh C-DAC. In recent times I've arrived at the conclusion that there's no upside thinking that TDIL/MinIT/C-DAC will eventually figure out that selling services around their products make for a better business case than trying to hawk the products themselves. Or, that LGPL licensing their products might make it easier to have an application developer network around it. -- sankarshan mukhopadhyay <http://sankarshan.randomink.org/blog>

Even though Central Government has adopted Unicode as the encoding standard, the case is not the same with most State Governments. As far as I know only few state goverments (Tamil Nadu, Punjab, Kerala,...) had adopted Unicode standard. Many are still in the ASCII era. On input methods - is there anything of a best practice or even a Government notification about an input standard? I haven't seen any notification regarding this yet. But InScript is officially/unofficially adopted as the default input scheme. That is why it is part school syllabus in some states. On Thu, Feb 24, 2011 at 9:54 AM, Gautam John &lt;gautam(a)prathambooks.org&gt;wrote;wrote: > Two things I meant to add: > 1. The eGov standards body for India has recently notified Unicode > 5.1.0 as the default standard for all eGov applications henceforth. > (Sadly, their website is DoA - http://egovstandards.gov.in/) I am > hopeful that this will be the start of some initiative within > Government and would, hopefully, spread. > A cache of their Approach Paper on Localization is here: > http://webcache.googleusercontent.com/search?q=cache:e28QCFBDI-cJ:egovstand… > And a cache their Character Encoding Standard For Indian Languages is here: > http://docs.google.com/viewer?a=v&q=cache:dYxnM6D7IMQJ:egovstandards.go… > 2. On input methods - is there anything of a best practice or even a > Government notification about an input standard? > Thank you. > Best, > Gautam > ________ > http://social.prathambooks.org/ > _______________________________________________ > Wikimediaindia-l mailing list > Wikimediaindia-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikimediaindia-l

On 24 February 2011 10:12, Shiju Alex &lt;shijualexonline(a)gmail.com&gt; wrote: Thank you, Shiju. A question - what are the hesitancies for Governments to move to Unicode as the encoding standards? Is it the tools they use? The workflow? A legacy issue - "we'll never be able to open our old files"? I'm trying to map this space out - it's just that I am coming to see it as being really really important and want to try and do something here. Also, the GoI is slowly making some noises about standards and openness etc. and I am hoping this are small points that can add up. For example, the TAGUP report: http://finmin.nic.in/reports/TAGUP_Report.pdf "Chapter 6 points out some key design considerations for the solution architecture. The solution architecture should be designed to be flexible, reusable, extensible by stakeholders, and free of vendor lock-in. Given that many Government projects touch end-users such as citizens and firms, the Government should also play an active role in promoting banking and accessibility for all. This can form the basis of a platform for delivery of services. Chapter 7 addresses openness in implementation of Government IT projects. It describes the relevance of open standards, open data, and open source. The Government should not only be a consumer, but also strive to produce and facilitate open standards, open data, and open source. It also suggests the creation of an open source foundation for open sourcing software from Government projects. Give me a little hope. Best, Gautam ________ http://social.prathambooks.org/

Dear gautam Thanks for those links. I am aware about that. But not get enough time to read it yet. But are you sure, it specified unicode 5.1 . I am curious becuase new rupee symbol getting encoded only in unicode 6.1. Usually govt standards does not specify versions. Anivar On 2/24/11, Gautam John &lt;gautam(a)prathambooks.org&gt; wrote: -- Sent from my mobile device "[It is not] possible to distinguish between 'numerical' and 'nonnumerical' algorithms, as if numbers were somehow different from other kinds of precise information." - Donald Knuth

Dear sankarshan Initial license of raghu font series was confusing. But later they changed it to gnu gpl, as per the insistance of RK joshi. Gnu gpl licensed fonts were released as a part of Indix project of cdacmumbai. Anivar On 2/24/11, sankarshan &lt;foss.mailinglists(a)gmail.com&gt; wrote: -- Sent from my mobile device "[It is not] possible to distinguish between 'numerical' and 'nonnumerical' algorithms, as if numbers were somehow different from other kinds of precise information." - Donald Knuth

On Thu, Feb 24, 2011 at 9:15 AM, Nikhil Sheth &lt;nikhil.js(a)gmail.com&gt; wrote: Getting things fixed at the 'plumbing' level is a hard climb but it is worth it since it would also ensure that offline devices can utilize what is technically correct (note, that this does not necessarily imply that the above choices are 'incorrect'). Doing it using web technologies is one thing, doing it for the desktop, especially the offline-desktop is another part of the same coin. We have come a long way since the days when one needed a recompiled Pango (the renderer) to even decently render Indic or, when input methods were flaky. Using standards and developing code pieces that comply with those standards make it easier for platforms across the spectrum to do Indic (and, other complex scripts) well. And, looking at all this discussion I now wish that I submitted a 'state of Indic' paper at some conference happening currently ;) -- sankarshan mukhopadhyay <http://sankarshan.randomink.org/blog>

sudden comparison between transliteration vs. InScript came here. Looking at all the solutions you provided, let me ask one thing. Have you really actively contributed/contributing to any Indian language wikipedia. A survey on the input methods used by Indian wikipedians will give a different answer. Shiju On Thu, Feb 24, 2011 at 9:15 AM, Nikhil Sheth &lt;nikhil.js(a)gmail.com&gt; wrote:

On Thu, Feb 24, 2011 at 9:36 AM, Anivar Aravind &lt;anivar.aravind(a)gmail.com&gt; wrote: The discussion started off with Unicode (Gautam was the OP if I recall correctly). And, then of course it has progressed into a discussion about the various pieces that are complex or, are work-in-progress towards a solution. Sometimes it isn't easy for everyone to see where it is going. Doesn't necessarily mean that we cannot be excellent to each other. -- sankarshan mukhopadhyay <http://sankarshan.randomink.org/blog>

On Thu, February 24, 2011 9:15 am, Nikhil Sheth wrote: Nikhil, all those tools you mentioned are for Transliteration based input methods. And *proprietary* solutions. But in any modern free desktop distributions - whether it is ubuntu, debian, fedora or any other distro, you have lot of inputs methods to select. All of these get installed in default installation itself and you can choose transliteration based input method or inscript or any input method which fits our "laziness-to-learn". And if you are too lazy to learn any key combination, for that also we have offline desktop based solutions which can do suggestions, give drop down list of alternatives etc. (see http://thottingal.in/blog/2008/10/27/swanalekha-m17n-based-input-method-for…) Just give a try on the various input methods available in a latest GNU/Linux distro for your language. If none fits, let the foss developers know what exactly you are looking for. You will surely get a solution. And it is worth to spent some time to learn one standard keyboard layout for your language. You learned to write(using pen) by spending some amount of time right? in Kerala, students at 7th standard learns typing in Malayalam. Once our language syllabus on other languages accept this model, I am sure that every student will be good in writing and typing in their language. Language proficiency will be measured as read+write+type+speak in future. -Santhosh