Oh yes, this would really be great. Just think about the money the Foundation gives out meanwhile for translation, plus the many many volunteers' work invested into translation. A free and open translation software is long overdue indeed. Great idea Erik. Greetings Ting Am 4/24/2013 8:29 AM, schrieb Erik Moeller: _______________________________________________ Wikimedia-l mailing list Wikimedia-l(a)lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l

Oh yes, this would really be great. Just think about the money the Foundation gives out meanwhile for translation, plus the many many volunteers' work invested into translation. A free and open translation software is long overdue indeed. Great idea Erik.

>>>

Though the Wikimedia community seems eager to add new projects (Wikidata, Wikivoyage), I wonder how it can be sensible or reasonable to focus on yet another project when the current projects are largely neglected (Wikinews, Wikisource, Wikiversity, Wikibooks, Wikiquote, Wiktionary, etc.).

There's a general trend currently within the Wikimedia Foundation to "narrow focus," which includes shelling out third-party MediaWiki release support to an outside contractor or group, because there are apparently not enough resources within the Wikimedia Foundation's 160-plus staff to support the Wikimedia software platform for anyone other than Wikimedia.

In light of this, it seems even more unreasonable and against good sense to pursue a new machine translation endeavor, virtuous as it may be.

I've stated before why I disagree with this characterization, and I reject this framing. Functionality like the Visual Editor, the mobile site improvements, Lua, and other core engineering initiatives aren't limited in their impact to Wikipedia. The recent efforts on mobile uploading are actually focused on Commons. Deploying new software every two weeks and continually making key usability improvements is not what neglect looks like.

3) Wiktionary could be an even more amazing resource if we would finally tackle the issue of structuring its content more appropriately. I think Wikidata opened a few venues to structure planning in this direction and provide some software, but this would have the potential to provide more support for any external project than many other things we could tackle

One idea I have been mulling over for years is basically how can we use this advantage for the task of creating content available in many languages. Wikidata is an obvious attempt at that, but it really goes only so far. The system I am really aiming at is a different one, and there has been plenty of related work in this direction: imagine a wiki where you enter or edit content, sentence by sentence, but the natural language representation is just a surface syntax for an internal structure.

Your editing interface is a constrained, but natural language.

Now, in order to really make this fly, both the rules for the parsers (interpreting the input) and the serializer (creating the output) would need to be editable by the community - in addition to the content itself. There are a number of major challenges involved, but I have by now a fair idea of how to tackle most of them (and I don't have the time to detail them right now).

Erik, all, sorry for the long mail. Incidentally, I have been thinking in this direction myself for a while, and I have come to a number of conclusions: 1) the Wikimedia movement can not, in its current state, tackle the problem of machine translation of arbitrary text from and to all of our supported languages 2) the Wikimedia movement is probably the single most important source of training data already. Research that I have done with colleagues based on Wikimedia corpora as training data easily beat other corpora, and others are using Wikimedia corpora routinely already. There is not much we can improve here, actually 3) Wiktionary could be an even more amazing resource if we would finally tackle the issue of structuring its content more appropriately. I think Wikidata opened a few venues to structure planning in this direction and provide some software, but this would have the potential to provide more support for any external project than many other things we could tackle Looking at the first statement, there are two ways we could constrain it to make it possibly feasible: a) constrain the number of supported languages. Whereas this would be technically the simpler solution, I think there is agreement that this is not in our interest at all b) constrain the kind of input text we want to support If we constrain b) a lot, we could just go and develop "pages to display for pages that do not exist yet based on Wikidata" in the smaller languages. That's a far cry from machine translating the articles, but it would be a low hanging fruit. And it might help with a desire which is evidently strongly expressed by the mass creation of articles through bots in a growing number of languages. Even more constraints would still allow us to use Wikidata items for tagging and structuring Commons in a language-independent way (this was suggested by Erik earlier). Current machine translation research aims at using massive machine learning supported systems. They usually require big parallel corpora. We do not have big parallel corpora (Wikipedia articles are not translations of each other, in general), especially not for many languages, and there is no reason to believe this is going to change. I would question if we want to build an infrastructure for gathering those corpora from the Web continuously. I do not think we can compete in this arena, or that is the best use of our resources to support projects in this area. We should use our unique features to our advantage. How can we use the unique features of the Wikimedia movement to our advantage? What are our unique features? Well, obviously, the awesome community we are. Our technology, as amazing as it is, running our Websites on the given budget, is nevertheless not what makes us what we are. Most processes on the Wikimedia projects are developed in the community space, and not implemented in bits. To summon Lessing, if code is law, Wikimedia projects are really good in creating a space that allows for a community to live in this space and have the freedom to create their own ecosystem. One idea I have been mulling over for years is basically how can we use this advantage for the task of creating content available in many languages. Wikidata is an obvious attempt at that, but it really goes only so far. The system I am really aiming at is a different one, and there has been plenty of related work in this direction: imagine a wiki where you enter or edit content, sentence by sentence, but the natural language representation is just a surface syntax for an internal structure. Your editing interface is a constrained, but natural language. Now, in order to really make this fly, both the rules for the parsers (interpreting the input) and the serializer (creating the output) would need to be editable by the community - in addition to the content itself. There are a number of major challenges involved, but I have by now a fair idea of how to tackle most of them (and I don't have the time to detail them right now). Wikidata had some design decision inside it that are already geared towards enabling the solution for some of the problems for this kind of wiki. Whatever a structured Wiktionary would look like, it should also be aligned with the requirements of the project outlined here. Basically, we take constrain b, but make it possible to push the constraint further and further through the community - that's how we could scale on this task. This would be far away from solving the problem of automatic translation of text, and even further away from understanding text. But given where we are and the resources we have available, I think it would be a more feasible path towards achieving the mission of the Wikimedia movement than tackling the problem of general machine learning. In summary, I see four calls for action right now (and for all of them this means to first actually think more and write down a project plan and gather input on that), that could and should be tackled in parallel if possible: I ) develop a structured Wiktionary II ) develop a feature that blends into Wikipedia's search if an article about a topic does not exist yet, but we have data on Wikidata about that topic III ) develop a multilingual search, tagging, and structuring environment for Commons IV ) develop structured Wiki content using natural language as a surface syntax, with extensible parsers and serializers None of these goals would require tens of millions or decades of research and development. I think we could have an actionable plan developed within a month or two for all four goals, and my gut feeling is we could reach them all by 2015 or 16, depending when we actually start with implementing them. Goal IV carries a considerable risk, but there's a fair chance it could work out. It could also fail utterly, but if it would even partially succeed ... Cheers, Denny 2013/4/24 Erik Moeller <erik(a)wikimedia.org> -- Project director Wikidata Wikimedia Deutschland e.V. | Obentrautstr. 72 | 10963 Berlin Tel. +49-30-219 158 26-0 | http://wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985. _______________________________________________ Wikimedia-l mailing list Wikimedia-l(a)lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l

In summary, I see four calls for action right now (and for all of them this means to first actually think more and write down a project plan and gather input on that), that could and should be tackled in parallel if possible: I ) develop a structured Wiktionary II ) develop a feature that blends into Wikipedia's search if an article about a topic does not exist yet, but we have data on Wikidata about that topic III ) develop a multilingual search, tagging, and structuring environment for Commons IV ) develop structured Wiki content using natural language as a surface syntax, with extensible parsers and serializers None of these goals would require tens of millions or decades of research and development. I think we could have an actionable plan developed within a month or two for all four goals, and my gut feeling is we could reach them all by 2015 or 16, depending when we actually start with implementing them.

to of So what would you want to enable with this? Faster bootstrapping of content? How would it work, and how would this be superior to an approach like the one taken in the Translate extension (basically, providing good interfaces for 1:1 translation, tracking differences between documents, and offering MT and translation memory based suggestions)? Are there examples of this approach being taken somewhere else?

You are blowing my mind, dude. :)

*awesome*. If we can tie natural-language statements directly to data nodes (rather than merely annotating vague references like we do today), then we'd be much better able to keep language versions in sync. How to make them sane to edit... sounds harder. :)

What would be the limits you would expect from your solution, because you can't expect to just "translate" everything. Form may be a part of the meaning. It's clear that you can't translate a poem for example. Sur wikipedia is not primary concerned about poetry, but it does treat the subject.

Not just bootstrapping the content. By having the primary content be saved in a language independent form, and always translating it on the fly, it would not merely bootstrap content in different languages, but it would mean that editors from different languages would be working on the same content. The texts in the different language is not a translation of each other, but they are all created from the same source. There would be no primacy of, say, English.

Le 2013-04-25 20:56, Theo10011 a écrit : To my mind, there's no such thing as "absolute" meaning. It's all about intrepretation in a given a context by a given interpreter. I mean, I do think that MT could probably be as good as a profesional translators. But even profesional translators can't make "perfect translations". I already gave the example of poetry, but you may also take example of humour, which ask for some cultural background, otherwise you have to explain why it's funny and you know that you have to explain a joke, it's not a joke. That's just how learning process work, you can't "understand" something you didn't experiment. Reading an algorithm won't give you the insight you'll get when you process it mentaly (with the help of pencil and paper) and a textual description of "making love" won't provide you the feeling it provide. English have many so called "non-neutral" problems. As far as I know, if the goal is to use syntactically unambiguous human language, lojban is the best current candidate. English as an international language is a very harmful situation. Believe it or not, but I sometime have to translate to English sentences which are written in French, because the writer was thinking with English idiomatic locution that he poorly translated to French, its native language in which it doesn't know the idiomatic locution. Even worst, I red people which where where using concepts with an English locution because they never matched it with the French locution that they know. And in the other way, I'm not sure that having millions of people speaking a broken English is a wonderful situation for this language. Search "why not english as international language" if you need more documentation. -- Association Culture-Libre http://www.culture-libre.org/ _______________________________________________ Wikimedia-l mailing list Wikimedia-l(a)lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l

Thanks to Jane for introducing CoSyne. But I feel all the wikis do not want to be synchronized to certain wikis. Rather than having identical articles, I hope they would have their own articles. I hope I could have two more tabs at right of the 'Article' and 'Talk' on English Wikipedia for Korean language. The two tabs are 'Article in Korean' and 'Talk in Korean'. The translations would have same information in originals and any editing on an article or a talk in translation pages would go back to the originals. In this case they need to be synchronized precisely. I mean these are done in the scope of English Wikipedia, not related to Korean Wikipedia. But the Korean Wikipedia linked to the left side of a page would be benefited from the translations in English Wikipedia eventually when an Korean Wikipedia editor find a good part of English Wikipedia article could be inserted to Korean Wikipedia. You can find the merits of the exact Korean translation of English Wikipedia or the scheme of the exact translation of big Wikipedias. It will help you reach to more potential contributors. It will make the language barrier lower for those who want to contribute to a Wikipedia they do not speak very well. Also, It could provide the better aligned corpora and it could could track how human translators or reviewers improve the translations. Cheol On 2013. 4. 26., at 오후 9:04, Jane Darnell <jane023(a)gmail.com> wrote: _______________________________________________ Wikimedia-l mailing list Wikimedia-l(a)lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l

Hoi, When we invest in MT it is to convey knowledge, information and primarily Wikipedia articles. They do not have the same problems poetry has. With explanatory articles on a subject there is a web of associated concepts. These concepts are likely to occur in any language if the subject exists in that other language. Consequently MT can work for Wikipedia and provide quite a solid interpretation of what the article is about. This is helped when the associated concepts are recognised as such and when the translations for these concepts are used in the MT. Thanks, GerardM

On 26 April 2013 10:38, Mathieu Stumpf <psychoslave(a)culture-libre.org>wrote;wrote: _______________________________________________ Wikimedia-l mailing list Wikimedia-l(a)lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l

At the moment, Wikisource could be a interesting corpora and laboratory for improving and enhancing OCR, as the OCR generated text is always proofread and corrected by humans. As part of our project ( http://wikisource.org/wiki/Wikisource_vision_development), Micru was looking for a GSoC candidate for studing the reinsertion of proofread text into djvus [1], but at the moment didn't find any interested student. We have some contacts with people at Google working on Tesseract, and they were available for mentoring.

[1] We thought about this both for OCR enhancement purposes and files updating on Commons and Internet Archive (which is off topic here).

We can't usefully help with internet access (and that's proceeding at good pace even in the third world), but language will remain a barrier when people get access. In a few situations politics / firewalling is as well (China, primarily), which is another strategic challenge. That, however, is political and geopolitical, and not an easy nut for WMF to crack. Of the three issues - net, firewalling, and language, one of them is something we can work on. We should think about how to work on that. MT seems like an obvious answer, but not the only possible one.

* Erik Moeller wrote: Wiktionary. If you want to help free software efforts in the area of machine translation, then what they seem to need most is high quality data about words, word forms, and so on, in a readily machine-usable form, and freely licensed.

< [ Andrea's ideas about using Wikisource to improve OCR tools ] I built various tools that could be fairly easily adapted for this, my http://www.google.com/search?q=site:lists.w3.org+intitle:hoehrmann+ocr notes are available. One of the tools for instance is a diff tool, see image at <http://lists.w3.org/Archives/Public/www-archive/2012Apr/0031>.

We could basically clone the frontend component of Google Translate, and use Moses as a backend. The work would be mostly JavaScript... the next job would be to develop a corpus sharing site, hosting any available freely-licensed output of the frontend tool.

On Fri, Apr 26, 2013 at 7:57 PM, Samuel Klein &lt;meta.sj(a)gmail.com&gt; wrote: OmegaWiki is a masterpiece from the perspective of one [computational] linguist. Erik made the structure so well, that it's the best starting point to create a contemporary multilingual dictionary. I didn't see anything better in concept. (And, yes, when I was thinking about creating such software by my own, I was always at the dead end of "but, OmegaWiki is already that".)

On Fri, Apr 26, 2013 at 1:24 PM, Bjoern Hoehrmann &lt;derhoermi(a)gmx.net&gt; wrote: Yes. Finding a way to capture and integrate the work OmegaWiki has done into a new Wikidata-powered Wiktionary would be a useful start. And we've already sort of claimed the space (though we are neglecting it) -- it's discouraging to anyone else who might otherwise try to build a brilliant free structured dictionary that we are *so close* to getting it right.