Re: [Abstract-wikipedia] NLP issues severely overlooked

On 4 Jul 2020, at 12:26, Louis Lecailliez &lt;louis.lecailliez(a)outlook.fr&gt; wrote:  Hello, my name is Louis Lecailliez, PhD student at Kyoto University in education technology. I'm a Computer Science and NLP graduate. One thing I do is working on language learner's knowledge modelling as graphs. The Abstract Wikipedia project is really interesting. There is however two very concerning issues I spotted when reading the associated paper draft (https://arxiv.org/abs/2004.04733). The following email could be read as negative, but please don't take it as such: my purpose is to avoid spending people efforts and money for things that can (need to!) be fixed upfront. 1. Issues with NLP The main issue is that the difficulty of the NLP task of generating natural text from an abstract representation is severely overlooked. This stems from the other main problem: the paper is not based on the decades of previous work in that space. As I understand it, the main value proposition of Abstract Wikipedia (AW) is a computer representation of encyclopedic knowledge that can be projected into different existing natural languages, with the goal of supporting a huge number of them. Plus, an editor to make this happen easily. This is in fact surprisingly extremely close to what the Universal Networking Language (UNL) project, which started 20 years ago, aims to do. UNL provides a language agnostic representation of text that uses hypergraph. Software (called EnConverter) produce UNL graphs from natural text in a given language. Another kind of software called DeConverter do the reverse, that is producing natural text from the abstract representation. This is exactly the same function of the "renderers" in the AW paper. The way of doing it is also similar: by applying successive transformations until the final text string is produced. In general, that kind of abstract meaning representation is called an Interlingua, and is widely used in Machine Translation (MT) systems. Disregarding two decades of work, in the UNL case, on the same problem space (rule-based machine translation, here from an abstract language as fixed source language), which was itself based on few other decades of work, doesn't seem to be a wise move to start a new project. For a start, the graph representation used in the AW will likely not be expressive enough to encode linguistic knowledge; this is why UNL uses hypergraphs instead of graphs. The problem is glaring when looking at the references list: the list is bloated with irrelevant references (such as those to programming languages [27, 37, 41, 77], Turing completeness being the worst offender [11, 17, 23, ...]) while containing only two references [7, 85] to the really hard part of the project: generating natural language from the abstract representation. There are few more relevant references about natural language generation, but this isn't enough. Interestingly, [85] is an UNL paper, but not the main one. Moreover, it is cited in Section 9 "Opening future research". This should be instead placed in a "Previous work" section which is missing from the paper. To fill a part of this section yet to be written, I propose the following references: [*1] Uchida, H., Zhu, M., & Della Senta, T. (1999). A gift for a millennium. IAS/UNU, Tokyo. https://www.researchgate.net/profile/Hiroshi_Uchida2/publication/239328725_… [*2] Wang-Ju Tsai (2004) La coédition langue-UNL pour partager la révision entre langues d'un document multilingue. [Language-UNL coedition to share revisions in a multilingual document] Thèse de doctorat. Grenoble. https://pdfs.semanticscholar.org/b030/ea4662e393657b9a134c006ca5b08e8a23b3.… [3*] Boitet, C., & Tsai, W. J. (2002). La coédition langue<—> UNL pour partager la révision entre les langues d'un document multilingue: un concept unificateur. Proc. TALN-02, Nancy, 22-26. http://www.afcp-parole.org/doc/Archives_JEP/2002_XXIVe_JEP_Nancy/talnrecita… [4*] Tomokiyo, M., Mangeot, M., & Boitet, C. (2019). Development of a classifiers/quantifiers dictionary towards French-Japanese MT. arXiv preprint arXiv:1902.08061. https://arxiv.org/pdf/1902.08061.pdf [5*] Boguslavsky, I. (2005). Some controversial issues of UNL: Linguistic aspects. Research on Computer Science, 12, 77-100. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.212.2058&rep=r… [6*] Boitet, C. (2002). A rationale for using UNL as an interlingua and more in various domains. In Proc. LREC-02 First International Workshop on UNL, other Interlinguas, and their Applications, Las Palmas (pp. 26-31). https://www.cicling.org/2005/unl-book/Papers/003.pdf [7*] Dhanabalan, T., & Geetha, T. V. (2003, December). UNL deconverter for Tamil. In International Conference on the Convergences of Knowledge, Culture, Language and Information Technologies. http://www.cfilt.iitb.ac.in/convergence03/all%20data/paper%20032-372.pdf [8*] Singh, S., Dalal, M., Vachhani, V., Bhattacharyya, P., & Damani, O. P. (2007). Hindi generation from Interlingua (UNL). Machine Translation Summit XI. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.78.979&rep=rep… [9*] Banarescu, L., Bonial, C., Cai, S., Georgescu, M., Griffitt, K., Hermjakob, U., ... & Schneider, N. (2013, August). Abstract meaning representation for sembanking. In Proceedings of the 7th linguistic annotation workshop and interoperability with discourse (pp. 178-186). https://www.aclweb.org/anthology/W13-2322.pdf [10*] Berment, V., & Boitet, C. (2012). Heloise—An Ariane-G5 Compatible Rnvironment for Developing Expert MT Systems Online. In Proceedings of COLING 2012: Demonstration Papers (pp. 9-16). https://www.aclweb.org/anthology/C12-3002.pdf [11*] Berment, V. (2005). Online Translation Services for the Lao Language. In Proceedings of the First International Conference on Lao Studies. De Kalb, Illinois, USA (pp. 1-11). https://www.researchgate.net/profile/Vincent_Berment/publication/242140227_… [*1] is the paper that describes UNL. [2*] is a doctoral thesis discussing a core problem AW is trying to address too. [3*] is a short paper done in the scope of [2*], even if you don't understand French you can have a look at the figures: two of them are about an editor similar in principe to what AW wants to incorporate. [5*] Insights about UNL expressivity issues, 10 years after the project's start. [6*] More UNL, with short history and context in which it is used. [4*] shows how deep natural language conversion goes: this paper addresses the issue of classifiers in French and Japanese. This is just one linguistic issue and there are dozens if not hundreds of such. An important point is that both of the languages involved need to be taken into account when modelling the abstract encoding, otherwise too much information is lost for producing a correct output. [7*] [8*] are very valuable examples of real world deconverter systems for UNL. As it's visible on [7*]'s Figure 1 and [8*]'s Figure 2, the process is *way* more complicated than a single "renderers" box. Moreover, there are very distinct identifiable steps, informed by linguistics. The AW does not describe any such structuration of natural text generation processing steps, everything is supposed to be happening in some unstructured "lambda" system. Also missing are the specialized resources (UNL-Hindi dictionary, Tamil Word dictionary, co-occurrence dictionary, etc.) required for the task. Nothing precise is said about linguistic resources in the AW paper except for "These function finally can call the lexicographic knowlegde stored in Wikidata.", which is not very convincing: first because Wiktionary projects themselves severely lacks content and structure for those who has some content at all, secondly since specialized NLP ressources are missing there too (note: I'm not familiar with Wikidata so I could be wrong, however I never saw it cited for the kind of NLP resources I'm talking about). [10*] is a translation system built with "specialised languages for linguistic programming (SLLPs)" which is the service Wikilambda is supposed to provide for Abstract Wikipedia. [11*] gives the estimation of 2500 hours for the development (by a specialist) of three linguistic modules for Lao processing. So, in regard to the difficulty of the task, and previous work in the literature, the AW paper does not provide any convincing evidence that the technology on which it is supposed to be built can even reach the state-of-art. Dismissing every existing formal and software systems on the ground of "no consensus commiting to any specific linguistic theory" is not gonna work: this will result in ad hoc implementation-driven formalism that will have hard time fullfilling its goal. The NLP part (generating sentences from abstract representation) is the hardest of the project, yet it’s by far the least convincing one. "Abstract Wikipedia is indeed firmly within this tradition, and in preparation for this project we studied numerous predecessors." I would like to believe so, but the lack of corresponding reference as well as lack of previous work section tends to prove the contrary. While I can't advice for a switch to UNL, as I'm not specialist of it, it would be smart to capitalize on the work done on it by highly skilled (PhD level) individuals. As the UNL system is built on hypergraphs, it probably could be made interoperable easily with RDF knowledge graphs if named graphs are used. By having a UNL/RDF specification (yet to be written), the vision exposed in the AW paper may be reached sooner by reusing existing software (we are speaking of thousands man-year of work as per [11*]), and almost as importantly, an existing formalism that has been "debugged" for decades. There are probably other systems I'm unaware of that are worth investigating too, some like [9*] having more specialized usage. In any case, there is a strong need to back the paper and the project on the existing (huge) literature. 2. Other issues "In order to evaluate a function call, an evaluator can choose from a multitude of backends: it may evaluate the function call in the browser, in the cloud, on the servers of the Wikimedia Foundation, on a distributed peer-to-peer evaluation platform, or natively on the user’s machine in a dedicated hosting runtime, which could be a mobile app or a server on the user’s computer." This part is big technical creep. There is no reason to turn the project into a distributed heterogenous computing platform with a dedicated runtime, which could be a research project on its own, when the stated goal is to provide abstract multilingual encyclopedic content. All the computation can be done on servers (cloud is servers too) and cached. This is way easier to implement, test and deliver than to implement 10 different backends with various progress in implementation, incompatibilities and runtime characteristics. The paper presents AW as sitting on top on WL. Both are big projects. Sitting a big project on top of another one is really risky, as it means a significant milestone must first be reached in the dependency (here WL), which would likely took some years, before even starting the work on the other project. AW can be realised with current tools and engineering practices. "One obstacle in the democratization of programming has been that almost every programming language requires first to learn some basic English." This strong affirmation needs to be sourced. Programming languages, save for a few keywords, doesn't rely much on English. The vast insuccess of localized version of programming languages (such as French Basic) as well as the heavy use of existing programming language in countries that doesn't even use the Latin alphabet (China, Russia) tends to prove that English is not all a bottleneck for the democratization of programming. [53] is cited later in the paper but is a pop-linguistic article from an online newspaper, not an academic article. 3. Final words To finish on a positive note, I would like to highlight the points I really like in the paper. First, its collaborative and open nature, like all Wikimedia projects, gives him a much higher chance of success than its predecessors. If UNL is not too well-known, it’s not because it didn't yield research achievements, but because one selected institution per language is working on it and keep the resources and software within the lab walls. Secondly, there are some very welcome out-of-scope features: conversion from natural language, restriction to encyclopedic style text. This will allow for more focused effort towards the end goal, making it more achievable. And finally, the choice to go with symbolic/rule-based system with a touch of other ML where useful. This is, as said in the paper, a big win for explainability and using human contributions to build the system. This will also keep the computing cost to a more sane baseline than what the current deep learning models require. I think the project can succeed thanks to its openess, yet there is are real dangers visible in the paper: on the NLP side to reinvent a wheel that took 40 years to build, and on the technical side to lose time and effort on a project not required per se for AW to be build. As I spend a significant time (~10 hours) gathering references and writing this email (which is 5 pages long in Word), I would like to be mentioned as co-author in the final paper if any idea or references presented here is used in it. Best regards, Louis Lecailliez PS: 4. Typos * "These two projects will considerably expand the capabilities of the Wikimedia platform to enable every single human being to freely share share in the sum of all knowledge." => duplicate share * "The content is than turned into" => The content is then turned into * "[26] Charles J Fillmore, Russell Lee-Goldman, and Russell Rhodes. The framenet constructicon. Sign-based construction grammar, pages 309–372, 2012." => The framenet construction * "These function finally can call the lexicographic knowlegde stored in Wikidata." => These function finally can call the lexicographic knowledge stored in Wikidata * "[102] George Kinsley Zipf. Human Behavior and the Pirnciple of Least Effort. Addison-Wesley, 1949." => [102] George Kinsley Zipf. Human Behavior and the Principle of Least Effort. Addison-Wesley, 1949. * "Allowing the individual language Wikipedias to call Wikilambda has an addtional benefit." => Allowing the individual language Wikipedias to call Wikilambda has an additional benefit. _______________________________________________ Abstract-Wikipedia mailing list Abstract-Wikipedia(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/abstract-wikipedia