Hello,
my name is Louis Lecailliez, PhD student at Kyoto University in education technology.
I'm a Computer Science and NLP graduate. One thing I do is working on language
learner's knowledge modelling as graphs.
The Abstract Wikipedia project is really interesting. There is however two very concerning
issues I spotted when reading the associated paper draft
(
https://arxiv.org/abs/2004.04733). The following email could be read as negative, but
please don't take it as such: my purpose is to avoid spending people efforts and money
for things that can (need to!) be fixed upfront.
1. Issues with NLP
The main issue is that the difficulty of the NLP task of generating natural text from an
abstract representation is severely overlooked. This stems from the other main problem:
the paper is not based on the decades of previous work in that space.
As I understand it, the main value proposition of Abstract Wikipedia (AW) is a computer
representation of encyclopedic knowledge that can be projected into different existing
natural languages, with the goal of supporting a huge number of them. Plus, an editor to
make this happen easily.
This is in fact surprisingly extremely close to what the Universal Networking Language
(UNL) project, which started 20 years ago, aims to do. UNL provides a language agnostic
representation of text that uses hypergraph. Software (called EnConverter) produce UNL
graphs from natural text in a given language. Another kind of software called DeConverter
do the reverse, that is producing natural text from the abstract representation. This is
exactly the same function of the "renderers" in the AW paper. The way of doing
it is also similar: by applying successive transformations until the final text string is
produced. In general, that kind of abstract meaning representation is called an
Interlingua, and is widely used in Machine Translation (MT) systems.
Disregarding two decades of work, in the UNL case, on the same problem space (rule-based
machine translation, here from an abstract language as fixed source language), which was
itself based on few other decades of work, doesn't seem to be a wise move to start a
new project. For a start, the graph representation used in the AW will likely not be
expressive enough to encode linguistic knowledge; this is why UNL uses hypergraphs instead
of graphs.
The problem is glaring when looking at the references list: the list is bloated with
irrelevant references (such as those to programming languages [27, 37, 41, 77], Turing
completeness being the worst offender [11, 17, 23, ...]) while containing only two
references [7, 85] to the really hard part of the project: generating natural language
from the abstract representation. There are few more relevant references about natural
language generation, but this isn't enough.
Interestingly, [85] is an UNL paper, but not the main one. Moreover, it is cited in
Section 9 "Opening future research". This should be instead placed in a
"Previous work" section which is missing from the paper.
To fill a part of this section yet to be written, I propose the following references:
[*1] Uchida, H., Zhu, M., & Della Senta, T. (1999). A gift for a millennium. IAS/UNU,
Tokyo.
https://www.researchgate.net/profile/Hiroshi_Uchida2/publication/239328725_…
[*2] Wang-Ju Tsai (2004) La coédition langue-UNL pour partager la révision entre langues
d'un document multilingue. [Language-UNL coedition to share revisions in a
multilingual document] Thèse de doctorat. Grenoble.
https://pdfs.semanticscholar.org/b030/ea4662e393657b9a134c006ca5b08e8a23b3.…
[3*] Boitet, C., & Tsai, W. J. (2002). La coédition langue<—> UNL pour partager
la révision entre les langues d'un document multilingue: un concept unificateur. Proc.
TALN-02, Nancy, 22-26.
http://www.afcp-parole.org/doc/Archives_JEP/2002_XXIVe_JEP_Nancy/talnrecita…
[4*] Tomokiyo, M., Mangeot, M., & Boitet, C. (2019). Development of a
classifiers/quantifiers dictionary towards French-Japanese MT. arXiv preprint
arXiv:1902.08061.
https://arxiv.org/pdf/1902.08061.pdf
[5*] Boguslavsky, I. (2005). Some controversial issues of UNL: Linguistic aspects.
Research on Computer Science, 12, 77-100.
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.212.2058&rep=r…
[6*] Boitet, C. (2002). A rationale for using UNL as an interlingua and more in various
domains. In Proc. LREC-02 First International Workshop on UNL, other Interlinguas, and
their Applications, Las Palmas (pp. 26-31).
https://www.cicling.org/2005/unl-book/Papers/003.pdf
[7*] Dhanabalan, T., & Geetha, T. V. (2003, December). UNL deconverter for Tamil. In
International Conference on the Convergences of Knowledge, Culture, Language and
Information Technologies.
http://www.cfilt.iitb.ac.in/convergence03/all%20data/paper%20032-372.pdf
[8*] Singh, S., Dalal, M., Vachhani, V., Bhattacharyya, P., & Damani, O. P. (2007).
Hindi generation from Interlingua (UNL). Machine Translation Summit XI.
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.78.979&rep=rep…
[9*] Banarescu, L., Bonial, C., Cai, S., Georgescu, M., Griffitt, K., Hermjakob, U., ...
& Schneider, N. (2013, August). Abstract meaning representation for sembanking. In
Proceedings of the 7th linguistic annotation workshop and interoperability with discourse
(pp. 178-186).
https://www.aclweb.org/anthology/W13-2322.pdf
[10*] Berment, V., & Boitet, C. (2012). Heloise—An Ariane-G5 Compatible Rnvironment
for Developing Expert MT Systems Online. In Proceedings of COLING 2012: Demonstration
Papers (pp. 9-16).
https://www.aclweb.org/anthology/C12-3002.pdf
[11*] Berment, V. (2005). Online Translation Services for the Lao Language. In Proceedings
of the First International Conference on Lao Studies. De Kalb, Illinois, USA (pp. 1-11).
https://www.researchgate.net/profile/Vincent_Berment/publication/242140227_…
[*1] is the paper that describes UNL. [2*] is a doctoral thesis discussing a core problem
AW is trying to address too. [3*] is a short paper done in the scope of [2*], even if you
don't understand French you can have a look at the figures: two of them are about an
editor similar in principe to what AW wants to incorporate.
[5*] Insights about UNL expressivity issues, 10 years after the project's start. [6*]
More UNL, with short history and context in which it is used.
[4*] shows how deep natural language conversion goes: this paper addresses the issue of
classifiers in French and Japanese. This is just one linguistic issue and there are dozens
if not hundreds of such. An important point is that both of the languages involved need to
be taken into account when modelling the abstract encoding, otherwise too much information
is lost for producing a correct output.
[7*] [8*] are very valuable examples of real world deconverter systems for UNL. As
it's visible on [7*]'s Figure 1 and [8*]'s Figure 2, the process is *way* more
complicated than a single "renderers" box. Moreover, there are very distinct
identifiable steps, informed by linguistics. The AW does not describe any such
structuration of natural text generation processing steps, everything is supposed to be
happening in some unstructured "lambda" system. Also missing are the specialized
resources (UNL-Hindi dictionary, Tamil Word dictionary, co-occurrence dictionary, etc.)
required for the task. Nothing precise is said about linguistic resources in the AW paper
except for "These function finally can call the lexicographic knowlegde stored in
Wikidata.", which is not very convincing: first because Wiktionary projects
themselves severely lacks content and structure for those who has some content at all,
secondly since specialized NLP ressources are missing there too (note: I'm not
familiar with Wikidata so I could be wrong, however I never saw it cited for the kind of
NLP resources I'm talking about).
[10*] is a translation system built with "specialised languages for linguistic
programming (SLLPs)" which is the service Wikilambda is supposed to provide for
Abstract Wikipedia. [11*] gives the estimation of 2500 hours for the development (by a
specialist) of three linguistic modules for Lao processing.
So, in regard to the difficulty of the task, and previous work in the literature, the AW
paper does not provide any convincing evidence that the technology on which it is supposed
to be built can even reach the state-of-art. Dismissing every existing formal and software
systems on the ground of "no consensus commiting to any specific linguistic
theory" is not gonna work: this will result in ad hoc implementation-driven formalism
that will have hard time fullfilling its goal.
The NLP part (generating sentences from abstract representation) is the hardest of the
project, yet it’s by far the least convincing one. "Abstract Wikipedia is indeed
firmly within this tradition, and in preparation for this project we studied numerous
predecessors." I would like to believe so, but the lack of corresponding reference as
well as lack of previous work section tends to prove the contrary.
While I can't advice for a switch to UNL, as I'm not specialist of it, it would be
smart to capitalize on the work done on it by highly skilled (PhD level) individuals. As
the UNL system is built on hypergraphs, it probably could be made interoperable easily
with RDF knowledge graphs if named graphs are used. By having a UNL/RDF specification (yet
to be written), the vision exposed in the AW paper may be reached sooner by reusing
existing software (we are speaking of thousands man-year of work as per [11*]), and almost
as importantly, an existing formalism that has been "debugged" for decades.
There are probably other systems I'm unaware of that are worth investigating too, some
like [9*] having more specialized usage. In any case, there is a strong need to back the
paper and the project on the existing (huge) literature.
2. Other issues
"In order to evaluate a function call, an evaluator can choose from a multitude of
backends: it may evaluate the function
call in the browser, in the cloud, on the servers of the Wikimedia Foundation, on a
distributed peer-to-peer evaluation
platform, or natively on the user’s machine in a dedicated hosting runtime, which could be
a mobile app or a server on the
user’s computer."
This part is big technical creep. There is no reason to turn the project into a
distributed heterogenous computing platform with a dedicated runtime, which could be a
research project on its own, when the stated goal is to provide abstract multilingual
encyclopedic content. All the computation can be done on servers (cloud is servers too)
and cached. This is way easier to implement, test and deliver than to implement 10
different backends with various progress in implementation, incompatibilities and runtime
characteristics.
The paper presents AW as sitting on top on WL. Both are big projects. Sitting a big
project on top of another one is really risky, as it means a significant milestone must
first be reached in the dependency (here WL), which would likely took some years, before
even starting the work on the other project. AW can be realised with current tools and
engineering practices.
"One obstacle in the democratization of programming has been that almost every
programming language requires first to learn some basic English."
This strong affirmation needs to be sourced. Programming languages, save for a few
keywords, doesn't rely much on English. The vast insuccess of localized version of
programming languages (such as French Basic) as well as the heavy use of existing
programming language in countries that doesn't even use the Latin alphabet (China,
Russia) tends to prove that English is not all a bottleneck for the democratization of
programming. [53] is cited later in the paper but is a pop-linguistic article from an
online newspaper, not an academic article.
3. Final words
To finish on a positive note, I would like to highlight the points I really like in the
paper. First, its collaborative and open nature, like all Wikimedia projects, gives him a
much higher chance of success than its predecessors. If UNL is not too well-known, it’s
not because it didn't yield research achievements, but because one selected
institution per language is working on it and keep the resources and software within the
lab walls. Secondly, there are some very welcome out-of-scope features: conversion from
natural language, restriction to encyclopedic style text. This will allow for more focused
effort towards the end goal, making it more achievable. And finally, the choice to go with
symbolic/rule-based system with a touch of other ML where useful. This is, as said in the
paper, a big win for explainability and using human contributions to build the system.
This will also keep the computing cost to a more sane baseline than what the current deep
learning models require.
I think the project can succeed thanks to its openess, yet there is are real dangers
visible in the paper: on the NLP side to reinvent a wheel that took 40 years to build, and
on the technical side to lose time and effort on a project not required per se for AW to
be build.
As I spend a significant time (~10 hours) gathering references and writing this email
(which is 5 pages long in Word), I would like to be mentioned as co-author in the final
paper if any idea or references presented here is used in it.
Best regards,
Louis Lecailliez
PS:
4. Typos
* "These two projects will considerably expand the capabilities of the Wikimedia
platform to enable every single human being to freely share share in the sum of all
knowledge." => duplicate share
* "The content is than turned into" => The content is then turned into
* "[26] Charles J Fillmore, Russell Lee-Goldman, and Russell Rhodes. The framenet
constructicon. Sign-based construction grammar, pages 309–372, 2012." => The
framenet construction
* "These function finally can call the lexicographic knowlegde stored in
Wikidata." => These function finally can call the lexicographic knowledge stored
in Wikidata
* "[102] George Kinsley Zipf. Human Behavior and the Pirnciple of Least Effort.
Addison-Wesley, 1949." => [102] George Kinsley Zipf. Human Behavior and the
Principle of Least Effort. Addison-Wesley, 1949.
* "Allowing the individual language Wikipedias to call Wikilambda has an addtional
benefit." => Allowing the individual language Wikipedias to call Wikilambda has an
additional benefit.