The on-wiki version of this newsletter can be found here: https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Updates/2023-02-08
----
More thoughts on the Evaluation by the Google.org fellows
During the fellowship, the Google.org fellows gained detailed insight into the Wikifunctions and Abstract Wikipedia project. With the goal to point out potential issues and to discuss potential alternatives to some of the project’s approaches, they wrote a detailed evaluation of the Wikifunctions and Abstract Wikipedia projects. The team read through the evaluation and wrote a detailed answer. Back in December, we published https://meta.wikimedia.org/wiki/Special:MyLanguage/Abstract_Wikipedia/Updates/2022-12-19 the full documents:
-
Wikifunctions and Abstract Wikipedia: An Evaluation https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Google.org_Fellows_evaluation -
Answer to Wikifunctions and Abstract Wikipedia: An Evaluation https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Google.org_Fellows_evaluation_-_Answer
The Signpost https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2023-01-01/Technology_report reported on the evaluation, and an episode of the podcast Between the Brackets https://betweenthebrackets.libsyn.com/episode-130-ariel-gutman discussed it, among other things. Today, we want to provide a condensed and not too technical summary of the documents. The documents were both long and technical. We understand that a two page summary of a forty page discussion will be necessarily simplifying and glance over a lot of nuance, but for everyone who wants to dive into the details, the full documents are available.
The evaluation describes many proposals and suggestions to change the project, and culminates in eight sweeping recommendations. We have accepted a large number of the suggestions the evaluation made. Some have already been implemented, others are tasks on our project task board and are awaiting implementation.
The main area of disagreement lies in the final eight recommendations. In summary, the recommendations suggest that the Foundation should treat Wikifunctions and Abstract Wikipedia as separate projects. Further, they recommend dramatically narrowing the scope of both projects
For Wikifunctions, the evaluation recommends dropping the function model, on which our multilingual user interface rests. They further recommend cutting features that enable non-developers to contribute functions, and focusing on a single programming language, Lua, instead of allowing for several languages.
For Abstract Wikipedia, the evaluation recommends dropping the idea of co-creating one or more natural language generation solutions with the community, suggesting to select either an existing natural language generation system, or to adopt the one that one of the fellows initiated during the fellowship. Instead of creating an all-purpose function repository that would allow the community to experiment and develop a solution, the recommendation is for the project team to decide on and hard-code a single solution to be a system external to the wiki integrating the contributions of the editors.
This is similar to early critiques of Wikidata, which faced a very similar suggestion: numerous voices declared that the decision to allow the community to create and maintain the ontology of Wikidata, instead of either choosing an existing ontology or hiring a team of ontologists, would doom Wikidata. We lost some major funding because of sticking with that decision.
The development team of Abstract Wikipedia and Wikifunctions rejected both sets of recommendations.
One simple reason for some of the rejections was that the evaluation indeed evaluated the technical plan, and not the current state of development. For example, one recommendation is to focus solely on Lua instead of allowing different programming languages, as the latter would take a long time to implement. But given that we have already implemented that feature, adopting the recommendation would slow us down and have us discard work that had already been done.
Some of the rejections for the recommendations were based on Wikimedia values. We think it crucial to have a project that people can contribute to and use in their own natural language. We think it important to enable non-programmers to contribute, and not just for the sake of diversity but also simply because the pool of potential contributors with the relevant skill set is very limited in many language communities. We need people to create and maintain natural language generation functions in many languages. Allowing for a wider skill set to contribute reduces this potential bottleneck. This is why Abstract Wikipedia needs Wikifunctions, and at the same time Wikifunctions benefits from Abstract Wikipedia as a use case. It is a symbiotic relation.
There is currently no symbolic natural language generation system that covers 300 languages, as the Wikimedia movement already does. We can not confidently choose an existing solution, nor the novel one developed during the fellowship. In fact, Prof. Maria Keet of the University of Cape Town, South Africa, joined the Abstract Wikipedia development as a volunteer. She criticized both the existing popular natural language generation systems as well as the one created during the fellowship, because they wouldn’t work well with Niger Congo B languages. The fellow’s proposal was updated to better support the needs of such languages. We were able to do this because our architecture is not tightly coupled to an external natural language generation system.
We do not, and mostly cannot know whether the languages that have not yet been implemented have features that would become a major problem for a given system. In a general purpose, flexible system, the community will have the power to fix any such issue. By selecting any single solution, we would considerably increase the barrier to fixing problems. Instead of contributing to a wiki, we would need developers contributing elsewhere, e.g. in the case of SimpleNLG, to an upstream Java project through git, or in the case of Grammatical Framework, to a project written in Haskell. And worst of all, the communities that will most likely be adversely affected by such a decision would be the ones that are already under-represented.
We are not against using an existing natural language generation system. We will and already are supporting the community in leveraging existing solutions (we are closely working with Grammatical Framework, which is strongly recommended by the fellows). But at this point we will not commit exclusively to any one of the existing solutions. Instead the plan was, from the beginning, to allow the community to build, maintain, and own their approaches, to enable the community to use and combine available solutions, and to co-create Abstract Wikipedia with the community together.
We hope this summary of the eight recommendations and our decision helps in understanding what we are building and why we made the decision.
What’s a good name for a process to improve?
Python has the PEP https://peps.python.org/ (Python Enhancement Proposal), Java the JSR https://jcp.org/en/jsr/overview (Java Specification Request), and many wikis have the RfC https://www.wikidata.org/wiki/Q4654740 (Request for Comment, inspired by IETF) or similar terms. Wikifunctions will be improving well after its launch, and we will need a process to discuss these improvements, either on wiki, on the mailing list, or in Phabricator. This process needs a name!
We won’t make a big vote or anything, but we would like to ask for ideas for the name, and whether you have preferences or concerns. Some internal suggestions have been
-
Community System Change (CSC) -
Little Lambda chat (LLC) -
Food for thought (FFT) -
Wikifunctions Lambda Delta (WILD) -
Wikifunctions System Consultation (WSC) -
Functional Model Proposal (FMP)
Also if you have suggestions on how to run and structure the process of changes to the formal Wikifunctions model initially, let us know. We expect that this will change considerably over time, but there is likely plenty of experience we can work from.
Weekly development update for February 3, 2023
Last week we had our two-monthly fix-it week, which is a meeting-light week focused on fixing bugs and paying down technical debt. Some examples of the work that was done include:
-
Fixing small bugs, e.g. fixWidth issues https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikiLambda/+/886004 or removing unnecessary checks https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikiLambda/+/886048 -
Improving test coverage, e.g. in serializer https://gerrit.wikimedia.org/r/c/mediawiki/services/function-schemata/+/885388 or on the string component https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikiLambda/+/760999, or re-enable temporarily disabled tests, e.g. for mapping empty lists https://gerrit.wikimedia.org/r/c/mediawiki/services/function-orchestrator/+/886079 -
Renaming components and functions to make them clearer and more consistent, e.g. rename ‘generic error’ to ‘unspecified error’ https://gerrit.wikimedia.org/r/c/mediawiki/services/function-schemata/+/885817 or add a prefix to components https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikiLambda/+/884990 -
Refactoring, e.g. restructure whole directories https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikiLambda/+/885393, consolidate metadata dialogs into a single component https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikiLambda/+/885846 or generalize existing functions https://gerrit.wikimedia.org/r/c/mediawiki/services/function-schemata/+/885792
Replacing components with more modern components, e.g. our own 'chip' with the one from Codex https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikiLambda/+/886143