The on-wiki version of this newsletter can be found here:
https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Updates/2022-08-19
--
Our Google.org fellow, Ariel Gutman
<https://meta.wikimedia.org/wiki/User:AGutman-WMF>, together with Prof. Maria
Keet <http://www.meteck.org/>, who is devoting part of her sabbatical year
to work with the Abstract Wikipedia's Natural Language Generation
workstream, have recently authored a detailed specification of a template
language
<https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Template_Language_for_Wi…>.
This aims to allow Wikifunctions contributors to easily create renderers
<https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Glossary#renderer>
of abstract
content
<https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Glossary#Content>. For
instance, entity Q7259 <https://www.wikidata.org/wiki/Q7259> has property
P106 <https://www.wikidata.org/wiki/Property:P106> pointing to Q5482740
<https://www.wikidata.org/wiki/Q5482740> asserted in Wikidata, and with all
the machinery in place, it may render as, e.g., *“Ada Lovelace was a
programmer.”* The template language seeks to assist with specifying the
structure for generating sentences so that the structured content will be
displayed as text in a natural language of one’s choice.
You may recall from the architecture proposal
<https://meta.wikimedia.org/wiki/Abstract_Wikipedia/NLG_system_architecture_…>
that
every constructor
<https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Glossary#Constructor>
(which
typically aims to capture the meaning of a single phrase or sentence
structure) will be matched with a specific template to render that
constructor as text. The templates will reside in Wikifunctions, and will
be parsed into Composition
<https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Glossary#composition>
syntax,
so that it can act as a Renderer. An initial version of this parser has
already been implemented
<https://gerrit.wikimedia.org/r/c/mediawiki/tools/wikilambda-cli/%2B/814724> as
part of the Wikifunctions CLI tool
<https://www.mediawiki.org/wiki/Extension:WikiLambda/CLI>, which you can
toy around with.
What do these templates look like? A template is a combination of text and
slots, where slots can refer to other templates or functions from
Wikifunctions, allowing for dynamic content. The specification of
grammatical constraints is done through dependency relations (using, for
instance, the UD <https://universaldependencies.org/> formalism for grammar
annotations) specified as labels within the slots. As for the text, it may
represent static text, which will be kept untouched throughout the
rendering, or it may represent lexemes that can assume different forms
according to the neighboring syntactic and phonological constraints.
For starters, let's look at an example template to generate a sentence
describing the age of a person, e.g. *"Dan is 20 years old."*, given a
constructor with two fields: entity (the Q-id of the person) and years (the
age). In English, this template may look like this:
{Person(entity)} is {nummod:Cardinal(years)} {root:Lexeme(L2505)} old.
There are three slots, which are delimited by curly brackets:
1. {Person(entity)} resolves to the name of the person.
2. {nummod:Cardinal(years)} resolves to the number of years. It is
marked as the "*num*eral *mod*ifier" of the third slot.
3. {root:Lexeme(L2505)} fetches from Wikidata Lexeme L2505
<https://www.wikidata.org/wiki/Lexeme:L2505>, which refers to the lemma
"year". Since the slot is marked as root, it will be linked to the
previous slot, allowing for the selection of the right form of the lexeme:
"year" or "years".
The remaining text in the template – "is" and "old" – is in this case
static text. In other cases, we might need to specify that the verb is can
inflect as well or the number may need some additional processing to render
it properly, and we would use similar dependency labels to mark
subject-verb agreement and other types of agreement across the sentence’s
constituents.
In the document, similar examples
<https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Template_Language_for_Wi…>
–
though more complex – are given for 4 other languages (Swedish, French,
Hebrew and isiZulu), each presenting its own peculiarities and challenges
but that still can be captured successfully with the proposed template
language. We invite you to read the document, provide feedback and try to
come up with challenging examples in other languages that may prove
difficult to render using this formalism, so we can improve on it and
achieve the broadest possible applicability to, ideally, all natural
languages used.
Wikimania video
Last week was Wikimania 2022
<https://meta.wikimedia.org/wiki/Wikimania_2022>, the annual event for
Wikimedians from all over the world to meet and discuss. There were two
sessions on Wikifunctions, one session on Wikifunctions
<https://wikimania.wikimedia.org/wiki/2022:Submissions/Wikifunctions_-_A_new…>
lead
by the team and one on Ninai and Udiron lead by Mahir Morshed
<https://wikimania.wikimedia.org/wiki/2022:Submissions/Ninai-Udiron:_Using_W…>
.
Our session consisted of a short introduction to Wikifunctions by Denny,
followed by a pre-recorded section, where several team members had short
deep dives into different topics.
We had:
- James Forrester on the technical architecture
- Amin Al Hazwani on the design language
- Genoveva Galarza Heredero on the content model
- Julia Kieserman on Codex
- Cory Massaro on knowledge equity
- Ariel Gutman on natural language generation, an intro to the first
part of the newsletter above
- Ali Assaf on formalizing the function model
You can watch this pre-recorded segment on Commons
<https://commons.wikimedia.org/wiki/File:Wikimania_2022_Wikifunctions_HD.webm>
.
As with all Wikimania sessions, collaborative note taking was enabled.
The notes
on the session
<https://wikimania.wikimedia.org/wiki/2022:Submissions/Wikifunctions_-_A_new…>
also
contain all questions that have been asked and answered in the closing part
of the session, following the video. A full video of the session is
available on YouTube <https://www.youtube.com/watch?v=Zasie41p1-U?t=35278s>,
but note that playing the pre-recorded video faced a number of technical
issues. You might want to skip to the Commons video instead. Uploads of
individual sessions are expected to become available later.
Mahir Morshed had a Wikimania session about Ninai and Udiron
<https://wikimania.wikimedia.org/wiki/2022:Submissions/Ninai-Udiron:_Using_W…>
and
the recording starts here
<https://www.youtube.com/watch?v=BCRi1VRtQXE&t=3431s>. Ninai and Udiron are
tools for natural language generation, and we have introduced them in earlier
newsletters
<https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Updates/2021-09-03>.
Workstream updates as of August 12, 2022
Performance
- Started performance analysis methodology documentation
- Set up health-check API endpoint for Wikilambda
Natural language generation
- Not too much progress due to team members' vacation time. Started
adding noun class information for isiZulu, Mboshi, Kiswahili
Meta-data
- Finished display of metadata dialog on tester page
- Created some new PHP utilities for ZMaps
Experience
- Fixed and merged Beta launch blockers
- Made great progress on fixing various bugs
- Began researching diffing options
The on-wiki version of this newsletter can be found here:
https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Updates/2022-08-09
--
We are pleased to invite you all to the Wikifunctions Beta
<https://wikifunctions.beta.wmflabs.org/wiki/Wikifunctions:Main_Page>!
We have been planning to set up a Wikifunctions Beta for a long time, but
we felt that our prototype was not yet stable enough for a public Beta.
With a concerted effort, our engineers together with the Google.org fellows
have pushed to get the system to a stage that works well enough to warrant
a Beta release. So, without further ado, you can visit the Beta of
Wikifunctions at wikifunctions.beta.wmflabs.org/wiki/Wikifunctions:Main_Page
.
*Thank you, Lucas!* We are thankful to Lucas Werkmeister
<https://meta.wikimedia.org/wiki/User:Lucas_Werkmeister>, who has been
running the test NotWikilambda
<https://notwikilambda.toolforge.org/wiki/Not_Wikilambda:Main_Page> instance
the whole time. Lucas has done an immeasurable service to the community,
and to the development team, by setting up a public instance of WikiLambda
early on, which allowed many to get an early feeling for our prototype.
Lucas’s service will always have a very special place in the history of
Wikifunctions, and not only because of NotWikilambda. We understand that
NotWikilambda is now going to be discontinued, having served us well. Lucas
is a living example of the best in our community.
*What is Wikifunctions?* Wikifunctions will be a new top-level project of
the Wikimedia Foundation, which we are currently developing. Wikifunctions
will be a place for everyone to collaboratively create and maintain a
library of code functions. The goal of Wikifunctions is to support the
Wikimedia projects and beyond, and to allow everyone to call and re-use
functions in the world's natural and programming languages. That is our
mission statement
<https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Updates/2021-04-08>.
Many of the core functionalities should be available for you right now.
Beta runs the newest version of the code, usually just a few minutes behind
development merges. This is also true for both the "WikiLambda" MediaWiki
code, and the orchestrator and evaluator back-end services, which execute
function calls for the users.
Note: All content in Beta may be deleted at any time, so don’t rely on Beta
having persistent content.
*What works?* What should you be able to do on the Wikifunctions Beta right
now? You should be able to
- Call functions and see the results of those function calls
- Create new and edit existing functions
- Create and edit implementations, in JavaScript or Python, and through
composition of other functions
- Create and edit testers
- Connect testers and implementations with the functions
- Create and edit types
- Create and edit instances of types
- Create functions for the user-defined types
All of these should work on any modern device, whether mobile or desktop,
and in different natural languages.
One thing that is currently a bit incomplete, and that will be improved, is
the workflow to create implementations in code for functions that are
taking or returning user-defined types. It is possible, but it might be a
bit difficult to write. We want to rewrite that part.
*What does not work?* Many things don’t work yet. Creating an
implementation or a tester is best done by creating a new object, and then
selecting “Implementation” or “Tester” as the type, and then selecting the
respective function. For connecting them, you would need to use the
“Fallback editor” that is visible once you are in edit mode on a function.
Other big missing pieces:
- Search. Currently, your best bet to find what’s already on the wiki
and to look through the existing pages is not by using the search, but
through the list of objects
<https://wikifunctions.beta.wmflabs.org/wiki/Special:ListZObjectsByType>.
The most useful lists here will likely be the list of functions
<https://wikifunctions.beta.wmflabs.org/wiki/Special:ListZObjectsByType/Z8>
and
the list of types
<https://wikifunctions.beta.wmflabs.org/wiki/Special:ListZObjectsByType/Z4>
.
- Language settings are not sticky. By adding the uselang parameter to a
URL you can see how a page looks in a different language, but once you
navigate to another page, the setting gets lost again. Log in to set your
display language to change it for good.
- User rights. This means that you can currently edit things that you
shouldn’t be able to. This means you can do edits which will break things
badly. Please don’t edit the core data model, or do things like add a third
Boolean value.
- Documentation. The place where we store documentation alongside each
Function etc. is not implemented yet. For now, as a work around, you can
add documentation on the respective Talk pages.
Besides these larger missing pieces, many smaller pieces are not right yet.
There's no recording of what functions are used, and so no listing of
popular functions. There's no documentation welcoming people to
Wikifunctions, or helping them know how to use the system. There's no
system to display the diff of an edit to a function. There are no
maintenance reports helping the community find functions that need fixing,
translating, or expanding. Feel free to report things that you're missing,
even though we are aware of many of those. We hope to make future
improvements much more tangible and visible from now on. We also wrote a
tutorial on using the Wikifunctions Beta
<https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Beta_tutorial>, noting
several current caveats and workarounds.
*Report issues and ideas.* What to do if you run into an issue or if you
have an idea for an improvement: either file a task
<https://phabricator.wikimedia.org/maniphest/task/edit/form/43/?tags=WikiLam…>,
or check if there is a task and file one if not. You can also chat with us
on IRC #wikipedia-abstractconnect
<https://web.libera.chat/?channel=#wikipedia-abstract> or Telegram group
<https://t.me/Wikifunctions>, or on the Meta-wiki talk page
<https://meta.wikimedia.org/wiki/Talk:Abstract_Wikipedia>. We’ll try to
bring all tasks and ideas together on the Phabricator board
<https://phabricator.wikimedia.org/project/view/4876/>. We do not promise
to get to any of the bugs and ideas swiftly.
*Contribute code.* If you want to contribute to the code, that would be
more than welcome. There are so many clean-up tasks to be done, and now
with the Beta, these clean-ups and their effects should be much more
visible than before. Your code contributions will be very welcome. Either
find something you want to improve by using the Beta, or select one of our
tasks marked as fix-it
<https://phabricator.wikimedia.org/maniphest/?project=PHID-PROJ-2d3tcr43q2ip…>.
These should be comparably easy and self-contained tasks.
Workstream updates (as of August 5, 2022)
Performance:
- Continued analyzing and fixing the performance slowness
- Ali Assaf wrote a document describing generic vs pre-generic function
types
<https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Generic_function_type>.
We mentioned this document in the latest weekly newsletter and asked for
the community feedback
NLG:
- Finalized the design document
<https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Template_Language_for_Wi…>
describing
the template language
- Drafted initial proposal of the Abstract Representation
Meta-data:
- Refined CPU & memory reporting for Python & JavaScript execution
processes
- Resolved remaining error condition in tester pipeline
- Finished JavaScript code to deconstruct tester pipeline results and
completed wiring tester pipeline results into Vuex
Experience:
- Fixed Beta launch blocker when no arguments were shown on an
implementation
- Fixed wrong titles in non-function pages
- [DESIGN] Handed off Table component designs
The on-wiki version of this newsletter can be found here:
https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Updates/2022-08-05
--
Launch Plan
As we are getting closer to launching Wikifunctions, we want to share our
launch plan with you.
Launch will go through four major stages; we will detail below what each of
them means:
1. Beta: get the beta in a working mode and let folks know about it
2. Locked-down: deploy WikiLambda to wikifunctions.org but with no
general editing
3. Limited: switch on public visibility and limited function editing on
wikifunctions.org
4. General: move to full editing for wikifunctions.org, declare it
launched
Let’s take a look at each of the stages.
Beta
We aim to invite external people to the Beta as soon as possible. The
content on Beta may be deleted at any time. This is true for individual
pages as well as for the wiki as a whole. Also, this situation will not
change: we do not plan to keep content on Beta persistent or to put
particular effort into migrating it.
Beta is auto-deployed from "head", which means whenever we merge a change
to any of the components of Wikifunctions, it will be deployed to Beta
within a few minutes.
We are still working on ensuring that the major workflows are possible and
that a rough monitoring infrastructure is in place.
Locked-down deploy
We will set up wikifunctions.org as a wiki — an instance of WikiLambda on
the production Wikimedia systems. This is to find hidden surprises in
moving it from beta to production. This will use the standard Wikimedia
accounts ("SUL <https://meta.wikimedia.org/wiki/Help:Unified_login>") but
will allow only a very small group to make edits or call the API while we
check that everything is plugged together as we expect.
For this system, we will already be following our future deployment
mechanism, i.e. we cut releases and are part of the weekly Wikimedia-wide
deployment train <https://wikitech.wikimedia.org/wiki/Deployments/Train>,
etc. We will begin to prioritize fixing issues in production, figuring out
a playbook on how to do this (ideally, we will be following existing
Foundation playbooks on this).
Content may still be deleted any time.
Limited launch
We may reset the content at this point. We will copy over the Main Page and
other community content from Meta, where we previously collaborated with
the community on the Wikifunctions main page (we will start a discussion on
that, soon). We switch on general editing for all logged in users, and
Wikifunctions-specific editing to users with the appropriate rights. We
will have a configurable rate-limiting in place, which limits the number of
functions a single user can run. Calling approved functions will work for
both logged-in as well as logged-out users.
We will follow the deployment train, and so we need to have processes to
support three versions (head, current deployment, next deployment), while
developing like other Wikimedia teams do. We will have the appropriate
playbooks in place for that.
From this point on, content (functions, implementations, documentation)
will be expected to be persistent. We still might need to start over, but
we will try not to.
General launch
We will declare a general launch. We will have to follow our deployment
playbooks. We will have to keep content persistent.
At this point we will invite external communities and increase our outreach.
Please let us know if you have suggestions or questions around the launch
plan.
Representing functions
We have recently published the start of a discussion about generic function
types
<https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Generic_function_type>.
The initial function model had expected to change the type of a function
from just being a Function to include the signature of the function, i.e.
the type of a function such as “length of string” would not be just “
function” but instead “function from a string to a positive integer”. This
would have considerable advantages, particularly for higher-level
functions, i.e. functions using other functions as arguments.
However, our recent experience with Benjamin arrays
<https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Updates/2022-07-29> showed
us that changing the function model is a rather complex and subtle process
that will likely take a long time. We are not sure about whether the
benefit is worth the effort and delay, and what our approach here should
be. Ali Assaf wrote this document detailing three options
<https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Generic_function_type>,
and we invite you to share your thoughts.
Wikimania 2022
We will be presenting at Wikimania! Wikimania 2022 will be from August 11
to 14 and will be primarily virtual. You can learn more about the event and
register from the Wikimania 2022 page
<https://meta.wikimedia.org/wiki/Wikimania_2022>.
Some of our team will be attending portions of the Hackathon. If you'd like
to work on one of the #good_first_tasks
<https://phabricator.wikimedia.org/maniphest/query/_kVbtV8W6la8/#R> or
another task, we would be very thankful. Reach out to our team members, and
say hi! We’re happy to help.
Reminder: You can also see an overview of our session and related sessions from
last year's Wikimania
<https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Updates/2021-08-20>.
Workstream updates as of July 29, 2022
- July 25 – 29 was a ‘Fix-it’ week for the Abstract Wikipedia team.
During this week, the team paused the development of new features, and
focused on tasks related to technical debt.
- Interviews for the QTE role are in progress
- We continued on-boarding the Google fellows who have recently joined
the team
Hello all,
this is just a friendly reminder that our Developers-Volunteers'
Corner will take place at 17:30 UTC (that is, in 22 minutes from
now!).
The link for joining us on Google is meet.google.com/evj-ktbq-hbn
You can also ask your questions in our Telegram-IRC channel. :)
See you there!
--
Luca Martinelli [Sannita] (he/him)
Community Relations Specialist