The on-wiki version of this newsletter can be found here:
https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Updates/2022-04-28
--
Once Wikifunctions launches, we currently plan to support implementations
of functions in two programming languages, Python and JavaScript. But
unfortunately, that doesn't mean that all the code out there, written in
Python or JavaScript, will become readily available to be copied and used
in Wikifunctions. The code has to fulfill certain requirements. In today's
newsletter we will discuss these requirements, and what you can do to
prepare code you want to make available through Wikifunctions.
First, it must be legal to bring the code in. As per the result of the
community discussion
<https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Updates/2021-12-21>,
the code will be published under the Apache-2 license. If you wrote the
code yourself, or otherwise own the rights to it, you are free to publish
it in Wikifunctions. If you are taking the code from an existing open
source project, you must make sure that it has a compatible license.
Second, the code must be “functional”. That means that, given a specific
input, the code must always return the same output. In particular, that
makes a number of classes of functions not available:
- The result must be determined by the input and have no random
component. This means we cannot have a function that simulates throwing
dice, that returns a newly minted GUID, or similar random results. A random
element would break our caching strategy, in particular the ability to
memoize any function call and replace it with its result if available.
- The result of the function cannot depend implicitly on the time of
day, or the current date, or the location of the user. This means we cannot
have a function that returns the current day of the week. If we want to
rely on such context, we have to make it explicit as an argument to the
function. A function such as “how high is the sun here right now?” would
need to be rephrased “how high is the sun at the given location and time?”
and take the location and time as arguments.
- A function call cannot have intentional side effects. There must be no
function calls that are expected to cause a specific change in the world,
e.g. a function call which instructs a robot to start a certain routine, or
that switches on a light. Yes, a function call will always have some effect
on the world (it may cause caches to change, it will use computing
resources, and it will turn some electricity into heat), but those are
incidental and may change in the future. Someone may write a system that
uses the results of Wikifunctions to control their robots or devices, but
the actual control would be implemented in that system, not in
Wikifunctions.
- A function cannot have a hidden state that is changeable by a function
call. This is a consequence of the previous point. This means, for example,
that we cannot have a function that keeps a count of how often it has been
called and returns that count.
- This also means there cannot be a function call that edits a Wikipedia
article or edits a Wikidata item. Editing a function, or a function
implementation, may eventually change the content of an article on
Wikipedia (that is, once we allow for Wikifunctions functions to be called
from articles in Wikipedia), but calling a function on Wikifunctions will
not cause an article content to change.
- A function may not call out to the Web or the wider Internet. No HTTP
requests or similar mechanisms will be allowed at launch. Any resources or
data a function wants to use must be provided as arguments.
- Initially, functions will not be able to access data from Wikimedia
projects. We plan to extend Wikifunctions in an early post-launch milestone
to allow access to items and lexemes in Wikidata, and later to meta-data
about media files on Commons.
- Functions cannot store or load files to a persistent file system, or
read from or write to a persistent database. They may not access any other
network or device.
Some of these use-cases may be tackled after launch (e.g. in order to allow
random results, or to use implicit arguments for functions such as “what is
the current time?”), but these will require careful planning, discussion,
and ultimately changes to the system.
These are further restrictions that Wikifunctions will initially have:
- Both Python and JavaScript have an extensive ecosystem of third party
libraries available. Initially it will only be possible to access the
Python standard library and the JavaScript standard built-in objects,
respectively. We plan to allow later for a process to add further libraries
and make them accessible from an implementation.
- It will initially not be possible to call another Wikifunctions
function from a code implementation (only from compositions). We plan to
allow that, although this might initially be restricted to cases where the
called function also has an implementation in the same programming language.
- Initially, only certain types will have built-in
serialization/deserialization logic (i.e., code that maps between the
ZObject representation and in-memory objects for each programming
language). These types are Boolean, string, List, Map (becomes a dict in
Python or a Map in JS), and Nothing (None in Python, null in JS). For every
other type, native code will work directly with the respective native JSON
object at first. We are working on a design to allow the community to add
serialization and deserialization for more types.
- This also means that there is no real support for 'objects' in the
sense of object-oriented languages. The interface with Wikifunctions will
be a function call, not a call to the method of an object that can rely on
internal state. There is also no information hiding in objects. Every value
in Wikifunctions must be entirely serializable and deserializable. Values
are immutable, which also is at odds with how objects are commonly designed
and used in practice in many object-oriented contexts.
- Initially, we won’t have a built-in mechanism to support dispatch or a
type hierarchy.
We are already planning to add more programming languages, but they will
also have similar restrictions. Both JavaScript and Python, as well as many
other languages, allow top-level functions to be defined. For other
languages, such as Java or Smalltalk, we would need to define a slightly
different pattern in order to interact with the functional interface that
Wikifunctions provides. Whenever we add a language, the process will
involve a design step in which we will discuss the appropriate mappings. We
also plan to document how further programming languages can be added so
that the effort becomes predictable.
This post has no examples and no how-to, but rather describes the
requirements for implementations. In the following weeks, we will follow up
with one or more posts that examine a few patterns and examples on how code
from libraries could be reused within Wikifunctions.
Thanks to Mahir256 <https://meta.wikimedia.org/wiki/User:Mahir256> for
providing comments on earlier drafts of this newsletter.
The on-wiki version of this newsletter can be found here:
https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Updates/2022-04-20
--
This week we are very happy to welcome Josh Minor to the wider team as our
Group Product Manager! As the team has grown, it is now the time to add a
dedicated Product Manager to ensure the launch and growth of Wikifunctions.
While we are hiring for this role, Josh will pitch in. His background in
linguistics and his years of experience with the Foundation make him the
perfect person for this role. But let’s have him introduce himself in his
own words.
Howdy!
I'm a 6.5 year veteran of the Wikimedia Foundation, working on Product
Management for the Wikipedia Apps, especially for iOS. I've also worked on
many other strategic projects and initiatives, including advocating for
making our content more structured across Wikimedia and easier for humans
and machines to access and understand. And before I moved into the consumer
app space, I got a BA in Linguistics and MA in Computational Linguistics,
and worked for many years in natural language technology and information
retrieval startups. So I've been following the Abstract Wikipedia concept
with keen interest from Denny's first proposal. Now I am helping him and
the team find a full time Senior Product Manager
<https://boards.greenhouse.io/wikimedia/jobs/4135630> who will help ensure
the successful launch of Wikifunctions and long term impact of Abstract
Wikipedia. In the meantime I'll be pitching in where I can and
collaborating on this amazing unique project. In my spare time I make art
and software and software art <https://imaginarycartography.com/>, walk
around San Francisco with my husband, and cuddle our cat.
I'm also a pretty nice boss. So if you know anyone who wants to play a key
role in the future of Abstract Wikipedia and knowledge equity and has
Product and relevant technical experience, please pass along our job posting
<https://boards.greenhouse.io/wikimedia/jobs/4135630>.
As Josh said, we are hiring for the role of a full time Senior Product
Manager for Wikifunctions. Please pass along this job posting to people who
might be interested, and if you are interested yourself, please apply. You
might also be interested in Josh's post about PMing in public
<https://medium.com/freely-sharing-the-sum-of-all-knowledge/pming-in-public-…>,
describing his experience with working for Wikimedia.
Please join me in welcoming Josh on the extended team!
It's been more and four days and you're getting an automatic confirmation
status (including admin). After the evaluation period, you can restore the
previous version of the deleted page from the previously deleted page and
return to (almost) the same page for other reasons: Long-term abuse, but
that is no longer in the discussion of deleting pages.
This is not an unblock request, but a request to recreate a deleted page.
You can nicely copy the article at https://nl.wikipedia.org/wiki/Oh_My_Girl.
So can you regenerate a page that has been deleted but no longer exists in
the delete page discussion?
The on-wiki version of this newsletter can be found here:
https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Updates/2022-04-12
--
We are very happy to share that Wikifunctions and Abstract Wikipedia will
be supported by a Google.org Fellowship. Let’s first introduce Google.org
and the Fellowship program in their own words:
*About Google.org* Google.org, Google's philanthropic arm, supports
nonprofits that address humanitarian issues and apply scalable, data-driven
innovation to solving the world's biggest challenges. They accelerate
progress by connecting nonprofits with a unique blend of support that
includes funding, products, and technical expertise from Google volunteers.
They engage with these believers-turned-doers who make a significant impact
on the communities they represent, and whose work has the potential to
produce meaningful change. Google.org wants a world that works for
everyone—and they believe technology and innovation can move the needle.
*Google.org Fellowship* The Google.org Fellowship enables Googlers to
complete up to 6 months of full-time pro bono technical work to accelerate
the social impact of nonprofits and civic entities. Fellows, including
engineers, product managers, UX researchers, and designers, roll up their
sleeves alongside the organization’s staff to help build open-source
solutions and equip the organization to maintain and implement these
solutions long after the Fellowship ends. Each year, 50+ Googlers provide
50,000+ hours of pro bono services assisting organizations to build
solutions for some of the world’s toughest challenges.
For nine months, up to ten Google.org Fellows will be supporting the
Abstract Wikipedia and Wikifunctions team. Most of their focus will be on
the backend of Wikifunctions: in other words, they will work towards making
the evaluation of functions far more efficient. This will enable
Wikifunctions to grow without immediately overwhelming the computing
resources we have available at the Wikimedia Foundation.
The Fellows will also work on a few other projects about which we will
share updates in the weeks and months to come. We will also publish a
detailed report in the end about the impact and the results of the work.
Several Fellows will start their work in April this month, and we’ll bring
on others to begin work in July.
As with all of our work, the work of the Fellows will be open source and
developed in the open, using our usual workflows and tools. The Fellows
will work under the direction of the Wikimedia Foundation, and will, during
the time of the Fellowship, interact with the community and others just
like every other employee or contractor.
The contributions of the Fellows are expected to allow us to launch sooner
and scale Wikifunctions much faster than we would otherwise be able to.
This also will allow us to focus on Abstract Wikipedia sooner than we had
expected over the last year, and thus ultimately speed up to deliver on our
mission of allowing more people to share knowledge in more languages.
In the following, we let the Fellows introduce themselves in their own
words.
*Ali Assaf, Software Engineer Fellow* (Apr.-Sep. 2022) *Location*: Zurich,
Switzerland Ali (he/him) has worked for more than 6 years as a software
engineer at Google Zurich, developing backend infrastructure for YouTube.
His work enables creators and partners to deliver content at scale. The cat
videos must flow! Before that, he was doing graduate studies in theoretical
computer science in Paris, on the subject of logic and computer-assisted
theorem proving. Although he completed his PhD, his quest for the ultimate
proof language is still incomplete. Ali grew up in Beirut, Lebanon, at the
intersection of cultures, languages, and beliefs. He is passionate about
education and access to information, and considers Wikipedia to be one of
the top 3 most important websites in the world.
*Ariel Gutman, Software Engineer Fellow* (May-Oct. 2022) *Location*:
Zurich, Switzerland Ariel (he/him) defended his PhD thesis at the
University of Konstanz in 2016, where he was researching Neo-Aramaic
dialects as an associate fellow of the Zukunftskolleg interdisciplinary
institute. His curriculum includes a master's degree in Linguistics awarded
by the Université Sorbonne Nouvelle and a master's degree in computer
Science awarded by the École Normale Supérieure, following a B.Sc. from the
Hebrew University in Jerusalem. He has conducted fieldwork on Neo-Aramaic
in France and Israel, as well as fieldwork on an Austronesian language in
West Papua, Indonesia. He has published numerous articles about Neo-Aramaic
and Language Acquisition. His first book (co-authored with Wido van
Peursen) entitled The Two Syriac Versions of the Prayer of Manasseh was
published by Gorgias Press in 2011. His second book, Attributive
Constructions in North-Eastern Neo-Aramaic, was published by Language
Science Press in 2018. Currently he is working as a software engineer
specialized in computational linguistics at Google, Zurich.
*Eunice Moon, Program Manager Fellow* (Apr.-Sep. 2022) *Location*: San
Francisco, CA, USA Eunice (she/her) is a Business Partner in Google's
Operational Effectiveness, responsible for contingent workforce strategy
and planning for Alphabet. Previously, Eunice was a Senior Manager at
Accenture, where she advised Fortune 500 companies on building data-driven
organizational change programs. Through Accenture Development Partnerships,
Eunice worked at the United Nations to develop a strategy to enable access
to family planning for 3 million women in the Philippines. As a consultant
at TechnoServe, she developed a strategy that improved the livelihoods of
smallholder farmers in Mozambique. Eunice holds an MBA from London Business
School and a BA in Economics from UC Berkeley.
*Mary Yang, Software Engineer Fellow* (Apr.-Sep. 2022) *Location*: Seattle,
WA, USA Mary (she/her) is a Software Engineer at Google Seattle for 2.5
years. At her home team in Ads, she primarily works on data pipelines. She
graduated college with a B.S. in Computer Science and a B.A. in Physics.
She enjoys learning and researching, and she dreams of going back to school
one day. With Chinese as her first language, Mary is passionate about the
mission of this project, making information available around the globe. In
her spare time, Mary likes baking, traveling, playing with ML models, and
her two cats.
*Olivia Zhang, Product Manager Fellow* (Apr.-Sep. 2022) *Location*: St
Louis, MO, USA Olivia (she/her) is a Google Cloud customer engineer,
focused on enabling & coaching enterprises in their infrastructure and
application modernization journey. Before Google, Olivia held roles in SaaS
solution architecture and technology consulting, driving large scale
technology transformations for customers across industry verticals. Olivia
has a Bachelor of Arts in Psychology from Washington University in St.
Louis, and an MBA from University of Missouri - Columbia.
*Ori Livneh, Software Engineer (TLM) Fellow* (Apr.-Sep. 2022) *Location*:
New York City, NY, USA Ori (he/him) is a Software Engineer at Google, where
he works on optimizing software to run efficiently on modern hardware. He
lives in New York City with his partner and their son. In his spare time,
he enjoys running and hacking on open-source software. He's been a
Wikipedian since 2005. Ori also worked for the Wikimedia Foundation as a
software engineer from 2012 to 2016.
*Sandy Woodruff, UX Designer Fellow* (Jul.-Dec. 2022) *Location*: San
Francisco, CA, USA Sandy (she/her) is a Bay Area-based interaction designer
with 9+ years of experience who strives to create experiences that are both
ethical and impactful. She has been at Google for 4.5 years on the Cloud AI
& Industry Solutions UX team designing products that make AI accessible to
users with limited machine learning experience. Outside of Google, Sandy
loves helping others break into the field of UX through career coaching.
Before that, she designed at Etsy, Rent the Runway, and Fab; advised for
Cornell Tech’s incubator and launched a startup that helped New York City
residents develop better recycling habits.
If you have further questions or comments, feel free to ask on wiki or on
the mailing list. We are very thankful to Google.org for offering us this
co-operative program. This relationship is not exclusive to them; if other
companies, universities, or other groups would be interested in making
similar donations in kind towards the Wikimedia movement, we would be
delighted to work with them, too. You can reach out to us at
partnerships(a)wikimedia.org.
Our work with the Fellows is part of an ongoing relationship
<https://meta.wikimedia.org/wiki/Overview_of_Wikimedia_Foundation_and_Google…>
between
the Foundation, Google, and Google.org. We’re happy to continue our work
with both entities to support a healthy web ecosystem, making knowledge
more accessible and representative of the diversity of the web’s users.
The on-wiki version is available here (and might be much easier to read):
https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Updates/2022-04-01
--
Today’s newsletter will be focusing on a technical detail, and you should
feel free to skip it if you don’t want to dive into the hairy details of
our data model.
Over the last few weeks we have repeatedly hit a blocker, and have not yet
resolved it, which is how to represent a typed list without the
representation becoming very long. A list in Wikifunctions is a type that
contains an unknown number of elements in a specific order.
In the Wikifunctions data model, a list is represented the same way as it
is in the programming language Lisp
<https://meta.wikimedia.org/w/index.php?title=Wp:en:Lisp_(programming_langua…>:
a list consists of two parts, a “head”, which is the first element of the
list, and a “tail”, which itself is a list and holds the rest of the list.
In addition, there is the empty list, which has no head or tail and
represents a list with zero elements. In Wikifunctions, the type for a list
is Z10/List.
So a list with two elements "a" and "b" would, represented in our usual
JSON notation, look like this:
{ "type": "List", "head": "a" "tail": { "type": "List",
"head": "b", "tail": { "type": "List" } }}
{
"Z1K1": "Z10",
"Z10K1": "a"
"Z10K2": {
"Z1K1": "Z10",
"Z10K1": "b",
"Z10K2": {
"Z1K1": "Z10"
}
}}
As this gets a bit long, we have introduced a shortcut by using JSON’s
array notation, and so the list can be represented as follows:
[ "a", "b" ]
The short notation (we call it canonicalized) can be easily and
mechanically translated into the longer notation seen above, and the other
way around.
In Phase η we introduced the possibility to have typed lists. Whereas in
the normal list we don’t say anything about the type of the elements of the
list, in a typed list we say “every element of the list must be of a
certain type”. Having typed lists allows us to have stronger guarantees on
typing: if you ask for an element of a typed list, you know what type the
result will be. This can be helpful with type checking and with providing
better interfaces.
A typed list is the result of a function call. We have a function,
Z881/Typed list, that takes a type as its sole argument, and returns a
type. The resulting type is a copy of Z10/List, with the difference that
instead of the head having a value of any type, it must have a value of the
type provided as the argument. And the tail is not of type Z10/List, but of
the type returned by calling Z881/Type list with the given argument type.
Let’s say we want to represent the list we had before, but instead of it
being a Z10/List it should be a list of strings. A list of strings is a
type which is created by calling the function Z881/Typed list on the type
Z6/String. This would look as follows in JSON notation.
{ "type": { "type": "Function call", "function": "Typed list",
"argument type": "String" }, "head": "a" "tail": { "type": {
"type": "Function call", "function": "Typed list",
"argument type": "String" }, "head": "b", "tail": {
"type": { "type": "Function call", "function": "Typed
list", "argument type": "String" } } }}
{
"Z1K1": {
"Z1K1": "Z7",
"Z7K1": "Z881",
"Z881K1": "Z6"
},
"K1": "a"
"K2": {
"Z1K1": {
"Z1K1": "Z7",
"Z7K1": "Z881",
"Z881K1": "Z6"
},
"Z10K1": "b",
"Z10K2": {
"Z1K1": {
"Z1K1": "Z7",
"Z7K1": "Z881",
"Z881K1": "Z6"
}
}
}}
The challenge is that we cannot have the same short notation for this, as
we would lose information.
We have tried to work around this by guessing the type: we look into the
list, and if everything has the same type, we declare it a typed list. But
first, it adds quite a bit of complexity to the code, and second it still
does not solve all issues: the short form of the above list could indeed be
a list of objects, or it could be a list of strings - there is no way for
us to know.
We are publishing our internal discussion and a few options
<https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Typed_lists>, and we
would like to hear your thoughts and gather some wider input.
The on-wiki version is available here:
https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Updates/2022-04-08
--
When we started the development effort towards the Wikifunctions site, we
sub-divided the work leading up to the launch of Wikifunctions into eleven
phases
<https://meta.wikimedia.org/wiki/Special:MyLanguage/Abstract_Wikipedia/Phases>,
named after the first eleven letters of the Greek alphabet.
- With Phase α (alpha) completed, it became possible to create instances
of the system-provided Types in the wiki.
- With Phase β (beta), it became possible to create new Types on-wiki
and to create instances of these Types.
- With Phase γ (gamma), all the main Types of the pre-generic function
model were available.
- With Phase δ (delta), it became possible to evaluate built-in
implementations.
- With Phase ε (epsilon), it became possible to evaluate
contributor-written implementations in any of our supported programming
languages.
- With Phase ζ (zeta), it became possible to evaluate implementations
composed of other functions.
- This week we declared Phase η (eta) as complete, in which we planned
to add support for generic types and functions.
We did that, but we also did so much more:
- We re-designed and re-built the error system entirely
- We moved from the preliminary function model
<https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Pre-generic_function_mod…>
to
the full function model
<https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Function_model>
- We implemented validators running on user-implemented types
- We designed the viewing and editing of functions
<https://design.wikimedia.org/blog/2022/03/14/wikifunctions.html>
- We moved the frontend to use Codex
<https://github.com/wikimedia/design-codex>, Wikimedia’s new front-end
toolkit
- We announced the logo
<https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Updates/2021-10-27> for
Wikifunctions
- We decided with the community on the licensing of Wikifunctions
<https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Updates/2021-12-21>
- We on-boarded quite a number of people since the start of the phase:
Cai
<https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Updates/2021-10-08>,
Adesoji
<https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Updates/2021-10-08>,
David
<https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Updates/2021-11-04>,
Julia
<https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Updates/2021-11-18>,
Mariya
<https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Updates/2021-12-16>,
and Luca
<https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Updates/2022-01-21>
We have learned a lot during this phase. Most notably, it took much longer
than anticipated: it took seven full months for this phase, whereas
previous phases took around two months. We held a retrospective on this,
and we identified a number of issues that we are aiming to considerably
improve. The scope creep, as witnessed by the number of things we have
accomplished, is one such issue. A lesson I certainly learned is the
real-world complexity of generic type processing, which presumably is why
so many programming languages only added support for generics later, and
did not have it from the start. And, particularly towards the end of the
phase, we were noticing an unsustainable and exhaustive working mode. We
will change our approach in the upcoming phase, by focusing on smaller,
more self-contained workstreams, and focus on these one by one.
Today, we have kicked off Phase θ. Originally called “thawing and
freezing”, the theme of this phase is to allow for the community and
technical processes on-wiki that allow the community to collaboratively
work on a library of functions in a stable and secure manner. This includes
deciding on and implementing relevant user-rights, features for
understanding edits done by others, having the system choose the right
implementation, and much more. The description in the phases page on Meta
<https://meta.wikimedia.org/wiki/Abstract_Wikipedia/Phases> still needs to
be updated, but here are the workstreams that we will work on in this phase:
- Decide and implement canonical form for typed lists
<https://meta.wikimedia.org/wiki/Special:MyLanguage/Abstract_Wikipedia/Updat…>
- Give users an intuitive user experience for functions by implementing
the designs for viewing and editing functions
- Allow the system to run function evaluations correctly and efficiently
by deciding which implementations to select
Provide users with the meta-data about individual function runs on the wiki
(e.g. how long it took, how many resources were used, etc.)
- Allow the system to run and evolve in a stable and secure manner, by
deciding on and implementing user rights, rate limiting and caches, and by
displaying edit diffs between revisions
- Ensure users have a comprehensible view experience for non-function
objects, as we redesign and reimplement texts with language fallbacks,
references, strings, and lists
- Help to start some related early community discussions about topics
such as user-group-rights, code of conduct, staff collaboration on
functions, and other new-wiki-preparation efforts
- Get ready for future phases, by designing for multilingual
documentation of objects and instrumenting the frontend
- Participate in the 2022 Wikimedia Hackathon
<https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2022>
You can see that even within the concept of stability and security, we have
a lot of things planned for this phase, but each of the workstreams are
much more self-contained than the big and somewhat open-ended goals of the
previous phase. This should also allow for more visibility into our
progress, and we will keep you updated in the weeklies on progress.
Once this phase is complete, we are getting very close to the finish line:
in Phase ι (iota) we plan to add multilingual documentation of objects.
Then, we have set Phase κ (kappa) for last minute clean up tasks, before
launching in Phase λ (lambda).
A huge thank you to the whole team, this is a major milestone. I am proud
of all the things achieved, and I am very excited to see the work in the
upcoming phase, which will be crucial to allow for Wikifunctions to not
just be a platform for running functions, but to allow it to grow as a
community.