---------- Forwarded message ---------
From: Adam Baso <abaso(a)wikimedia.org>
Date: Wed, Mar 22, 2023 at 4:45 AM
Subject: Service Decommission Notice: Mobile Content Service - July 2023
To: Wikimedia developers <wikitech-l(a)lists.wikimedia.org>
TL;DR: The legacy Mobile Content Service is going away in July 2023. Please
switch to Parsoid or another API before then to ensure service continuity.
Hello World,
I'm writing about a service decommission we hope to complete mid-July 2023.
The service to be decommissioned is the legacy Mobile Content Service
("MCS"), which is maintained by the Wikimedia Foundation's Content
Transform Team. We will be marking this service as deprecated soon.
We hope that with this notice, people will have ample time to update their
systems for use of other endpoints such as Parsoid [1] (n.b., MCS uses
Parsoid HTML).
The MCS endpoints are the ones with the relative URL path pattern
/page/mobile-sections* on the Wikipedias. For examples of the URLs see the
"Mobile" section on the online Swagger (OpenAPI) specification
documentation with matching URLs here:
https://en.wikipedia.org/api/rest_v1/#/Mobile
== History ==
The Mobile Content Service ("MCS") is the historical aggregate service that
originally provided support for the article reading experience on the
Wikipedia for Android native app, as well as some other experiences. We
have noticed that there are other users of the service. We are not able to
determine all of the users, as it's hard to tell with confidence from the
web logs.
The Wikimedia Foundation had already transitioned the Wikipedia for
Android and iOS apps to the newer Page Content Service ("PCS") several
years ago. PCS has some similarities with MCS in terms of its mobility
focus, but it also has different request-response signatures in practice.
PCS, as with MCS, is intended to primarily satisfy Wikimedia
Foundation-maintained user experiences only, and so this is classified with
the "unstable" moniker.
== Looking ahead ==
Generally, as noted in the lead, we recommend that folks who use MCS (or
PCS, for that matter) switch over to Parsoid for accessing Wikipedia
article content programmatically for the most predictable service.
The HTML produced by Parsoid has a versioned specification [2] and because
Parsoid is accessed regularly by a number of components across the globe
tends to have fairly well cached responses. However, please note that
Parsoid may be subject to stricter rate limits that can apply under certain
traffic patterns.
At this point, I do also want to note that in order to keep up with
contemporary HTML standards, particularly those favoring accessibility and
machine readability enhancements, Parsoid HTML will undergo change as we
further converge parsing stacks [3]. Generally, you should expect iteration
on the Parsoid HTML spec, and of course as you may have come to appreciate
that the shape of HTML in practice can vary nontrivially wiki-by-wiki as
practices across wikis vary.
You may also want to consider Wikimedia Enterprise API options, which range
from no cost to higher volume access paid options.
https://meta.wikimedia.org/wiki/Wikimedia_Enterprise#Access
== Forking okay, but not recommended ==
Because MCS acts as a service aggregate and makes multiple backend API
calls, caveats can apply for those subresources - possibility of API
changes, deprecation, and the like. We do not recommend a plain fork of MCS
code because of the subresource fetch behavior. This said, of course you
are welcome to fork in a way compatible with MCS's license.
== Help spread the word ==
Although we are aware of the top two remaining consumers of MCS, we also
are not sure who else is accessing MCS and anticipate that some downstream
tech may break when MCS is turned off. As we are cross-posting this
message, we hope most people who have come to rely upon MCS will see this
message. Please feel free to forward this message to contacts if you know
they are using MCS.
== Help ==
Although we intend to decommission MCS in July 2023, we would like to share
resources if you need some help. We plan to hold office hours in case you
would like to meet with us to discuss this or other Content Transform Team
matters. We will host these events on Google Meet. We will provide notice
of these office hours on the wikitech-l mailing list in the coming weeks
and months.
Additionally, if you would like to discuss your MCS transition plans,
please visit the Content Transform Team talk page:
https://www.mediawiki.org/wiki/Talk:Content_Transform_Team
Finally, some Content Transform Team members will also be at the Wikimedia
Hackathon [4] if you would like some in-person support.
Thank you.
Adam Baso (he/him/his/Adam), on behalf of the Content Transform Team
Director of Engineering
Wikimedia Foundation
[1] https://www.mediawiki.org/wiki/Parsoid
[2] https://www.mediawiki.org/wiki/Specs/HTML
[3] https://www.mediawiki.org/wiki/Parsoid/Parser_Unification/Updates
[4] https://www.mediawiki.org/wiki/Wikimedia_Hackathon_2023
Yesterday, the MobileView Action API was removed from all Wikimedia
production servers [1].
The API was originally built to service our apps but over time that has
been replaced with the Page Content Service [2]. The original service was
not being maintained, and usage was very low. If you were using the
MobileView API we urge you to check this API out as it is a more powerful,
better cached alternative.
We marked the API as hard deprecated in December 2019 [3], after which it
was marked in ApiSandbox and in the response as deprecated, and support has
been dwindling since then, for example, the noimages parameter was dropped
in September 2020 [4].
In March 2022 we removed the remaining production blocker for removing the
code: language variant support [5]. The Page content service was
prreviously using the mobile view API for language variant views but has
now been rewired to use the better supported core action=parse API. If you
need to support language variant views and previously couldn't because of
the lack of support in the page content service this should no longer be an
obstacle..
Since April 2022 users of the API would have been seeing an inline banning
warning them of the upcoming breakage [6]. We got no feedback from these
banners so were comfortable with pushing forward.
Impact on user scripts was judged as low [7] with only 14 scripts impacted
so this has not been announced on tech news.
If you are impacted by this change, I apologise that previous
communications have failed you and I'd love to hear from you about how we
could have done this deprecation better. If I can support you in any way in
making sure your apps/gadgets/scripts are working again please feel free to
reply to this email either privately or publicly or to raise a topic on the
MobileFrontend talk page [8].
Finally, thanks to the engineers who helped make this deprecation possible.
Jon
[1] https://phabricator.wikimedia.org/T186627
[2] https://www.mediawiki.org/wiki/Page_Content_Service
[3] https://phabricator.wikimedia.org/T210808
[4] https://phabricator.wikimedia.org/T262580,
[5] https://phabricator.wikimedia.org/T236733
[6] https://phabricator.wikimedia.org/T286836
[7]
https://global-search.toolforge.org/?q=%5B%27%22%5C%3D%5Dmobileview®ex=1…
[8]
https://www.mediawiki.org/wiki/Extension_talk:MobileFrontend?tableofcontent…
This message is a notice about forthcoming removal of a response field in
the REST API’s /page/summary endpoint. [1] This message was posted to
wikitech-l and is being cross-posted on mediawiki-api-announce.
The endpoint is used in the Page Previews (“hovercards”) functionality on
the classic web (desktop) and Android & iOS experiences for Wikipedia, in
addition to numerous external experiences. We don't anticipate impacts on
the mainline Wikipedia experiences from this forthcoming field removal, as
the endpoint will still be operational.
The REST API's /page/summary endpoint provides an api_urls field in its
response with links to several other REST API endpoints supported by the
Page Content Service.
Three of the api_urls subfields refer to experimental endpoints that have
been removed from the REST API altogether in favor of newer API endpoints.
1. media (/page/media)
2. references (/page/references)
3. metadata (/page/metadata).
api_urls also contains subfields referring to stable endpoints that
continue to exist in the REST API:
1. summary (which is self referential)
2. edit_html (/page/html - the Parsoid HTML)
3. talk_html (/page/html/Talk:<title> - the Parsoid HTML for the
corresponding Talk page)
Wikimedia’s Product Infrastructure team intends to remove the api_urls
field of the /page/summary response.
A cursory review at https://codesearch.wmflabs.org/ and Wikimedia Git
mirrors suggests api_urls isn’t in use in consuming code.
Review of web logs suggests traffic for the following endpoints:
- The media endpoint is at about 5% of its original traffic before its
decommission and it appears to be from old Wikipedia for Android clients.
This is expected.
- The references endpoint’s Wikipedia for Android traffic does not seem
present and its other traffic appears to be from non-user application
software based on User-Agent header components.
-The metadata endpoint’s traffic seems to have all but stopped.
This change is being announced in advance of the change because the
endpoint is advertised as stable. [2]
Please update your clients if you rely on the presence of the api_urls
field. If this change poses a problem for your clients, please do let us
know as soon as possible at the tracking task:
https://phabricator.wikimedia.org/T247991
We plan to remove the api_urls field described here on or after 14 July
2020.
Thank you.
Adam Baso
Director of Engineering
Wikimedia Foundation
[1]
https://en.wikipedia.org/api/rest_v1/#/Page%20content/get_page_summary__tit…
[2] Refer to
https://www.mediawiki.org/wiki/API_versioning#End_point_stability and
https://www.mediawiki.org/wiki/Wikimedia_Product/Wikimedia_Product_Infrastr…
for more information on stability designations.
Since April 2010,[1] when no lgtoken is passed to the Action API
action=login it will return a "NeedToken" response including the token to
use. While this method of fetching the login token was deprecated in
January 2016,[2] it is still present for the benefit of clients that have
not yet been updated and is not (yet) being removed.
The NeedToken response was also being returned when an lgtoken was supplied
but could not be validated due to session loss. While this made sense back
in 2010 when the NeedToken response was the only way to fetch the login
token, these days it is mainly confusing[3] and a way for clients with
broken cookie handling to wind up in a loop.
With the merge of Gerrit change 586448,[4] the API will no longer return
NeedToken when lgtoken was supplied. If the token cannot be validated due
to session loss, a "Failed" response will be returned with a message
referring to session loss as the problem.
This change should be deployed to Wikimedia sites with 1.35.0-wmf.28 or
later, see https://www.mediawiki.org/wiki/MediaWiki_1.35/Roadmap for a
schedule.
Note that the change HAS NOT been deployed to Wikimedia sites as of the
time of this email. If your client's ability to log in broke on 6 April
2020, the cause is most likely an unrelated change to Wikimedia's
infrastructure that caused some HTTP headers to be output with HTTP/2
standard casing, i.e. "set-cookie" rather than the traditional
"Set-Cookie". See https://phabricator.wikimedia.org/T249680 for details and
further discussion of that situation.
[1]: https://www.mediawiki.org/wiki/Special:Code/MediaWiki/64677
[2]:
https://lists.wikimedia.org/pipermail/mediawiki-api-announce/2016-January/0…
[3]: https://phabricator.wikimedia.org/T249526
[4]: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/586448
--
Brad Jorsch (Anomie)
Senior Software Engineer
Wikimedia Foundation
Various unusual values for integer-type parameters to the Action API will
no longer be accepted. Acceptable values will consist of an optional sign
(ASCII `+` or `-`) followed by 1 or more ASCII digits.
Values that were formerly allowed, and will now result in a "badinteger"
error, include:
- Values with extraneous whitespace, such as " 1".
- "1.9", formerly interpreted as "1".
- "1e1", formerly interpreted as either "1" or "10" at various times.
- "1foobar", formerly interpreted as "1"
- "foobar", formerly interpreted as "0".
Most clients should already be using correct formats, unless they are
taking direct user input without validation.
This change will most likely go out to Wikimedia wikis with 1.35.0-wmf.19.
See https://www.mediawiki.org/wiki/MediaWiki_1.35/Roadmap for a schedule.
--
Brad Jorsch (Anomie)
Senior Software Engineer
Wikimedia Foundation
The error codes that may be changing are some of those representing invalid
values for API parameters. Notably, the following will change:
- "noX", indicating that a required parameter was not specified, becomes
"missingparam".
- "unknown_X", indicating that an unrecognized value was specified for
an enumerated-value parameter, becomes "badvalue".
- "too-many-X", indicating that too many values were supplied to a
multi-valued parameter, becomes "toomanyvalues".
- "baduser_X", "badtimestamp_X", and so on become "baduser",
"badtimestamp", and so on.
Note this is not a comprehensive list, other codes may be changing as well.
These changes make the error codes more predictable, at the expense of not
indicating in the code which parameter specifically triggered the error. If
you have a use case where knowing which parameter triggered the error is
needed, please let us know (by replying to this message or by filing a
request in Phabricator) and we'll add the necessary error metadata.
The human-readable text is also changing for some of these errors (with or
without error code changes), and for a few the error metadata may be
changing (e.g. "botMax" changes to "highmax" for limit-type warnings in
non-back-compat error formats).
This change will most likely go out to Wikimedia wikis with 1.35.0-wmf.19.
See https://www.mediawiki.org/wiki/MediaWiki_1.35/Roadmap for a schedule.
--
Brad Jorsch (Anomie)
Senior Software Engineer
Wikimedia Foundation
Hi all,
action=helppanelquestionposter is an API meant for internal use by the
GrowthExperiments extension's web UI, but it has not been formally marked
as internal, so I'm announcing the recent changes [1] to it just in case:
- the 'email' and 'resendconfirmation' parameters have been removed;
- the 'email' response field has been removed;
- the API is now marked as internal.
The functionality provided by the API is not really useful outside of the
web-based workflow it supports, and can be approximated with other, public
APIs, so this is not expected to cause problems to any clients.
[1]
https://gerrit.wikimedia.org/r/c/mediawiki/extensions/GrowthExperiments/+/5…
The format of block expiry timestamps returned from list=users and
list=allusers has long been inconsistent. It was being returned in an
internal format (e.g. "20190918201053"), rather than the ISO 8601 format
used by the rest of the API (e.g. "2019-09-18T20:10:53Z").
The 'blockexpiry' property from these two modules will be changing to the
standard ISO 8601 format with 1.34.0-wmf.24. See
https://www.mediawiki.org/wiki/MediaWiki_1.34/Roadmap for a schedule of
deployment to Wikimedia wikis.
This change also brings the block information returned by list=users and
list=allusers fully in line with that already used for meta=userinfo and
'blocked' errors from various actions.
--
Brad Jorsch (Anomie)
Senior Software Engineer
Wikimedia Foundation
When saving an edit is prevented by the AbuseFilter or SpamBlacklist
extensions, the error is currently reported as a successful API response
with a 'failure' code in the body.[1][2]
In the future, these will be reported as standard API errors.[3][4]
This change should be deployed to Wikimedia wikis with 1.34.0-wmf.23. See
https://www.mediawiki.org/wiki/MediaWiki_1.34/Roadmap for a schedule.
Clients that do not need to specially handle failures due to AbuseFilter or
SpamBlacklist will likely need no changes, as they probably already include
code to generically handle API error responses.
Clients that do handle AbuseFilter or SpamBlacklist failures specially will
need to be updated to check for error codes 'abusefilter-warning',
'abusefilter-disallowed', and/or 'spamblacklist' and handle them as they do
the current AbuseFilter and SpamBlacklist failures, if they want to
preserve their current special handling.
Note that edit failures due to CAPTCHAs from ConfirmEdit are not being
changed at this time. They will continue to be reported as before.[5]
[1]: AbuseFilter: https://phabricator.wikimedia.org/P8988
[2]: SpamBlacklist: https://phabricator.wikimedia.org/P8990
[3]: AbuseFilter: https://phabricator.wikimedia.org/P8989
[4]: SpamBlacklist: https://phabricator.wikimedia.org/P8991
[5]: ConfirmEdit: https://phabricator.wikimedia.org/P9076
--
Brad Jorsch (Anomie)
Senior Software Engineer
Wikimedia Foundation