After a lot of work, we're ready to provide a more sensible data layout for format=json results (and also format=php). The changes are generally backwards-compatible for API clients, but extension developers might have some work to do. If your extension is maintained in Gerrit, much of the necessary conversion has already been done for you (the major exception being booleans that were violating the old API output conventions).
The general theme is that the ApiResult arrays now have more metadata, which is used to apply a backwards-compatible transformation for clients that need it and optional transformation so JSON output needn't be limited by restrictions of XML. At the same time, improvements were made to ApiResult and ApiFormatXml to hopefully make it easier for developers to use.
Relevant changes include:
- Several ApiResult methods were deprecated. If your extension is maintained in Gerrit, these should have already been taken care of for you (with the exception of T95168 https://phabricator.wikimedia.org/T95168 where work is ongoing), but new code will need to avoid the deprecated methods. - All ApiResult methods that operate on a passed-in array (rather than internal data) are now static, and static versions of all relevant data- and metadata-manipulation methods are provided. This should reduce the need for passing ApiResult instances around just to be able to set metadata. - Properties with names beginning with underscores are reserved for API metadata (following the lead of existing "_element" and "_subelements"), and will be stripped from output. Such properties may be marked as non-metadata using ApiResult::setPreserveKeysList(), if necessary. - PHP-arrays can now be tagged with "array types" to indicate whether they should be output as arrays or hashes. This is particularly useful to fix T12887 https://phabricator.wikimedia.org/T12887. - The "*" property is deprecated in favor of a properly-named property and special metadata to identify it for XML format and for back-transformation. Use ApiResult::setContentValue() instead of ApiResult::setContent() and all the details are handled for you. - ApiFormatXml will no longer throw an exception if you forget to call ApiResult::setIndexedTagName()! - ApiFormatXml will now reversibly mangle tag and attribute names that are not valid XML, instead of irreversibly mangling spaces and outputting invalid XML for other stuff. - ApiResult will now validate data added (e.g. adding resources or non-finite floats will throw an exception) and auto-convert objects. The ApiSerializable interface can be used to control object conversion, if __toString() or cast-to-array is inappropriate. - Actual booleans should now be added to ApiResult, and will be automatically converted to the old convention (empty-string for true and absent for false) when needed for backwards compatibility. Code that was violating the old convention will need to use the new ApiResult::META_BC_BOOLS metadata property to prevent this conversion. - Modules outputting as {"key":{"*":"value"}} to avoid large strings in XML attributes can now output as {"key":"value"} while still maintaining <container><key>value</key></container> in XML format, using ApiResult::META_BC_SUBELEMENTS. New code should use ApiResult::setSubelementsList() instead. - Modules outputting hashes as [{"name":"key1","*":"value1"},{"name":"key2","*":"value2"}] (due to the keys being invalid for XML) can now output as {"key1":"value1","key2":"value2"} in JSON while maintaining <container><item name="key1">value1</item><item name="key2">value2</item></container> in XML format, using array types "kvp" or "BCkvp".
I apologize for forgetting to announce this sooner. If developers need assistance with API issues or code review for API modules, please do reach out to me.
On Fri, May 1, 2015 at 7:53 AM, Brad Jorsch (Anomie) bjorsch@wikimedia.org wrote:
The general theme is that the ApiResult arrays now have more metadata, which is used to apply a backwards-compatible transformation for clients that need it
How does ApiResult know if a client needs this transformation? Do you mean clients requesting in formats other than json?
Is there anything API client developers should check in their code? It seems all the possible update work is in API module implementors.
Relevant changes include: ...
- Actual booleans should now be added to ApiResult, and will be
automatically converted to the old convention (empty-string for true and absent for false) when needed for backwards compatibility. Code that was violating the old convention will need to use the new ApiResult::META_BC_BOOLS metadata property to prevent this conversion.
Is this the cause of the EventLogging outage [1], [2]? It seems EventLogging was just naively returning a JSON schema as its results, could other API modules that return JSON structures containing binary values also break clients?
If developers need assistance with API issues or code review for API modules, please do reach out to me.
Let me know off-list how I can help update the docs. To start, I think this change description should go on the wiki.
[1] https://wikitech.wikimedia.org/wiki/Incident_documentation/20150428-EventLog... [2] https://gerrit.wikimedia.org/r/#/c/207297/
On Fri, May 1, 2015 at 4:41 PM, S Page spage@wikimedia.org wrote:
On Fri, May 1, 2015 at 7:53 AM, Brad Jorsch (Anomie) < bjorsch@wikimedia.org> wrote:
The general theme is that the ApiResult arrays now have more metadata, which is used to apply a backwards-compatible transformation for clients that need it
How does ApiResult know if a client needs this transformation?
If they don't use formatversion=2 to format=json or format=php, they get the BC transformation.
Is there anything API client developers should check in their code? It seems all the possible update work is in API module implementors.
https://lists.wikimedia.org/pipermail/mediawiki-api-announce/2015-May/000082...
[bools]
Is this the cause of the EventLogging outage [1], [2]?
Probably, if the linked Gerrit change fixed it.
It seems EventLogging was just naively returning a JSON schema as its results, could other API modules that return JSON structures containing binary values also break clients?
It's possible, yes.
Wow, this list of changes fixes a lot of things that I've hated about our API for a long time (no more star properties! no more setIndexedTagName errors! consistent JSON formatting!). I feel like it's Christmas! Thanks for all your hard work on this Brad! It is much appreciated.
Kaldari
On Fri, May 1, 2015 at 7:53 AM, Brad Jorsch (Anomie) bjorsch@wikimedia.org wrote:
After a lot of work, we're ready to provide a more sensible data layout for format=json results (and also format=php). The changes are generally backwards-compatible for API clients, but extension developers might have some work to do. If your extension is maintained in Gerrit, much of the necessary conversion has already been done for you (the major exception being booleans that were violating the old API output conventions).
The general theme is that the ApiResult arrays now have more metadata, which is used to apply a backwards-compatible transformation for clients that need it and optional transformation so JSON output needn't be limited by restrictions of XML. At the same time, improvements were made to ApiResult and ApiFormatXml to hopefully make it easier for developers to use.
Relevant changes include:
- Several ApiResult methods were deprecated. If your extension is
maintained in Gerrit, these should have already been taken care of for you (with the exception of T95168 <https://phabricator.wikimedia.org/T95168
where work is ongoing), but new code will need to avoid the deprecated methods.
- All ApiResult methods that operate on a passed-in array (rather than
internal data) are now static, and static versions of all relevant data- and metadata-manipulation methods are provided. This should reduce the need for passing ApiResult instances around just to be able to set metadata.
- Properties with names beginning with underscores are reserved for API
metadata (following the lead of existing "_element" and "_subelements"), and will be stripped from output. Such properties may be marked as non-metadata using ApiResult::setPreserveKeysList(), if necessary.
- PHP-arrays can now be tagged with "array types" to indicate whether
they should be output as arrays or hashes. This is particularly useful to fix T12887 https://phabricator.wikimedia.org/T12887.
- The "*" property is deprecated in favor of a properly-named property
and special metadata to identify it for XML format and for back-transformation. Use ApiResult::setContentValue() instead of ApiResult::setContent() and all the details are handled for you.
- ApiFormatXml will no longer throw an exception if you forget to call
ApiResult::setIndexedTagName()!
- ApiFormatXml will now reversibly mangle tag and attribute names that
are not valid XML, instead of irreversibly mangling spaces and outputting invalid XML for other stuff.
- ApiResult will now validate data added (e.g. adding resources or
non-finite floats will throw an exception) and auto-convert objects. The ApiSerializable interface can be used to control object conversion, if __toString() or cast-to-array is inappropriate.
- Actual booleans should now be added to ApiResult, and will be
automatically converted to the old convention (empty-string for true and absent for false) when needed for backwards compatibility. Code that was violating the old convention will need to use the new ApiResult::META_BC_BOOLS metadata property to prevent this conversion.
- Modules outputting as {"key":{"*":"value"}} to avoid large strings in
XML attributes can now output as {"key":"value"} while still maintaining <container><key>value</key></container> in XML format, using ApiResult::META_BC_SUBELEMENTS. New code should use ApiResult::setSubelementsList() instead.
- Modules outputting hashes as
[{"name":"key1","*":"value1"},{"name":"key2","*":"value2"}] (due to the keys being invalid for XML) can now output as {"key1":"value1","key2":"value2"} in JSON while maintaining <container><item name="key1">value1</item><item name="key2">value2</item></container> in XML format, using array types "kvp" or "BCkvp".
I apologize for forgetting to announce this sooner. If developers need assistance with API issues or code review for API modules, please do reach out to me.
-- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
When does this go live?
On Friday, May 1, 2015, Ryan Kaldari rkaldari@wikimedia.org wrote:
Wow, this list of changes fixes a lot of things that I've hated about our API for a long time (no more star properties! no more setIndexedTagName errors! consistent JSON formatting!). I feel like it's Christmas! Thanks for all your hard work on this Brad! It is much appreciated.
Kaldari
On Fri, May 1, 2015 at 7:53 AM, Brad Jorsch (Anomie) < bjorsch@wikimedia.org javascript:;> wrote:
After a lot of work, we're ready to provide a more sensible data layout
for
format=json results (and also format=php). The changes are generally backwards-compatible for API clients, but extension developers might have some work to do. If your extension is maintained in Gerrit, much of the necessary conversion has already been done for you (the major exception being booleans that were violating the old API output conventions).
The general theme is that the ApiResult arrays now have more metadata, which is used to apply a backwards-compatible transformation for clients that need it and optional transformation so JSON output needn't be
limited
by restrictions of XML. At the same time, improvements were made to ApiResult and ApiFormatXml to hopefully make it easier for developers to use.
Relevant changes include:
- Several ApiResult methods were deprecated. If your extension is
maintained in Gerrit, these should have already been taken care of for you (with the exception of T95168 <
https://phabricator.wikimedia.org/T95168
where work is ongoing), but new code will need to avoid the deprecated methods.
- All ApiResult methods that operate on a passed-in array (rather than
internal data) are now static, and static versions of all relevant
data-
and metadata-manipulation methods are provided. This should reduce the need for passing ApiResult instances around just to be able to set
metadata.
- Properties with names beginning with underscores are reserved for
API
metadata (following the lead of existing "_element" and
"_subelements"),
and will be stripped from output. Such properties may be marked as non-metadata using ApiResult::setPreserveKeysList(), if necessary.
- PHP-arrays can now be tagged with "array types" to indicate whether
they should be output as arrays or hashes. This is particularly useful to fix T12887 https://phabricator.wikimedia.org/T12887.
- The "*" property is deprecated in favor of a properly-named property
and special metadata to identify it for XML format and for back-transformation. Use ApiResult::setContentValue() instead of ApiResult::setContent() and all the details are handled for you.
- ApiFormatXml will no longer throw an exception if you forget to call
ApiResult::setIndexedTagName()!
- ApiFormatXml will now reversibly mangle tag and attribute names that
are not valid XML, instead of irreversibly mangling spaces and outputting invalid XML for other stuff.
- ApiResult will now validate data added (e.g. adding resources or
non-finite floats will throw an exception) and auto-convert objects.
The
ApiSerializable interface can be used to control object conversion, if __toString() or cast-to-array is inappropriate.
- Actual booleans should now be added to ApiResult, and will be
automatically converted to the old convention (empty-string for true
and
absent for false) when needed for backwards compatibility. Code that
was
violating the old convention will need to use the new ApiResult::META_BC_BOOLS metadata property to prevent this conversion.
- Modules outputting as {"key":{"*":"value"}} to avoid large strings
in
XML attributes can now output as {"key":"value"} while still
maintaining
<container><key>value</key></container> in XML format, using ApiResult::META_BC_SUBELEMENTS. New code should use ApiResult::setSubelementsList() instead.
- Modules outputting hashes as
[{"name":"key1","*":"value1"},{"name":"key2","*":"value2"}] (due to
the
keys being invalid for XML) can now output as {"key1":"value1","key2":"value2"} in JSON while maintaining <container><item name="key1">value1</item><item name="key2">value2</item></container>
in
XML format, using array types "kvp" or "BCkvp".
I apologize for forgetting to announce this sooner. If developers need assistance with API issues or code review for API modules, please do
reach
out to me.
-- Brad Jorsch (Anomie) Software Engineer Wikimedia Foundation _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org javascript:; https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org javascript:; https://lists.wikimedia.org/mailman/listinfo/wikitech-l
John wrote:
When does this go live?
My reading of https://phabricator.wikimedia.org/T76728 and https://gerrit.wikimedia.org/r/182858 is that this change is likely already live on Wikimedia wikis. It also looks like this change will be part of MediaWiki 1.25 and later.
MZMcBride
On Sat, May 2, 2015 at 10:39 PM, MZMcBride z@mzmcbride.com wrote:
My reading of https://phabricator.wikimedia.org/T76728 and https://gerrit.wikimedia.org/r/182858 is that this change is likely already live on Wikimedia wikis. It also looks like this change will be part of MediaWiki 1.25 and later.
Are you sure? https://gerrit.wikimedia.org/r/182858 was merged April 15.
formatversion is in 1.25 but RELEASE-NOTES-1.25 say it's experimental, they say * Many modules have changed result data formats. While this shouldn't affect clients not using the experimental formatversion=2015, code using ApiResult::getResultData() and not using ApiResult::transformForBC() may need updating.
Brad, is it OK to use formatversion=2 in extensions running on 1.25?
On Mon, May 4, 2015 at 4:51 AM, S Page spage@wikimedia.org wrote:
On Sat, May 2, 2015 at 10:39 PM, MZMcBride z@mzmcbride.com wrote:
My reading of https://phabricator.wikimedia.org/T76728 and https://gerrit.wikimedia.org/r/182858 is that this change is likely already live on Wikimedia wikis. It also looks like this change will be part of MediaWiki 1.25 and later.
Are you sure? https://gerrit.wikimedia.org/r/182858 was merged April 15.
Yes, it's currently live on all Wikimedia wikis and has been backported to 1.25.
Brad, is it OK to use formatversion=2 in extensions running on 1.25?
Yes, generally, although it might mean an extra change or two later if something it depends on gets improved. The "experimental" warning is because I don't want to have to announce breaking changes every time someone finds an instance of T12887 https://phabricator.wikimedia.org/T12887 that we can fix, or something that's still using a "*" key, or some other bit of output formatting that could be cleaned up.
wikitech-l@lists.wikimedia.org