Some of the Mobile Apps Team engineers met to discuss the scope of the MVP for a mobile apps content service.
Firstly, we enumerated our goals for this content service:
- Minimise number of requests the app has to make to generate a page - Reduce payload size of requests - Abstract away from of the API logic contained in the client into the server instead - Do some of the transforms that are currently done on the client on the server instead, where appropriate
We decided that the scope of the service should be feature parity; the service should be a drop-in replacement for our current API calls. More concretely, this means it should give us:
- Data from mobileview - Wikidata description - Revision ID of page - Wikibase ID - Image meta data for all images in the page - License - Description - Includes the URL for the high resolution version used for the lead image - Exclude unused HTML - Navboxes, etc. - Exclude hidden elements - CSS: display: none; unless if we display later - Related articles suggestions from full text search API
Things which we want to add, but are out of scope for the first iteration:
- Arbitrary Wikidata properties - Disambiguation (already "transcluded", following the chain of disambiguation pages and returning a JSON array of those links) - Page issues - Interwiki links - Table of contents (as a JSON array)
Open questions:
- Versioning? What if we add or remove a field from the response? We should version this. Can we do this easily?
Thanks, Dan
There are two main approaches to versioning y'all can take:
One is the method used by most MediaWiki API modules where if you want a new bit of info that wasn't in the default before, you have to ask for it... and you never remove anything from defaults for fear of breaking things.
The other is to use a nice linear versioning scheme, where you go to "/api/v1/" or "/api/v2/" etc and the version number controls the business logic of which elements get fetched and inserted into the response.
I tend to favor the latter as it's conceptually simpler and fits with the concept of versioning up the apps along with their associated service. (The former is more flexible but also means your requests get longer and longer as you add features.)
-- brion
On Thu, Feb 5, 2015 at 12:00 PM, Dan Garry dgarry@wikimedia.org wrote:
Some of the Mobile Apps Team engineers met to discuss the scope of the MVP for a mobile apps content service.
Firstly, we enumerated our goals for this content service:
- Minimise number of requests the app has to make to generate a page
- Reduce payload size of requests
- Abstract away from of the API logic contained in the client into the
server instead
- Do some of the transforms that are currently done on the client on
the server instead, where appropriate
We decided that the scope of the service should be feature parity; the service should be a drop-in replacement for our current API calls. More concretely, this means it should give us:
- Data from mobileview
- Wikidata description
- Revision ID of page
- Wikibase ID
- Image meta data for all images in the page
- License
lead image
- Description
- Includes the URL for the high resolution version used for the
- Exclude unused HTML
- Navboxes, etc.
- Exclude hidden elements
- CSS: display: none; unless if we display later
- Related articles suggestions from full text search API
Things which we want to add, but are out of scope for the first iteration:
- Arbitrary Wikidata properties
- Disambiguation (already "transcluded", following the chain of
disambiguation pages and returning a JSON array of those links)
- Page issues
- Interwiki links
- Table of contents (as a JSON array)
Open questions:
- Versioning? What if we add or remove a field from the response? We
should version this. Can we do this easily?
Thanks, Dan
-- Dan Garry Associate Product Manager, Mobile Apps Wikimedia Foundation
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
On Thu, Feb 5, 2015 at 12:04 PM, Brion Vibber bvibber@wikimedia.org wrote:
- Versioning? What if we add or remove a field from the response? We should version this. Can we do this easily?
In RESTBase we generally use mime type versioning like this:
application/json; profile=mediawiki.org/specs/app_page_bundle/1.0.0
This is specced in the Swagger spec & enforced by RESTBase. On version mismatch from stored content, the backend service is called to re-generate or upgrade the content, which is then saved back. It can be passed the stored content to make re-generation more efficient.
If needed, clients could also signal the content-type they expect with an accept header. The details of how things would then be upgraded / downgraded would need to be worked out in case you really need this. In general, you can go a long way by only adding properties while remaining backwards-compatible with the old properties.
We also distinguish stability per end point:
- *experimental *end points can change at any time (effectively private, use at your own risk) - *unstable *end points can change, but we make an effort to avoid breakage and notify users - any change to *stable* end points will increment the major API version (/v1/) following semver
The mobile API should probably be marked as experimental, as it is primarily intended as an API for apps we control.
Gabriel
In an effort to avoid yet another mega thread, I went ahead and created a Wiki page under the apps space which holds the problems & requirements for app services https://www.mediawiki.org/wiki/Wikimedia_Apps/App_Services*. There's even a discussion page https://www.mediawiki.org/wiki/Talk:Wikimedia_Apps/App_Services where I've cherry-picked Brion & Gabriel's comments, organized by topic. I hope this proves useful and will give us something to go on in the future. I've put my comments so far on the talk page.
On Thu, Feb 5, 2015 at 3:15 PM, Gabriel Wicke gwicke@wikimedia.org wrote:
On Thu, Feb 5, 2015 at 12:04 PM, Brion Vibber bvibber@wikimedia.org wrote:
- Versioning? What if we add or remove a field from the response? We should version this. Can we do this easily?
In RESTBase we generally use mime type versioning like this:
application/json; profile=mediawiki.org/specs/app_page_bundle/1.0.0
This is specced in the Swagger spec & enforced by RESTBase. On version mismatch from stored content, the backend service is called to re-generate or upgrade the content, which is then saved back. It can be passed the stored content to make re-generation more efficient.
If needed, clients could also signal the content-type they expect with an accept header. The details of how things would then be upgraded / downgraded would need to be worked out in case you really need this. In general, you can go a long way by only adding properties while remaining backwards-compatible with the old properties.
We also distinguish stability per end point:
- *experimental *end points can change at any time (effectively
private, use at your own risk)
- *unstable *end points can change, but we make an effort to avoid
breakage and notify users
- any change to *stable* end points will increment the major API
version (/v1/) following semver
The mobile API should probably be marked as experimental, as it is primarily intended as an API for apps we control.
Gabriel
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
I forgot to fill in my "*" next to "app services." The name is not a great one, and I'm open to suggestions that are more specific or intuitive.
On Thu, Feb 5, 2015 at 4:04 PM, Brian Gerstle bgerstle@wikimedia.org wrote:
In an effort to avoid yet another mega thread, I went ahead and created a Wiki page under the apps space which holds the problems & requirements for app services https://www.mediawiki.org/wiki/Wikimedia_Apps/App_Services*. There's even a discussion page https://www.mediawiki.org/wiki/Talk:Wikimedia_Apps/App_Services where I've cherry-picked Brion & Gabriel's comments, organized by topic. I hope this proves useful and will give us something to go on in the future. I've put my comments so far on the talk page.
On Thu, Feb 5, 2015 at 3:15 PM, Gabriel Wicke gwicke@wikimedia.org wrote:
On Thu, Feb 5, 2015 at 12:04 PM, Brion Vibber bvibber@wikimedia.org wrote:
- Versioning? What if we add or remove a field from the response? We should version this. Can we do this easily?
In RESTBase we generally use mime type versioning like this:
application/json; profile=mediawiki.org/specs/app_page_bundle/1.0.0
This is specced in the Swagger spec & enforced by RESTBase. On version mismatch from stored content, the backend service is called to re-generate or upgrade the content, which is then saved back. It can be passed the stored content to make re-generation more efficient.
If needed, clients could also signal the content-type they expect with an accept header. The details of how things would then be upgraded / downgraded would need to be worked out in case you really need this. In general, you can go a long way by only adding properties while remaining backwards-compatible with the old properties.
We also distinguish stability per end point:
- *experimental *end points can change at any time (effectively
private, use at your own risk)
- *unstable *end points can change, but we make an effort to avoid
breakage and notify users
- any change to *stable* end points will increment the major API
version (/v1/) following semver
The mobile API should probably be marked as experimental, as it is primarily intended as an API for apps we control.
Gabriel
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- EN Wikipedia user page: https://en.wikipedia.org/wiki/User:Brian.gerstle IRC: bgerstle
On Thu, Feb 5, 2015 at 12:00 PM, Dan Garry dgarry@wikimedia.org wrote:
- Image meta data for all images in the page
- License
lead image
- Description
- Includes the URL for the high resolution version used for the
Wouldn't that bloat the response size?
On 5 February 2015 at 13:20, Max Semenik maxsem.wiki@gmail.com wrote:
Wouldn't that bloat the response size?
The rationale for including this is because that way the image viewer will work even if you've not tapped on any of the images. This is pretty important functionality that doesn't work offline right now, even if you save the page for offline reading.
That said, we may want to punt this until after the MVP as being possibly outside of scope.
Dan
Saving a page for offline is not that speed sensitive, however, serving this information as part of normal page views would definitely impact overall latency.
On Thu, Feb 5, 2015 at 1:24 PM, Dan Garry dgarry@wikimedia.org wrote:
On 5 February 2015 at 13:20, Max Semenik maxsem.wiki@gmail.com wrote:
Wouldn't that bloat the response size?
The rationale for including this is because that way the image viewer will work even if you've not tapped on any of the images. This is pretty important functionality that doesn't work offline right now, even if you save the page for offline reading.
That said, we may want to punt this until after the MVP as being possibly outside of scope.
Dan
-- Dan Garry Associate Product Manager, Mobile Apps Wikimedia Foundation
Saving pages is not the primary use case for the image meta data portion - users tapping on an image and not having to wait for another server round trip to determine the high res url and meta data is the primary use case.
Having all this data as part of one request means image taps can be much lower latency from the user's perspective. The panel can pop up immediately with meta data overlay over the already downloaded low res image and can immediately begin lazy loading the high res version. It will seem much more responsive, which is really important.
As a side benefit, saving page data for offline access also becomes easier, but users tap on images (I think) more than they save pages.
On Feb 5, 2015, at 1:40 PM, Max Semenik maxsem.wiki@gmail.com wrote:
Saving a page for offline is not that speed sensitive, however, serving this information as part of normal page views would definitely impact overall latency.
On Thu, Feb 5, 2015 at 1:24 PM, Dan Garry dgarry@wikimedia.org wrote: On 5 February 2015 at 13:20, Max Semenik maxsem.wiki@gmail.com wrote:
Wouldn't that bloat the response size?
The rationale for including this is because that way the image viewer will work even if you've not tapped on any of the images. This is pretty important functionality that doesn't work offline right now, even if you save the page for offline reading.
That said, we may want to punt this until after the MVP as being possibly outside of scope.
Dan
-- Dan Garry Associate Product Manager, Mobile Apps Wikimedia Foundation
-- Best regards, Max Semenik ([[User:MaxSem]]) _______________________________________________ Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Max,
For regular page views we split it into two requests: one for lead section and another for the remaining sections. As a general rule of thumb we want to put only the information that is needed to display above the fold (section 0, lead image URL, Wikidata description, ...), plus maybe some minor metadata fields that help us with housekeeping.
The gallery data for should go into the second request.
As far as housekeeping goes, I think it would be great to have the revision ID and some hash for the things outside the pure pagedata (gallery items, ...) so we could determine after the first request if we could skip the second request because we already have that as a saved page or in a cache. Ideally, those would go into an ETag, and the client uses an If-None-Match header the next time we request it, so that in the case we already have the data there wouldn't be a payload in the response, just the header.
Bernd
On Thu, Feb 5, 2015 at 9:59 PM, Monte Hurd mhurd@wikimedia.org wrote:
Saving pages is not the primary use case for the image meta data portion - users tapping on an image and not having to wait for another server round trip to determine the high res url and meta data is the primary use case.
Having all this data as part of one request means image taps can be much lower latency from the user's perspective. The panel can pop up immediately with meta data overlay over the already downloaded low res image and can immediately begin lazy loading the high res version. It will seem much more responsive, which is really important.
As a side benefit, saving page data for offline access also becomes easier, but users tap on images (I think) more than they save pages.
On Feb 5, 2015, at 1:40 PM, Max Semenik maxsem.wiki@gmail.com wrote:
Saving a page for offline is not that speed sensitive, however, serving this information as part of normal page views would definitely impact overall latency.
On Thu, Feb 5, 2015 at 1:24 PM, Dan Garry dgarry@wikimedia.org wrote:
On 5 February 2015 at 13:20, Max Semenik maxsem.wiki@gmail.com wrote:
Wouldn't that bloat the response size?
The rationale for including this is because that way the image viewer will work even if you've not tapped on any of the images. This is pretty important functionality that doesn't work offline right now, even if you save the page for offline reading.
That said, we may want to punt this until after the MVP as being possibly outside of scope.
Dan
-- Dan Garry Associate Product Manager, Mobile Apps Wikimedia Foundation
-- Best regards, Max Semenik ([[User:MaxSem]])
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
I think it will be more than offset by the reduction in response size gained by the "Exclude unused HTML" bullet point. Also, gzip encoding is pretty good at text compression.
On Feb 5, 2015, at 1:20 PM, Max Semenik maxsem.wiki@gmail.com wrote:
On Thu, Feb 5, 2015 at 12:00 PM, Dan Garry dgarry@wikimedia.org wrote: Image meta data for all images in the page License Description Includes the URL for the high resolution version used for the lead image
Wouldn't that bloat the response size?
-- Best regards, Max Semenik ([[User:MaxSem]]) _______________________________________________ Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l