Now that we have new shiny native apps and at the same time we're about to deprecate dynamic page views, it might be a good time to think for a while about the API module that powers it, action=mobileview.
Having been written using assumptions about workflow that didn't come true, it has accumulated some cruft and technical debt. I therefore propose to form a long-term vision. I'd like to hear your thoughts on how it can be changed, can its interface be simplified (e.g. [1]), how fast do we want it to perform, what do we expect from Parsoid migration.
--------- [1] https://bugzilla.wikimedia.org/show_bug.cgi?id=60485
Okay, it is so perfect that I don't need to care about it? :P
On Fri, Jul 25, 2014 at 4:26 PM, Max Semenik maxsem.wiki@gmail.com wrote:
Now that we have new shiny native apps and at the same time we're about to deprecate dynamic page views, it might be a good time to think for a while about the API module that powers it, action=mobileview.
Having been written using assumptions about workflow that didn't come true, it has accumulated some cruft and technical debt. I therefore propose to form a long-term vision. I'd like to hear your thoughts on how it can be changed, can its interface be simplified (e.g. [1]), how fast do we want it to perform, what do we expect from Parsoid migration.
[1] https://bugzilla.wikimedia.org/show_bug.cgi?id=60485
-- Best regards, Max Semenik ([[User:MaxSem]])
Max, it's been the weekend give me and others time to reply ;-) You can expect a mail from me soon :)
On Mon, Jul 28, 2014 at 1:30 PM, Max Semenik maxsem.wiki@gmail.com wrote:
Okay, it is so perfect that I don't need to care about it? :P
On Fri, Jul 25, 2014 at 4:26 PM, Max Semenik maxsem.wiki@gmail.com wrote:
Now that we have new shiny native apps and at the same time we're about to deprecate dynamic page views, it might be a good time to think for a while about the API module that powers it, action=mobileview.
Having been written using assumptions about workflow that didn't come true, it has accumulated some cruft and technical debt. I therefore propose to form a long-term vision. I'd like to hear your thoughts on how it can be changed, can its interface be simplified (e.g. [1]), how fast do we want it to perform, what do we expect from Parsoid migration.
[1] https://bugzilla.wikimedia.org/show_bug.cgi?id=60485
-- Best regards, Max Semenik ([[User:MaxSem]])
-- Best regards, Max Semenik ([[User:MaxSem]])
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
A few notes offhand:
On a slow connection pulling down multiple sections in one blob is tricky -- JSON decoders don't normally stream so we end up being pretty slow with that "second section and beyond" request.
I would love to be able to include action=query stuff along with a mobile view request, such as grabbing the current user and site metadata.
Handling reference popups is dependent on loading the refs section, which appears somewhere near the end. See above about slow connections. Could we preextract them and ship them with the first section as metadata?
There are some oddities with remote file pages not returning a mod timestamp.
Exposing CSS and scripting modules for extensions used would be nice. Alternately we can try to retool things intoself contained embeddable I frames.
-- brion On Jul 25, 2014 4:26 PM, "Max Semenik" maxsem.wiki@gmail.com wrote:
Now that we have new shiny native apps and at the same time we're about to deprecate dynamic page views, it might be a good time to think for a while about the API module that powers it, action=mobileview.
Having been written using assumptions about workflow that didn't come true, it has accumulated some cruft and technical debt. I therefore propose to form a long-term vision. I'd like to hear your thoughts on how it can be changed, can its interface be simplified (e.g. [1]), how fast do we want it to perform, what do we expect from Parsoid migration.
[1] https://bugzilla.wikimedia.org/show_bug.cgi?id=60485
-- Best regards, Max Semenik ([[User:MaxSem]])
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
Thanks, Brion!
On Tue, Jul 29, 2014 at 10:32 AM, Brion Vibber bvibber@wikimedia.org wrote:
On a slow connection pulling down multiple sections in one blob is tricky
-- JSON decoders don't normally stream so we end up being pretty slow with that "second section and beyond" request.
You still can request separate sections.
I would love to be able to include action=query stuff along with a mobile
view request, such as grabbing the current user and site metadata.
Sigh, without core support we can only do one-offs to return select parts in mobileview. Is there anything particular that you want? Also, it's not a browser environment and you can actually make a couple reqests in parallel to e.g. decouple siteinfo retrieval.
Handling reference popups is dependent on loading the refs section, which
appears somewhere near the end. See above about slow connections. Could we preextract them and ship them with the first section as metadata?
Already possible: §ions=0|references
There are some oddities with remote file pages not returning a mod
timestamp.
That's an interesting problem: the wiki itself doesn't know if a remote repository page has changed, so we have either to not cache the information about such pages (will be slow) or expect this information to be outdated.
Exposing CSS and scripting modules for extensions used would be nice.
Alternately we can try to retool things intoself contained embeddable I frames.
This is something worth investigating (as well as returning mw.config values related to page being retrieved), however I suspect that there will be a few wwonderful obstacles to work around, as a lot of extensions just add their modules/variables to OutpuPage in hooks scarily close to page display, making a lot of assumptions that are not true for API page views.
I think we should look at how our users are using the API? Is there a way to get a dump of the moment common requests? It would be great to separate these out into 3rd party usage and Wikimedia usage.
In terms of sensible defaults - what queries can we simplify? Personally I think the more data we return in a response the better. We should be minimising the number of HTTP requests.
I think how people use it is the best way to learn and improve.
On Tue, Jul 29, 2014 at 1:40 PM, Max Semenik maxsem.wiki@gmail.com wrote:
Thanks, Brion!
On Tue, Jul 29, 2014 at 10:32 AM, Brion Vibber bvibber@wikimedia.org wrote:
On a slow connection pulling down multiple sections in one blob is tricky -- JSON decoders don't normally stream so we end up being pretty slow with that "second section and beyond" request.
You still can request separate sections.
I would love to be able to include action=query stuff along with a mobile view request, such as grabbing the current user and site metadata.
Sigh, without core support we can only do one-offs to return select parts in mobileview. Is there anything particular that you want? Also, it's not a browser environment and you can actually make a couple reqests in parallel to e.g. decouple siteinfo retrieval.
Handling reference popups is dependent on loading the refs section, which appears somewhere near the end. See above about slow connections. Could we preextract them and ship them with the first section as metadata?
Already possible: §ions=0|references
There are some oddities with remote file pages not returning a mod timestamp.
That's an interesting problem: the wiki itself doesn't know if a remote repository page has changed, so we have either to not cache the information about such pages (will be slow) or expect this information to be outdated.
Exposing CSS and scripting modules for extensions used would be nice. Alternately we can try to retool things intoself contained embeddable I frames.
This is something worth investigating (as well as returning mw.config values related to page being retrieved), however I suspect that there will be a few wwonderful obstacles to work around, as a lot of extensions just add their modules/variables to OutpuPage in hooks scarily close to page display, making a lot of assumptions that are not true for API page views.
-- Best regards, Max Semenik ([[User:MaxSem]])
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
spelling correction: * the most /common/ requests
On Tue, Jul 29, 2014 at 2:48 PM, Jon Robson jdlrobson@gmail.com wrote:
I think we should look at how our users are using the API? Is there a way to get a dump of the moment common requests? It would be great to separate these out into 3rd party usage and Wikimedia usage.
In terms of sensible defaults - what queries can we simplify? Personally I think the more data we return in a response the better. We should be minimising the number of HTTP requests.
I think how people use it is the best way to learn and improve.
On Tue, Jul 29, 2014 at 1:40 PM, Max Semenik maxsem.wiki@gmail.com wrote:
Thanks, Brion!
On Tue, Jul 29, 2014 at 10:32 AM, Brion Vibber bvibber@wikimedia.org wrote:
On a slow connection pulling down multiple sections in one blob is tricky -- JSON decoders don't normally stream so we end up being pretty slow with that "second section and beyond" request.
You still can request separate sections.
I would love to be able to include action=query stuff along with a mobile view request, such as grabbing the current user and site metadata.
Sigh, without core support we can only do one-offs to return select parts in mobileview. Is there anything particular that you want? Also, it's not a browser environment and you can actually make a couple reqests in parallel to e.g. decouple siteinfo retrieval.
Handling reference popups is dependent on loading the refs section, which appears somewhere near the end. See above about slow connections. Could we preextract them and ship them with the first section as metadata?
Already possible: §ions=0|references
There are some oddities with remote file pages not returning a mod timestamp.
That's an interesting problem: the wiki itself doesn't know if a remote repository page has changed, so we have either to not cache the information about such pages (will be slow) or expect this information to be outdated.
Exposing CSS and scripting modules for extensions used would be nice. Alternately we can try to retool things intoself contained embeddable I frames.
This is something worth investigating (as well as returning mw.config values related to page being retrieved), however I suspect that there will be a few wwonderful obstacles to work around, as a lot of extensions just add their modules/variables to OutpuPage in hooks scarily close to page display, making a lot of assumptions that are not true for API page views.
-- Best regards, Max Semenik ([[User:MaxSem]])
Mobile-l mailing list Mobile-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mobile-l
-- Jon Robson
On Wed, Jul 30, 2014 at 3:18 AM, Jon Robson jdlrobson@gmail.com wrote:
I think we should look at how our users are using the API? Is there a way to get a dump of the moment common requests? It would be great to separate these out into 3rd party usage and Wikimedia usage.
API logs are kept in full for last 30 days, so someone with prod access should be able to get them for you.
In terms of sensible defaults - what queries can we simplify? Personally I think the more data we return in a response the better. We should be minimising the number of HTTP requests.
Also remember to be backwards compatible :)
I think how people use it is the best way to learn and improve.
Sure.. easily solved problem though with versioning :)
On Tue, Jul 29, 2014 at 2:51 PM, Yuvi Panda yuvipanda@gmail.com wrote:
On Wed, Jul 30, 2014 at 3:18 AM, Jon Robson jdlrobson@gmail.com wrote:
I think we should look at how our users are using the API? Is there a way to get a dump of the moment common requests? It would be great to separate these out into 3rd party usage and Wikimedia usage.
API logs are kept in full for last 30 days, so someone with prod access should be able to get them for you.
In terms of sensible defaults - what queries can we simplify? Personally I think the more data we return in a response the better. We should be minimising the number of HTTP requests.
Also remember to be backwards compatible :)
I think how people use it is the best way to learn and improve.
-- Yuvi Panda T http://yuvi.in/blog
Hi Max & Mobile folks!
On 07/26/2014 01:26 AM, Max Semenik wrote:
I'd like to hear your thoughts on how it can be changed, can its interface be simplified (e.g. [1]), how fast do we want it to perform, what do we expect from Parsoid migration.
The main benefit from using Parsoid HTML is that it enables you to more easily re-style or massage the content with a stable HTML spec [DOMSpec]. As an example, it is straightforward to identify transclusion expansions in Parsoid HTML, which lets you pull out this content or massage it to better display on mobile. The full HTML for each revision will be available as static HTML.
The other area of interest to you should be the upcoming storage service & REST API. This infrastructure will help to provide a low-latency REST interface backed by storage. First performance data shows a request latency in the single-digit ms area at request rates of around 2k req/s, using a single CPU thread. There will also be a mechanism to pre-generate custom content after an edit, so that a later request will typically just retrieve a stored blob.
On Tue, Jul 29, 2014 at 10:32 AM, Brion Vibber <bvibber@wikimedia.org mailto:bvibber@wikimedia.org> wrote:
On a slow connection pulling down multiple sections in one blob is tricky
-- JSON decoders don't normally stream so we end up being pretty slow with that "second section and beyond" request.
I guess it would make sense to benchmark different solutions, including loading the full HTML & letting it render progressively. Images etc could perhaps be disabled by default outside the first section, and then uncovered / re-enabled progressively by JS.
I would love to be able to include action=query stuff along with a mobile
view request, such as grabbing the current user and site metadata.
Sigh, without core support we can only do one-offs to return select parts in mobileview. Is there anything particular that you want? Also, it's not a browser environment and you can actually make a couple reqests in parallel to e.g. decouple siteinfo retrieval.
There will also be SPDY support in the foreseeable future, which will further cheapen individual requests. Ops currently has this scheduled for Q2, and I'll push for the REST API to enable it from the start. Yuvi also has been running Nginx with SPDY for the labs proxy for a long time now.
Exposing CSS and scripting modules for extensions used would be nice.
Alternately we can try to retool things intoself contained embeddable I frames.
This is something worth investigating (as well as returning mw.config values related to page being retrieved), however I suspect that there will be a few wwonderful obstacles to work around, as a lot of extensions just add their modules/variables to OutpuPage in hooks scarily close to page display, making a lot of assumptions that are not true for API page views.
This is somewhat related to https://bugzilla.wikimedia.org/show_bug.cgi?id=67540. We plan to expose this metadata through the REST API, so that clients can load the right modules for the page content.
Gabriel
[DOMSpec]: https://www.mediawiki.org/wiki/Parsoid/MediaWiki_DOM_spec