So I was running into the problem of localizing the messages for the add_media_wizard & mv_embed & associated libraries. So I have taken a first pass at witting the script server (that I had previously described) http://tinyurl.com/ae44vd Below is a description of how it works. the code is in the svn http://tinyurl.com/darmme ...
I think it would be good to consider wider usage or at least turning it on sometime soon so I can deploy the add_media_wizard as a multi-lingual gadget. I am happy to volunteer to write a basic patch for putting it in the root. MediaWiki folder. (since its another entry point I imagine it should go in the root)
I have put a copy of this desc on http://www.mediawiki.org/wiki/ScriptLoader ... let me know if there is a better place to put that on mediaWiki
== Primary Motivation: == We need a script server to package language text into javascript. When a user invokes a interface component say the add media wizard we don't want to pull the entire interface code base and all the messages at once. Instead we just want to grab enough to display the current interface interaction. Once the user click on some tab say the 'archive.org search' we then want to import the code to run the archive.org search and the localized messages specific to that interface component. In other words we don't want to package all the message text in the html output of the initial page because in their are lots of possible interface interactions from any given point of invoking the user interface.
== Secondary Benefits == If we are ruining all the javascript though a php script might as well group requests, minify; gzip; use the local file cache system; send headers to cache forever on the squids, and we make each javascript request unique based on svn version and avoid re-validate requests. All these combined do speed things up a good deal on the first time a user loads page.
== How to use== In your Global settings you add to the global wgJSAutoloadClasses variable something like for every javascript class you want to load. $wgJSAutoloadClasses['mv_embed'] = '/local/path/to/my/javascript/mv_embed.js';
then at the top of mv_embed.js you would put something like: loadGM({ "msg_key":"Fall-back text if script server is disabled" ... } note the json variable that you pass to loadGM has to be valid JSON (not javascript) or php will complain.
Then in your message file you put the normal msg key pairs: "msg_key":"My localized text",
Right now we only support basic $1 replace ... obviously things get tricky with contextual wiki-text type replacements. (don't use that in js messages)
Then you can modify your head javascript include: with something like: if( $wgEnableScriptLoader ){ $unique_req_param = SVN_VERSION $debug_param = ($mvgJSDebug) ? '&debug=true' : ''; $wgOut->addScript(... mvwScriptLoader.php?" .
"class=all_the_class_i_need_separated_by_commas" . '&urid=' .$unique_req_param . $debug_param . ""></script>" );
Note the use of a few config vars...here are the config vars:
===Configuration vars=== $wgEnableScriptLoader = true; //if the script loader should be used at all $mvgJSDebug=false; // if set to true will send raw, fresh un-minified, localized javascript $wgEnableScriptLoaderJsFile = true; //if you can use the script loader with js files based on the root script folder ie: ?file=/skin/common.js (rather than only js classes looked up with $wgEnableScriptLoader) $wgEnableScriptMinify = true;
peace, michael.
On 2/20/09 12:32 PM, Michael Dale wrote:
So I was running into the problem of localizing the messages for the add_media_wizard& mv_embed& associated libraries. So I have taken a first pass at witting the script server (that I had previously described) http://tinyurl.com/ae44vd Below is a description of how it works. the code is in the svn http://tinyurl.com/darmme ...
I think it would be good to consider wider usage or at least turning it on sometime soon so I can deploy the add_media_wizard as a multi-lingual gadget. I am happy to volunteer to write a basic patch for putting it in the root. MediaWiki folder. (since its another entry point I imagine it should go in the root)
I have put a copy of this desc on http://www.mediawiki.org/wiki/ScriptLoader ... let me know if there is a better place to put that on mediaWiki
I'm getting some warm fuzzies from this... :D not sure I like every detail but I very much like the ideas of providing a standard interface for handling localized messages, and of providing a batch-load system so we don't have to fetch every little .js bit individually (hence reducing the HTTP request count, and thus the latency for getting things up and running).
== Primary Motivation: == We need a script server to package language text into javascript. When a user invokes a interface component say the add media wizard we don't want to pull the entire interface code base and all the messages at once. Instead we just want to grab enough to display the current interface interaction. Once the user click on some tab say the 'archive.org search' we then want to import the code to run the archive.org search and the localized messages specific to that interface component. In other words we don't want to package all the message text in the html output of the initial page because in their are lots of possible interface interactions from any given point of invoking the user interface.
On the other hand we don't want to delay those interactions; it's probably cheaper to load 15 messages in one chunk after showing the wizard rather than waiting until each tab click to load them 5 at a time.
But that can be up to the individual component how to arrange its loads...
== Secondary Benefits == If we are ruining all the javascript though a php script might as well group requests, minify; gzip; use the local file cache system; send headers to cache forever on the squids, and we make each javascript request unique based on svn version and avoid re-validate requests. All these combined do speed things up a good deal on the first time a user loads page.
*nod*
Much as minification feels "impure", it's probably a component we'll end up making use of for performance. And if we decide not to, it's easy to just keep it disabled. :)
In your Global settings you add to the global wgJSAutoloadClasses variable something like for every javascript class you want to load. $wgJSAutoloadClasses['mv_embed'] = '/local/path/to/my/javascript/mv_embed.js';
This I like!
then at the top of mv_embed.js you would put something like: loadGM({ "msg_key":"Fall-back text if script server is disabled" ... }
I'm a bit less certain about this; primarily it creates redundant copies of message text between the .js and the .i18n.php files, which is a maintenance problem.
Secondarily, it means you lose localization support if you disable the JS serving. Currently we do JS localization by doing the lookups in PHP and exporting those as JS variables or writing them directly into calls to JS functions in the HTML output. This kinda sucks because of the degree of manual intervention, but a more basic 'list out which vars you need and export them' should be straightforward and avoids the duplication.
There may also be difficulties with handling the distinction between message lookups which need to go to content language and those which need to be in the UI language... we also would rather not lose the ability to use language-specific features like {{PLURAL:}} and {{GENDER:}} which are needed for proper localization of many languages.
I'm not sure how best to handle those cases.
$wgEnableScriptMinify = true;
(Note for those who didn't peek at the code -- this is overridable at runtime with ?debug=true on the script load; we'd presumably want to be able to pass that on from web page views as well.)
-- brion
On Fri, Feb 20, 2009 at 5:51 PM, Brion Vibber brion@wikimedia.org wrote: [snip]
On the other hand we don't want to delay those interactions; it's probably cheaper to load 15 messages in one chunk after showing the wizard rather than waiting until each tab click to load them 5 at a time.
But that can be up to the individual component how to arrange its loads...
Right. It's important to keep in mind that in most cases the user is *latency bound*. That is to say that the RTT between them and the datacenter is the primary determining factor in the load time, not how much data is sent.
Latency determines the connection time, it also influences how quickly rwin can grow and get you out of slow-start. When you send more at once you'll also be sending more of it with a larger rwin.
So in terms of user experience you'll usually improve results by sending more data if doing so is able to save you a second request.
Even ignoring the users experience— connections aren't free. There is byte-overhead in establishing a connection. Byte-overhead in lost compression by working with smaller objects. Byte-overhead in having more partially filled IP packets. CPU overhead from processing more connections, etc.
Obviously there is a line to be drawn— You wouldn't improve performance by sending the whole of Wikipedia on the first request. But you will most likely not be conserving *anything* by avoiding sending another kilobyte of compressed user interface text for an application a user has already invoked, even if only a few percent use the additional messages.
... I was looking at commons upload form JavaScript load profile is like a long waterfall or rather a steep river :( http://metavid.org/promo/round_trips.png
Would be much nicer do 1 or 2 request instead of 27 ... true... a lot are gadgets and what not.. but even on a not logged in Main page visit on En we hit 6 scripts all loading one after another...
The patch to switch over to the script loader involves a lot of conditionals in a lot of skins... the majority of the scripts get piped through page serving so a fix to import & group js per wikiTitles would be added ... (and we would pass along their revision id in the url for caching support)... all the existing code would run "as is" where the scriptLoader config var is turned off. While I am touch all of those parts of the code I might as well pass the css loading to a loader/grouper as well.
if we are going this way... I will take a pass at making a patch since it would make other pieces I am working on a lot easier to integrate with localization.
comments in-line:
Brion Vibber wrote:
On the other hand we don't want to delay those interactions; it's probably cheaper to load 15 messages in one chunk after showing the wizard rather than waiting until each tab click to load them 5 at a time.
But that can be up to the individual component how to arrange its loads...
right ... I used the "tab" example since its a natural load point since we have to load the JSON search results for the requested query the interface already displays a little loading animation ... but as Mr. Maxwell points out given latency its better to front load a lot more ... that can easily be tweaked towards more front loading ... but if we have a dozen remote archive integrations maybe not so fun to load them all at initial run time or maybe you don't want to load the video player js code for people just interested in grabbing images...or in video sequences loading all the transitions and all the filters when only fade to black is used.
anyway .... the idea is it gives developers more flexibility in choosing what js classes they want to load at any point in time without inuring HTTP latency request costs for logical modularization of js code into separate files. Likewise we don't want to /have to/ front load all the msg text even if we choose to do that for /most/ cases for performance reasons.
I'm a bit less certain about this; primarily it creates redundant copies of message text between the .js and the .i18n.php files, which is a maintenance problem.
Secondarily, it means you lose localization support if you disable the JS serving. Currently we do JS localization by doing the lookups in PHP and exporting those as JS variables or writing them directly into calls to JS functions in the HTML output. This kinda sucks because of the degree of manual intervention, but a more basic 'list out which vars you need and export them' should be straightforward and avoids the duplication.
I would like to eventually take out the msg from the javascript files and only store a key list in js and the full Msgs only in the php. ... but for now will just have to deal with duplication... yes it makes things a bit bothersome... but retains portability and flexibility while we iron out and standardize the script loader. Once the script loader is solid it should be adaptable to being used in other CMSs so we can retain portability and localization and just require the script loader.
There may also be difficulties with handling the distinction between message lookups which need to go to content language and those which need to be in the UI language... we also would rather not lose the ability to use language-specific features like {{PLURAL:}} and {{GENDER:}} which are needed for proper localization of many languages.
yea these parts are a bit tricky... I imagine we can detect if we ever touch those magic words in the msg key list and then export some additional json keys for a script to use in cases of contextual swapping.... meaning our js will have to parse those magic words mirror their logic and then swap based on additional keys proved in the language packing.
(Note for those who didn't peek at the code -- this is overridable at runtime with ?debug=true on the script load; we'd presumably want to be able to pass that on from web page views as well.)
added debug var check from web page views in r47713 ...
I did not mention it last time but we also include js code to take advantage of the script loader system. It passes along any debug request or unique svn id, auto checks all the libraries as to not request anything that is already loaded, and groups the requests using the script loader. So once you have mv_embed lib in place you can "auto load" libraries with:
mvJsLoader.doLoad({ 'libname' : 'lib_location_in_case_script_loader_is_off', 'lib2' : 'lib2_location'}, function(){ //code to run now that the all the libs are ready });
Like the localization msg system this logic is hampered by the fact the script loader is /optional/. If we required the script loader it could be simpler:
requireOnce( {'list', 'of', 'libs', 'to', 'load'}, function(){ //code to run now that the libs are ready });
peace, --michael
On 2/23/09 11:35 AM, Michael Dale wrote: *snip*
Like the localization msg system this logic is hampered by the fact the script loader is /optional/. If we required the script loader it could be simpler:
*nod* I don't think it's really feasible to make use of the loader optional and still keep the flexibility. As long as processing like minification is optional, running through the bundling script should be fine.
-- brion
On Mon, Feb 23, 2009 at 2:35 PM, Michael Dale mdale@wikimedia.org wrote:
... I was looking at commons upload form JavaScript load profile is like a long waterfall or rather a steep river :( http://metavid.org/promo/round_trips.png
Note that this should be dramatically improved in next-gen browsers like Firefox 3.1, IE8, and whatever the next stable branch of WebKit is. All of these three can now load scripts in parallel, last I heard.
Yes performance does improve a great deal as the browsers pre-fetch in parallel. Its definitely a long overdue feature for browsers :) But it will still should be more efficient / faster to do one request instead of many in parallel. Even with parallel fetching the default in the Firefox nightlies its 5 scripts at a time. So you still end up doing a few round trips when you have a high script count.
--michael
Aryeh Gregor wrote:
On Mon, Feb 23, 2009 at 2:35 PM, Michael Dale mdale@wikimedia.org wrote:
... I was looking at commons upload form JavaScript load profile is like a long waterfall or rather a steep river :( http://metavid.org/promo/round_trips.png
Note that this should be dramatically improved in next-gen browsers like Firefox 3.1, IE8, and whatever the next stable branch of WebKit is. All of these three can now load scripts in parallel, last I heard.
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
wikitech-l@lists.wikimedia.org