There seems to be some confusion about how ResourceLoader works, which has been leading people to make commits like r73196 and report bugs like #25362. I would like to offer some clarification.
ResourceLoader, if you aren't already aware, is a new system in MediaWiki 1.17 which allows developers to bundle collections of *resources* (like JavaScript and CSS files, or localized messages) into *modules*. Modules may represent any number of scripts, styles and messages, which are read from the file system, the database, or generated by software.
When a request is made for one or more modules, each resource is packaged together and sent back to the client as a response. The way in which these requests and responses are performed depends on whether debug is on or off.
When debug mode is off:
* Modules are requested in batches * Resources are combined into modules * Modules are combined into a response * The response is minified
When debug mode is on:
* Modules are requested individually * Resources are combined into modules
I think it's debatable whether debug=true mode goes far enough, since it still combines resources into modules, and I am open to contributions that can make debug=true mode even more debugging friendly by delivering the resources to the client as unchanged as possible. I also think it's debatable if debug=false mode goes far enough, since things like Google Closure Compiler have been proven to even further reduce the size of JavaScript resources, so I am also open to contributions which can make debug=false even more production friendly by improving front-end performance.
The commits and bugs that I'm contending here are ones which are aiming to dilute the optimized nature of debug=false mode, when debug=true mode is really what they should be using or improving. These kinds of changes and suggestions result in software that is neither optimized for debugging or for production, making the front-end performance of the site in production slower without making it any easier to debug than it would have been by using debug=true.
If you are a developer, working on your localhost, you probably want to code with...
$wgResourceLoaderDebug = true;
.. and then test that things work in debug=false mode before committing your code. This will result in more requests but less processing, which will be much faster when developing on localhost.
I hope this helps clarify this situation.
- Trevor
Trevor Parscal wrote:
There seems to be some confusion about how ResourceLoader works, which has been leading people to make commits like r73196 and report bugs like #25362. I would like to offer some clarification.
ResourceLoader, if you aren't already aware, is a new system in MediaWiki 1.17 which allows developers to bundle collections of *resources* (like JavaScript and CSS files, or localized messages) into *modules*. Modules may represent any number of scripts, styles and messages, which are read from the file system, the database, or generated by software.
When a request is made for one or more modules, each resource is packaged together and sent back to the client as a response. The way in which these requests and responses are performed depends on whether debug is on or off.
When debug mode is off:
* Modules are requested in batches * Resources are combined into modules * Modules are combined into a response * The response is minifiedWhen debug mode is on:
* Modules are requested individually * Resources are combined into modulesI think it's debatable whether debug=true mode goes far enough, since it still combines resources into modules, and I am open to contributions that can make debug=true mode even more debugging friendly by delivering the resources to the client as unchanged as possible. I also think it's debatable if debug=false mode goes far enough, since things like Google Closure Compiler have been proven to even further reduce the size of JavaScript resources, so I am also open to contributions which can make debug=false even more production friendly by improving front-end performance.
I might be a little tired but are you saying that debug mode does not preserve original line numbers? So this is debug mode as in "a bit easier to debug in Firebug then fully minified code"?
And I still don't see how minification options are evil. One "if" won't really change performance.
Regards, Nux.
Maciej Jaros wrote:
I might be a little tired but are you saying that debug mode does not preserve original line numbers? So this is debug mode as in "a bit easier to debug in Firebug then fully minified code"?
And I still don't see how minification options are evil. One "if" won't really change performance.
Regards, Nux.
No. The files are served 'as is' in debug mode. What he says is "Do not try to keep line numbers in minified mode (use debug mode instead)".
It's not a matter of performance, it's a matter of complexity. More configurable does not always mean better.
- Trevor
On 9/29/10 3:46 PM, Maciej Jaros wrote:
Trevor Parscal wrote:
There seems to be some confusion about how ResourceLoader works, whichhas been leading people to make commits like r73196 and report bugs like #25362. I would like to offer some clarification.
ResourceLoader, if you aren't already aware, is a new system in MediaWiki 1.17 which allows developers to bundle collections of *resources* (like JavaScript and CSS files, or localized messages) into *modules*. Modules may represent any number of scripts, styles and messages, which are read from the file system, the database, or generated by software.
When a request is made for one or more modules, each resource is packaged together and sent back to the client as a response. The way in which these requests and responses are performed depends on whether debug is on or off.
When debug mode is off:
* Modules are requested in batches * Resources are combined into modules * Modules are combined into a response * The response is minifiedWhen debug mode is on:
* Modules are requested individually * Resources are combined into modulesI think it's debatable whether debug=true mode goes far enough, since it still combines resources into modules, and I am open to contributions that can make debug=true mode even more debugging friendly by delivering the resources to the client as unchanged as possible. I also think it's debatable if debug=false mode goes far enough, since things like Google Closure Compiler have been proven to even further reduce the size of JavaScript resources, so I am also open to contributions which can make debug=false even more production friendly by improving front-end performance.
I might be a little tired but are you saying that debug mode does not preserve original line numbers? So this is debug mode as in "a bit easier to debug in Firebug then fully minified code"?
And I still don't see how minification options are evil. One "if" won't really change performance.
Regards, Nux.
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Trevor Parscal wrote:
ResourceLoader, if you aren't already aware, is a new system in MediaWiki 1.17 which allows developers to bundle collections of *resources* (like JavaScript and CSS files, or localized messages) into *modules*. Modules may represent any number of scripts, styles and messages, which are read from the file system, the database, or generated by software.
Speaking of debugging and breakages... is there an audit page anywhere currently for what ResourceLoader has broken? Ideally a page with a specific look at extensions that Wikimedia uses. Something like, "Wikimedia has X extensions installed. Of these, the following are currently broken with ResourceLoader...."
I looked at http://www.mediawiki.org/wiki/ResourceLoader and didn't see anything off-hand. Could/Should something like this be made? I've heard a lot of rumors about various extensions being broken (FlaggedRevs, e.g.), but I don't know what the current status is (whether these problems have been addressed, whether there are other extensions that are broken, whether this particular rumor turned out to not be true).
Thanks!
MZMcBride
That is all in our roadmap - which needs to be updated and put onto the wiki. Ideally this will happen next week when Alolita returns from Paris.
- Trevor
On 9/29/10 4:27 PM, MZMcBride wrote:
Trevor Parscal wrote:
ResourceLoader, if you aren't already aware, is a new system in MediaWiki 1.17 which allows developers to bundle collections of *resources* (like JavaScript and CSS files, or localized messages) into *modules*. Modules may represent any number of scripts, styles and messages, which are read from the file system, the database, or generated by software.
Speaking of debugging and breakages... is there an audit page anywhere currently for what ResourceLoader has broken? Ideally a page with a specific look at extensions that Wikimedia uses. Something like, "Wikimedia has X extensions installed. Of these, the following are currently broken with ResourceLoader...."
I looked at http://www.mediawiki.org/wiki/ResourceLoader and didn't see anything off-hand. Could/Should something like this be made? I've heard a lot of rumors about various extensions being broken (FlaggedRevs, e.g.), but I don't know what the current status is (whether these problems have been addressed, whether there are other extensions that are broken, whether this particular rumor turned out to not be true).
Thanks!
MZMcBride
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On 30/09/10 08:27, Trevor Parscal wrote:
There seems to be some confusion about how ResourceLoader works, which has been leading people to make commits like r73196 and report bugs like #25362. I would like to offer some clarification.
I made that change because it was requested by multiple people in discussions before the resource loader was implemented. It's not because I actually had any actual problem with debugging, I'm well aware of the existence of the debug mode.
Concerns were raised that it may be necessary to interpret minified code in cases such as:
* Where it is suspected that, due to caching issues, the debug code may not be the same as the minified code.
* To debug the minifier itself, which is far short of a full JavaScript parser and may well introduce functional differences.
* Where end users report platform-specific JavaScript errors, it may be useful to be able to match the line number of the error with something meaningful.
Since the cost of adding line breaks is fairly small, the contention was that it was a fair compromise between size and developer (and tech supporter) sanity. So I took those comments on board and implemented it shortly after the branch merge.
-- Tim Starling
Tim,
First off, thank you for taking the time to respond here, I know you have a lot on your plate right now, and I hope things are going well for you. I did not mean to call you out as having made a mistake as much as I wanted to point to examples of what kinds of changes have been made based on what I feel are misunderstandings.
On 9/29/10 6:28 PM, Tim Starling wrote:
On 30/09/10 08:27, Trevor Parscal wrote:
There seems to be some confusion about how ResourceLoader works, which has been leading people to make commits like r73196 and report bugs like #25362. I would like to offer some clarification.
I made that change because it was requested by multiple people in discussions before the resource loader was implemented. It's not because I actually had any actual problem with debugging, I'm well aware of the existence of the debug mode.
Your responsiveness to the community is admirable, but I feel this may be a case where more collaborative discussion should have taken place before action was taken. Nonetheless, I appreciate your boldness, even if I disagree with the action taken.
Concerns were raised that it may be necessary to interpret minified code in cases such as:
- Where it is suspected that, due to caching issues, the debug code
may not be the same as the minified code.
Shift+refresh?
- To debug the minifier itself, which is far short of a full
JavaScript parser and may well introduce functional differences.
If JSMin is causing the bug, then it will be best detected by seeing that debug=true/false have different behavior, not that debug=true contains line breaks.
- Where end users report platform-specific JavaScript errors, it may
be useful to be able to match the line number of the error with something meaningful.
The usefulness of this is attached to the idea the most important part of the error message is the line number. In some browsers (such as many versions of IE) the line numbers aren't even correct. Besides, as I have said already, over and over, combination is going to throw off line numbers anyways, not just making them higher, but depending on the user's preferences they may be totally different from one user to another. Line numbers in production mode (debug=false) are useless no matter how much white-space is preserved.
Since the cost of adding line breaks is fairly small, the contention was that it was a fair compromise between size and developer (and tech supporter) sanity. So I took those comments on board and implemented it shortly after the branch merge.
We should probably calculate just how small that cost is before we start making asumptions based on it being "fairly small". Also, as I have been saying, the production mode (debug=false) is only going to become more optimized in the future, further reducing the value of line numbers.
I am not meaning to argue with *you* on a point-by-point basis as much as I'm arguing with the originators of these points. I get the feeling that, while you may understand how resource loader works, they may not.
My top priorities are to make the site as high-performance as possible while maintaining a reasonable level of developer friendliness. These are on opposite ends of the spectrum, and middle ground serves neither well.
- Trevor
-- Tim Starling
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
2010/9/30 Trevor Parscal tparscal@wikimedia.org:
- Where it is suspected that, due to caching issues, the debug code
may not be the same as the minified code.
Shift+refresh?
Or change the URL in some way that doesn't alter the meaning of the URL. There's various ways to get around caching.
- To debug the minifier itself, which is far short of a full
JavaScript parser and may well introduce functional differences.
If JSMin is causing the bug, then it will be best detected by seeing that debug=true/false have different behavior, not that debug=true contains line breaks.
JSMin causing a bug sounds very unlikely to me, but I agree that we should be paranoid and account for the possibility anyway. Still, I think we can agree that JSMin breaking something should be a very, very rare occurrence (it's been around for a long time and has been used on a large body of code), and that debugging the minifier would be done locally anyway (to be sure caching is not to blame), so line breaks in production won't help. All you'd need to debug JSMin locally without going insane is to be able to preserve line breaks locally, which can be accommodated with a setting (off by default) or even just requiring commenting out of code (as long as it's clear how to do this and not cumbersome, this should be fine).
but depending on the user's preferences they may be totally different from one user to another.
This is false in almost all cases. Because we make our requests as cacheable as possible, the vast majority of RL responses will *not* depend on user preferences, or in fact on anything not in the URL. For these requests, you'll be able to pull up the same URL that the user is pulling and see the exact same thing (provided you both clear your caches). User preferences and other factors may be used to *build* the URL, so whether you see the issue may depend on your preferences, but once you find out which URL the user is getting their stuff from you'll have a meaningful line number.
There should only be one request whose contents depend on user preferences: the one getting the user's preferences plus the user scripts. The former is really just one call to a setter with a JSON-encoded preferences map, so it consists of one simple statement; inspecting that is doable. The rest is the user's own user JS, which can contain almost anything. This is the only thing you won't reasonably be able to debug based on just a line number, but then there's lots of other easy ways to nail down user JS as being the culprit.
Line numbers in production mode (debug=false) are useless no matter how much white-space is preserved.
Per the above this is not entirely true, but the one case in which they are useful (culprit is in user JS+prefs request and all you have is a line number) doesn't warrant preserving line breaks IMO.
We should probably calculate just how small that cost is before we start making asumptions based on it being "fairly small".
I agree that we should get our numbers first, but some preliminary playing around seems to indicate Tim's right about this one.
Roan Kattouw (Catrope)
On 9/30/10 10:11 AM, Roan Kattouw wrote:
2010/9/30 Trevor Parscaltparscal@wikimedia.org:
- Where it is suspected that, due to caching issues, the debug code
may not be the same as the minified code.
Shift+refresh?
Or change the URL in some way that doesn't alter the meaning of the URL. There's various ways to get around caching.
- To debug the minifier itself, which is far short of a full
JavaScript parser and may well introduce functional differences.
If JSMin is causing the bug, then it will be best detected by seeing that debug=true/false have different behavior, not that debug=true contains line breaks.
JSMin causing a bug sounds very unlikely to me, but I agree that we should be paranoid and account for the possibility anyway. Still, I think we can agree that JSMin breaking something should be a very, very rare occurrence (it's been around for a long time and has been used on a large body of code), and that debugging the minifier would be done locally anyway (to be sure caching is not to blame), so line breaks in production won't help. All you'd need to debug JSMin locally without going insane is to be able to preserve line breaks locally, which can be accommodated with a setting (off by default) or even just requiring commenting out of code (as long as it's clear how to do this and not cumbersome, this should be fine).
but depending on the user's preferences they may be totally different from one user to another.
This is false in almost all cases. Because we make our requests as cacheable as possible, the vast majority of RL responses will *not* depend on user preferences, or in fact on anything not in the URL. For these requests, you'll be able to pull up the same URL that the user is pulling and see the exact same thing (provided you both clear your caches). User preferences and other factors may be used to *build* the URL, so whether you see the issue may depend on your preferences, but once you find out which URL the user is getting their stuff from you'll have a meaningful line number.
There should only be one request whose contents depend on user preferences: the one getting the user's preferences plus the user scripts. The former is really just one call to a setter with a JSON-encoded preferences map, so it consists of one simple statement; inspecting that is doable. The rest is the user's own user JS, which can contain almost anything. This is the only thing you won't reasonably be able to debug based on just a line number, but then there's lots of other easy ways to nail down user JS as being the culprit.
Yes, thank you for correcting me here. In fact, we encode the user's name into the URL (but still limit access to the information to those with a matching session) so the URLs are all always unique to the response.
Line numbers in production mode (debug=false) are useless no matter how much white-space is preserved.
Per the above this is not entirely true, but the one case in which they are useful (culprit is in user JS+prefs request and all you have is a line number) doesn't warrant preserving line breaks IMO.
We should probably calculate just how small that cost is before we start making asumptions based on it being "fairly small".
I agree that we should get our numbers first, but some preliminary playing around seems to indicate Tim's right about this one.
Roan Kattouw (Catrope)
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On 9/30/10 9:31 AM, Trevor Parscal wrote:
Tim,
First off, thank you for taking the time to respond here, I know you have a lot on your plate right now, and I hope things are going well for you. I did not mean to call you out as having made a mistake as much as I wanted to point to examples of what kinds of changes have been made based on what I feel are misunderstandings.
On 9/29/10 6:28 PM, Tim Starling wrote:
On 30/09/10 08:27, Trevor Parscal wrote:
There seems to be some confusion about how ResourceLoader works, whichhas been leading people to make commits like r73196 and report bugs like #25362. I would like to offer some clarification.
I made that change because it was requested by multiple people in discussions before the resource loader was implemented. It's not because I actually had any actual problem with debugging, I'm well aware of the existence of the debug mode.
Your responsiveness to the community is admirable, but I feel this may be a case where more collaborative discussion should have taken place before action was taken. Nonetheless, I appreciate your boldness, even if I disagree with the action taken.
Concerns were raised that it may be necessary to interpret minified code in cases such as:
- Where it is suspected that, due to caching issues, the debug code
may not be the same as the minified code.
Shift+refresh?
- To debug the minifier itself, which is far short of a full
JavaScript parser and may well introduce functional differences.
If JSMin is causing the bug, then it will be best detected by seeing that debug=true/false have different behavior, not that debug=true contains line breaks.
- Where end users report platform-specific JavaScript errors, it may
be useful to be able to match the line number of the error with something meaningful.
The usefulness of this is attached to the idea the most important part of the error message is the line number. In some browsers (such as many versions of IE) the line numbers aren't even correct. Besides, as I have said already, over and over, combination is going to throw off line numbers anyways, not just making them higher, but depending on the user's preferences they may be totally different from one user to another. Line numbers in production mode (debug=false) are useless no matter how much white-space is preserved.
Since the cost of adding line breaks is fairly small, the contention was that it was a fair compromise between size and developer (and tech supporter) sanity. So I took those comments on board and implemented it shortly after the branch merge.
We should probably calculate just how small that cost is before we start making asumptions based on it being "fairly small". Also, as I have been saying, the production mode (debug=false) is only going to become more optimized in the future, further reducing the value of line numbers.
OK, now I've calculated it...
On a normal page view with the Vector skin and the Vector extension turned on there's a 2KB difference. On an edit page with the Vector skin and Vector and WikiEditor extensions there's a 4KB difference.
While adding 2KB to a request for a person in a remote corner of the world on a 56k modem will only add about 0.3 seconds to the download, sending 2,048 extra bytes to 350 million people each month increases our bandwidth by about 668 gigabytes a month. I don't know what that kind of bandwidth costs the foundation, but it's not free.
I am not meaning to argue with *you* on a point-by-point basis as much as I'm arguing with the originators of these points. I get the feeling that, while you may understand how resource loader works, they may not.
My top priorities are to make the site as high-performance as possible while maintaining a reasonable level of developer friendliness. These are on opposite ends of the spectrum, and middle ground serves neither well.
- Trevor
-- Tim Starling
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
At 2010-09-30 20:35, Trevor Parscal wrote:
[...]
- Where end users report platform-specific JavaScript errors, it may
be useful to be able to match the line number of the error with something meaningful.
The usefulness of this is attached to the idea the most important part of the error message is the line number. In some browsers (such as many versions of IE) the line numbers aren't even correct. Besides, as I have said already, over and over, combination is going to throw off line numbers anyways, not just making them higher, but depending on the user's preferences they may be totally different from one user to another. Line numbers in production mode (debug=false) are useless no matter how much white-space is preserved.
To my experience something is better then nothing. IE (old one that is) is usually wrong when you have either long lines (ekhem ;-)) and have code inside script tags (instead of outer files). Picture this:
1. A user says his browser reports an error. 2. I ask for the browser and other stuff... 3. I don't see the problem. 4. He tries out debug mode and it's fine. 5. I search through his code...
The problem with fully minified without any vertical white space code to me is that you cannot read this. Well I can't and Firebug can't and any other debugger AFAIK. If something comes up it will be almost impossible to catch.
Since the cost of adding line breaks is fairly small, the contention was that it was a fair compromise between size and developer (and tech supporter) sanity. So I took those comments on board and implemented it shortly after the branch merge.
We should probably calculate just how small that cost is before we start making asumptions based on it being "fairly small". Also, as I have been saying, the production mode (debug=false) is only going to become more optimized in the future, further reducing the value of line numbers.
OK, now I've calculated it...
On a normal page view with the Vector skin and the Vector extension turned on there's a 2KB difference. On an edit page with the Vector skin and Vector and WikiEditor extensions there's a 4KB difference.
While adding 2KB to a request for a person in a remote corner of the world on a 56k modem will only add about 0.3 seconds to the download, sending 2,048 extra bytes to 350 million people each month increases our bandwidth by about 668 gigabytes a month. I don't know what that kind of bandwidth costs the foundation, but it's not free.
OK. It's not free, the question is it worth the effort and are you really calculating this correctly? I think it would be worth it as a just-in-case thing. Also JS is usually cached and actually get stuck in cache for days. And just to be sure - did you calculated gziped version difference or plain?
Plus developers will probably stick to debug mode if won't provide something in the middle. Yes, I can't speak for all, but personally I work on code (small tweaks) from time to time while I'm not doing other stuff in the minute. Always changing to debug mode to debug code will not be very productive to me. That's why I wrote my loader and I was hoping that you will add some stuff from it to the loader so I can use it and so the RL can be useful to me (in non-wikimedia use cases too).
Again I'm sure you are confident with your code and code of your colleagues and I'm not saying you screwed something up. I'm just saying something will eventually get screwed up just because it always does. That's why programmers are always needed isn't it ;-).
Regards, Nux.
On 9/30/10 1:55 PM, Maciej Jaros wrote:
At 2010-09-30 20:35, Trevor Parscal wrote:
[...]
- Where end users report platform-specific JavaScript errors, it may
be useful to be able to match the line number of the error with something meaningful.
The usefulness of this is attached to the idea the most important part of the error message is the line number. In some browsers (such as many versions of IE) the line numbers aren't even correct. Besides, as I have said already, over and over, combination is going to throw off line numbers anyways, not just making them higher, but depending on the user's preferences they may be totally different from one user to another. Line numbers in production mode (debug=false) are useless no matter how much white-space is preserved.
To my experience something is better then nothing. IE (old one that is) is usually wrong when you have either long lines (ekhem ;-)) and have code inside script tags (instead of outer files). Picture this:
1. A user says his browser reports an error. 2. I ask for the browser and other stuff... 3. I don't see the problem. 4. He tries out debug mode and it's fine. 5. I search through his code...The problem with fully minified without any vertical white space code to me is that you cannot read this. Well I can't and Firebug can't and any other debugger AFAIK. If something comes up it will be almost impossible to catch.
There's this assumption here that when a bug is reported that a JavaScript developer is going to be lost without a line number. My experience is that when bugs are reported, the first step is reproduction, and then investigation. If we shouldn't be relying on users with little or no expertise to be participating in investigation. Also, error reporting with Firebug and most browsers is much more verbose than "error on line ##".
Since the cost of adding line breaks is fairly small, the contention was that it was a fair compromise between size and developer (and tech supporter) sanity. So I took those comments on board and implemented it shortly after the branch merge.
We should probably calculate just how small that cost is before we start making asumptions based on it being "fairly small". Also, as I have been saying, the production mode (debug=false) is only going to become more optimized in the future, further reducing the value of line numbers.
OK, now I've calculated it...
On a normal page view with the Vector skin and the Vector extension turned on there's a 2KB difference. On an edit page with the Vector skin and Vector and WikiEditor extensions there's a 4KB difference.
While adding 2KB to a request for a person in a remote corner of the world on a 56k modem will only add about 0.3 seconds to the download, sending 2,048 extra bytes to 350 million people each month increases our bandwidth by about 668 gigabytes a month. I don't know what that kind of bandwidth costs the foundation, but it's not free.
OK. It's not free, the question is it worth the effort and are you really calculating this correctly? I think it would be worth it as a just-in-case thing. Also JS is usually cached and actually get stuck in cache for days. And just to be sure - did you calculated gziped version difference or plain?
Compressed of course. It would be 5-10 times that much without compression.
Plus developers will probably stick to debug mode if won't provide something in the middle. Yes, I can't speak for all, but personally I work on code (small tweaks) from time to time while I'm not doing other stuff in the minute. Always changing to debug mode to debug code will not be very productive to me. That's why I wrote my loader and I was hoping that you will add some stuff from it to the loader so I can use it and so the RL can be useful to me (in non-wikimedia use cases too).
This is what debug mode is for. Developers. Developers should always use debug mode while debugging. Switching back and forth is not necessary for every tiny change, just a good thing to verify before checking in code.
Again I'm sure you are confident with your code and code of your colleagues and I'm not saying you screwed something up. I'm just saying something will eventually get screwed up just because it always does. That's why programmers are always needed isn't it ;-).
It's not a matter of me thinking our code is so good it won't break. It's a matter of wanting to send fully optimized code to clients, not partially optimized code.
I agree that something will always break - I just disagree that there's much merit in your approach to solving this.
If we are talking about allowing the ResourceLoader to be configurable with LocalSettings.php adjustments, then you are already assuming a level of expertise about the developer debugging the problem that these middle-ground approaches will be useless.
If we are talking about adding yet another URL parameter to be able to turn minification off, or otherwise customize the processing of the modules beyond turning debug on and off, I think you are adding complexity without reasonable benefit.
- Trevor
Regards, Nux. _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
At 2010-09-30 23:13, Trevor Parscal wrote:
On 9/30/10 1:55 PM, Maciej Jaros wrote:
At 2010-09-30 20:35, Trevor Parscal wrote:[...]
- Where end users report platform-specific JavaScript errors, it may
be useful to be able to match the line number of the error with something meaningful.
The usefulness of this is attached to the idea the most important part of the error message is the line number. In some browsers (such as many versions of IE) the line numbers aren't even correct. Besides, as I have said already, over and over, combination is going to throw off line numbers anyways, not just making them higher, but depending on the user's preferences they may be totally different from one user to another. Line numbers in production mode (debug=false) are useless no matter how much white-space is preserved.
To my experience something is better then nothing. IE (old one that is) is usually wrong when you have either long lines (ekhem ;-)) and have code inside script tags (instead of outer files). Picture this:
1. A user says his browser reports an error. 2. I ask for the browser and other stuff... 3. I don't see the problem. 4. He tries out debug mode and it's fine. 5. I search through his code...The problem with fully minified without any vertical white space code to me is that you cannot read this. Well I can't and Firebug can't and any other debugger AFAIK. If something comes up it will be almost impossible to catch.
There's this assumption here that when a bug is reported that a JavaScript developer is going to be lost without a line number. My experience is that when bugs are reported, the first step is reproduction, and then investigation. If we shouldn't be relying on users with little or no expertise to be participating in investigation. Also, error reporting with Firebug and most browsers is much more verbose than "error on line ##".
It's not an assumption. I AM one of the main developers working on bugs reported by users on pl.wiki. And yes line number is very helpful if accurate. Also note that I was picturing a situation where I have to debug the minified code when I don't get the error in my browser(s) at hand. Usually the problem is some extension/plug-in, but I can't tell unless I investigate. And reading minified code at it's current state is impossible for me.
I understand the need to minify code I really do. I just don't get it why do you want to minify this that much. HTML which is usually served at every page load is not minified. Further more even minified HTML can be readable for debuging purposes as every debugger (inspector) shows the structure of the code rather then a code pulp. Also why are there are explicitly added comments like "<!-- /navigation -->". People don't need comments and whitespace to read code, right? Also I have a PHP minifier if you are interested. Let's serve Mediawiki encode and if some developer want to debug the code he can always switch folders.
Plus developers will probably stick to debug mode if won't provide something in the middle. Yes, I can't speak for all, but personally I work on code (small tweaks) from time to time while I'm not doing other stuff in the minute. Always changing to debug mode to debug code will not be very productive to me. That's why I wrote my loader and I was hoping that you will add some stuff from it to the loader so I can use it and so the RL can be useful to me (in non-wikimedia use cases too).
This is what debug mode is for. Developers. Developers should always use debug mode while debugging. Switching back and forth is not necessary for every tiny change, just a good thing to verify before checking in code.
Again I'm sure you are confident with your code and code of your colleagues and I'm not saying you screwed something up. I'm just saying something will eventually get screwed up just because it always does. That's why programmers are always needed isn't it ;-).
It's not a matter of me thinking our code is so good it won't break. It's a matter of wanting to send fully optimized code to clients, not partially optimized code.
This is impossible. Fully optimized code would only contain one byte per variable name. Yo need to make sacrifices. And this seems to be needed as shown before and above.
I agree that something will always break - I just disagree that there's much merit in your approach to solving this.
If we are talking about allowing the ResourceLoader to be configurable with LocalSettings.php adjustments, then you are already assuming a level of expertise about the developer debugging the problem that these middle-ground approaches will be useless.
If we are talking about adding yet another URL parameter to be able to turn minification off, or otherwise customize the processing of the modules beyond turning debug on and off, I think you are adding complexity without reasonable benefit.
Well we differentiate on that. I'm wondering of what is you scenario on debugging the minified code.
Regards, Nux.
On 9/30/10 4:31 PM, Maciej Jaros wrote:
At 2010-09-30 23:13, Trevor Parscal wrote:
On 9/30/10 1:55 PM, Maciej Jaros wrote:At 2010-09-30 20:35, Trevor Parscal wrote:[...]
- Where end users report platform-specific JavaScript errors, it may
be useful to be able to match the line number of the error with something meaningful.
The usefulness of this is attached to the idea the most important part of the error message is the line number. In some browsers (such as many versions of IE) the line numbers aren't even correct. Besides, as I have said already, over and over, combination is going to throw off line numbers anyways, not just making them higher, but depending on the user's preferences they may be totally different from one user to another. Line numbers in production mode (debug=false) are useless no matter how much white-space is preserved.
To my experience something is better then nothing. IE (old one that is) is usually wrong when you have either long lines (ekhem ;-)) and have code inside script tags (instead of outer files). Picture this:
1. A user says his browser reports an error. 2. I ask for the browser and other stuff... 3. I don't see the problem. 4. He tries out debug mode and it's fine. 5. I search through his code...The problem with fully minified without any vertical white space code to me is that you cannot read this. Well I can't and Firebug can't and any other debugger AFAIK. If something comes up it will be almost impossible to catch.
There's this assumption here that when a bug is reported that a JavaScript developer is going to be lost without a line number. My experience is that when bugs are reported, the first step is reproduction, and then investigation. If we shouldn't be relying on users with little or no expertise to be participating in investigation. Also, error reporting with Firebug and most browsers is much more verbose than "error on line ##".
It's not an assumption. I AM one of the main developers working on bugs reported by users on pl.wiki. And yes line number is very helpful if accurate. Also note that I was picturing a situation where I have to debug the minified code when I don't get the error in my browser(s) at hand. Usually the problem is some extension/plug-in, but I can't tell unless I investigate. And reading minified code at it's current state is impossible for me.
How can you possibly resolve an issue if you can't reproduce it? Developers have access to multiple browsers for this exact reason. Otherwise, you are debugging and programming blind, which is extremely unproductive.
I understand the need to minify code I really do. I just don't get it why do you want to minify this that much. HTML which is usually served at every page load is not minified. Further more even minified HTML can be readable for debuging purposes as every debugger (inspector) shows the structure of the code rather then a code pulp. Also why are there are explicitly added comments like "<!-- /navigation -->". People don't need comments and whitespace to read code, right? Also I have a PHP minifier if you are interested. Let's serve Mediawiki encode and if some developer want to debug the code he can always switch folders.
I think that our outgoing HTML *should* be minified, and those comments should be stripped, I just have yet to get to that as of yet. Go to www.google.com and view source. Tell me what you see. Why are we so different from them? They are taking advantage of aggressive minification, and we should too.
To be honest, I can't quite wrap my head around why I am getting any friction on making the front-end as fast as possible. I was expecting people to say I had not gone far enough.
Plus developers will probably stick to debug mode if won't provide something in the middle. Yes, I can't speak for all, but personally I work on code (small tweaks) from time to time while I'm not doing other stuff in the minute. Always changing to debug mode to debug code will not be very productive to me. That's why I wrote my loader and I was hoping that you will add some stuff from it to the loader so I can use it and so the RL can be useful to me (in non-wikimedia use cases too).
This is what debug mode is for. Developers. Developers should always use debug mode while debugging. Switching back and forth is not necessary for every tiny change, just a good thing to verify before checking in code.
Again I'm sure you are confident with your code and code of your colleagues and I'm not saying you screwed something up. I'm just saying something will eventually get screwed up just because it always does. That's why programmers are always needed isn't it ;-).
It's not a matter of me thinking our code is so good it won't break. It's a matter of wanting to send fully optimized code to clients, not partially optimized code.
This is impossible. Fully optimized code would only contain one byte per variable name. Yo need to make sacrifices. And this seems to be needed as shown before and above.
Google Closure Compiler is on the horizon, and with it, single letter names for temporary variables and functions within closures can be achieved without obfuscating the original source code. Closure can do more tricks than that, but that's a start, and usually yields 10% reductions or more depending on how verbose your naming tends to be.
I agree that something will always break - I just disagree that there's much merit in your approach to solving this.
If we are talking about allowing the ResourceLoader to be configurable with LocalSettings.php adjustments, then you are already assuming a level of expertise about the developer debugging the problem that these middle-ground approaches will be useless.
If we are talking about adding yet another URL parameter to be able to turn minification off, or otherwise customize the processing of the modules beyond turning debug on and off, I think you are adding complexity without reasonable benefit.
Well we differentiate on that. I'm wondering of what is you scenario on debugging the minified code.
Why would you want to debug code in production mode?
- Trevor
Regards, Nux.
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
At 2010-10-01 01:50, Trevor Parscal wrote:
On 9/30/10 4:31 PM, Maciej Jaros wrote:
At 2010-09-30 23:13, Trevor Parscal wrote:On 9/30/10 1:55 PM, Maciej Jaros wrote:At 2010-09-30 20:35, Trevor Parscal wrote:> [...] > * Where end users report platform-specific JavaScript errors, it may > be useful to be able to match the line number of the error with > something meaningful. The usefulness of this is attached to the idea the most important part of the error message is the line number. In some browsers (such as many versions of IE) the line numbers aren't even correct. Besides, as I have said already, over and over, combination is going to throw off line numbers anyways, not just making them higher, but depending on the user's preferences they may be totally different from one user to another. Line numbers in production mode (debug=false) are useless no matter how much white-space is preserved.
To my experience something is better then nothing. IE (old one that is) is usually wrong when you have either long lines (ekhem ;-)) and have code inside script tags (instead of outer files). Picture this:
1. A user says his browser reports an error. 2. I ask for the browser and other stuff... 3. I don't see the problem. 4. He tries out debug mode and it's fine. 5. I search through his code...The problem with fully minified without any vertical white space code to me is that you cannot read this. Well I can't and Firebug can't and any other debugger AFAIK. If something comes up it will be almost impossible to catch.
There's this assumption here that when a bug is reported that a JavaScript developer is going to be lost without a line number. My experience is that when bugs are reported, the first step is reproduction, and then investigation. If we shouldn't be relying on users with little or no expertise to be participating in investigation. Also, error reporting with Firebug and most browsers is much more verbose than "error on line ##".
It's not an assumption. I AM one of the main developers working on bugs reported by users on pl.wiki. And yes line number is very helpful if accurate. Also note that I was picturing a situation where I have to debug the minified code when I don't get the error in my browser(s) at hand. Usually the problem is some extension/plug-in, but I can't tell unless I investigate. And reading minified code at it's current state is impossible for me.
How can you possibly resolve an issue if you can't reproduce it? Developers have access to multiple browsers for this exact reason. Otherwise, you are debugging and programming blind, which is extremely unproductive.
I'm very productive with inspecting code with my eyes. Ask my boss ;-).
I understand the need to minify code I really do. I just don't get it why do you want to minify this that much. HTML which is usually served at every page load is not minified. Further more even minified HTML can be readable for debuging purposes as every debugger (inspector) shows the structure of the code rather then a code pulp. Also why are there are explicitly added comments like "<!-- /navigation -->". People don't need comments and whitespace to read code, right? Also I have a PHP minifier if you are interested. Let's serve Mediawiki encode and if some developer want to debug the code he can always switch folders.
I think that our outgoing HTML *should* be minified, and those comments should be stripped, I just have yet to get to that as of yet. Go to www.google.com and view source. Tell me what you see. Why are we so different from them? They are taking advantage of aggressive minification, and we should too.
To be honest, I can't quite wrap my head around why I am getting any friction on making the front-end as fast as possible. I was expecting people to say I had not gone far enough.
Two reasons probably: 1. The gain is not big for average user (loading big JS is only painful at first visit in a day or even more). 2. Wikipedia is not Google (you go to a Google's website and use it; you go to Wikipedia and change it).
Plus developers will probably stick to debug mode if won't provide something in the middle. Yes, I can't speak for all, but personally I work on code (small tweaks) from time to time while I'm not doing other stuff in the minute. Always changing to debug mode to debug code will not be very productive to me. That's why I wrote my loader and I was hoping that you will add some stuff from it to the loader so I can use it and so the RL can be useful to me (in non-wikimedia use cases too).
This is what debug mode is for. Developers. Developers should always use debug mode while debugging. Switching back and forth is not necessary for every tiny change, just a good thing to verify before checking in code.
Again I'm sure you are confident with your code and code of your colleagues and I'm not saying you screwed something up. I'm just saying something will eventually get screwed up just because it always does. That's why programmers are always needed isn't it ;-).
It's not a matter of me thinking our code is so good it won't break. It's a matter of wanting to send fully optimized code to clients, not partially optimized code.
This is impossible. Fully optimized code would only contain one byte per variable name. Yo need to make sacrifices. And this seems to be needed as shown before and above.
Google Closure Compiler is on the horizon, and with it, single letter names for temporary variables and functions within closures can be achieved without obfuscating the original source code. Closure can do more tricks than that, but that's a start, and usually yields 10% reductions or more depending on how verbose your naming tends to be.
If I could be honest I would say I hope no one will ever have time to implement it.
I agree that something will always break - I just disagree that there's much merit in your approach to solving this.
If we are talking about allowing the ResourceLoader to be configurable with LocalSettings.php adjustments, then you are already assuming a level of expertise about the developer debugging the problem that these middle-ground approaches will be useless.
If we are talking about adding yet another URL parameter to be able to turn minification off, or otherwise customize the processing of the modules beyond turning debug on and off, I think you are adding complexity without reasonable benefit.
Well we differentiate on that. I'm wondering of what is you scenario on debugging the minified code.
Why would you want to debug code in production mode?
I'm asking you how are you going to cope with a situation in which some brand new, minified code is buggy. How are you going to fix it? Two scenarios: you are able to reproduce this only in Opera Mobile, you are only able to reproduce it in IE6.
Regards, Nux.
Trevor Parscal wrote:
I think that our outgoing HTML *should* be minified, and those comments should be stripped, I just have yet to get to that as of yet. Go to www.google.com and view source. Tell me what you see. Why are we so different from them? They are taking advantage of aggressive minification, and we should too.
Yes, I've looked at www.google.com's page source. It doesn't have a closing <body> tag or a closing <html> tag. It's also largely unreadable. I'm not sure why this is a something to strive for. Bandwidth on Wikimedia's end is cheap. The emerging bandwidth issues come largely from mobile platforms, which have separate mobile sites to address this. That isn't to say bandwidth should be needlessly wasted, but there's no reason to be paranoid about it. Whether or not you remove every newline from the HTML output is going to have a negligible impact for mobile and non-mobile platforms. It will have a demonstrable impact on developers and users, though, which I'll discuss below.
To be honest, I can't quite wrap my head around why I am getting any friction on making the front-end as fast as possible. I was expecting people to say I had not gone far enough.
You're getting friction because of three different issues.
The first issue is that the front-end is currently pretty damn fast for _most_ people. It's serving cached HTML most of the time with very little JavaScript executed for a huge majority of users because (1) these users are not editing, they're just reading; and (2) these users are not logged in. I imagine there would be a lot more support for making, for example, page rendering faster. Anyone who has edited a large article logged in knows how painful it is. If ResourceLoader is going to address _that_ problem, then this is a marketing and public relations issue.
The second issue is that you're focusing on the mess that was of your creation. There's a huge effort to clean up the insane amount of JavaScript that the UsabilityInitiative introduced. There isn't going to be much appreciation for cleaning up your own mess. Don't take that the wrong way, please. I'm not saying that the UsabilityInitiative's work wasn't good, but it came at a cost that now has to be addressed. You're asking why people aren't too excited about the clean-up and I'm offering my theory.
The third issue is that, unlike www.google.com, Wikimedia wikis are editable sites. People customize their experience via gadgets, user scripts, and other things of that nature. The same isn't true for Google's homepage. The interactivity of Wikimedia wikis means that users need to be able to read the page source easily. This includes the HTML page source, the CSS, and the JavaScript that's loaded. Minification and obfuscation come at a cost when trying to develop scripts that modify the user experience. People have been trying to say this to you in so many ways, but you don't seem to be quite getting that. Yes, there's a debug mode via a URL parameter, but anyone who has ever dealt with settings set temporarily by URL parameter (?useskin, ?uselang, etc.) know what a pain in the ass these are. And, as with every feature ever introduced, there _will_ be bugs that come from minification and obfuscation. There doesn't seem to be an acknowledgement of this in a lot of your writing, which seems to be frustrating a lot of people.
MZMcBride
On 9/30/10 6:04 PM, MZMcBride wrote:
You're getting friction because of three different issues.
The first issue is that the front-end is currently pretty damn fast for _most_ people.
The front end that we have *now* is fast for some people. The point of writing a new platform is to be able to do more, if we want.
The second issue is that you're focusing on the mess that was of your creation. There's a huge effort to clean up the insane amount of JavaScript that the UsabilityInitiative introduced.
I think we have a large difference of vision here.
By contemporary standards the amount of JS in the UsabilityInitiative, and related projects, is a mere trifle.
Our competitors invest a lot in Javascript infrastructure because they find that every last efficiency is usually worth it. If stripping newlines gives us 1K more per page, we can squeeze 1K more of usability or other interface niceties in.
Now, we might decide not to do certain kinds of minification for other very good reasons. But I say this to put the Resource Loader in context. The point here, at least as I see it, is to bring Wikimedia sites up to the standards of the rest of the web, not to "clean up" some "mess".
The third issue is that, unlike www.google.com, Wikimedia wikis are editable sites. People customize their experience via gadgets, user scripts, and other things of that nature. The same isn't true for Google's homepage.
I agree this is a very good reason why we should hesitate before doing anything that would obfuscate our pages. "View Source" doesn't work as well on the web any more, and it should on Wikipedia. (I personally would set the balance at a different point -- I'd like there to be a note right in the page source explaining how to view an unoptimized version.)
That said, your assertion that Google doesn't allow customizations of its page is just not true. Google has offered "skins" for something like a year or more now, and their home page widget platform is literally called Gadgets.
The great grandaddy of all site-Javascript modifiers is GreaseMonkey, and the guy that wrote that is even a Google employee (Aaron Boodman). Mihai Parparita, another Google employee, circulates a very popular set of Greasemonkey improvements to Gmail.
So, the argument that Google etc. just don't get developer hacks, or that they do them badly, is quite wrong. Their platforms for doing this are far, far more successful than ours. We should be learning from them on this, not vice versa.
Here's how they do it: Google guarantees that certain Javascript functions or DOM objects are reliably available for use and modification, and others are not. So they use minification for efficiency while explicitly preventing some things from being altered (that's a feature of Google's Closure Compiler).
This is a story as old as computing; to enable a community, create a rich platform with a stable public API.
Merely leaving the pages open to any modification is actually worse for gadget writers, since anything could change.
Anyway, I think that's somewhat in the flavor of what the Resource Loader people are trying to achieve here. Efficiency *and* community. Minification *and* openness. Gadget-writers are a big part of their targeted use cases. Otherwise, they would have just used something off the shelf. There are a lot of good JS libraries out there, but none that quite fit the needs of our community.
On 9/30/2010 10:03 PM, Neil Kandalgaonkar wrote:
Here's how they do it: Google guarantees that certain Javascript functions or DOM objects are reliably available for use and modification, and others are not. So they use minification for efficiency while explicitly preventing some things from being altered (that's a feature of Google's Closure Compiler).
This is a story as old as computing; to enable a community, create a rich platform with a stable public API.
As a gadget developer, I would love that. However, that seems to be, in part, the opposite of what's happening. We're taking a public API that's been used for years (wikibits.js) and deprecating the whole thing, with no documentation (that I can find in a couple minutes of searching) for the replacements.
When the new skin/editor was rolled out, there was basically no documentation for anything and there still isn't very much. When I rewrote the reftoolbar gadget[1] to support the new toolbar, in advance of the rollout, I had to do it by reading the Wikieditor code and testing on a personal sandbox wiki running trunk, which changed often and was frequently broken. Now I'm probably going to have to do that again to make sure it won't break with ResourceLoader. That's not a very good development process.
Merely leaving the pages open to any modification is actually worse for gadget writers, since anything could change.
Anyway, I think that's somewhat in the flavor of what the Resource Loader people are trying to achieve here. Efficiency *and* community. Minification *and* openness. Gadget-writers are a big part of their targeted use cases. Otherwise, they would have just used something off the shelf. There are a lot of good JS libraries out there, but none that quite fit the needs of our community.
That's all great, but if you're going to go that route, you have to make damn sure you get everything right, else we're going to end up with a bunch of frustrated gadget developers and a ton of angry users with broken gadgets.
[1] http://en.wikipedia.org/wiki/Wikipedia:RefToolbar_2.0
Neil Kandalgaonkar wrote:
The front end that we have *now* is fast for some people. The point of writing a new platform is to be able to do more, if we want.
Do more, such as?
If there's a broader outlook here, a roadmap for the future, what is it and where is it posted? I think there's a reasonable expectation that if you're going to make claims like this and push forward on projects like ResourceLoader, there be a list of all the gains that are going to be made as a result of it, or at least a list of all the foreseeable gains from it.
You're talking about "doing more." Be specific. I've looked at http://www.mediawiki.org/wiki/ResourceLoader and Trevor has said that the non-public roadmap will soon be public, though it's my understanding that the roadmap is more about deployment, not about the grand vision of what ResourceLoader will enable in the future. Can you shed some light on this?
The second issue is that you're focusing on the mess that was of your creation. There's a huge effort to clean up the insane amount of JavaScript that the UsabilityInitiative introduced.
I think we have a large difference of vision here.
By contemporary standards the amount of JS in the UsabilityInitiative, and related projects, is a mere trifle.
Our competitors invest a lot in Javascript infrastructure because they find that every last efficiency is usually worth it. If stripping newlines gives us 1K more per page, we can squeeze 1K more of usability or other interface niceties in.
Which competitors are you referring to here?
You seem to be suggesting that a lot of other sites bog down their users' experience with an obscene amount of JavaScript, so it's okay for Wikimedia to do the same, albeit with a bit less JavaScript. This doesn't follow.
You also seem to be creating a mostly false trade-off through the idea of premature optimization. There isn't a fixed size a page can be. Making the HTML output unreadable and adding a feature aren't mutually exclusive. To suggest that they are isn't fair.
The third issue is that, unlike www.google.com, Wikimedia wikis are editable sites. People customize their experience via gadgets, user scripts, and other things of that nature. The same isn't true for Google's homepage.
I agree this is a very good reason why we should hesitate before doing anything that would obfuscate our pages. "View Source" doesn't work as well on the web any more, and it should on Wikipedia. (I personally would set the balance at a different point -- I'd like there to be a note right in the page source explaining how to view an unoptimized version.)
That said, your assertion that Google doesn't allow customizations of its page is just not true. Google has offered "skins" for something like a year or more now, and their home page widget platform is literally called Gadgets.
I was talking about www.google.com, as was the post I was replying to. Google has iGoogle, that much is certain, but I don't see how that's relevant to the broader discussion, sharing the use of the term "gadgets" notwithstanding. I think it's possible that your answer to the "what are the future benefits of ResourceLoader?" question might make what you were trying to say here clearer.
Anyway, I think that's somewhat in the flavor of what the Resource Loader people are trying to achieve here. Efficiency *and* community. Minification *and* openness. Gadget-writers are a big part of their targeted use cases. Otherwise, they would have just used something off the shelf. There are a lot of good JS libraries out there, but none that quite fit the needs of our community.
The broad point of my previous reply was that minification and obfuscation come at a real cost and the acknowledgement of this cost seems to be non-existent from the people pushing ResourceLoader forward. That's my take from reading the past discussion on this list, but perhaps I'm completely off the mark. When people talk about "friction" but can't understand the source of this friction, that indicates a problem to me. And it seems to be directly incompatible with your stated goal of "efficiency *and* community."
MZMcBride
2010/9/30 Maciej Jaros egil@wp.pl:
The problem with fully minified without any vertical white space code to me is that you cannot read this. Well I can't and Firebug can't and any other debugger AFAIK.
Minified code does not contain zero line breaks. Some line breaks are left in, although I don't understand exactly what the logic is there. Looking at a large minified file, the longest line I see is 1700ish characters, and that one contains a long array of strings; 'real' code lines don't seem to exceed 1000 characters, and most seem to be below 600.
It's true that debuggers struggle with this. Firebug has no problem with jumping from one statement to the next if there's a dozen statements on one line, but of course setting breakpoints is still line-based so that's not very helpful.
However, Trevor's main point is that you shouldn't be /trying/ to read or debug minified code, because we have a debug mode to give you unminified (and uncombined) code.
Roan Kattouw (Catrope)
On 01/10/10 07:27, Roan Kattouw wrote:
Minified code does not contain zero line breaks. Some line breaks are left in, although I don't understand exactly what the logic is there.
The logic is that line breaks can have functional significance in two cases. One is where statement-ending semicolons are omitted:
foo() bar()
A semicolon is implied where the result of joining the lines would not be valid JS, in this case:
foo() bar()
This is not valid so a semicolon is implicitly added. The other case is where a line break acts like whitespace:
x- -y
Here the first "-" is the subtraction operator, and the second "-" is the unary negation operator. This is the same as "x- -y" but not the same as "x--y", which is a syntax error since the "--" is counted as the unary postfix decrement operator.
Determining all of these cases precisely would be difficult, so JSMin just leaves the line break in where it follows any potentially problematic token.
-- Tim Starling
On 01/10/10 04:35, Trevor Parscal wrote:
OK, now I've calculated it...
On a normal page view with the Vector skin and the Vector extension turned on there's a 2KB difference. On an edit page with the Vector skin and Vector and WikiEditor extensions there's a 4KB difference.
While adding 2KB to a request for a person in a remote corner of the world on a 56k modem will only add about 0.3 seconds to the download, sending 2,048 extra bytes to 350 million people each month increases our bandwidth by about 668 gigabytes a month.
We don't pay by volume (GB per month), we pay by bandwidth (megabits per second at the 95th percentile). They should be roughly proportional to each other, but to calculate a cost we have to convert that 668GB figure to a percentage of total volume.
I took this graph:
http://www.nedworks.org/~mark/reqstats/trafficstats-monthly.png
And I used the GIMP histogram tool to integrate the outgoing part for 30 days between week 34 and week 37. The result was 31,824 pixels of blue and 20,301 pixels of green, which I figure is about 2113 TB/month. So on your figure, the cost of adding line breaks would be about 0.03% of whatever the bandwidth bill for that month is. I don't have that number to hand, but I suspect 0.03% of it is not going to be very much. For 2009-2010 there was a budget of about $1M for "internet hosting", of which bandwidth is a part, and 0.03% of that entire budget category is only $25 per month.
I think your 668GB figure is too low, because current uniques is more like 390M per month, and because some unique visitors will request the JS more than once. You can double it if you think it would help you make your case.
I don't know what that kind of bandwidth costs the foundation, but it's not free.
Developer time is not free either.
-- Tim Starling
I was hardly making a case for how amazingly expensive it was. I was running some basic calculations that seemed to support your concept of "fairly cheap", but chose to mention that it's still "not free".
- Trevor
On 9/30/10 9:30 PM, Tim Starling wrote:
On 01/10/10 04:35, Trevor Parscal wrote:
OK, now I've calculated it...
On a normal page view with the Vector skin and the Vector extension turned on there's a 2KB difference. On an edit page with the Vector skin and Vector and WikiEditor extensions there's a 4KB difference.
While adding 2KB to a request for a person in a remote corner of the world on a 56k modem will only add about 0.3 seconds to the download, sending 2,048 extra bytes to 350 million people each month increases our bandwidth by about 668 gigabytes a month.
We don't pay by volume (GB per month), we pay by bandwidth (megabits per second at the 95th percentile). They should be roughly proportional to each other, but to calculate a cost we have to convert that 668GB figure to a percentage of total volume.
I took this graph:
http://www.nedworks.org/~mark/reqstats/trafficstats-monthly.png
And I used the GIMP histogram tool to integrate the outgoing part for 30 days between week 34 and week 37. The result was 31,824 pixels of blue and 20,301 pixels of green, which I figure is about 2113 TB/month. So on your figure, the cost of adding line breaks would be about 0.03% of whatever the bandwidth bill for that month is. I don't have that number to hand, but I suspect 0.03% of it is not going to be very much. For 2009-2010 there was a budget of about $1M for "internet hosting", of which bandwidth is a part, and 0.03% of that entire budget category is only $25 per month.
I think your 668GB figure is too low, because current uniques is more like 390M per month, and because some unique visitors will request the JS more than once. You can double it if you think it would help you make your case.
I don't know what that kind of bandwidth costs the foundation, but it's not free.
Developer time is not free either.
-- Tim Starling
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
This is getting a little out of hand? People are going to spend more time talking about the potential for minification errors or sub optimisation cost criteria then we will ever actually be running into real minification errors or any real readability issue. Reasonable efforts are being made to make the non-minified versions easily accessible. We can add additional comment to every outputted file that says request me with ?debug=true to get the source code version, add a user preference, add a cookie, and a view non-minfied source link to the bottom every page...
But try to keep in mind the whole point of minification is identical program execution while minimising the file size! If we are to optimise with random adjustments for "readability" we are optimising in two opposite directions, and every enhacment in the direction of optimized pacakge code delivery could potentially go against the 'readability' optimisation. We are already commited to supporting two modes one that optimized readability with raw source code delivery and one that is optimised for small packaged delivery. No sense in setting readability as criteria for the packaged mode since the 'readability' mode will always do 'readability' better.
--michael
On 10/01/2010 12:38 AM, Trevor Parscal wrote:
I was hardly making a case for how amazingly expensive it was. I was running some basic calculations that seemed to support your concept of "fairly cheap", but chose to mention that it's still "not free".
- Trevor
On 9/30/10 9:30 PM, Tim Starling wrote:
On 01/10/10 04:35, Trevor Parscal wrote:
OK, now I've calculated it...
On a normal page view with the Vector skin and the Vector extension turned on there's a 2KB difference. On an edit page with the Vector skin and Vector and WikiEditor extensions there's a 4KB difference.
While adding 2KB to a request for a person in a remote corner of the world on a 56k modem will only add about 0.3 seconds to the download, sending 2,048 extra bytes to 350 million people each month increases our bandwidth by about 668 gigabytes a month.
We don't pay by volume (GB per month), we pay by bandwidth (megabits per second at the 95th percentile). They should be roughly proportional to each other, but to calculate a cost we have to convert that 668GB figure to a percentage of total volume.
I took this graph:
http://www.nedworks.org/~mark/reqstats/trafficstats-monthly.png
And I used the GIMP histogram tool to integrate the outgoing part for 30 days between week 34 and week 37. The result was 31,824 pixels of blue and 20,301 pixels of green, which I figure is about 2113 TB/month. So on your figure, the cost of adding line breaks would be about 0.03% of whatever the bandwidth bill for that month is. I don't have that number to hand, but I suspect 0.03% of it is not going to be very much. For 2009-2010 there was a budget of about $1M for "internet hosting", of which bandwidth is a part, and 0.03% of that entire budget category is only $25 per month.
I think your 668GB figure is too low, because current uniques is more like 390M per month, and because some unique visitors will request the JS more than once. You can double it if you think it would help you make your case.
I don't know what that kind of bandwidth costs the foundation, but it's not free.
Developer time is not free either.
-- Tim Starling
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Thank you. +1.
- Trevor
On 9/30/10 10:04 PM, Michael Dale wrote:
This is getting a little out of hand? People are going to spend more time talking about the potential for minification errors or sub optimisation cost criteria then we will ever actually be running into real minification errors or any real readability issue. Reasonable efforts are being made to make the non-minified versions easily accessible. We can add additional comment to every outputted file that says request me with ?debug=true to get the source code version, add a user preference, add a cookie, and a view non-minfied source link to the bottom every page...
But try to keep in mind the whole point of minification is identical program execution while minimising the file size! If we are to optimise with random adjustments for "readability" we are optimising in two opposite directions, and every enhacment in the direction of optimized pacakge code delivery could potentially go against the 'readability' optimisation. We are already commited to supporting two modes one that optimized readability with raw source code delivery and one that is optimised for small packaged delivery. No sense in setting readability as criteria for the packaged mode since the 'readability' mode will always do 'readability' better.
--michael
On 10/01/2010 12:38 AM, Trevor Parscal wrote:
I was hardly making a case for how amazingly expensive it was. I was running some basic calculations that seemed to support your concept of "fairly cheap", but chose to mention that it's still "not free".
- Trevor
On 9/30/10 9:30 PM, Tim Starling wrote:
On 01/10/10 04:35, Trevor Parscal wrote:
OK, now I've calculated it...
On a normal page view with the Vector skin and the Vector extension turned on there's a 2KB difference. On an edit page with the Vector skin and Vector and WikiEditor extensions there's a 4KB difference.
While adding 2KB to a request for a person in a remote corner of the world on a 56k modem will only add about 0.3 seconds to the download, sending 2,048 extra bytes to 350 million people each month increases our bandwidth by about 668 gigabytes a month.
We don't pay by volume (GB per month), we pay by bandwidth (megabits per second at the 95th percentile). They should be roughly proportional to each other, but to calculate a cost we have to convert that 668GB figure to a percentage of total volume.
I took this graph:
http://www.nedworks.org/~mark/reqstats/trafficstats-monthly.png
And I used the GIMP histogram tool to integrate the outgoing part for 30 days between week 34 and week 37. The result was 31,824 pixels of blue and 20,301 pixels of green, which I figure is about 2113 TB/month. So on your figure, the cost of adding line breaks would be about 0.03% of whatever the bandwidth bill for that month is. I don't have that number to hand, but I suspect 0.03% of it is not going to be very much. For 2009-2010 there was a budget of about $1M for "internet hosting", of which bandwidth is a part, and 0.03% of that entire budget category is only $25 per month.
I think your 668GB figure is too low, because current uniques is more like 390M per month, and because some unique visitors will request the JS more than once. You can double it if you think it would help you make your case.
I don't know what that kind of bandwidth costs the foundation, but it's not free.
Developer time is not free either.
-- Tim Starling
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Thu, Sep 30, 2010 at 8:27 AM, Trevor Parscal tparscal@wikimedia.org wrote:
... When debug mode is off:
* Modules are requested in batches * Resources are combined into modules * Modules are combined into a response * The response is minified
When debug mode is on:
* Modules are requested individually * Resources are combined into modules
There are two features here. modules optimising traffic
It seems everyone realises that the serving of raw files has to remain.
Why is this old and remaining functionality going to be named 'debug mode'?
Why not give the new feature a name, put it in the prefs, and turn that pref on by default after sufficient road testing. As happened with the new skin.
There are going to be mediawiki implementers and wikimedia users who want to disable this feature for reasons other than debugging.
(I've always considered the 'Misc' section of the prefs to be 'Performance')
-- John Vandenberg
Last night I made a change to how ResourceLoader's debug mode works; one based on an idea which has been hoped for since the inception of the project but had not yet been fleshed out well enough to implement. It's for the most part an implementation detail, but it changes something I posted on this list so I wanted to provide an update.
On 9/29/10 3:27 PM, Trevor Parscal wrote:
When a request is made for one or more modules, each resource is packaged together and sent back to the client as a response. The way in which these requests and responses are performed depends on whether debug is on or off.
This now happens, no matter what the debug mode is.
When debug mode is off:
* Modules are requested in batches * Resources are combined into modules * Modules are combined into a response * The response is minified
Production mode (debug=true) is unchanged.
When debug mode is on:
* Modules are requested individually * Resources are combined into modules
Development mode (debug=true) has changed a bit. Modules are still batched, but now the code from ResourceLoaderFileModule-based modules is sent in the form of a document.write( '<script src="path/to/raw/file.js"></script>' ) instead of being embedded directly. This means that we can take advantage of the loading system, but also have direct file includes, making 100% line/character matching with the original source files (because you ARE loading the original source files).
Thanks to Michael Dale for helping me sort out the best way to implement this.
I think it's debatable whether debug=true mode goes far enough, since it still combines resources into modules, and I am open to contributions that can make debug=true mode even more debugging friendly by delivering the resources to the client as unchanged as possible. I also think it's debatable if debug=false mode goes far enough, since things like Google Closure Compiler have been proven to even further reduce the size of JavaScript resources, so I am also open to contributions which can make debug=false even more production friendly by improving front-end performance.
This is still my position - in r75137 I took development mode (debug=true) a step further by making it easier to understand where the the client has received came from on the server, and supporting tools that can connect a browser to an IDE by thus preserving line-numbers and character-offsets. I am considering ways to continue to make improvements to how easy we can make development mode for developers - ideas are welcome, commits and patches are appreciated.
I also still feel strongly that we should not hold back on making production mode (debug=false) as optimized as possible. http://www.mediawiki.org/wiki/Special:Code/MediaWiki/75137
If you are a developer, working on your localhost, you probably want to code with...
$wgResourceLoaderDebug = true;
.. and then test that things work in debug=false mode before committing your code. This will result in more requests but less processing, which will be much faster when developing on localhost.
More information about how ResourceLoader will affect your development process, and how to take control of it has been added to MediaWiki.org:
http://www.mediawiki.org/wiki/ResourceLoader/Documentation/Developing_with_R...
I would also like to just take a second to thank everyone, staff and volunteers alike, for all of the patience and support you have all shown Roan and I while we have been developing ResourceLoader. We really believe that this is a big step forward for MediaWiki towards making it a modern wiki platform, and look forward to seeing it in action on high and low traffic wikis alike.
If you are wondering what's next for ResourceLoader, take a peek at the task list on MediaWiki.org:
http://www.mediawiki.org/wiki/ResourceLoader/Status#Task_List
- Trevor
W dniu 2010-10-21 20:01, Trevor Parscal pisze:
Last night I made a change to how ResourceLoader's debug mode works; one based on an idea which has been hoped for since the inception of the project but had not yet been fleshed out well enough to implement. It's for the most part an implementation detail, but it changes something I posted on this list so I wanted to provide an update.
On 9/29/10 3:27 PM, Trevor Parscal wrote:
When a request is made for one or more modules, each resource is packaged together and sent back to the client as a response. The way in which these requests and responses are performed depends on whether debug is on or off.
This now happens, no matter what the debug mode is.
When debug mode is off:
* Modules are requested in batches * Resources are combined into modules * Modules are combined into a response * The response is minifiedProduction mode (debug=true) is unchanged.
When debug mode is on:
* Modules are requested individually * Resources are combined into modulesDevelopment mode (debug=true) has changed a bit. Modules are still batched, but now the code from ResourceLoaderFileModule-based modules is sent in the form of a document.write( '<script src="path/to/raw/file.js"></script>' ) instead of being embedded directly. This means that we can take advantage of the loading system, but also have direct file includes, making 100% line/character matching with the original source files (because you ARE loading the original source files).
Thanks to Michael Dale for helping me sort out the best way to implement this.
I think it's debatable whether debug=true mode goes far enough, since it still combines resources into modules, and I am open to contributions that can make debug=true mode even more debugging friendly by delivering the resources to the client as unchanged as possible. I also think it's debatable if debug=false mode goes far enough, since things like Google Closure Compiler have been proven to even further reduce the size of JavaScript resources, so I am also open to contributions which can make debug=false even more production friendly by improving front-end performance.
This is still my position - in r75137 I took development mode (debug=true) a step further by making it easier to understand where the the client has received came from on the server, and supporting tools that can connect a browser to an IDE by thus preserving line-numbers and character-offsets. I am considering ways to continue to make improvements to how easy we can make development mode for developers - ideas are welcome, commits and patches are appreciated.
I also still feel strongly that we should not hold back on making production mode (debug=false) as optimized as possible. http://www.mediawiki.org/wiki/Special:Code/MediaWiki/75137
If you are a developer, working on your localhost, you probably want to code with...
$wgResourceLoaderDebug = true;
.. and then test that things work in debug=false mode before committing your code. This will result in more requests but less processing, which will be much faster when developing on localhost.
More information about how ResourceLoader will affect your development process, and how to take control of it has been added to MediaWiki.org:
http://www.mediawiki.org/wiki/ResourceLoader/Documentation/Developing_with_R...
I would also like to just take a second to thank everyone, staff and volunteers alike, for all of the patience and support you have all shown Roan and I while we have been developing ResourceLoader. We really believe that this is a big step forward for MediaWiki towards making it a modern wiki platform, and look forward to seeing it in action on high and low traffic wikis alike.
If you are wondering what's next for ResourceLoader, take a peek at the task list on MediaWiki.org:
http://www.mediawiki.org/wiki/ResourceLoader/Status#Task_List
What about setting a cookie to go into a debug mode? I thought you liked the idea?
Regards, Nux.
@2010-10-22 00:53, Trevor Parscal:
On 10/21/10 3:39 PM, Maciej Jaros wrote:
What about setting a cookie to go into a debug mode? I thought you liked the idea?
I have no problem with that, it's just not implemented yet. Want to help?
Sure. I see new line characters are there, too. Cool!
Nux.
wikitech-l@lists.wikimedia.org