Bandwidth usage

List overview All Threads
Download

newer

older

Semi-protection: require edit...

Fwd: [Foundation-l] Mail archives

Brion Vibber

7 Feb 2007 7 Feb '07

1:17 a.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I took a quick peek at the sampled squid log and found that CSS and JS files together are eating a lot of bandwidth; together they make up about 20% of what's served: https://wikitech.leuksman.com/view/Squid_bandwidth_breakdown (May be inaccurate due to coding mistakes in my counter or weird dupe caching effects.) It should be possible to serve these files compressed through Apache with mod_gzip set up, which should squish them by probably 2/3. - -- brion vibber (brion @ pobox.com / brion @ wikimedia.org)

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFySi4wRnhpk1wk44RAm1tAJ9vgpvqYBvN+l6sX6whsyaGVlw26gCgkD2E VrBaaLtgtVSwLxU5zA/gbKE= =8Bky -----END PGP SIGNATURE-----

Show replies by date

Nick Jenkins

7 Feb 7 Feb

4:14 a.m.

...

I took a quick peek at the sampled squid log and found that CSS and JS files together are eating a lot of bandwidth; together they make up about 20% of what's served: https://wikitech.leuksman.com/view/Squid_bandwidth_breakdown (May be inaccurate due to coding mistakes in my counter or weird dupe caching effects.)

Please forgive me if this is a dumb question, but if I check the headers returned for two successive requests, like so: ====================================================== root@bling:/var/www/hosts/mediawiki/wiki# curl --silent --include --head http://en.wikipedia.org/skins-1.5/monobook/main.css?55 HTTP/1.0 200 OK Date: Wed, 07 Feb 2007 02:56:31 GMT Server: Apache Cache-Control: max-age=2592000 Expires: Fri, 09 Mar 2007 02:56:31 GMT Last-Modified: Tue, 06 Feb 2007 20:04:40 GMT ETag: "60874b-709d-45c8df58" Accept-Ranges: bytes Content-Length: 28829 Content-Type: text/css Age: 2 X-Cache: HIT from sq30.wikimedia.org X-Cache-Lookup: HIT from sq30.wikimedia.org:80 Via: 1.0 sq30.wikimedia.org:80 (squid/2.6.STABLE9) Connection: close root@bling:/var/www/hosts/mediawiki/wiki# curl --silent --include --head http://en.wikipedia.org/skins-1.5/monobook/main.css?55 HTTP/1.0 200 OK Date: Wed, 07 Feb 2007 02:56:26 GMT Server: Apache Cache-Control: max-age=2592000 Expires: Fri, 09 Mar 2007 02:56:26 GMT Last-Modified: Tue, 06 Feb 2007 20:04:40 GMT ETag: "15c02de-709d-45c8df58" Accept-Ranges: bytes Content-Length: 28829 Content-Type: text/css Age: 9 X-Cache: HIT from sq20.wikimedia.org X-Cache-Lookup: HIT from sq20.wikimedia.org:80 Via: 1.0 sq20.wikimedia.org:80 (squid/2.6.STABLE9) Connection: close root@bling:/var/www/hosts/mediawiki/wiki# ====================================================== ... then I have two questions: 1) Does it matter that the ETag varies between successive requests? Reason I ask is that the http://www.web-caching.com/mnot_tutorial/how.html page says: "HTTP 1.1 introduced a new kind of validator called the ETag. ETags are unique identifiers that are generated by the server and changed every time the object does. Because the server controls how the ETag is generated, caches can be surer that if the ETag matches when they make a If-None-Match request, the object really is the same." I.e. if the ETag changes between requests, as it did in the above example, could that make requesters think that the object has changed too, thus reducing caching? 2) Would it help using "Cache-Control: max-age=2592000, public" instead of "Cache-Control: max-age=2592000" ? Public is defined as "marks the response as cacheable, even if it would normally be uncacheable. For instance, if your pages are authenticated, the public directive makes them cacheable." I.e. I'm not sure if the Wikipedia cookie is being treated as authentication for the purposes of this definition, but if it is, caching the site-wide CSS or JS seems unlikely to hurt (since it really is "public") - but obviously caching the user-specific CSS or JS would be bad.

...

It should be possible to serve these files compressed through Apache with mod_gzip set up, which should squish them by probably 2/3.

Last time I looked mod_gzip seemed to be losing favour somewhat - the new "in" compression method for apache2 seems to be mod_deflate ( http://httpd.apache.org/docs/2.0/mod/mod_deflate.html ), partially because it's an apache module, rather than a 3rd-party module; however it got a few percentage points less efficient compression than mod_gzip, but the suggestion was that it caused less CPU load to do its compression. It was about a year ago that I looked at this stuff though, so the state of play may have changed since. All the best, Nick.

Nick Jenkins

5:37 a.m.

...

ETag: "60874b-709d-45c8df58" [...snip...] ETag: "15c02de-709d-45c8df58" [...snip...] 1) Does it matter that the ETag varies between successive requests? Reason I ask is that the http://www.web-caching.com/mnot_tutorial/how.html page says: "HTTP 1.1 introduced a new kind of validator called the ETag. ETags are unique identifiers that are generated by the server and changed every time the object does. Because the server controls how the ETag is generated, caches can be surer that if the ETag matches when they make a If-None-Match request, the object really is the same." I.e. if the ETag changes between requests, as it did in the above example, could that make requesters think that the object has changed too, thus reducing caching?

Just found how to stop this - The ETag comes from 3 sources of information - the inode, the file size, and the file modified time. That's why these two are different yet similar:

...

ETag: "60874b-709d-45c8df58" ETag: "15c02de-709d-45c8df58"

I.e. the format currently is ETag: "<inode value>-<date value>-<file size value>". [which is the apache2 default, I think]. The inodes differ for the different systems, but the date and the file values are the same, as you can see. The solution therefore is to add this line to the cluster's apache2.conf: ------------------------- FileETag MTime Size ------------------------- Then the result will be: ETag: "709d-45c8df58" for all boxes serving this example file. All the best, Nick.

Matthew Flaschen

5:54 a.m.

Nick Jenkins wrote:

...

I'm not sure if the Wikipedia cookie is being treated as authentication for the purposes of this definition, but if it is, caching the site-wide CSS or JS seems unlikely to hurt (since it really is "public") - but obviously caching the user-specific CSS or JS would be bad.

Why? User-specific CSS and JS is still public. Matthew Flaschen

Nick Jenkins

7:14 a.m.

...

Why? User-specific CSS and JS is still public.

That's true - I had assumed it was private (not that I've ever put anything there that I particularly cared about being private) - but you're correct, it's not ... which in turn raises another question: ------------------------------------------------------------------- root@bling:/var/www/hosts/mediawiki/wiki/skins# curl --silent --include --head "http://en.wikipedia.org/w/index.php?title=User:Nickj/monobook.js&action=raw&ctype=text/javascript" HTTP/1.0 200 OK Date: Wed, 07 Feb 2007 06:17:28 GMT Server: Apache X-Powered-By: PHP/5.1.2 Cache-Control: private, s-maxage=0, max-age=2678400 Last-Modified: Wed, 31 Jan 2007 05:56:29 GMT Content-Type: text/javascript; charset=utf-8 X-Cache: MISS from sq16.wikimedia.org X-Cache-Lookup: MISS from sq16.wikimedia.org:80 Via: 1.0 sq16.wikimedia.org:80 (squid/2.6.STABLE9) Connection: close root@bling:/var/www/hosts/mediawiki/wiki/skins# curl --silent --include --head "http://en.wikipedia.org/w/index.php?title=User:Nickj/monobook.js&action=raw&ctype=text/javascript" [ ... snip 3 more tests and responses from servers that's aren't sq16.wikimedia.org ... ] root@bling:/var/www/hosts/mediawiki/wiki/skins# curl --silent --include --head "http://en.wikipedia.org/w/index.php?title=User:Nickj/monobook.js&action=raw&ctype=text/javascript" HTTP/1.0 200 OK Date: Wed, 07 Feb 2007 06:17:41 GMT Server: Apache X-Powered-By: PHP/5.1.2 Cache-Control: private, s-maxage=0, max-age=2678400 Last-Modified: Wed, 31 Jan 2007 05:56:29 GMT Content-Type: text/javascript; charset=utf-8 X-Cache: MISS from sq16.wikimedia.org X-Cache-Lookup: MISS from sq16.wikimedia.org:80 Via: 1.0 sq16.wikimedia.org:80 (squid/2.6.STABLE9) Connection: close ------------------------------------------------------------------- So, the question is: Why is the cache-control line for raw custom JS and CSS marked as "private" ?

...

From http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9.1 , it defines the

"private" directive as: ------------------------------- private Indicates that all or part of the response message is intended for a single user and MUST NOT be cached by a shared cache. This allows an origin server to state that the specified parts of the response are intended for only one user and are not a valid response for requests by other users. A private (non-shared) cache MAY cache the response. ------------------------------- So, if the web page is public anyway, and people *do* use other people's custom CSS & JS (as I'm doing), why mark it as private when action=raw, and thus defeat shared caching? All the best, Nick.

Matthew Flaschen

9:37 a.m.

Nick Jenkins wrote:

...

Why? User-specific CSS and JS is still public.

That's true - I had assumed it was private (not that I've ever put anything there that I particularly cared about being private) - but you're correct, it's not ... which in turn raises another question: root@bling:/var/www/hosts/mediawiki/wiki/skins# curl --silent --include --head "http://en.wikipedia.org/w/index.php?title=User:Nickj/monobook.js&action=raw&ctype=text/javascript" HTTP/1.0 200 OK Date: Wed, 07 Feb 2007 06:17:41 GMT Server: Apache X-Powered-By: PHP/5.1.2 Cache-Control: private, s-maxage=0, max-age=2678400 Last-Modified: Wed, 31 Jan 2007 05:56:29 GMT Content-Type: text/javascript; charset=utf-8 X-Cache: MISS from sq16.wikimedia.org X-Cache-Lookup: MISS from sq16.wikimedia.org:80 Via: 1.0 sq16.wikimedia.org:80 (squid/2.6.STABLE9) Connection: close ------------------------------------------------------------------- So, the question is: Why is the cache-control line for raw custom JS and CSS marked as "private" ?

From http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.9.1 , it defines the

I have no idea. I suspect with widely used scripts like popups that could significantly degrade performance. P.S. To clarify, user css and js can't be *edited* by other users (admins can edit everything) Matthew Flaschen

Matthew Flaschen

5:55 a.m.

Brion Vibber wrote:

...

Is there a reason everything shouldn't be compressed? Matthew Flaschen

jf＠mormo.org

7:14 a.m.

On Wed, Feb 07, 2007 at 12:55:35AM -0500, Matthew Flaschen wrote:

...

Brion Vibber wrote:

It should be possible to serve these files compressed through Apache with mod_gzip set up, which should squish them by probably 2/3.

Is there a reason everything shouldn't be compressed?

MediaWiki already compresses all wiki pages. But static files are being served from apache directly and thus aren't compressed currently. jens

jf＠mormo.org

7:19 a.m.

On Tue, Feb 06, 2007 at 05:17:44PM -0800, Brion Vibber wrote:

...

All projects have the same codebase, so that main.css is the same for de.wikipedia.org and en.wikipedia.org. Can we move the common CSS and JS to a common address? This would reduce the number of identical copies that we store in the squids - and of course it would reduce the load time for users accessing different projects. Regards, jens

Brion Vibber

5:37 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jens Frank wrote:

...

All projects have the same codebase, so that main.css is the same for de.wikipedia.org and en.wikipedia.org. Can we move the common CSS and JS to a common address?

In theory. 1) Where? Upload cluster or something else? 2) How to set up compression for it? - -- brion vibber (brion @ pobox.com / brion @ wikimedia.org)

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFyg5awRnhpk1wk44RAhPYAKDTGT4+aTT4RIvEd9CCPvZIk6tgygCfWHeV t5DYPEBF9DXhm1qCIsy9n90= =qfBD -----END PGP SIGNATURE-----

Evan Martin

9:16 p.m.

On 2/6/07, Brion Vibber <brion(a)pobox.com> wrote:

...

Another option (which won't require more sysadminning or server CPU) is to compress them "manually" by stripping all whitespace and comments. For example, looking at http://en.wikipedia.org/skins-1.5/monobook/main.css?55 it appears that comments and whitespace comprise more than 20% of the bytes in the file:

...

maincss.gsub(/\/\*.*?\*\//m, '').gsub(/\s{2,}/, ' ').length / maincss.length.to_f

=> 0.79468590655243 You could likely do better with a real CSS parser.

Mark Clements

9:38 p.m.

"Evan Martin" <evanm(a)google.com> wrote in message news:9f43d19d0702071316k29d43f31y3a2cfe8a18d7859d@mail.gmail.com...

...

On 2/6/07, Brion Vibber <brion(a)pobox.com>

wrote:

...

maincss.length.to_f

...

=> 0.79468590655243 You could likely do better with a real CSS parser.

However, that will make debugging very hard indeed. Not sure how big a consideration that is, but I know that I have often looked into the MW stylesheets to see how particular stylings are acheived when making my own css rules. - Mark Clements (HappyDog)

Tels

9:34 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Moin, On Wednesday 07 February 2007 22:38:48 Mark Clements wrote:

...

"Evan Martin" <evanm(a)google.com> wrote in message news:9f43d19d0702071316k29d43f31y3a2cfe8a18d7859d@mail.gmail.com...

On 2/6/07, Brion Vibber <brion(a)pobox.com>

wrote:

[snip]

...

For example, looking at http://en.wikipedia.org/skins-1.5/monobook/main.css?55 it appears that comments and whitespace comprise more than 20% of the bytes in the file: > maincss.gsub(/\/\*.*?\*\//m, '').gsub(/\s{2,}/, ' ').length /

maincss.length.to_f

=> 0.79468590655243 You could likely do better with a real CSS parser.

But shouldn't be a CSS-tidy being able to parse the file and spit it out nicely formatted again? best wishes, tels - -- Signed on Wed Feb 7 22:33:58 2007 with key 0x93B84C15. View my photo gallery: http://bloodgate.com/photos PGP key on http://bloodgate.com/tels.asc or per email. "I have always wished that my computer would be as easy to use as my telephone. My wish has come true. I no longer know how to use my telephone." - Bjarne Stroustrup, inventor of the C++ programming language

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (GNU/Linux) iQEVAwUBRcpF+3cLPEOTuEwVAQLazgf+ObEtFUhNB/KYdcAANcV4KoW7G29by85L 1wERkBtRiHYHe2IWhMuNEkTVj1055qg0cF82O7eYIo7Iq90TDxTjpPvkgtCXSwe6 1peSIKFREffyW8Af0M769M+4GuQsxDj5kdQ8E/UDPKyriESxE1k3bOrFdb2hAh+T tj/sl4MsZZpmL6pAUmb/V11pNSSRxXpC+htTY/GF2d/ta19NBxk0NMA2BnYfyeYs wQCp57hPFP2AbaOOdBD7PsWi2HKqA1C2sRYjb2NZ2NsVfB5QcNb+BgNTrfQWsUqa m5ANDYE5IsjfwXrW9fhCkPVs/bajaKF8l8o/TYqdNlDyAJgueClP/A== =bLwo -----END PGP SIGNATURE-----

Mark Clements

10:08 p.m.

"Tels" <nospam-abuse(a)bloodgate.com> wrote in message news:200702072234.51476@bloodgate.com...

...

On Wednesday 07 February 2007 22:38:48 Mark Clements wrote:

But shouldn't be a CSS-tidy being able to parse the file and spit it out nicely formatted again?

That's not what was being suggested. The suggestion was to remove _all_ whitespace and all comments. - Mark Clements (HappyDog)

Mets501

9:49 p.m.

...

Just from a quick check of http://en.wikipedia.org/skins-1.5/monobook/main.css?55, removing all the possible white space will reduce it in size from 30882 bytes to 23411 bytes, saving 7471 bytes (24%). That's a drastic change in size, and if done would save a ton of bandwidth (over time). If you're concerned about readability, perhaps two files are warranted, a "readable" file and a "usable" file, and the readable one could be linked from a comment at the top of the usable one.

Brion Vibber

10:47 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Mets501 wrote:

...

It's small compared to actual compression, though. - -- brion vibber (brion @ pobox.com / brion @ wikimedia.org)

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFylb6wRnhpk1wk44RAqLrAJ9y0oQaPkcfZt7KQvv1cjuU6fVolQCgtXOa UnHB8274HVPUB1dBVH+mBFg= =3CbN -----END PGP SIGNATURE-----

Mets501

10:50 p.m.

...

> Just from a quick check of > http://en.wikipedia.org/skins-1.5/monobook/main.css?55, removing all the > possible white space will reduce it in size from 30882 bytes to 23411

bytes,

...

> saving 7471 bytes (24%). That's a drastic change in size, and if done

would

...

save a ton of bandwidth (over time).

It's small compared to actual compression, though.

...

-- brion vibber (brion @ pobox.com / brion @ wikimedia.org)

Yes, but it may be a good temporary measure. --Mets501

Gregory Maxwell

10:56 p.m.

On 2/7/07, Mets501 <mets501wiki(a)gmail.com> wrote:

...

Yes, but it may be a good temporary measure. --Mets501

We've survived this long... Has anyone tried to figure out why it seems on average users load only slightly more than two pages for every time they load css? Is really so much of our traffic "one click from a search engine and then depart" that it is shifting so much of our hits towards CSS loads? ... if so, it may be an indication that something is wrong with our service from the perspective of our readers. (Perhaps they are so used to seeing spam with Wikipedia content that they just hit "back". :) )

Mets501

11:01 p.m.

...

I think most people just search google, and then encounter Wikipedia for one or two pages. I see my friends working in the library or other places doing research, and they search first on Google, click the first or second result (which is often Wikipedia), read that page, and then move on to other sites. --Mets501

Rob Church

10:58 p.m.

On 07/02/07, Mets501 <mets501wiki(a)gmail.com> wrote:

...

Yes, but it may be a good temporary measure.

Well, no, not if it takes us more effort to integrate this whitespace stripping into the software than it does to configure things so that this stuff is compressed properly. Rob Church

Mets501

11:03 p.m.

...

Yes, but it may be a good temporary measure.

Well, no, not if it takes us more effort to integrate this whitespace stripping into the software than it does to configure things so that this stuff is compressed properly. Rob Church

I actually wasn't thinking of working it into the software, but rather just removing the white spaces in the main monobook css file (which is all most users load anyway). --Mets501

Rob Church

11:04 p.m.

On 07/02/07, Mets501 <mets501wiki(a)gmail.com> wrote:

...

I actually wasn't thinking of working it into the software, but rather just removing the white spaces in the main monobook css file (which is all most users load anyway).

Well, from a developer's point of view, that's not actually acceptable, because we have to be able to maintain the damn file. Rob Church

Mets501

11:10 p.m.

...

> I actually wasn't thinking of working it into the software, but rather

just

...

> removing the white spaces in the main monobook css file (which is all

most

...

users load anyway).

Well, from a developer's point of view, that's not actually acceptable, because we have to be able to maintain the damn file. Rob Church

Listen, it was just a suggestion. If it's not acceptable, no problem. You're the programming professional, I'm not. --Mets501

Nick Jenkins

8 Feb 8 Feb

3:48 a.m.

...

I actually wasn't thinking of working it into the software, but rather just removing the white spaces in the main monobook css file (which is all most users load anyway).

... Yeah, and the rest! ;-) -------------------------------- root@bling:~/tmp# curl --silent --include "http://en.wikipedia.org/wiki/Main_Page" | grep -i "css" <style type="text/css" media="screen,projection">/*<![CDATA[*/ @import "/skins-1.5/monobook/main.css?55"; /*]]>*/</style> <link rel="stylesheet" type="text/css" media="print" href="/skins-1.5/common/commonPrint.css?55" /> <link rel="stylesheet" type="text/css" media="handheld" href="/skins-1.5/monobook/handheld.css?55" />     <style type="text/css">/*<![CDATA[*/ @import "/w/index.php?title=MediaWiki:Common.css&usemsgcache=yes&action=raw&ctype=text/css&smaxage=2678400"; @import "/w/index.php?title=MediaWiki:Monobook.css&usemsgcache=yes&action=raw&ctype=text/css&smaxage=2678400"; @import "/w/index.php?title=-&action=raw&gen=css&maxage=2678400"; root@bling:~/tmp# -------------------------------- That's six CSS files for most desktop browsers, and seven if you're using IE. All the best, Nick.

Simetrical

3:42 a.m.

On 2/7/07, Nick Jenkins <nickpj(a)gmail.com> wrote:

...

That's six CSS files for most desktop browsers, and seven if you're using IE.

Actually just four and five, respectively. The media-specific stylesheets will probably not be loaded by clients that aren't using that medium.

Nick Jenkins

5:45 a.m.

...

That's six CSS files for most desktop browsers, and seven if you're using IE.

Actually just four and five, respectively. The media-specific stylesheets will probably not be loaded by clients that aren't using that medium.

I can only speak to what Firefox and IE are actually doing. My proxy logs for doing a shift-F5 refresh on a local test wiki install include the following get/response headers for Firefox: ======================================================== +++GET 1595+++ GET /wiki/skins/monobook/main.css?55 HTTP/1.1 Host: 192.168.0.64 User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1 Accept: text/css,*/*;q=0.1 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Referer: http://192.168.0.64/wiki/index.php/Test Cookie: wikidbUserName=Nickj; wikidb_session=178df0fe68c75834643af65dec9ec98a; wikidbLoggedOut=20070208020154 Pragma: no-cache Cache-Control: no-cache Connection: keep-alive Browser reload detected... +++RESP 1595+++ HTTP/1.1 200 OK Date: Thu, 08 Feb 2007 05:02:41 GMT Server: Apache Last-Modified: Mon, 05 Feb 2007 23:57:09 GMT ETag: "709d-428c37389ff40" Accept-Ranges: bytes Vary: Accept-Encoding,User-Agent Content-Encoding: gzip Cache-Control: max-age=2592000, public Content-Length: 7763 Content-Type: text/css +++CLOSE 1595+++ BlockList 1596: in Bypass, line 69 +++GET 1596+++ GET /wiki/skins/common/commonPrint.css?55 HTTP/1.1 Host: 192.168.0.64 User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1 Accept: text/css,*/*;q=0.1 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Referer: http://192.168.0.64/wiki/index.php/Test Cookie: wikidbUserName=Nickj; wikidb_session=178df0fe68c75834643af65dec9ec98a; wikidbLoggedOut=20070208020154 Pragma: no-cache Cache-Control: no-cache Connection: keep-alive Browser reload detected... +++RESP 1596+++ HTTP/1.1 200 OK Date: Thu, 08 Feb 2007 05:02:41 GMT Server: Apache Last-Modified: Mon, 05 Feb 2007 23:57:09 GMT ETag: "1479-428c37389ff40" Accept-Ranges: bytes Vary: Accept-Encoding,User-Agent Content-Encoding: gzip Cache-Control: max-age=2592000, public Content-Length: 1731 Content-Type: text/css +++CLOSE 1596+++ BlockList 1597: in Bypass, line 69 +++GET 1597+++ GET /wiki/skins/monobook/handheld.css?55 HTTP/1.1 Host: 192.168.0.64 User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1 Accept: text/css,*/*;q=0.1 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Referer: http://192.168.0.64/wiki/index.php/Test Cookie: wikidbUserName=Nickj; wikidb_session=178df0fe68c75834643af65dec9ec98a; wikidbLoggedOut=20070208020154 Pragma: no-cache Cache-Control: no-cache Connection: keep-alive Browser reload detected... +++RESP 1597+++ HTTP/1.1 200 OK Date: Thu, 08 Feb 2007 05:02:41 GMT Server: Apache Last-Modified: Mon, 05 Feb 2007 23:57:09 GMT ETag: "5bc3-428c37389ff40" Accept-Ranges: bytes Vary: Accept-Encoding,User-Agent Content-Encoding: gzip Cache-Control: max-age=2592000, public Content-Length: 6631 Content-Type: text/css +++CLOSE 1597+++ BlockList 1598: in Bypass, line 69 ======================================================== ... which is also reflected in the apache logs: ======================================================== 192.168.0.2 - - [08/Feb/2007:16:20:38 +1100] "GET /wiki/skins/monobook/main.css?55 HTTP/1.1" 200 7763 "http://192.168.0.64/wiki/index.php/Test" "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1" 192.168.0.2 - - [08/Feb/2007:16:20:39 +1100] "GET /wiki/skins/common/commonPrint.css?55 HTTP/1.1" 200 1731 "http://192.168.0.64/wiki/index.php/Test" "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1" 192.168.0.2 - - [08/Feb/2007:16:20:39 +1100] "GET /wiki/skins/monobook/handheld.css?55 HTTP/1.1" 200 6631 "http://192.168.0.64/wiki/index.php/Test" "Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1" ======================================================== Similarly for IE 5.01: ======================================================== +++GET 1610+++ GET /wiki/skins/common/commonPrint.css?55 HTTP/1.1 Accept: */* Referer: http://192.168.0.64/wiki/index.php/Test Accept-Language: en-au Accept-Encoding: gzip, deflate User-Agent: Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; .NET CLR 1.1.4322) Host: 192.168.0.64 Pragma: no-cache Connection: keep-alive +++GET 1611+++ GET /wiki/skins/monobook/main.css?55 HTTP/1.1 Accept: */* Referer: http://192.168.0.64/wiki/index.php/Test Accept-Language: en-au Accept-Encoding: gzip, deflate User-Agent: Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; .NET CLR 1.1.4322) Host: 192.168.0.64 Pragma: no-cache Connection: keep-alive BlockList 1612: in Bypass, line 69 Browser reload detected... +++GET 1612+++ GET /wiki/skins/monobook/handheld.css?55 HTTP/1.1 Accept: */* Referer: http://192.168.0.64/wiki/index.php/Test Accept-Language: en-au Accept-Encoding: gzip, deflate User-Agent: Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; .NET CLR 1.1.4322) Host: 192.168.0.64 Pragma: no-cache Connection: keep-alive ======================================================== ... which is also reflected in the apache logs: ======================================================== 192.168.0.2 - - [08/Feb/2007:16:21:53 +1100] "GET /wiki/skins/common/commonPrint.css?55 HTTP/1.1" 200 1731 "http://192.168.0.64/wiki/index.php/Main_Page" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; .NET CLR 1.1.4322)" 192.168.0.2 - - [08/Feb/2007:16:21:53 +1100] "GET /wiki/skins/monobook/handheld.css?55 HTTP/1.1" 200 6631 "http://192.168.0.64/wiki/index.php/Main_Page" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; .NET CLR 1.1.4322)" 192.168.0.2 - - [08/Feb/2007:16:21:53 +1100] "GET /wiki/skins/common/IEFixes.js?55 HTTP/1.1" 200 1344 "http://192.168.0.64/wiki/index.php/Main_Page" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; .NET CLR 1.1.4322)" 192.168.0.2 - - [08/Feb/2007:16:21:53 +1100] "GET /wiki/skins/monobook/main.css?55 HTTP/1.1" 200 7763 "http://192.168.0.64/wiki/index.php/Main_Page" "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0; .NET CLR 1.1.4322)" ======================================================== I.e. both browsers fetch all 3 media variant stylesheets, at least on a full refresh. ... and exactly why both browsers feel the need to load a media="handheld" stylesheet whilst running on a distinctly non-handheld device is a question that's probably best directed at the authors of those browsers ;-) All the best, Nick.

Tels

6:23 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Moin, On Thursday 08 February 2007 06:45:07 Nick Jenkins wrote:

...

That's six CSS files for most desktop browsers, and seven if you're using IE.

Actually just four and five, respectively. The media-specific stylesheets will probably not be loaded by clients that aren't using that medium.

I can only speak to what Firefox and IE are actually doing.

[snip]

...

I.e. both browsers fetch all 3 media variant stylesheets, at least on a full refresh. ... and exactly why both browsers feel the need to load a media="handheld" stylesheet whilst running on a distinctly non-handheld device is a question that's probably best directed at the authors of those browsers ;-)

Have you filed bug reports with them? :) best wishes, tels - -- Signed on Thu Feb 8 19:22:58 2007 with key 0x93B84C15. View my photo gallery: http://bloodgate.com/photos PGP key on http://bloodgate.com/tels.asc or per email. "HOT PACKET ON SERVER ACTION! Click here for FREE ACCESS to streaming video of dirty packets penetrating badly-configured firewalls!!!" aanand (705284) on 2004-03-20 on /. about the Adult Bit and the Evil bit

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (GNU/Linux) iQEVAwUBRctqnHcLPEOTuEwVAQJp9wf/QD+3jQVn5qysoN8CTv8q5gBcnip7ApMO Xmuiynbk0r9xZeaDVed7rpL0xiOyi3WvMuGtIPT9PtLa7cCKIk8saEnTGUvhlx7q 3vOqrLgwagX8oc90LaRc3KXTZqpY1CZvuLFph+tTUWqR152VfsO/18ivyVDsuPrf bkjx1mw+KKQFerHfZIcYJOG6gd1P8UW1nsImEogHoAO+MsstxeeVG1fPNGAOz18S rdkWI8NGq8Dppce9Nb3NIqCiCTRVC/GuEtyFto/WPG3Lb8B59Mt1lhfYf4gzn/Mk tHDh4tCPopWjBtwUMIlUykNaWdpaeosN2kFHq/OBhn/GQ85apceI7g== =JnOc -----END PGP SIGNATURE-----

Nick Jenkins

11:25 p.m.

...

That's six CSS files for most desktop browsers, and seven if you're using IE.

Actually just four and five, respectively. The media-specific stylesheets will probably not be loaded by clients that aren't using that medium.

I can only speak to what Firefox and IE are actually doing.

[snip]

Have you filed bug reports with them? :)

For Firefox: See: "Bug 79021 - All stylesheets are loaded regardless of rendering media" https://bugzilla.mozilla.org/show_bug.cgi?id=79021 [Open 5.5 years, no activity in past 3.5 years] For IE: Don't care - reporting bugs to MS is an exercise in pain and frustration, involving having to provide your credit card details to do their QA for them on their products (and then they say "sorry, we can't help" - completely useless), or if you use the web feedback forms they just ignore you for 9 years (at which point they discontinue the product) even though it was clearly a 1-line trivial fix ... or at least these have been *my* experiences reporting bugs to MS. And as the piece-de-resistance, their IE bug database is now closed ( https://connect.microsoft.com/IE ) to both reads and writes. Nope - I'm leaving this one to a masochist. All the best, Nick.

Gregory Maxwell

11:16 p.m.

On 2/8/07, Nick Jenkins <nickpj(a)gmail.com> wrote:

...

At least this explains why we have half as many CSS loads as page views. :)

Nick Jenkins

9 Feb 9 Feb

9:08 a.m.

...

At least this explains why we have half as many CSS loads as page views. :)

Actually, I think 20% is an *underestimate* of the CSS & JS loads. The way it's being measured is like this: ======================================================= + ("upload", re.compile(r"^http://upload.wikimedia.org/"))g/")), + ("article", re.compile(r"^http://[^/]+/wiki/")), + ("wikifiles", re.compile(r"^http://[^/]+/w/")), + ("skins-css", re.compile(r"^http://[^/]+/skins-1.5/.*\.css")), + ("skins-js", re.compile(r"^http://[^/]+/skins-1.5/.*\.js")), + ("skins-image", re.compile(r"^http://[^/]+/skins-1.5/.*\.(?:png|gif|jpg)"))] ======================================================= Now, for me, 4 of the 7 CSS stylesheets I'm loading would be counted as "wikifiles" : ======================================================= @import "/w/index.php?title=MediaWiki:Common.css&usemsgcache=yes&action=raw&ctype=text/css&smaxage=2678400"; @import "/w/index.php?title=MediaWiki:Monobook.css&usemsgcache=yes&action=raw&ctype=text/css&smaxage=2678400"; @import "/w/index.php?title=-&action=raw&gen=css&maxage=2678400&smaxage=0&ts=20070207033933"; @import "/w/index.php?title=User:Nickj/monobook.css&action=raw&ctype=text/css"; ======================================================= (The reason it's 7 rather than 6 stylesheets is that I have a custom stylesheet; in IE it'd be 8 stylesheets) An extra rule for "css-wikifiles" that matched on "/w/ + "action=raw&ctype=text/css" or "action=raw&gen=css" could be useful. I recognise and acknowledge though that the original figures were just a very rough quick back-of-the-envelope thing, and any figures are good and welcome :-) My very very crude guess though is that *maybe* 2/3rds of the "wikifiles" category is CSS or JS files - which would make the total bandwidth usage around 26% for CSS and JS. And what's more, that figure would certainly be higher, if it were not for the dynamic content compression already in place for those files (but not for the skins/ files). All the best, Nick.

Tim Starling

12:43 p.m.

Nick Jenkins wrote:

...

At least this explains why we have half as many CSS loads as page views. :)

The MIME type in the response is logged, to me that would be the obvious way to distinguish between CSS and HTML. -- Tim Starling

Brion Vibber

7:49 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Nick Jenkins wrote:

...

At least this explains why we have half as many CSS loads as page views. :)

Irrelevant, since I was measuring the CSS and JS *files* specifically. Those put out through the wiki are already compressed. - -- brion vibber (brion @ pobox.com / brion @ wikimedia.org)

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFzNArwRnhpk1wk44RAluKAJ9JcsCGkIZmZPGw4FZtO2JMYh6yxgCggK7q Ho03URXjJP1V/rkCiW1pnmI= =yQgl -----END PGP SIGNATURE-----

Simetrical

8 Feb 8 Feb

11:24 p.m.

On 2/8/07, Nick Jenkins <nickpj(a)gmail.com> wrote:

...

For Firefox: See: "Bug 79021 - All stylesheets are loaded regardless of rendering media" https://bugzilla.mozilla.org/show_bug.cgi?id=79021 [Open 5.5 years, no activity in past 3.5 years]

Well, of course, if you don't vote for it. :)

Edward Z. Yang

9 Feb 9 Feb

12:43 a.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Simetrical wrote:

...

On 2/8/07, Nick Jenkins <nickpj(a)gmail.com> wrote:

For Firefox: See: "Bug 79021 - All stylesheets are loaded regardless of rendering media" https://bugzilla.mozilla.org/show_bug.cgi?id=79021 [Open 5.5 years, no activity in past 3.5 years]

Well, of course, if you don't vote for it. :)

Bah, voting is useless. It looks like fixing that bug will be tough design-wise, so cash would be better.

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFy8OiqTO+fYacSNoRAmQvAJ4lB86z0y6mswMzubhty7W5xmbsmACghJcd AMJYraGO2lgyXfJ91LIRe2s= =5keP -----END PGP SIGNATURE-----

Tels

9:16 a.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Moin, On Friday 09 February 2007 00:25:19 Nick Jenkins wrote:

...

> That's six CSS files for most desktop browsers, and seven if > you're using IE. Actually just four and five, respectively. The media-specific stylesheets will probably not be loaded by clients that aren't using that medium.

I can only speak to what Firefox and IE are actually doing.

[snip]

Have you filed bug reports with them? :)

For Firefox: See: "Bug 79021 - All stylesheets are loaded regardless of rendering media" https://bugzilla.mozilla.org/show_bug.cgi?id=79021 [Open 5.5 years, no activity in past 3.5 years]

I didn't want to sound rude, just making sure it's actually filed as bug :) Maybe someone should reply to this bug report along the lines of "this wastes a lot of bandwidth on sites like wikipedia"

...

For IE: Don't care - reporting bugs to MS is an exercise in pain and frustration, involving having to provide your credit card details to do their QA for them on their products (and then they say "sorry, we can't help" - completely useless), or if you use the web feedback forms they just ignore you for 9 years (at which point they discontinue the product) even though it was clearly a 1-line trivial fix ... or at least these have been *my* experiences reporting bugs to MS. And as the piece-de-resistance, their IE bug database is now closed ( https://connect.microsoft.com/IE ) to both reads and writes. Nope - I'm leaving this one to a masochist.

I completely understand. You might as well talk to the trees. At least the wind makes them sway understandingly. Just for a laugh I tried the page you gave, and it redirects you to: http://www.microsoft.com/windows/products/winfamily/ie/iefaq.mspx And when you click there on "Internet Explorer Support", it goes to: http://www.microsoft.com/windows/products/winfamily/ie/tab2 Which can not be found. I rest my case. Best wishes, Tels - -- Signed on Fri Feb 9 10:13:45 2007 with key 0x93B84C15. View my photo gallery: http://bloodgate.com/photos PGP key on http://bloodgate.com/tels.asc or per email. "Look, I'm about to buy me a double barreled sawed off shotgun and show Linus what I think about backspace and delete not working." -- some anonymous .signature

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (GNU/Linux) iQEVAwUBRcw75HcLPEOTuEwVAQIuvQf/RxRxdVmw/li3hjbBXyWxJjIBvCq9C013 CE5hCv85BsoYbAMNuKQOxrAJSQz0i/5+W+MATCgGuu3YWirpFz+AYI2S9aIMY9DM mbH1ZZV8gwVUmTbz6hl6EocMpi5+pahX1Jm2RZQ5LtHDVLq8LoR+dyG31nzfmQ0+ 4q5btnNO3Cb1IAcj9fm6s6RH7boQtIm2BXf6211YF1LkQQLXBFIkedyZK9d6RSUC p1E0wJ+JwwW+OxNl6qnPUBISdxT2fKOhdnsYPomiH25M6gyaK6V00DT8R2m4hWE7 LYOU8WStEaEaMrd2bGERFuvwQznemhitEcYKCiIr7VJpiKPmMn+Eog== =gl7g -----END PGP SIGNATURE-----

Simetrical

4:55 p.m.

On 2/9/07, Tels <nospam-abuse(a)bloodgate.com> wrote:

...

Maybe someone should reply to this bug report along the lines of "this wastes a lot of bandwidth on sites like wikipedia"

No! Bad Tels! No bugspam! On 2/9/07, Phil Boswell <phil.boswell(a)gmail.com> wrote:

...

No, because the IE comment conditionals only work in HTML comments, not CSS comments. (Ref: http://www.quirksmode.org/css/condcom.html.)

Phil Boswell

12:34 p.m.

Nick Jenkins wrote:

...

That's six CSS files for most desktop browsers, and seven if you're using IE.

Is there a particular reason why the "IE Fixes" files are loaded separately? Would it be feasible to conflate them into a single ifle, with the selectors for particular versions *inside* that file, and then simply load that one file for IE users? If I've understood the rest of this thread, the combined file could be compressed quite a lot, so the question would be whether loading one bigger file is more efficient than possibly loading 4 smaller files. HTH HAND -- Phil -- View this message in context: http://www.nabble.com/-Wikitech-l--Bandwidth-usage-tf3184309.html#a8884916 Sent from the Wikipedia Developers mailing list archive at Nabble.com.

Brion Vibber

7:51 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Phil Boswell wrote:

...

Nick Jenkins wrote:

That's six CSS files for most desktop browsers, and seven if you're using IE.

a) there isn't a CSS selector for "MSIE versions 5.5 and above but below 6.0" etc, whereas there are conditional comments in MSIE's HTML parser b) there's no reason to make everyone else load four or five MSIE-version-specific style sets c) it's easier to maintain with them separate - -- brion vibber (brion @ pobox.com / brion @ wikimedia.org)

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFzNC+wRnhpk1wk44RAoCHAKDQnyodWohYpzu0GirGBR4e17W2tACg09Ir Sz3Txh3DZpArpowlqYoSPmg= =HuN3 -----END PGP SIGNATURE-----

Nick Jenkins

8 Feb 8 Feb

3:07 a.m.

...

It's small compared to actual compression, though.

Exactly. To quantify with mod_deflate : ------------------------------------ root@bling:/var/www/hosts/mediawiki/wiki# curl --silent --include --head http://192.168.0.64/wiki/skins/monobook/main.css?55 HTTP/1.1 200 OK Date: Thu, 08 Feb 2007 01:20:04 GMT Server: Apache Last-Modified: Mon, 05 Feb 2007 23:57:09 GMT ETag: "709d-428c37389ff40" Accept-Ranges: bytes Content-Length: 28829 Vary: Accept-Encoding,User-Agent Cache-Control: max-age=2592000, public Content-Type: text/css root@bling:/var/www/hosts/mediawiki/wiki# curl --silent --include --head --header "Accept-Encoding: gzip,deflate" http://192.168.0.64/wiki/skins/monobook/main.css?55 HTTP/1.1 200 OK Date: Thu, 08 Feb 2007 01:20:12 GMT Server: Apache Last-Modified: Mon, 05 Feb 2007 23:57:09 GMT ETag: "709d-428c37389ff40" Accept-Ranges: bytes Vary: Accept-Encoding,User-Agent Content-Encoding: gzip Cache-Control: max-age=2592000, public Content-Length: 7763 Content-Type: text/css root@bling:/var/www/hosts/mediawiki/wiki# ------------------------------------ I.e. 28829 bytes reduced to 7763 bytes saves 21066 bytes, which means a saving of 73%. And you don't have shadow versions of files, and the version that gets sent to clients is still legible (and it's the same one that developers know and are familiar with). [P.s. please ignore the "public" directive in the cache-control line above, it's not there by default, I've just manually overridden it to test stuff. ] All the best, Nick.

Jared Williams

7 Feb 7 Feb

10:49 p.m.

...

-----Original Message----- From: wikitech-l-bounces(a)lists.wikimedia.org [mailto:wikitech-l-bounces@lists.wikimedia.org] On Behalf Of Mets501 Sent: 07 February 2007 21:50 To: 'Wikimedia developers' Subject: Re: [Wikitech-l] Bandwidth usage

However, that will make debugging very hard indeed. Not

sure how big a

consideration that is, but I know that I have often looked

into the MW

stylesheets to see how particular stylings are acheived

when making my own

css rules. - Mark Clements (HappyDog)

Waste of time if your gzipping. 7,772 bytes for main.css(a)55.gz, yet a whole load of hassle if want to try and read it. Jared

6323

days inactive

6325

days old

wikitech-l@lists.wikimedia.org

Manage subscription

39 comments

15 participants

tags (0)

participants (15)

Brion Vibber
Edward Z. Yang
Evan Martin
Gregory Maxwell
Jared Williams
jf＠mormo.org
Mark Clements
Matthew Flaschen
Mets501
Nick Jenkins
Phil Boswell
Rob Church
Simetrical
Tels
Tim Starling