The last batch of fun upgrades included use of gzip compression on cached pages where browsers accept it. I'm happy to report that this seems to have decreased the bandwidth usage of the English-language Wikipedia by up to roughly 25%.
Data from http://www.wikipedia.org/stats/usage_200306.html
kilobytes / hits = kb/hit 2003-06-01 6392951 / 619534 = 10.319 \ 2003-06-02 7908928 / 793065 = 9.973 | 2003-06-03 8267879 / 822025 = 10.058 | mean 10.1 2003-06-04 7513917 / 755482 = 9.946 | range 0.37 2003-06-05 7347843 / 723717 = 10.153 | 2003-06-06 6300476 / 614552 = 10.252 | 2003-06-07 5159151 / 503097 = 10.255 | 2003-06-08 5732741 / 566484 = 10.120 / -- gzip cache activated: -- 2003-06-09 5376987 / 726971 = 7.396 \ 2003-06-10 5442685 / 732897 = 7.426 | mean 7.6 2003-06-11 5735325 / 765204 = 7.495 | range 0.85 2003-06-12 6362049 / 772002 = 8.241 /
These counts include pages, images, css, everything. (But not the other languages, mailing list, or database dump downloads.)
The bandwidth usage did go up a bit today, so it remains to be seen just how stable the effect is. A number of things can affect it:
* Since caching, and thus sending of gzipped cached pages, is currently only done for anonymous users, an increase in activity by registered users would tend to reduce the overall percentage of savings
* Lots of editing and loading of dynamic pages, which are not compressed, would do the same
* A large increase in brief visits by newbies drawn in by a link or news mention would increase the relative bandwidth used by linked items (logo, style sheet, etc) which are not additionally compressed
* Lots of work with new images might increase bandwidth usage.
Other thoughts: - So far, no one's complained about being unable to read pages due to pages being sent compressed when they shouldn't. (There was in fact a bug to this effect which made the wiki unusable with Safari, but I don't know if anyone but me noticed before I fixed it. :)
- Since gzipping is done only at cache time, this should use very little CPU. IIRC the gzip was one of the faster steps when I was test profiling this ;) and the number of times gzipping is done should not generally exceed the number of edits * some factor regarding creation/deletion rates and number of links per page. (More of course when the cache has to be regenerated en masse.)
- The page cache presently stores both uncompressed and compressed copies of pages, which is space inefficient, though we're not presently hurting for space on larousse. Someone suggested storing just the compressed pages, then in the relatively rare case a browser won't accept gzipped pages, we can unzip it on the fly.
- We could offer either as default or as an option to compress dynamically generated pages as well, which could shave some more percentage points off the bandwidth usage. Might be a help for the modem folks who do log in. :) However I'm not sure how much this would affect CPU usage; in any case there's no urgency for this, it's just something we might do if we have the cycles to burn (I don't think we do just now, but we might one day).
-- brion vibber (brion @ pobox.com)