---------- Forwarded message ---------- From: Brent Palmer bop@brentopalmer.com Date: 2010/2/2 Subject: [Mediawiki-l] Wikipedia mirror speedup To: mediawiki-l@lists.wikimedia.org
Hi,
We are creating an off-line English Wikipedia mirror. By off-line, I only mean that it is not available from the Internet; only from a LAN which is not connected to the Internet. This will be deployed in locations where there is little or no Internet access (schools and universities in Africa for example). More info: http://en.wikipedia.org/wiki/EGranary_Digital_Library
The machines hosting the mirror are often lower-end machines without a lot of spare memory and we are tight on disk space. Generally there won't be a lot of traffic though. Also, the Wikipedia mirror is read-only; The main problem is that it is incredibly slow (even on our relatively fast server). Articles like Abraham_Lincoln can take several minutes. The longer the article and the more templates it uses, the longer it takes. I know that this is a common problem that people ask about for mediawiki installations and Wikipedia uses various levels of caching. Because of the above constraints, using Squid and the file cache don't really seem like viable options. We tried PHP accelerators without much benefit. Looking at the profiler log, it seems as if the the most time is spent by the parser. Pages speed up considerably after the initial access when we have the parser cache set to the CACHE_DB.
The plan: We want to deliver the Wikipedia in a form that makes it as fast as possible for the end user. The plan is to try and pre-cache all the articles. In other words, set the parser cache not to expire at all and try to hit all the articles one time to set the parser cache before we deploy the mirror. The assumption is that this will ultimately be less disk space than just creating a static copy of all the pages or even just creating a huge file cache (I think there is a management script that allows you to do this). This will also serve the problem of generating all the thumbnails ahead of time so that we don't have to do it on the fly. (We currently rewrite all requests to thumbnails to the original and let the browser resize them--this also makes the mirror appear to run slowly).
Any comments? Are our assumptions correct? Is there another way to go about this? Are there options we haven't thought of? Any comments on how to go about it?
Thanks to donations, we have all the machinery, storage, time, and processing power to do any pre-processing of Wikipedia assets -- we're seeking any ideas of things we can do in advance to make it run fast on end-user machines. Generally we plan on setting up multiple machines with the Wikipedia running on each one, all accessing one database and running multiple clients requesting articles simultaneously. I don't know if it will be feasible to have all the machines share the same filespace for thumbnail generation (NFS/SMB) but I might try that too.
Thanks in advance for your thoughts. Brent Widernet.org
MediaWiki 1.14.0 PHP 5.2.8 (apache2handler) MySQL 5.0.41-community-nt-log Apache 2.1 OS: various
_______________________________________________ MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/mediawiki-l