I have been thinking about the performance of Wikipedia, and how it
might be improved.
Before I go off and investigate in detail, I'd just like to check my
basic concept of how the code works,
(based on reading this list -- I haven't pulled down the CVS to look at
it yet).
=== Total guesswork follows ===
Am I right in thinking that, for each ordinary page request,
* the raw text is pulled out of the database
* the taxt is parsed and reformatted
* links are looked up to see if they are linked and treated appropriately
* final page generation to HTML, with page decorations as per theme is added
My general impressions about activity rate is:
* about 100 pages per day are created or deleted
* roughly one edit every 30 seconds
* roughly one page hit every second
Packet loss seems negligible, so you don't seem to be running out of
bandwidth.
Although I guesstimate the hit rate at around one-per-second, pages seem
to be taking around 5 seconds to serve,
suggesting that the system is probably running at a loadav of say 5 or so.
My best guess is that the parsing and lookups on regular pages are
currently the main load, not editing or exotic database queries -- is
this right?
Jimbo has mentioned that the machine has a lot of RAM, so disk I/O is
unlikely to be the bottleneck: it's more likely to be CPU and
inter-process locking problems.
If so, I think careful page content caching could greatly improve
performance, by reducing the number of page parsings, renderings and
lookups across the board, at the cost of a slight increase in the cost
of page deletion and creation. However, by freeing up resources,
performance should improve across the board on all operations.
If I'm right, I think suitably intelligent caching could be applied not
only to ordinary pages, but also to some special pages, without any
major redesign or excessive complexity.
Before I start to look at things in more detail, could anyone confirm
whether I am even vaguely making sense?
-- Neil