I have been thinking about the performance of Wikipedia, and how it might be improved.

Before I go off and investigate in detail, I'd just like to check my basic concept of how the code works, (based on reading this list -- I haven't pulled down the CVS to look at it yet).

=== Total guesswork follows ===

Am I right in thinking that, for each ordinary page request,

* the raw text is pulled out of the database * the taxt is parsed and reformatted * links are looked up to see if they are linked and treated appropriately * final page generation to HTML, with page decorations as per theme is added

My general impressions about activity rate is:

* about 100 pages per day are created or deleted * roughly one edit every 30 seconds * roughly one page hit every second

Packet loss seems negligible, so you don't seem to be running out of bandwidth.

Although I guesstimate the hit rate at around one-per-second, pages seem to be taking around 5 seconds to serve, suggesting that the system is probably running at a loadav of say 5 or so.

My best guess is that the parsing and lookups on regular pages are currently the main load, not editing or exotic database queries -- is this right?

Jimbo has mentioned that the machine has a lot of RAM, so disk I/O is unlikely to be the bottleneck: it's more likely to be CPU and inter-process locking problems.

If so, I think careful page content caching could greatly improve performance, by reducing the number of page parsings, renderings and lookups across the board, at the cost of a slight increase in the cost of page deletion and creation. However, by freeing up resources, performance should improve across the board on all operations.

If I'm right, I think suitably intelligent caching could be applied not only to ordinary pages, but also to some special pages, without any major redesign or excessive complexity.

Before I start to look at things in more detail, could anyone confirm whether I am even vaguely making sense?

-- Neil