With the recent increases in traffic (which fit well with my estimates earlier) I think we should be giving some thought now to how the Wikipedia can be made massively scalable. If we start now, we should have a solution available before it is needed, rather than hitting a crunch point and then fixing things in a hurry.

Some thoughts on this:

* The single image of the database is the ultimate bottleneck. This should be held in RAM as far as possible. Thought should be given to using PostgreSQL rather than MySQL, which should scale better in heavily loaded cases.

* Wikipedia is read an order or magnitude more often than it is written. Caching can give a major performance boost, and will help palliate underlying scaling problems.

* Wikipedia read accesses appear to obey [[Zipf's law]]: most pages are read relatively infrequently, and a few very frequently. However, most of the load is from the aggregated accesses of the many low-traffic pages. Therefore, a 'hot spot' cache will not work: the cache has to be big enough to encompass the whole Wikipedia. Again, the cache will have to be held in RAM to avoid the 1000-times slower performance of disk accesses.

* Web serving consumes RAM: a slow modem transfer of a page will lock down resources in RAM for the duration of a page load. At one end of the scale, this is simply socket buffers. At the other end of the scale, it can be a page-rendering thread and its stack, unable to progress to do other work because it can't commit its output to the lower levels of the web server. On low-traffic sites, this is insignificant. On high-traffic sites, it can become a bottleneck. Machines with vast amounts of RAM are very expensive: it is cheaper to buy a number of smaller machines with reasonable amounts of RAM in -- and you get the benefit of extra CPUs for nothing.

=== The grand vision ===

For all these reasons, I propose a solution in which the database is kept on a separate server from the user-facing servers. The extra overhead involved in doing this should be repaid many times over in the long run by the fact that the front-end servers will take more than 90% of the load off the central server.

The front-end servers should be kept _dumb_, and all the control logic, program code, and configuration files should reside on the master server. This removes the need to configure vast numbers of machines in sync.

The front-end server code can also be kept small and tight, with most of the work (page rendering, parsing, etc.) being done by the existing code. (Later, rendering and parsing could be moved to the front-end servers if needed).

=== How to get there easily ===

The first step is to implement a separate caching server.

The system can even be trialled now, by running the master and slave servers on the same box! The existing Wikipedia just needs to have a special skin that serves up 'raw' rendered pages, without any page decorations, CSS, or whatever.

The caching server will deal with all user interaction: cookies, skins, etc. It should serve up 'raw' pages decorated with user-specific content and style sheets.

The caching system could be put on an experimental server, or later put on the main site, as a beta test.

There is only one other thing needed to implement this. Each page in the central DB needs to have a 'last changed' time, and a way to get hold of it. Now, this is where things get slightly tricky. This 'last changed' timestamp should be updated, not only when the page itself is edited, but also when any page linked in the page is created or deleted. That is to say: when creating or deleting a page, timestamps on all pages that link to that page should be touched.

Note that there is no need to touch other pages' timestamps when a page is edited. This is good, as otherwise the extra overhead of all that marking would be unreasonably high.

Now, this last-modified timestamp can be determined using a special request format: the underlying HTTP mechanism could be used, but this is too awkward to fiddle with as a way of supporting experimental software. Better to do something like:

http://www.wikipedia.com/wiki/last_changed?Example_article

(which even avoids invoking the main code at all)

result: a page with the content:

20020417120254

=== Criticisms ===

But, I hear you say -- surely you have added another bottleneck? For every page hit, you still generate a hit on the central DB for the 'last changed' timestamp. The answer is: yes, but -- the timestamp db is very small, performs a very simple lightweight operation, has a single writer and multiple readers, and can therefore later be made separate from the main DB, replicated, etc.

So, there are some ideas.

* Client/server database split * Front-end/back-end script split

Does anyone have any idea how reasonable this is as an idea, based on refactoring retrofitting the existing script? I think that if things were done slowly, the front-end code could be made about 10% of the overall complexity, mostly refactored from existing code.

-- Neil