Lee,
I don't think we should completely redesign things from scratch. See http://www.joelonsoftware.com/articles/fog0000000069.html about rewriting in general.
We are getting to the point where we know what the performance bottlenecks are, and we are fixing them. Brion has built some basic profiling into the code, and we've checked the slow query log. We still haven't fully understood everything, but I think we are pretty certain about the following:
1) PHP is not and has never been a problem. Virtually all our performance problems have been related to specific SQL queries (either a very high number of them, or complex ones). I do not see any reason at all to stop using PHP.
2) Getting MySQL to perform properly largly depends on using indexes the right way. This means providing composite indexes where needed. In the case of timestamps, we had to add a reverse timestamp column for it to be index-sorted fast, but while this is a hack, it is a needed hack until MySQL4.
It is specifically complex SQL queries which require ordering the whole result set that create headaches. These have been disabled for the moment, but I believe we can fix them. None of them are mission critical.
As for MySQL4, I support trying it out on our test server, test.wikipedia.org, and possibly on meta.wikipedia.org as well. We shouldn't switch the main site(s) until these two have run on it for a while.
We also need to keep in mind that we are growing very fast. We now have several highly active Wikipedias, all of them residing on the same server. While I think our server still has some room, at some point we will have to upgrade and no amount of hacking will prevent that. Separating web and database server, as is planned, should help, but I don't know how much.
I think our priorities should be this:
1) Get some of the other language Wikipedias up that people are waiting for. If there are motivated users who want to start a Wikipedia in their language, we should not let them wait.
2) Fix known bugs and try to improve the speed in case of remaining bottlenecks.
3) Implement suggested improvements. - Improve search + redirect handling - Finish Magnus' interlanguage links redesign - Fix Recent changes layout - Redesign image pages - Redesign talk pages - Improved edit conflict handling (CVS style merge) - Backends (SVG, Lilypond), syntax improvements etc.
Aside from this, an entirely new project is the dedicated Wikipedia client, for offline reading and, hopefully, ultimately for editing as well. Magnus has started working on this.
What I understand to be "Phase IV" is, then, a point where we have finished all the important fixes and improvements and then decide to move on to "nice to have" stuff. Among this is the much requested multilanguage portal for Wikipedia, with a multilanguage search, RC etc., and possibly merging the databases of the different Wikipedias (at least the user data). I do think the current software can and should be used as a basis for the next phase(s).
Regards,
Erik