Hi Taku,
we *know* that the site is slow. And we *know* that it's not because our server is too small. Our problems are database/lock related, and putting stuff on an even bigger server will not help much when dealing with O(n^x) problems. What we need to figure out is:
- When are our tables/rows locked and why (this behavior has changed drastically with the recent update to InnoDB). - When are our queries using indices and when are they not, and why (MySQL index behavior can be very hard to predict).
Solving these two problems should make Wikipedia very fast. If we cannot optimize some queries, we need to think about making them simpler, or caching them. (Also note that MySQL now supports subqueries, which we don't use yet.) We are dealing with *particular* queries in *particular* situations that make Wikipedia slow.
In other words, if I had the option to invest $10k in a new server or in a experienced database engineer, I'd pick the latter.
Now then, here is my proposal. Can we distribute database and related stuff?
Not practical. Too many queries require access to a single centralized article database, even an index alone won't suffice. Think of stuff like Most wanted, Orphaned pages etc. Besides, it won't make things any faster because our problem is not a too small server.
Regards,
Erik