Nick,
Your idea assumes that the "lag" problem is due to overloading a single machine, which plays double roles: database backeand and web server. So, if we divide the work amoung 2 or more machines, you expect faster throughput. Right?
(I'm just repeating the obvious to make sure that what's obvious to me, is what you really meant!)
I guess if we all pitch in $50 each we can buy another machine. Where should I send my money?
Ed Poor
On Mon, 25 Nov 2002 19:21:57 -0500 "Poor, Edmund W" Edmund.W.Poor@abc.com wrote:
Nick,
Your idea assumes that the "lag" problem is due to overloading a single machine, which plays double roles: database backeand and web server. So, if we divide the work amoung 2 or more machines, you expect faster throughput. Right?
My idea is not to divide the roles of web server and backend. It is to divide the workload of the one server between many servers. This includes search queries and other functionality.
I envisage many wikipedia servers around the world, supported by private individuals, companies and universities. Much like the system of mirror FTP and mirror web sites. All these servers are updated in real time from the core wikipedia server. From the user's perspective, all are equivalent. Each of these servers can do everything the current wikipedia server can do except for accepting update submissions. Updates from users are accepted only by the core wiki server.
Reasons for such an architecture: 1) Growth of bandwidth useage may put financial pressure on Wikipedia. Growth may follow a non-linear growth curve.
2) The cost of implementing one very fast, reliable, redundant machine is more than the cost of farming out work to many quite fast, unreliable systems none of which are mission critical. Especially true where there are people willing to donate part of their hard drive, CPU and net connection (or even an entire system) to a good cause such as wikipedia. (Overall system reliability can be guaranteed by using DNS tricks to ensure users and queries are only directed to working machines).
Given wikipedias limited technical resources, we need to make a choice between making small scale changes to the system (which may make 30-50% change in availability) or to make architectural changes which can scale to improvements of 100x or 1000x magnitude.
The core server takes the update submissions. These are integrated into the core database. Changes to the core database are reflected in all mirror servers in real time by using a system of pushing the database update to the mirror servers.
The core server will implement access control, ip blocking and other such administration and policy.
I accept the technological suggestion I made is by no means the only way to achieve this goal, although it may be a good one for it's scalability potential.
Before mirrors are implemented in the way I suggested, It would be wise to introduce meta-fields and records into the database. Fields which have no current use but may be used in future wikipedia software releases. Future wikipedia software releases for the mirror servers is guaranteed and extra database fields are almost certainly going to be required. Adding mata fields can help forward compatibility of databeses. This would be necessary in advance as not all mirror servers will update their whole database and software simaultaneously.
Nick Hill wrote:
I envisage many wikipedia servers around the world, supported by private individuals, companies and universities. Much like the system of mirror FTP and mirror web sites. All these servers are updated in real time from the core wikipedia server. From the user's perspective, all are equivalent. Each of these servers can do everything the current wikipedia server can do except for accepting update submissions. Updates from users are accepted only by the core wiki server.
Reasons for such an architecture:
- Growth of bandwidth useage may put financial pressure on Wikipedia.
Growth may follow a non-linear growth curve.
- The cost of implementing one very fast, reliable, redundant machine is
more than the cost of farming out work to many quite fast, unreliable systems none of which are mission critical. Especially true where there are people willing to donate part of their hard drive, CPU and net connection (or even an entire system) to a good cause such as wikipedia. (Overall system reliability can be guaranteed by using DNS tricks to ensure users and queries are only directed to working machines).
Given wikipedias limited technical resources, we need to make a choice between making small scale changes to the system (which may make 30-50% change in availability) or to make architectural changes which can scale to improvements of 100x or 1000x magnitude.
I confess to complete ignorance with the technical aspects of these proposals. Be that as it may, there still needs to be a review of the administrative structure of the project. There was some discussion a couple months ago about forming a non-profit corporation, but that was quickly forgotten.
A project that continues to depend on funding from a single source is always at risk. Similarly, depending on ad hoc demands of $50 from everybody may work once or twice, but it does not give any kind of financial security. Aside from any moral issues about depending on handouts from a single benefactor, there is the reality that no person's pockets are bottomless and we have no idea where the bottom is. It is irresponsible for the group as a whole to wait for a message from Jimbo like "I want to contribute more, but I can't." Messages like that never come at convenient times; they often coincide with major equipment breakdowns or necessary technical expansion.
The brand new dedicated server is less than a year old, and it already has difficulties coping with its volume of work. I think that Nick is conscious of the potential costs that we could be facing. This is evident from his willingness to look for lower cost alternatives. I would like to see the budgetting alternatives in terms of costs and in terms of what will be needed from participants over an extended length of time.
It is my experience that most volunteers find corporate boards, budgetting and issues of long term fiscal planning to be frightfully boring, but facing these issues is a question of fiscal responsibility.
Eclecticology
Ray Saintonge wrote:
There was some discussion a couple months ago about forming a non-profit corporation, but that was quickly forgotten.
Not forgotten at all. In the works. Announcements forthcoming soon.
A project that continues to depend on funding from a single source is always at risk. Similarly, depending on ad hoc demands of $50 from everybody may work once or twice, but it does not give any kind of financial security. Aside from any moral issues about depending on handouts from a single benefactor, there is the reality that no person's pockets are bottomless and we have no idea where the bottom is. It is irresponsible for the group as a whole to wait for a message from Jimbo like "I want to contribute more, but I can't." Messages like that never come at convenient times; they often coincide with major equipment breakdowns or necessary technical expansion.
That's all 100% correct. With the layoff of Larry Sanger and Toan Vo, both excellent people working full-time on Nupedia, this has already happened. I've gone from support at a level exceeding $100k per year, to current levels, which are difficult to measure exactly. (My time, Jason sometimes, the server, the bandwidth).
What I am current planning is a nonprofit corporation with a separate bank account, with the ability to accept credit cards, whereby people can sign up for various levels of monthly or quarterly or annual support. The contributors will be able, within some limits, to earmark contributions for particular uses (server, bandwidth, promotion, etc.).
I think that many regulars will be willing and able to sign up at the $20 per month level. Just 100 people at that level would give us $2000 a month, which would fairly quickly translate into a pretty serious server farm.
--Jimbo
Nick Hill wrote:
On Mon, 25 Nov 2002 19:21:57 -0500 "Poor, Edmund W" Edmund.W.Poor@abc.com wrote:
Nick,
Your idea assumes that the "lag" problem is due to overloading a single machine, which plays double roles: database backeand and web server. So, if we divide the work amoung 2 or more machines, you expect faster throughput. Right?
My idea is not to divide the roles of web server and backend. It is to divide the workload of the one server between many servers. This includes search queries and other functionality.
<snip brave and expensive vision>
While the proposal of Nick will no doubt be the long-term future, separating database server and webserver is both relatively cheap and easy, and should increase performance. It is even recommended in either the MySQL or the PHP online manual (or was it apache? Brain, where are you?).
Even a smaller server might do, though I'm not sure wether to run apache or MySQL on it. Probably the latter.
Oh, and we need a place for the sifter project, while we're at it ;-)
Magnus
On Tue, Nov 26, 2002 at 08:12:05PM +0100, Magnus Manske wrote:
Nick Hill wrote:
On Mon, 25 Nov 2002 19:21:57 -0500 "Poor, Edmund W" Edmund.W.Poor@abc.com wrote:
Your idea assumes that the "lag" problem is due to overloading a single machine, which plays double roles: database backeand and web server. So, if we divide the work amoung 2 or more machines, you expect faster throughput. Right?
My idea is not to divide the roles of web server and backend. It is to divide the workload of the one server between many servers. This includes search queries and other functionality.
<snip brave and expensive vision>
While the proposal of Nick will no doubt be the long-term future, separating database server and webserver is both relatively cheap and easy, and should increase performance. It is even recommended in either the MySQL or the PHP online manual (or was it apache? Brain, where are you?).
Even a smaller server might do, though I'm not sure wether to run apache or MySQL on it. Probably the latter.
Hi,
do we have any numbers related to the CPU usage of the machine? Throwing hardware at a performance issue is a solution that's chosen very often, but it's a) most of the time expensive b) too often not the solution
In case of a high lag situation, which process is blocking the CPU? Is it apache (that is: PHP) or mysql? How is the memory utilization? What is the I/O rate?
Do we have this kind of data already available?
Best regards,
jens frank
Jens Frank wrote:
do we have any numbers related to the CPU usage of the machine? Throwing hardware at a performance issue is a solution that's chosen very often, but it's a) most of the time expensive b) too often not the solution
In case of a high lag situation, which process is blocking the CPU? Is it apache (that is: PHP) or mysql? How is the memory utilization? What is the I/O rate?
The CPUs seem to have a fair amount of idle time whenever I check... I'm not sure how to measure CPU usage usefully (I can pop into 'top', but that corrupts the data -- top itself is usually the biggest user of CPU time!)
Load average when things are running smoothly runs between 1 and 2 (2 is ideal usage for a 2-CPU system). Right now there's an old zombie process that bumps up the load average by 1. During busier times during the day, 3-5 is not uncommon. The busiest periods can push us into the teens or very occasionally more.
Very roughly from top:
Apache processes take about 15-25 MB, with 10-25 MB of shared memory. MySQL has a resident memory size of around 234 MB, plus ~88MB of shared memory. I think most of this is shared between processes.
And more generally: ~14 MB buffers 800-900 MB disk cache (dips to 100-300 during very high load times) ~150-200 MB free (dips much lower during very high load times)
That probably doesn't quite cover everything, as we've got 2GB to fill up with those figures.
And: 120-180 MB of rarely used stuff sitting in swap
If you want to see load and memory stats from "uptime" and "free" updated every 10 minutes starting mid-day yesterday, see http://www.wikipedia.org/tools/uptimelog
How can we measure i/o rate?
-- brion vibber (brion @ pobox.com)
Nick Hill wrote:
I envisage many wikipedia servers around the world, supported by private individuals, companies and universities. Much like the system of mirror FTP and mirror web sites. All these servers are updated in real time from the core wikipedia server. From the user's perspective, all are equivalent.
My experience from situations like the one you describe tells me that the designed system can easily get more complex and cause more overhead than the needed performance gain, and that Moores law will give us the speed that we need in time when we need it.
Do you have any experience from designing systems like this? Would you write a prototype for this system that could be tested? The vision sounds like science fiction to me, but a prototype that I can run is not science fiction, so that would make all the difference.
Here is another vision: I envision a system where I can synchronize my laptop or PDA with a wiki, then go offline and use it, update it, and when I return to my office I can resynchronize the two again. I have no idea on how to implement this vision. I think it would be a lot of work. But I think the result could be really useful.
I also see there are similarities between your vision and mine. The idea is to express the update activity as a series of transactions (update submits) that can be transfered to another instance or multiple instances and be applied there. In either case, one must take care of the case that the transmission of updates gets interrupted or delayed, and the potential "edit conflicts" that would result. It doesn't seem trivial to me.
Nick Hill wrote:
I envisage many wikipedia servers around the world, supported by private individuals, companies and universities. Much like the system of mirror FTP and mirror web sites. All these servers are updated in real time from the core wikipedia server. From the user's perspective, all are equivalent.
Having read-only database copies with MySQL's replication sounds doable, at least in theory. All edits would be directed to the mama website. User accounts (ie for watchlists) are maybe a trickier matter, as we'd rather not send passwords and e-mail addresses around willy-nilly.
Lars Aronsson wrote:
Here is another vision: I envision a system where I can synchronize my laptop or PDA with a wiki, then go offline and use it, update it, and when I return to my office I can resynchronize the two again. I have no idea on how to implement this vision. I think it would be a lot of work. But I think the result could be really useful.
I also see there are similarities between your vision and mine. The idea is to express the update activity as a series of transactions (update submits) that can be transfered to another instance or multiple instances and be applied there. In either case, one must take care of the case that the transmission of updates gets interrupted or delayed, and the potential "edit conflicts" that would result. It doesn't seem trivial to me.
Distributed editing is, indeed, a rather bit tricker than distributed reading, and not something I really want to touch with a ten foot pole right now. :)
See also discussion on a client-side Wikipedia reader/editor at: http://meta.wikipedia.org/wiki/Dedicated_Wikipedia_editor
-- brion vibber (brion @ pobox.com)
On Thu, 28 Nov 2002 03:24:43 +0100 (CET) Lars Aronsson lars@aronsson.se wrote:
Nick Hill wrote:
I envisage many wikipedia servers around the world, supported by private individuals, companies and universities. Much like the system of mirror FTP and mirror web sites. All these servers are updated in real time from the core wikipedia server. From the user's perspective, all are equivalent.
My experience from situations like the one you describe tells me that the designed system can easily get more complex
Systems can always become complex in an unworkable sense. If the implementation is carefully managed, the complexity can be kept under control.
The system I suggested, in effect, distributes chunks of data. The issue is will these chunks of data, at some point, become incompatible with the systems which are supposed to read them? Is a degree of non-fatal incompatibility allowed?
example: As the definition for tags changes, the front end will interpret them differently, making the pages look different between newer and older implementations.
and cause more overhead than the needed performance gain,
Technical or computing overhead? How do you convert technical overhead to computing overhead? What is the needed performance gain?
and that Moores law will give us the speed that we need in time when we need it.
Several variables in the wikipedia system self-multiply. As the size of database multiplies, the demands on the database system grow. As the size of the database grows, the system becomes more attractive, bringing more people to Wikipedia. We may currently be in a situation where we have latent demand, which has been held back by system overload. Whilst the size of wikipedia may approximate moore's law, the demand will probably exceed it. The demand x size product is likely to far exceed moore's law.
We need more analysis on these issues. We need forecasts which the architecture can be moulded around.
Do you have any experience from designing systems like this? Would you write a prototype for this system that could be tested?
I have designed database systems. I have not designed a system of exactly this sort. I don't think the system I proposed is complex. It uses known database tecniques combined with public key and email. All of which are well matured and understood technologies which have extensibility built into the current free software code base. If it becomes clear to me no-one else is prepared to pick up the gauntlet, I will do so if I get time. A lot of my time is being spent on the GNU project.
The vision sounds like science fiction to me, but a prototype that I can run is not science fiction, so that would make all the difference.
'still haven't made the transporter! :-(
Here is another vision: I envision a system where I can synchronize my laptop or PDA with a wiki, then go offline and use it, update it, and when I return to my office I can resynchronize the two again. I have no idea on how to implement this vision. I think it would be a lot of work. But I think the result could be really useful.
The system I mentioned would work for this purpose. The PDA could collect the update emails then integrate them into the database. Porting Wiki and the supporting technology to the PDA would be a lot of work.
I also see there are similarities between your vision and mine. The idea is to express the update activity as a series of transactions (update submits) that can be transfered to another instance or multiple instances and be applied there. In either case, one must take care of the case that the transmission of updates gets interrupted or delayed, and the potential "edit conflicts" that would result. It doesn't seem trivial to me.
The solution I proposed is 1) To have edits serialised. They can only be applied in the specific order they were generated. 2) The pipermail mailing list will give sysadmins the facility of downloading missed updates. 3) Spoof edits would be filtered- the attachments from the main wiki server would be signed using a private/public key pair and verified at the receiving end.
Nick Hill wrote:
I envisage many wikipedia servers around the world, supported by private individuals, companies and universities. Much like the system of mirror FTP and mirror web sites. All these servers are updated in real time from the core wikipedia server. From the user's perspective, all are equivalent. Each of these servers can do everything the current wikipedia server can do except for accepting update submissions. Updates from users are accepted only by the core wiki server.
Organizationally, this would be a nightmare. Having a central server farm really eases everything, because the volunteer admins can be given root passwords, etc., and the ability to make changes on all the servers very quickly. It's not easy to get that kind of access if we are co-ordinating across many servers around the world.
Co-ordinating across many servers around the world *is* a solution we can ponder *if* it appears to be absolutely necessary for some reason. But it strikes me as unlikely for this to be the case.
Reasons for such an architecture:
- Growth of bandwidth useage may put financial pressure on Wikipedia.
Growth may follow a non-linear growth curve.
With the continued fall in bandwidth costs, this is very unlikely to exceed my capacity to support Wikipedia in the short run. *Bandwidth* per se, is not a major issue.
- The cost of implementing one very fast, reliable, redundant machine is
more than the cost of farming out work to many quite fast, unreliable systems none of which are mission critical. Especially true where there are people willing to donate part of their hard drive, CPU and net connection (or even an entire system) to a good cause such as wikipedia. (Overall system reliability can be guaranteed by using DNS tricks to ensure users and queries are only directed to working machines).
This is easy to say, but harder to implement well. I don't know of any really successful implementations of the kind you are discussing.
--Jimbo
wikitech-l@lists.wikimedia.org