I figured as much. The way the spec is worded, I'm not sure it needs to be valid XML. They basically say that anything aside from "</CONTENT>" can be between the CONTENT tags. Jason Brion Vibber wrote: -- "Jason C. Richey" &lt;jasonr(a)bomis.com&gt;

On Apr 2, 2004, at 00:53, Jimmy Wales wrote: It would take hours just to run a complete dump, which would be the equivalent of a sizeable fraction of our total daily page views. (Best case might be 100ms per page for 240,000 pages =~ 6 hours 40 minutes) If we're going to run something like this daily, some sort of incremental updates are a must, though we can probably get away with stuffing the saved data per page in a database or such and slurping it back out fairly quickly. -- brion vibber (brion @ pobox.com)

Brion Vibber wrote: I can't see where the proposal that some have made to have all the projects on one database would help this situation. Ec

On Apr 2, 2004, at 00:35, Timwi wrote: That's kinda the point, yeah. It might be better to skip redirects, though; otherwise they should be handled in some distinct way. That's obviously a bit inefficient, but yes. Incremental updates of only changed pages could hypothetically lead to faster output generation after the first run, though this would require some intermediate storage (since we don't yet have a running parser cache). (It would produce a series of files up to about 12.5 megabytes in length, not one big file.) A text base without the unnecessary UI elements could improve search results, and I suppose can be kept more complete more easily than constant spidering of a 200k+ page site. *shrug* If that's the data format they want, hey fine, though having to download the entire set of a couple hundred megabytes for every update doesn't sound ideal. Jason, would each output need to be self-contained, or can they accept incremental updates in IDIF? How often would they pull updates? -- brion vibber (brion @ pobox.com)

MySQL has a facility for distributing databases. It sounds as though you all are trying to solve a problem at the application level, when it is a data distribution, caching and integrity problem. And this is exactly the kind of problem that databases can solve. If the wiki database is synchronized out, clients of that synchronization can do stuff for Yahoo or do whatever they want. MySQL handles all this "changed since" checking and such. It also does it more efficiently than a PHP script. So, what problem are you all trying to solve? - ray On Apr 2, 2004, at 1:19 AM, Alfio Puglisi wrote:

On Apr 3, 2004, at 1:38 PM, Jimmy Wales wrote: But there are people at Yahoo who know MySQL. Jeremy Zawodny comes to mind. He works there and his O'Reilly book called 'High Performance MySQL' is coming out soon. The word "Replication" is in the title of the book, so they could probably even help set this up so as not to impact the wikipedia over-all. It will probably not be too difficult to come up with a feed oriented to them, but what happens when another organization wants the same feed. But, oh yes, in a slightly different form. And another. And another. Again, the people you might be talking to might not be the database people, so maybe they think that the solution they need is not a database solution. But you can let them replicate the database. It will be as efficient as any application-level solution and they can do anything they want with that data, spiced to their tastes. Respectfully, I would say that if there are two things that wiki people could spend their time on: 1) growing the wiki community and the database that keeps the collective corpus together, and 2) feeding this to various corporate clients, again and again and again, with all their ever-changing requirements then one of these seems to be closer to the core mission than the other. - ray

Brion Vibber wrote: If the problem is that each article has navigational elements on the top, bottom, and left, which multiplies the time required to crawl things, then how about allowing something like &raw=1 which would output only the parsed page, without a skin or anything, with all the links in it also pointing to &raw=1, and then arranging with Yahoo to have them spider these pages, but still send people to the URL without &raw=1?

On Fri, 02 Apr 2004 01:55:31 -0800, Brion Vibber wrote: What about adding a squid running at yahoo to our cache purge list? They could constantly crawl that one, and only purged pages would be fetched (from the wp caches configured as parent to the yahoo box, no extra db access). That's zero extra db usage and no scripting on our part required. And the wp html should be pretty easy to parse for them by cutting out the content of <div id='content'>. -- Gabriel Wicke