Timwi wrote: I agree with you, even though I also agreed with Brion. He was just joking, and so was I. His point was that some discussions of optimizations really are premature until we spend some time working out just where bottlenecks actually are. --Jimbo

Brion Vibber wrote: This is assuming that the developers whose time you have to your disposition are all equally interested in various things. I, for example, am interested in database optimisation, but not in profiling. This is why I discuss database optimisation, but not profiling. Similarly: Why not? - seeing as I'm not going to help with the profiling either way, I might as well help with the database optimisation. Timwi

Brion Vibber wrote: Hm. Looks okay for short-to-middle term. I suppose this suggestion is based on the thought of having to convert the current database easily. It doesn't account for several things we really really want to do in the middle-to-long term, such as combining all language Wikipedias in a single Wiki, so that people don't have to register hundreds of usernames, or more flexible handling of user properties and article properties. The database schema I propsed last year: http://meta.wikipedia.org/wiki/Experimental_new_database_schema takes these into account, but is (obviously) a little dated by now. For example, it does not know about the MediaWiki namespace and how it now handles translatable site text. Timwi

On Fri, Feb 27, 2004 at 09:45:50AM -0500, Evan Prodromou wrote: Have you hired some people to do work on MediaWiki ? If not, such roadmaps aren't very useful - people will work on whatever they want.

Agon S. Buchholz wrote: I think that's right. Having a consensus goal or direction can help us to move coherently, while of course the volunteer nature of the work means that people will do or not do whatever they like. But lots of time, people may be perfectly satisfied and interested in any number of different problems, and if we have a roadmap, they can pick one that's most sensible. I'm not the right person to set this roadmap, though. I'm not close enough to the software development process. --Jimbo

Ivo Köthnig wrote: 1) easy backup 2) easy construction of queries (SQL) 3) flexibility with construction of *new* queries that we haven't thought of before 4) don't (well, shouldn't) need to update hundreds of data structures upon every edit 5) databases are designed to do this sort of thing (esp. indexing) There are probably *loads* more. Timwi

Tomasz Wegrzanowski wrote: There is MySQL replication. I don't know much about it though (nor do I know much about rsync other than what it's used for), so I don't know. I suppose I could argue that you could couple this with CVS, which can produce patches to the dumps, but that's getting hacky :-) Ah, but that's not an indication that MySQL, or RDBMS in general, is a bad idea. I just shows that the current database schema is ill-designed. I'm not sure I fully understand what you mean by this. Apparently we do update something like the "recentchanges" table on every edit, because some people seem to have thought that it would make for better performance, but I'm pretty much convinced that this assumption is fallacious. Updating that table on every edit, I believe, is more costly than the benefit it brings in reading it. I'll gladly let an expert correct me on this, but this is what my past experience told me. Timwi

Ivan Krstic wrote: Do what? Recent Changes? Currently, on every edit, among lots of other things, this happens: - a row is inserted into "old" (involves a BLOB and several varchars) - the row in "cur" is updated (also involves a BLOB & several varchars) - another row is inserted into "recentchanges" (several varchars) This is three operations for something that can be done in one. (Simply insert a row into some generic table, 'articlerevisions' or somesuch.) There is neither a reason for the cur/old split, nor is there a reason for the 'recentchanges' table. Now, of course the database schema I proposed would indeed require two inserts (rather than just one) for this operation: - one insert for the metadata (edit summary being the only varchar) - one insert for the actual article text (the BLOB and nothing else) However, this is not noticeably more costly than adding the same data into one table that contains both (which is bad because then reading the metadata when you don't need the article text is inefficient). Furthermore, it means you don't lock an entire table with central importance for quite as long. Again, please let me stress that I do not claim to be an expert in this. Although I have created database schemata in the past, I have not had the benefit of testing them under load as heavy as that of Wikipedia. However, a major contribution to my experience is my involvement at LiveJournal, which is also a pretty heavily loaded website. Timwi

Lars Aronsson wrote: I know that; but you also need to consider that a write locks a database table, a read does not. Timwi

Ivo- This is an eternal debate. You may want to look at MoinMoin, which stores all its pages in plaintext and still manages to be very powerful: http://moin.sourceforge.net/ Also see my review of several DB and non-DB based wiki engines in c't 25/03. Using a database allows us to run a set of highly complex queries to examine the relationships between articles and the metadata about articles. In fact we can even examine the relationship between articles and users (watchlist), or between the metadata of different articles (redirect status in "What links here"). All that would only be possible using ugly hacks without a database -- you would effectively have to build your own mini-DB to store this metadata, and most text-based wikis do this. They get extremely slow when it comes to very basic stuff such as listing backlinks or doing a fulltext search. Not on a small wiki, but on one of the size of Wikipedia. I have imported large datasets into TWiki etc., and many operations became practically unusable. The MediaWiki FT search has locking issues. It's very fast for a normal wiki, but for a huge one like Wikipedia it tends to get in a deadlock state. That's why we have to disable it on Wikipedia. Some other queries could also be made faster. Many of our queries have been optimized now and every single one should eventually return results with a response time of milliseconds. It is a fallacy that a database makes things more complicated. It makes them much easier because you don't have to worry where to store something before implementing it. How do you want to store your user prefs, the redirect info, the recent changes table, the page history, always keeping quick access in mind? Usemod, Twiki etc. all use ugly hacks for that kind of stuff. In short, a DB is faster, cleaner, and more convenient. We benefit from the thousands of man hours that the MySQL developers have invested in building a scalable, fast, powerful multi-user database instead of having to reinvent the wheel whenever we want to do something that goes beyond "show me that file". Regards, Erik

Ivo- True. Go ahead, do a rewrite and show us how much better you can do with text files. Or if you don't want to do that yet, try loading all of the English Wikipedia (CUR and OLD) into MoinMoin and then try to mirror MediaWiki's functionality in MoinMoin. Really, this is not the place for philosophical discussions. Show us the code, please. Regards, Erik

Ivo Köthnig wrote: You have not provided examples of "overhead calculations" that a DB would have to perform that a file system wouldn't. You also haven't provided evidence of your claim that DBMSs cannot represent the required range of data structures. There is a huge number of organisatorial tasks that you would need to do with your files (when reading, that would be parsing the file, locating data, possibly caching and perhaps sorting; when writing, that would be updating indexes, managing "holes" in the file, and a huge host of other horrendous things). Next, if your data structures are spread over several files, you will have huge problems implementing atomicity, and you will run into a number of race conditions. DBMSs are specifically designed to do exactly these things. People often tend to forget that DBMSs aren't really much more than systems that manage files! But they have all these incredibly complicated and advanced things in it that you need and you really really don't want to have to re-implement from scratch. For 99.9% of data you want to store, you *can* create *exactly* the data structure you need using an RDBMS. It is pretty bold to think that you can do so much better (i.e. faster) than the hundreds of thousands of experienced developers who have put their effort into MySQL. Timwi

Ivo Köthnig wrote/schrieb/a écrit/skribis/etc.: I think you are greatly underestimating the powerful optimisations that can be done (and are done) in MySQL. Parsing an SQL statement is trivial (in terms of required computational power); I don't know much about relational algebra, but it seems that you would somehow have to express and then parse statements in that algebra too. If you don't have an index, then operations on huge tables are slow no matter what data structure you use. Any kind of indexing alleviates this problem, and RDMBSs do indexing really well. I think you haven't quite understood what I was getting at here. This is independent of what data structure you use. Yes, you do. You do need to manage your files. You do need to store your data structures somehow, and parse them, and update them, etc.etc. You never specifically need [[heap]]s. A heap is a special way of implementing a [[priority queue]] - if you need a priority queue, then you can do that by creating an indexed table. Whether the index is implemented as a heap should not be your concern (it should be the DMBS's). It is well possible that MySQL uses heaps, I don't know. Timwi

On Sat, Feb 28, 2004 at 08:37:44PM +0100, Ivo Köthnig wrote: No, it wouldn't use heaps. MySQL would use B+tree index. It would, if the indexes were set up right. As they aren't, it has to go through entire database (including cur_text) to find largest articles, project it so it contains only sizes, then use generalized merge sort to find the largest articles.

On Sun, Feb 29, 2004 at 12:53:46PM +0100, Ivo Köthnig wrote: If you think "heap" as in "priority queue", then no. Don't think O(). n on Wikipedia is known. By separating cur_text from the rest of cur we can make all searches order of magnitude faster - without changing O().