On Tue, Nov 18, 2008 at 3:22 AM, Tim Starling <tstarling(a)wikimedia.org> wrote: I do have the time to maintain DB2 support for the foreseeable future. There's a couple sites I'm involved with that would use it, so the long-term interest is there. What do I need to do to get commit access? My public key is: http://lpetr.org/personal/leo-public.pub That's a fair bit of work. Moving the code wouldn't solve the maintenance issue by itself, though. Would you make Database an abstract class? That would get us this: - DB-agnostic methods could still be defined in Database - DB-specific methods would have to be defined in any child class - there would be a clear error if a required DB-specific method was not implemented This would be a definite improvement over the current situation where, say, the Oracle layer could have a missing method, escalate to a MySQL-specific one in Database, and fail with varying degrees of mystery. Ah, that would be quite useful. If done right, it could also be used for installation in place of the DBMS-specific tables.sql files. What I am envisioning is an abstraction over Data Definition Language -- create table, alter table, rename column stuff -- that's similar in architecture to the Database classes. There'd be a root class that defines a DB-agnostic interface, and DB-specific child classes that implement vendor-specific things. This would solve the maintenance headache of having several different database schema definitions -- a separate tables.sql to maintain for MySQL, Postgres, SQLite, etc. A rework of updaters.inc would be a good start. -- Leons Petrazickis http://lpetr.org/blog/

Hello. I'm a new poster to this list, and I find this topic fascinating. Tim Starling escribió: I believe a better solution is to design a domain-specific language, an idea not very different from your first one. This DSL would model the interaction between the application and the DB as it is now, and would be designed to evolve. That's it. Your other ideas aim to more general-purpose solutions. I believe those would be harder to develop, more complex, and not as useful. If the DSL is well designed, new features would potentially require new domain-specific language elements to be added, and none to be removed. If it is badly designed, however, the end result is usually like you describe. But it's not an inescapable end. If you like the idea, I can develop a throwaway testbed for an embryonic DSL, to clearly see if it is suitable. Best regards.

On Wed, Dec 24, 2008 at 3:17 AM, Nikola Smolenski <smolensk(a)eunet.yu> wrote: This is not inherently different from people having to learn what MediaWiki classes and so on do. If it's a well-designed language, it will be as easy for people to pick up as alternative ways of writing the same info. The first step absolutely must be to look at what other open-source products in our situation are doing, though. If we can take the entire system wholesale from phpBB, say, then there's no reason for us to reinvent the wheel.

On Wed, Dec 24, 2008 at 2:19 PM, Daniel Friesen &lt;dan_the_man(a)telus.net&gt; wrote: I just looked, and you're right, they seem to rely on things like this for multi-DB support: /** * Oracle specific code to handle it's lack of sanity * @access private */ function _rewrite_where($where_clause) { preg_match_all('/\s*(AND|OR)?\s*([\w_.]++)\s*(?:(=|<[=>]?|>=?)\s*((?>\'(?>[^\']++|\'\')*+\'|[\d-.]+))|((NOT )?IN\s*\((?>\'(?>[^\']++|\'\')*+\',? ?|[\d-.]+,? ?)*+\)))/', $where_clause, $result, PREG_SET_ORDER); $out = ''; foreach ($result as $val) Probably not a model we want to follow, although I don't think that using raw SQL is necessarily bad in principle (using string concatenation to include variables certainly is, though).

That's disgraceful. I might have to boycott phpBB just because of that... Soxred93 On Dec 24, 2008, at 2:36 PM [Dec 24, 2008 ], Chad wrote:

On Wed, Dec 24, 2008 at 2:36 PM, Chad &lt;innocentkiller(a)gmail.com&gt; wrote: On Wed, Dec 24, 2008 at 2:41 PM, Soxred93 &lt;soxred93(a)gmail.com&gt; wrote: There's no need to flame other open-source projects here. We have code that's at least as awful if not more so, including *plenty* of places where we still call $dbr->query() directly (like everything based on QueryPage). As far as I know, our abstraction layer was devised by Domas early on in MediaWiki's existence. If he hadn't decided to go in that direction, we'd likely be using something similar. Besides, ugly or not, their multi-DB support looks to be a lot better than ours. As far as I know, MediaWiki barely works at all on anything other than MySQL and pgsql, and doesn't work fully on the latter either (various maintenance scripts, extensions, etc.).

On Wed, Dec 24, 2008 at 3:25 PM, Nikola Smolenski &lt;smolensk(a)eunet.yu&gt; wrote: If the domain-specific language is SQL-like, that benefit would exist for it too, in fact even more so. Syntax errors in a domain-specific language would often be logic errors in something shoehorned on top of PHP arrays, which your editor won't be able to highlight anyway. Plus, if it's SQL-like (and why shouldn't it be?), your editor probably *will* be able to highlight the code.

Hello. After a few days of pondering the issues, I would like to explain what I suggested in my previous message, in more detail and (hopefully) more clearly. What I'm about to say is pretty abstract, so it's difficult to convey the right meaning. Please forgive me if I say something you already know, or just nonsense :-) Jesús Quiroga escribió: The problem I discuss is how to best access the data store from an application. I believe the right answer is different for each project, but it's not difficult to evaluate the alternatives, one by one, in a given context. I think it is worthwhile to do that in the context of MediaWiki. I will refer to wiki modules and databases as if they were 'hosts' connected to a 'network', to highlight the role of languages in the operation of the system at runtime. The first way to access the data store is the 'direct' one: [polyglot wiki] <--- mysDataL ---> [mysql] [polyglot wiki] <--- posDataL ---> [postgresql] [polyglot wiki] <--- db2DataL ---> [db2] Here, the polyglot wiki module talks to every database using the proper languages. 'mysDataL' means 'the data language understood by MySQL', 'posDataL' means 'the data language understood by PostgreSQL', etc. The polyglot wiki promises to learn several languages and to speak them correctly forever, so, if a new database comes along or any of their data languages evolves, the polyglot wiki is forced to adapt at a potentially great cost. Besides, any change to the database schema can trigger lots of updates to the wiki code, and be very costly too. The advantages of this way are well known: it is fast, no need to do design, easy to understand. The drawbacks are apparently few, but devastating: verbose and complex code in multiple places in the wiki module, very costly to maintain, even more costly to evolve. All changes cost a lot, in time and effort. The second way to access the data store that is usually considered is the 'indirect' one: [wiki] <--- wikiDataL ---> [polyglot translator] [polyglot translator] <--- mysDataL ---> [mysql] [polyglot translator] <--- posDataL ---> [postgresql] [polyglot translator] <--- db2DataL ---> [db2] Here, wikiDataL means 'some relational data definition and manipulation language suitable for use by the wiki'. The polyglot translator promises to learn wikiDataL and the other dialects and to evolve with them, so it has all the problems the wiki had in the direct way, but now the cost is lower because a lot of complexity is 'hidden' inside the translator and can't reach the wiki. As a result, wiki code is not updated as much, and it's much cleaner and less verbose. The advantages of this way are: wiki module code is simpler, cost of evolution is reduced. The drawbacks are apparently many: it's slower, design is needed, harder to understand, a new language (wikiDataL), translator can be very complex. However, the need to reduce the cost to achieve change is usually so great that these inconveniences are minor in comparison. Now the interesting bit begins. A third possible way to access the data store, the 'interpreted' one: [wiki] <--- wikiNeedL ---> [polyglot interpreter] [polyglot interpreter] <--- mysDataL ---> [mysql] [polyglot interpreter] <--- posDataL ---> [postgresql] [polyglot interpreter] <--- db2DataL ---> [db2] Here, wikiNeedL means 'some language adequate for the wiki to express its data access needs and nothing else'. wikiNeedL is the domain-specific language I wrote about in my previous message. The differences between wikiDataL and wikiNeedL are mainly these: - wikiNeedL would contain just enough wiki concepts to express the wiki's needs, so it's effectively confined to that domain. wikiDataL belongs to the relational data model domain, which is quite different. - in general, wikiNeedL would have different semantics than the dialects understood by the databases, so the translation step becomes more like interpretation, rather than just syntactic transformations. wikiDataL usually has the same semantics than the dialects. - wikiNeedL would contain just enough concepts to satisfy current needs, and will be open to extension. wikiDataL aims to be general-purpose and to fulfill current and future needs. The main reason to consider the 'interpreted' way is, of course, that it helps reduce even more the cost to achieve change. So that's what I was talking about. I will say more about the differences between the indirect and the interpreted ways in a future message. Thanks for your attention.

Aryeh Gregor escribió: I understand you mean the indirect way is the 'official' way now, and that's is quite understandable. However, there are parts that 'lag behind' because they still use the obsolete direct way, and at the same time there are wikiNeedL phrases already in some other places, so I see that MediaWiki uses parts of all three. Usually, transitions from one way to other happen slowly, and feel like gentle evolutions, either towards improvement or towards degradation. I agree, the indirect way can support many backends, no question. In any case, it will be a comparison between 'good' and 'better', or to be clearer, between 'cheap' and 'cheaper', or perhaps 'successful' and 'wildly successful'. I agree completely, some wikiNeedL is already in there, and it's implemented using method calls and their arguments, which is just fine and it works. I'm confident you'll see it's not that complicated. For example, I found these in Article.php, method updateRedirectOn(): $dbw->replace( 'redirect', array( 'rd_from' ), $set, __METHOD__ ); $dbw->delete( 'redirect', $where, __METHOD__); These I consider wikiNeedL phrases, and they're (almost) perfect. Some helper agent is located and told about some need that ought to be fulfilled, and what is needed is expressed clearly using wiki concepts. No details about how to do it are included, except for that string 'rd_from' which seems to be a column name in some relational schema, and that would be forbidden in wikiNeedL proper. Of course, simple expressions are easier to manage, even trivially so, but they are good examples of the ideas I described. Another example, consider the query service (https://wiki.toolserver.org/view/Query_service), where someone with a need to access the data store manages to do so by delegation, expressing the need using wiki concepts. Of course, they are humans, but the model is the same. The usefulness of the interpreted way can be compellingly argued, and I plan to do so in a separate message. Finally, to get the full benefits of wikiNeedL, if it is deemed to be a good idea, I believe the first step should be to begin to steer its evolution in a top-down fashion, to analyze what is there, to bring it forward, and to make it more 'official'. Best regards.

I think you are way overthinking things in general, and failing to understand the existing code in the specific two examples above. Both contain a lot more "details about how to do it" than just the 'rd_from' column (see: 'redirect', $set, $where). It might be helpful if you took the relevant surrounding code (e.g. where $set is defined) and rewrote it in the newly proposed system so we can see exactly what you are talking about. -- Greg Sabino Mullane greg(a)endpoint.com End Point Corporation PGP Key: 0x14964AC8

On Tue, Dec 23, 2008 at 3:08 AM, Tim Starling &lt;tstarling(a)wikimedia.org&gt; wrote: leonsp, please.:-) -- Leons Petrazickis http://lpetr.org/blog/