First of all, thanks for your replay and your suggestions.

CC'ing to Wikitech-l. Some more review:

• The way NUMBEROFBOOKS is calculated (scanning the page namespace) is

not acceptable, performance-wise. It needs to work like NUMBEROFPAGES, etc., with a site_stats row or similar.

OK, I'll learn how use caching :-D

• The allbooks-regex message might not be a great way of doing things:

** Since results will have to be stored in various ways, like the count being stored in site_stats, it's probably not feasible to change the regex used without running a maintenance script. A config option might therefore be better. ** Directly inputting the regex into a query is VERY VERY BAD! It's an immediate SQL injection attack. Also, it will cause the query to break in the presence of things like single quotes. Use Database::addQuotes() here.

Ok i'm going to correct it

• Generating AllBooks using a REGEXP query is . . . maybe not ideal,

altogether. Usually we have a flag in the table, for instance, page_is_redirect. It might be okay given that you'd "just" be scanning a few times as many rows, in the average case, and only for views of a certain special page. But a flag would be nice.

What do you mean? A "is_book" flag? I saw that exists the a kind of "is supbage" flag but unfortuately not every Wikibooks use the subpage convention :(

• If you're using the Xml functions, you don't have to use

htmlspecialchars explicitly. It will double-escape the variable.

• Variable names that are in Italian are kind of funny, but not really

in accordance with our coding standards. :) $conta -> $count, $numero -> $number

Yes, it was a beta, I'll fix them (italian was for developing :D )

This is only a quick glance, mind you. I haven't actually tested it and probably won't find the time to do so.

Overall, I'm not sure this is the best way to tackle the problem. Has Wikibooks considered, for instance, having the books' "main pages" be in the main namespace, and the various pages be in a namespace like "Page"? Then the problem is partly solved right away: you can use Special:AllPages set to the main namespace to get a list of books. For NUMBEROFBOOKS, you just need PAGESINNAMESPACE to be enabled, which is fairly feasible at some point if someone's willing to do a little optimization work, since it's a generally-requested feature.

I think it is not very intuitive -- the real content should be in the main namespace. The best thing would be that each Wikibooks followed the subpages naming convention (book title/chapter). IT may be a multilingual policy or something like that.

By the way, is there any developer who can help me? I have knowledge of PHP but i'm not a very great mediawiki developer. It would be better having some help by any expert developer, mostly for caching and other advanced stuff :-).

--Ramac

On Thu, Apr 17, 2008 at 5:33 PM, mike.lifeguard mike.lifeguard@gmail.com wrote:

...

If you remember back in November we were asked what would be on a technical wishlist for Wikibooks. One of the things we said[2] we wanted was a way to list all books & enumerate them (all *books* as opposed to all pages). Ramac and Pietrodn have been working on an extension[2] which does this.

Darklama and I have taken an initial look at things, and Simetrical found one error that has been corrected. Anyone who is technically-minded is invited to review & test the code. Everyone else is invited to discuss improvements.

The extension is documented at mediawiki.org, but here's a quick overview for the sake of convenience:

*Adds the {{NUMBEROFBOOKS}} variable

*Adds [[Special:AllBooks]], which lists all books in specified namespaces

*List books beginning at some prefix using ?offset=whatever

*The regex used to determine what is a book and what isn't is a system message, so administrators may edit it, though the default should work for most (all?) Wikibooks languages

After this much time with apparently no interest in developing anything for Wikibooks it's exciting to finally see such active development on something needed specifically for this project. Many thanks to Ramac and Pietrodn for their work, and also those who have helped them along the way. Hopefully this is the beginning of a trend, and more tools will be developed to fulfill our wishlist!

-Mike.lifeguard

[1] http://lists.wikimedia.org/pipermail/textbook-l/2007-November/001164.html

[2] http://www.mediawiki.org/wiki/Extension:AllBooks

---

Textbook-l mailing list Textbook-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/textbook-l

---

Textbook-l mailing list Textbook-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/textbook-l

On Fri, Apr 18, 2008 at 12:07 PM, Simetrical Simetrical+wikilist@gmail.com wrote:

On Fri, Apr 18, 2008 at 11:54 AM, Raffaele raffaelemac@tiscali.it wrote:

...

I think it is not very intuitive -- the real content should be in the main namespace.

Not necessarily. Content that is logically different and that you want to be distinguished technically should go in separate namespaces.

Content from a sub-page is not logically different from the content of the book's base page. "Books" are more of a concept then an actually entity, they are a collection of pages. It is up to the author to determine how those pages are organized, and what is the logical relationship between them. Sometimes the base page is a title page, sometimes it is a table of contents, sometimes it is merely a redirect to one of these. Sometimes we have "Book/Chapter/Page", sometimes we have "Book/Page", sometimes we just have a one-page "Book". The basepage has no special significance besides being the most common landing point for new readers, and a prefix to differentiate the pages of one book from all the others.

Having subpages in a different namespace from the book's main page is completely illogical and, from my standpoint, unacceptable. The vast majority of all content pages on our site would need to be moved to a new namespace. Links would need to be updated, including links which are dynamically generated for in-book navigation. Consider the common snippet from a navigation template:

{{#if:{{{next|}}}|[[Bookname/{{{next|}}}|Next Page]]|}}

A bot likely can't edit all these links automatically, it is going to involve a huge waste of manpower resources to perform a task that could be easily performed by a single programmer writing a small extension instead.

Moving subpages to a new namespace would cause us to lose the automatically-generated back-links at the top of the page, which many books still rely on for navigation. Writing an extension or a javascript to replace these backlinks for pages which are written cross-namespace would be no easier then writing this allbooks extension.

Books are already separated from their subpages technically, using slash delimiters. Different books have different prefixes.

Books are single self-contained units, not to be constructed from bits and peices strewn across multiple namespaces.

For instance, mediawiki.org has most of its content in non-main namespaces, such as Manual:, Extension:, and Help:. In fact, enwikibooks already has a Cookbook namespace where a substantial amount of content resides.

The cookbook is a single book. Prior to creating the namespace, most of the pages in the cookbook were already titled using the pseudo-namespace "Cookbook:Subpage", which was deemed problematic. we created the new namespace for it in order to leverage searching and listing tools. If you think that this is the way to go, then every single book on Wikibooks should get it's own namespace. We would also need a form for easily creating new namespaces, since we don't want to have to wait on developers everytime we want to start a new book. We would also need a {{NUMBEROFNAMESPACES}} variable to count the number of books.

What works for one project, like Wikipeda or Mediawiki.org is not going to work for all projects. We want to do this the way that is right for us. If we wanted to do it the wrong way, we would just stay as we are now.

If you preferred, you could have the books in a Book: namespace and the pages in the main namespace.

This doesn't work either, for the same reasons. All content on Wikibooks is a "book", We shouldn't have to write [[Book:Bookname]] because all top-level pages are already books. You're trying to draw an unnatural distinction between a "book" and a "page" which is just not supported in the way we work. The pages in a book belong together.

The point is, don't try writing an extension when reorganization would do the same thing to better effect, using already-existent core software features.

The effect would be much worse, and I would consider either situation you suggested to be intolerable. All Wikibookians know what Wikibooks is being written on the "software that runs Wikipedia", we've been a round peg in the square hole since our project was created. We've had to come up with lots of hack-job solutions because what is best for our project is rarely well-supported in the software. We were asked for a wishlist of things that we would like as a project, not a list of more hack-jobs that we could throw together ourselves at the expense of doing things the right way.

Deciding on how to organize the site without any consideration for how well that works with the software you're using, then trying to patch the software to match your preordained specifications, is not the sensible way to do things.

The way we are organized is logical, and well-supported by the software. Using forward slashes to separate subpages is used on most Wikimedia projects, including occasional usage on Wikipedia. the recently added {{#titleparts:}} parser function supports this convention. The existence of backlinks at the top of the page support this convention. The only thing we don't have is an automatically-generated count and list of base pages. If the answer is, as it usually turns out to be "too bad, nobody cares about Wikibooks", then we'll just do things our own way.

--Andrew Whitworth

On Fri, Apr 18, 2008 at 1:43 PM, Simetrical Simetrical+wikilist@gmail.com wrote:

On Fri, Apr 18, 2008 at 1:02 PM, Andrew Whitworth wknight8111@gmail.com wrote:

...

Content from a sub-page is not logically different from the content of the book's base page.

Of course they are. Otherwise you wouldn't want to list and count it separately. If you prefer, the book *page* is logically different from the book's *subpages*, even if their respective contents may not be logically much different.

We're not interested in counting the individual pages (although I suspect that's one of the easiest ways to go about this), we're interested in counting groups of pages with the same prefix name. Logically, it doesn't matter whether a particular page is named "Bookname" or "Bookname/Subpagename". Consider a book with slashes in the title, like "P/L programming". We wouldn't have a page named "P" at all to be counted, but "P/" still serves as a unique prefix for all pages in that book. Or the book we had about Oracle, which until just recently didn't have a base page at all, there were [[Oracle/Contents]] and [[Oracle/Cheatsheet]], but no [[Oracle]].

What we are after here is a simple count, a single integer value that is probably in the neighborhood of 3000-4000 for en.wikibooks, and is significantly lower for most other wikibooks or wikiversity projects (since wikiversity is likely to want a feature like this too). Massive across-the-board reorganization of the entire project is simply not an acceptable solution for this problem.

We would also like a listing of books, although most of our books are already well-categorized, and we can obtain mostly-accurate lists from these categories (although it would be nice to know which "books" are not properly categorized, that's a small issue).

Consider that a solution from our point of view is to create two bots: 1) Bot 1 makes repeated calls to api?action=query?list=allpages to download the complete wikibooks page list. 2) Bot 2 applies a simple regex to that page list, counts the number of unique prefixes. Posts an edit to the page [[Template:NUMBEROFBOOKS]] with that number, and posts an edit to [[Wikibooks:List of all books]] with the complete list.

I doubt this would be an ideal solution, but it would give us exactly what we want.

An alternative approach to fix the problem would be to improve support for subpages. This is, from a software point of view, a path of greater resistance, since namespaces are already well-distinguished and subpages are not. It would have to be thought out very carefully. One thing that occurs to me is that theoretically, it doesn't have to be set in stone that different namespaces correspond to different colon-delimited prefixes. Hypothetically one could have "Book" in the main namespace and "Book/subpage" in another namespace, even though they have the same prefix. But this would require a considerable amount of effort to implement.

Again, measure the amount of implementation effort against the desired goal. All we want is a single integer value. A massive programming effort is no more desired on our end then a massive reorganization of our books.

Wikibooks is, comparatively, in the excellent position that if anyone wanted to become a developer, they could fairly easily become one, code up any features that they wanted, and have them incorporated into the main product without having to maintain hacks.

This is exactly what we are trying to do. Ramac is trying to code this feature up as an extension and have it added for Wikibooks.

Only if a feature is intrinsically of very narrow interest should it be put in an extension.

I would venture to guess that this feature is of particularly narrow interest. I can only imagine that, based on the way I know other projects to handle naming, only Wikibooks and possibly Wikiversity would use it. Hence, the extension.

--Andrew Whitworth