This works too, but its slower than mollasses on a cold Utah day .... :-) Jeff Brion Vibber wrote:

Jeff wrote: Shouldn't be realley needed. You parse < && > tags. The problem is that some tags can be splitted. You get "..long long line</te" and on next line "xt>" and *if* you're looking for "</text>", you have problems. </text> is tricky, because most tags start on their own line, but </text> doesn't (unless article ends with its own blank line).

On 1/17/07, Harish TM &lt;harish.tmh(a)gmail.com&gt; wrote: Any category link without a leading colon will add the category. With a leading colon, it links. So [[Category:Foo]] categorizes, [[:Category:Foo]] links. You can use that to your advantage. You may wish to look over the parser at http://svn.wikimedia.org/viewvc/mediawiki/trunk/phase3/includes/Parser.php?…. Be aware that the parser is not short, simple, or clean, because (as you've discovered) neither is the markup language. The legal title characters, by default, are " %!\"$&'()*,\\-.\\/0-9:;=?@A-Z\\\\^_`a-z~\\x80-\\xFF+". Therefore, a reasonable PCRE regex (with no string escaping of backslashes et al) might be: /\[\[Category:([ %!"$&'()*,\-.\/0-9:;=?@A-Z\\^_`a-z~\x80-\xFF+]+)\]\]/i Which would give you the category name. Hopefully. In most cases. Because other things can make a title invalid. Basically, it's a mess, and the only surefire way around it is using MediaWiki itself to do the parsing. Basically: install MediaWiki and hack it to do what you want. The parser is not well-defined or application-independent. You can make your own simplified parser, but it *will* fail in corner-cases (unless your simplifications consist of hacking out stuff you don't need and inlining stuff from other files). Use your own script to parse the XML part of the dumps and reformat them how you like. Then run each article's text through the parser, and grab the category list it spits back. It will not be very easy.

Simetrical <Simetrical+wikitech@...> writes: Note that one also needs to cope with leading and trailing spaces, and the "Category" is case-insensitive. So probably something more like: \[\[ *[Cc][Aa][Tt][Ee][Gg][Oo][Rr][Yy] *: * ... Or, to download categorylink.sql file from the same dump, and use the entries from that. A bit redundant if one needs to parse the markup anyway, though... A good first step is probably to use an XML toolkit/parser library of some kind for whatever PL one might be using for the task. Unless one happens to be using Haskell, I can't offer any specific advice on that... (Did the original querant specify?) Not a complete solution, since the markup is something of an XML/HTML mish-mash, but it'll still cope with much of the structure. Slan, Alai.

On 1/17/07, Harish TM &lt;harish.tmh(a)gmail.com&gt; wrote: The easiest way would probably be: 1) download the dumps 2) install mediawiki 3) have a bot scrape from *your own* mediawiki installation (printable version) 4) use the categorylinks table to sort by category (or scrape the categories at the bottom of each article) On the other hand, you could always write your own parser :). Anthony

Alai wrote: the Or just use an "ignore case" option like /i (in pcre)... -- chris

Jeffrey V. Merkey wrote: Well... what a surprise </sarcasm>

You could also just modify this code (released unde GPLv3) and use it to strip out titles. Stuff it into a file under linux called "parsetitle.c" and type: gcc parsetitle.c -o parsetitle ./parsetitle < enwiki<date>.xml > titles.txt Jeff #include "platform.h" #ifdef WINDOWS #define strncasecmp strnicmp #include "windows.h" #include "winioctl.h" #include "winuser.h" #include "stdarg.h" typedef UCHAR BYTE; typedef USHORT WORD; #include "stdio.h" #include "stdlib.h" #include "ctype.h" #include "conio.h" #endif #ifdef LINUX #include <unistd.h> #include <stdio.h> #include <stdlib.h> #include <fcntl.h> #include <ctype.h> #include <string.h> //#include <ncurses.h> #include <termios.h> #include <sys/ioctl.h> #include <sys/stat.h> #include <pthread.h> #include <sys/types.h> #include <sys/socket.h> #include <netinet/in.h> #include <net/if.h> #include <stdio.h> #include <errno.h> #include <stdlib.h> #include <string.h> #include <unistd.h> #include <sched.h> #include <ctype.h> #include <openssl/md5.h> #endif char buffer[0x10000]; char title[4096]; int main(int argc, char *argv[]) { register char *s, *p; register int intitle = 0, i, f, inpage = 0; register int titlefound = 0, revision = 0, inrev = 0; while (s = fgets(buffer, 0x10000, stdin)) { while (*s) { if (!*s || *s == '\n') { if (*s) { // putc(*s, stdout); s++; } break; } if (!memcmp(s, "<page>", 6)) { s += 6; inpage++; titlefound = 0; revision = 0; continue; } if (!memcmp(s, "</page>", 7)) { s += 7; if (!titlefound) fprintf(stdout, "no article title?\n"); if (!revision) fprintf(stdout, "no revision?\n"); titlefound = 0; revision = 0; if (inpage) inpage--; continue; } if (!memcmp(s, "</revision>", 11)) { if (inrev) inrev--; s += 11; continue; } if (!memcmp(s, "<revision>", 10)) { inrev++; revision = 1; s += 10; continue; } if (!memcmp(s, "<title>", 7)) { intitle++; s += 7; p = strstr(s, "</title>"); if (p) { if (intitle) intitle--; if (p - s) { strncpy(title, s, p - s); title[p - s] = '\0'; s += (p - s); for (f=i=0; i < (p - s); i++) { if (!isspace(*p++)) f = 1; } if (f) fprintf(stdout, "[%s] SPACES?\n", title); else fprintf(stdout, "[%s]\n", title); } else fprintf(stdout, "[%s] NULL?\n", s); titlefound = 1; continue; } if (intitle) { intitle--; printf("state error: title spanned segments [%s]\n", s); continue; } } // putc(*s, stdout); s++; } } return 0; } Matthew Flaschen wrote:

zetawoof wrote: If they span buffers, the xml parsing libs choke as well, just for your information. I know, I've seen them do it. I have a version that does not choke on buffer spanning, it buffers underneath. I just posted that as an example. Jeff