Michael,

Thanks for the reply .. a couple notes

".. Word - for one example - is a "standard" that is changed by MS every so often for no apparent purpose other than to screw the competition .. "

I get it. You are a MS hater and that is cool. I am not that fond of them either which is why we use OpenOffice in our business and on my systems at home. But the simple fact is that there are millions of users. Some of them are clients.

" Unless the standard is open, published, unrestricted and not under the

control of a single commercial entity, it is not a standard - it is proprietary. You may like the idea of chasing a moving, camouflaged target, but I don't."

Never said it was a standard, I said it was 'universal' which, of course, is a very different thing. Also I never suggested that '.. chasing a moving, camouflaged target ..'. Unless MS removed every 'text' word from their document I don't see where an extension that could index the words would be a moving target. Now possibly MW has been designed in a way that would make this not possible, I haven't gotten that far. But I have written, other environments, methods of scanning word docs.

"Supporting an open standard and encouraging companies like MS to allow export of their products' files in the open standard format as an alternative to their own is a better solution. MS already exports to RTF, txt and other formats. Adding an open format shouldn't be a big deal."

Do you think that MS gives a crap whether MW is or is not capable of indexing word documents? Unless, of course you feel able to calling Bill and having him change his format.

I believe that the market place will decide, rightly or wrongly (and who decides that?), what tools and "standards" will be used. It is my job as a consultant to give the user the best information and let them decide. Then either support the client in their decision or go find another client.

Of course right now it doesn't really matter as the only TYPE is IMAGE and MW doesn't appear to be able to search on it (which makes sense if the only type I wanted to store was IMAGE.

Thanks for the note. It is much like discussions I have with Linux friends <G>

Dave Sigafoos wrote:

There are a couple formats that seem to be 'universal' whether we like it or not. Word, excel, ppt, pdf then emerging standards like those coming from open office.

Word - for one example - is a "standard" that is changed by MS every so often for no apparent purpose other than to screw the competition. The latest Office 2007 version apparently breaks a lot of other products because of changes to the "standard".

Unless the standard is open, published, unrestricted and not under the control of a single commercial entity, it is not a standard - it is proprietary. You may like the idea of chasing a moving, camouflaged target, but I don't.

Supporting an open standard and encouraging companies like MS to allow export of their products' files in the open standard format as an alternative to their own is a better solution. MS already exports to RTF, txt and other formats. Adding an open format shouldn't be a big deal.

Mike

_______________________________________________ MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/mediawiki-l

On 09/04/07, Rob Church robchur@gmail.com wrote:

On 09/04/07, Dave Sigafoos davesigafoos@sanmar.com wrote:

...

I get it. You are a MS hater and that is cool. I am not that fond of them either which is why we use OpenOffice in our business and on my systems at home. But the simple fact is that there are millions of users. Some of them are clients.

No, it's true - Microsoft are known to keep changing their proprietary formats, and it is fair to state that this interferes with anybody trying to interpret them for any purpose, legitimate or otherwise.

...

Of course right now it doesn't really matter as the only TYPE is IMAGE and MW doesn't appear to be able to search on it (which makes sense if the only type I wanted to store was IMAGE.

David, from what I gather, you keep stating that we can't "search on (images)" but you haven't said what you'd actually like the software to be able to do. It may be possible to add some sort of extensible mechanism for MediaWiki to be able to index uploads in some fashion; if that's what you want, then endorse it.

All I can see is a bunch of "MediaWiki can't even do this, it sucks" type comments from various quarters, none of which are making the problem clear enough for us to understand.

Rob Church

---

MediaWiki-l mailing list MediaWiki-l@lists.wikimedia.org http://lists.wikimedia.org/mailman/listinfo/mediawiki-l

Dave Sigafoos:

I have stated that I want to be able to index other types of documents. That just having and IMAGE type, while probably fine in the beginning and probably find for the majority of use, might be *limiting*. I believe that these 2 things are still there so that is what I am looking at.

I don't think you're quite getting the problem here. As I said before, identifying the type is not the problem. The fact that the Image: namespace is called Image: is totally irrelevant. There is no "IMAGE" type in MediaWiki, in the sense in which I think you mean it; there's just a place where uploaded files of *all* types get stored, and it happens to be called "Image:".

Right now, if you upload a Word or PPT document, we can easily identify the type, either by running something like Unix's "file" on it, or simply by looking at the suffix (.doc, .ppt, etc.) of the filename. But as I stated before, there are two reasons why we cannot search on these documents, and neither of them has anything to do with identifying their type.

1. Right now, MediaWiki's searches are implemented by using the MySQL search feature. Unlike regular Wiki pages, uploaded documents do not go into MySQL, and therefore cannot be searched in this way.

2. You can only search for a text string in a file if the file is in a format you understand. That means writing a custom decoder for every format you want to handle. For plain text files, this is trivial, but for other files, it is not. For example, here's an excerpt from a PDF file:

ZÊü'¾h‹žu;ý,½‰Ÿóƒ‡·1oµ÷¯ù/g>QÁP€$IabþƺžUБ›ÈsF± _ÚæcçY~W¬%ó?åÈC‹›œ¯¼ÐO8 Ccý"š0~”œ%ò<GÃyì ÉeºœŸ¸ c+±j5[©J²WW ŒýDDÑ)Dp

Do you see the words "Inspection of clamps and of flexible piping" in there? They're in there.... somewhere. Simply knowing that this is a PDF doesn't help us to find them, though. Like it or not, formats like Word and Excel are proprietary, and that does make writing third-party tools for them harder.

So, in an attempt to take the heat out of this and get to facts, what I think you're looking for is an extension to: 1. allow admins to configure decoders for specific document types 2. run the right decoder (if any) when a document is uploaded 3. add the resulting plain text to the "searchindex" table. You would then have to find, install and configure decoders for your most-used document types.

I haven't had time to look into this in detail, but maybe more knowledgeable folk here could comment on whether this strategy makes sense; or, indeed, whether something like this already exists or is in progress.

Ian

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Moin,

On Monday 09 April 2007 20:59:36 Joshua Yeidel wrote:

The approach suggested by Ian Smith below is the one adopted by a couple of systems I work with. For example, Microsoft SharePoint 2007 uses "iFilters" for each document type to extract indexable information. Mac OS X's "Spotlight" feature also has per-filetype "importers" for extracting indexable text.

So the concept is not unworkable, but it seems to me to be a stretch for MediaWiki. MW is a Wiki-Page Management System™ <grin>, at which it excels; it's not a very good Document Management System, which is where Dave Sigafoos is apparently being driven (perhaps in slow stages) by his users.

Perhaps Dave should investigate other document management approaches and a metasearch engine to search across multiple systems.

Well, or write an extension that implements his idea, e.g.:

* upon upload, run the document through an index-generator (per file type) * add that index-text to some searchable index, or store it in the mysql

for each file type, you can do something like:

pdftotext $pdf_file exif $image_file etc.

There are very very probably filters for doc, ppt, xls, etc. If not, one can always whip one up with a Perl module (I know there exist modules for office and excel) While these might not get all the formatting etc, they will be able to extract the bulk (if not all) of the text and you can then easily index & search this text.

Another option would be to just let the webserver handle this, by running htdig (or google appliance?) over the uploaded files (which end up in wiki/images, anyway) and present the user with a search box to search all these files. The second option wouldn't integrate with mediawiki that nicely, tho.

All the best,

Tels

- -- Signed on Mon Apr 9 23:34:55 2007 with key 0x93B84C15. Get one of my photo posters: http://bloodgate.com/posters PGP key on http://bloodgate.com/tels.asc or per email.

"One man in a thousand is a leader of men, the other 999 follow women"

-- Groucho Marx

I've thrown some very experimental initial work on such an extension into Subversion; trunk/extensions/FileSearch.

Sounds great, Rob! Unfortunately, I can't access SVN at the moment*, so I'll just have to take your word for it now...

* I keep getting "Forbidden. You were denied access because: Access denied by access control list." for the SVN Web interface, as well as for Meta and images on Wikipedia. I'll try again from home tonight.

...

add that index-text to some searchable index, or store it in the mysql

Maybe I'm missing something here, but why not just throw that "index-text" into the file's description field?

-- F.

...

Maybe I'm missing something here, but why not just throw that

"index-text" into the file's description field?

Do you really want a 200kb Word document transcribed into the page itself?

It could always be enclosed in a "<DIV style='display: none;'>" wrapper...

However, loading times might indeed be an issue - though probably less so for intranet networks.

-- F.

On 09/04/07, Ian Smith ismith@good.com wrote:

So, in an attempt to take the heat out of this and get to facts, what I think you're looking for is an extension to:

1. allow admins to configure decoders for specific document types
2. run the right decoder (if any) when a document is uploaded
3. add the resulting plain text to the "searchindex" table.

You would then have to find, install and configure decoders for your most-used document types.

Sounds good. e.g. Antiword is a quick way to turn a Word document into indexable text.

Now all MediaWiki needs is a text search that doesn't suck ;-p

- d.

On 10/04/07, Platonides Platonides@gmail.com wrote:

David Gerard wrote:

...

Sounds good. e.g. Antiword is a quick way to turn a Word document into indexable text.

AbiWord --to=txt http://wvware.sourceforge.net/

There's lots of ways, yes :-) Something to extract indexable text from any document there's a filter for, and feed it to the indexer. That'd be just what we need to index Word documents added to a MediaWiki. Does anything like this exist already, or is it a Simple Matter Of Programming?

- d.

Dave Sigafoos wrote:

Unless MS removed every 'text' word from their document I don't see where an extension that could index the words would be a moving target.

I have a lot of old documents in IBM's Bookmanager format. They are encrypted in such a way that no one can scan past the formating info and find the text. I rely on some old, buggy Bookmanager software to access them and expect that with another change of OS version I will lose the ability to use them. IBM has never released the internal format specification for the documents and nothing I've been able to do has wrested the info from them.

I keep expecting MS to pull a similar stunt with Word. They could sell the encryption as a "security" feature.

Do you think that MS gives a crap whether MW is or is not capable of indexing word documents? Unless, of course you feel able to calling Bill and having him change his format.

MS might give a crap about the laws currently being pushed out that force open standards for document storage. These governments don't want to have docs stored that become obsolete because of one vendor's decision to change their format. This was what I was thinking of when I mentioned saving in an open standard. Of course, the battle right now is MS's version of it's "open standard" versus the open source community's desire for a truly open standard.

It's been in the computer news so much lately I thought you'd get the drift of the comments. I guess I shouldn't have been so obscure. Sorry!

I believe that the market place will decide, rightly or wrongly (and who decides that?), what tools and "standards" will be used.

It looks like the elected reps will beat the market to it. That of course brings its own risks/rewards.

Of course right now it doesn't really matter as the only TYPE is IMAGE and MW doesn't appear to be able to search on it (which makes sense if the only type I wanted to store was IMAGE.

Searching on images is a major problem. Do you search only on names of images, on descriptions or on content? Searching content is still a significant research effort in image processing and recognition.

The only restriction on what you can upload is in the extension list in Localsettings.php. My wiki allows specific text uploads. There's an extension I'm working on that processes them (mod of an existing extension). I don't see any reason why you couldn't make an extension that does the same with any chosen doc format.

Mike