Karl Matthias wrote:
I'm one of the authors of the Kiwi parser and will
be presenting it at
the Data Summit on Friday. The parser is pretty complete but certainly
we could use some community support and we encourage feedback and
participation! It is a highly functional tool already but it can use
some polish. It does actually handle most wikitext, though not
absolutely everything.
From your post I can see that you are experiencing a couple of design
decisions we made in writing this parser. We did not set out to match
the exact HTML output of MediaWiki, only to output something that will
look the same in the browser. This might not be the best approach, but
right now this is the case. Our site doesn't have the same needs as
Wikipedia so when in doubt we leaned toward what suited our needs and
not necessarily ultimate tolerance of poor syntax (though it is somewhat
flexible).
I felt bad for pointing out issues just after first try. I understand
that you have a much smaller content than wikipedia, and can use just a
subset of the markup without about corner cases.
I approach it as a tool which could work for the bigger parser, though.
Currently, it looks as just another wiki syntax, looking similar to
MediaWiki one.
Another design decision is that everything that you
put in
comes out wrapped in paragraph tags. Usually this wraps the whole
document, so if your whole document was just a heading, then yes it is
wrapped in paragraph tags. This is probably not the best way to handle
this but it's what it currently does. Feel free to contribute a
different solution.
It doesn't seem to be legal html*, so I wouldn't justify it just as a
"design decision". Same could be argued for nested <p> tags.
* opening the <hX> seems to implicitely close the previous <p>, leading
to an unmatched </p>.
Templates, as you probably know, require full
integration with an
application to work in the way that MediaWiki handles them, because they
require access to the data store, and possibly other configuration
information. We built a parser that works independently of the data
store (indeed, even on the command line in a somewhat degenerate form).
In order to do that, we had to decouple template retrieval from the
parse. If you take a look in the Ruby FFI examples, you will see a more
elegant handling of templates(though it needs work). When a document is
parsed, the parser library makes available a list of templates that were
found, the arguments passed to the template, and the unique replacement
tag in the document for inserting the template once rendered. Those
underscored tags that come out are not a bug, they are those unique
tags.
I supposed that it was somehting like that, but it was odd that it did
such conversion instead of leaving them as literals in such case.
I used just the parser binary. I have been looking at the ruby code, and
despite of the foreign language, understanding a bit more of its work.
Like templates, images require some different
solutions if the parser is
to be decoupled. Our parser does not re-size images, store them, etc.
It just works with image URLs. If your application requires images to
be regularized, you would need to implement resizing them at upload, or
lazily at load time, or whatever works in your scenario.
A parser shouldn't really need to handle images. At most it would
provide a callback so that the app could do something with the image urls.
More work is
needed in this area, though if you check out
http://kiwi.drasticcode.com
you can see that most image support is working (no resizing). You can
also experiment with the parser there as needed.
The url mapping used there, make some titles impossible to use, such as
making an entry for [[Edit]] -
http://en.wikipedia.org/wiki/Edit
Hope that at least helps explain what we've done.
Again, feedback and
particularly code contributions are appreciated!
Cheers,
Karl
Just code lurking for now :)