Brion Vibber this week told me about what he, Neil, and Trevor are
working on regarding parser/visual editor, so here's a snapshot. Please
correct it if it's inaccurate.
Brion focusing on the parser and visual editor, as well as MediaWiki
code review. Brion, Trevor, and Neil are still working on the early parts!
Brion is doing preliminary test work with CodeEditor, and says
"ParserPlayground gadget will add more of that code soon".
Trevor's investigating the editing surface work and some early DOM tests.
Neil's on combining DOM transforms & planning for the editor
communication connection.
And Erik Moeller and I are grabbing some community folks, several of
whom are from Wikia, to coordinate contributions. Inez Korczyński, for
example, is interested in contributing. Maciej Brencz has just put up a
short description of how Wikia's editor internals work -
http://www.mediawiki.org/wiki/Future/Wikia_Reverse_Parser .
Right now we're strongly looking for parser test cases and Abstract
Syntax Trees. Once we have a stabler base, maybe in August or
September, there'll be more opportunity to implement plugins and UI
extensions. More info to come via this list and also live on
http://www.mediawiki.org/wiki/Future and its subpages.
--
Sumana Harihareswara
Volunteer Development Coordinator
Wikimedia Foundation
Maciej Brencz, thanks for updating me! Maciej told me that Mike Schwartz from
Wikia recently added a list of Wikia's "test cases and situations we need to
fallback to source mode":
<http://www.mediawiki.org/wiki/Future/Parser_test_cases>.
I'll put together a short description of how we handle parsing of
wikitext to HTML and reverse parsing of HTML back to wikitext in our
Rich Text Editor.
I hope that together we will make significant improvement to the MW Parser!
I hope so too, Maciej. Thanks!
Welcoming parser test cases from all interested parties; stick 'em on that
page.
--
Sumana Harihareswara
Volunteer Development Coordinator
Wikimedia Foundation
We had a whole bunch of folks who've had their hands in the world of
MediaWiki parsing & rich text editing here at the Berlin Hackathon, and made
some great progress on setting out some ideas for how to start actually
working on it.
Tomorrow I'll distill our session notes into a clearer description of the
core ideas & next steps (dare I say... a manifesto? :)
In the meantime, if you're brave you can peek at the raw session notes:
http://etherpad.wikimedia.org/mwhack11Sat-Parser
We're reviving the wikitext-l mailing list for people interested in the
project; it's gotten some traffic about interesting projects but we'll be
making it an active working group. I'll also be making regular posts here on
wikitech-l, on the Wikimedia tech blog, and on the wikis -- but I don't want
to clutter wikitech-l *too* much with the nitty-gritty details. ;)
Project hub pages will go up tomorrow at
http://www.mediawiki.org/wiki/Future
-- brion vibber (brion @ wikimedia.org / brion @ pobox.com)
Hi everyone,
We are happy to announce the general availability of the first public
release of the Sweble Wikitext parser, available from http://sweble.org.
The Sweble Wikitext parser
* can parse all complex MediaWiki Wikitext, incl. tables and templates
* produces a real abstract syntax tree (AST); a DOM will follow soon
* is open source made available under the Apache Software License 2.0
* is written in Java utilizing only permissively licensed libraries
You can find all relevant information and code at http://sweble.org –
this also includes demos, in particular the CrystalBall demo, which lets
you query a Wikipedia snapshot using XQuery.
The Sweble Wikitext parser intends to be a complete parser for MediaWiki
Wikitext. That said, plenty of work remains to be done.
At this stage, we are hoping for your help. You can help us by
* playing with the CrystalBall demo and pointing out to us wiki
pages that look particularly bad or faulty
* simply using the parser in your projects and telling us what
works and what doesn’t (bug reports)
* getting involved in the open source project by contributing
code, documentation, and good humor
If you have questions, please don’t hesitate to use the sweble.org
facilities or send email to info(a)sweble.org.
Cheers
Hannes
Hi all!
Wikimedia Germany invites anyone interested in improving MediaWiki to come and
join us at or third developer meet-up. Like the last two years, it's going to be
awesome! Unlike the last two years, there will be more hacking and less talking
- it'll be a Hackathon, not a BarCamp.
We'll meet on May 13 to 15, in Berlin, on the 4th floor of the betahaus
coworking space <http://betahaus.de/>.
There will not be an entrance fee, but registration is mandatory and now open:
<http://de.amiando.com/hackathon2011>.
Registration will close on April 10. If you like to attend, please register in
time!
More information can be found at
<http://www.mediawiki.org/wiki/Berlin_Hackathon_2011>.
The Berlin Hackathon 2011 is an opportunity for MediaWiki hackers to come
together, squash bugs and write crazy new features. Our main focus this time
around will probably be:
* Improving usability / accessibility
* Interactive Maps
* Fixing the parser
* WMF Ops (new data center, virtualization)
* Supporting the Wiki Loves Monuments image hunt
* Squashing bugs
If you have different ideas, please let us know:
<http://www.mediawiki.org/wiki/Berlin_Hackathon_2011#Topics>
The Hackathon will be hosting the Language committee and Wiki loves Monuments
group. There is a limited number of seats reserved for these groups and if you
belong to one of them, you should receive an invitation code soon.
If you have any doubts or questions, contact us at <hackathon(a)wikimedia.de>.
We’re excited to see you in Berlin, your Hackathon Team
Daniel Kinzler (Program Coordinator)
Nicole Ebber (Logistics)
Cornelius Kibelka (Assistant)
You're right that the routes on kiwi.drasticcode.com leave something
to be desired. I was mainly focused on getting a demo of the parser
working, and didn't put a lot of thought into the urls, or try to
follow any wiki best practices. I did try to avoid some of the
MediaWiki conventions, like putting colons in routes, or indicating
the action with a GET param. I've found these can be tricky to
duplicate in other frameworks like Rails, which doesn't easily support
those.
I would like to find the time to address this (although Karl's right
that I would welcome contributions.) I'm considering that a routing
scheme like this might work:
GET /wiki/a-page-name
GET /wiki/another/page
GET /edit/another/page
GET /upload/someFile.jpg
POST /wiki/another/page # update or create
Are there any obvious problems with this approach I might want to consider?
Thanks
Sam Goldstein
On Thu, Feb 3, 2011 at 12:41 AM, Platonides <platonides at gmail.com> wrote:
> Karl Matthias wrote:
>>> The url mapping used there, make some titles impossible to use, such as
>>> making an entry for [[Edit]] - http://en.wikipedia.org/wiki/Edit
>>
>> You are right about that. I'm sure Sam would be happy to accept
>> contributions to change that. The site does support double-click to
>> edit, though, so making links to Edit is kind of unnecessary.
>
> It's not just edit, but all actions, such as upload. The real solution
> is to have the wiki items inside a "folder" and the actions outside. You
> could prefix actions, like mediawiki does (eg. Action:Edit, and
> forbidding pages starting with Action:) but you would still have the
> classic problems for root folder items such as favicon.ico.
> See
> http://www.mediawiki.org/wiki/Manual:Wiki_in_site_root_directory#Reasons_wh…
>
>
>
> Alan Post wrote:
> >* Interesting. Is the PEG grammar available for this parser?
> *>*
> *>* -Alan
> *
> It's at https://github.com/AboutUs/kiwi/blob/master/src/syntax.leg
>
> Get peg/leg from http://piumarta.com/software/peg/
>
> I just tried it and already found a bug on the first Hello World (it
> surrounds headers inside paragraphs).
> It strangely converts templates into underscored words. They may be
> expecting some other parser piece to restore it. I'm pretty sure there
> are corner cases in the preprocessor (eg. just looking at the peg file
> they don't handle mixed case noincludes), but I don't think that should
> need to be handled by the parser itself.
>
> The grammar looks elegant. I doubt it can really handle full wikitext.
> But it would be so nice if it did...
>
>
I'm one of the authors of the Kiwi parser and will be presenting it at the
Data Summit on Friday. The parser is pretty complete but certainly we could
use some community support and we encourage feedback and participation! It
is a highly functional tool already but it can use some polish. It does
actually handle most wikitext, though not absolutely everything.
>From your post I can see that you are experiencing a couple of design
decisions we made in writing this parser. We did not set out to match the
exact HTML output of MediaWiki, only to output something that will look the
same in the browser. This might not be the best approach, but right now
this is the case. Our site doesn't have the same needs as Wikipedia so when
in doubt we leaned toward what suited our needs and not necessarily ultimate
tolerance of poor syntax (though it is somewhat flexible). Another design
decision is that everything that you put in comes out wrapped in paragraph
tags. Usually this wraps the whole document, so if your whole document was
just a heading, then yes it is wrapped in paragraph tags. This is probably
not the best way to handle this but it's what it currently does. Feel free
to contribute a different solution.
Templates, as you probably know, require full integration with an
application to work in the way that MediaWiki handles them, because they
require access to the data store, and possibly other configuration
information. We built a parser that works independently of the data store
(indeed, even on the command line in a somewhat degenerate form). In order
to do that, we had to decouple template retrieval from the parse. If you
take a look in the Ruby FFI examples, you will see a more elegant handling
of templates(though it needs work). When a document is parsed, the parser
library makes available a list of templates that were found, the arguments
passed to the template, and the unique replacement tag in the document for
inserting the template once rendered. Those underscored tags that come out
are not a bug, they are those unique tags. There is a switch to disable
templates and in that case it just swallows them instead. So the template
handling work flow (simplistically) is:
1. Parse original document and generate list of templates, arguments,
replacement tags
2. Fetch first template, if there is no recursion needed, insert into
original document
3. Fetch next template, etc
We currently recurse 6 templates deep in the bindings we built for
AboutUs.org (sysop-only at the moment). Template arguments don't work right
now, but it's fairly trivial to do it. We just haven't done it yet.
Like templates, images require some different solutions if the parser is to
be decoupled. Our parser does not re-size images, store them, etc. It just
works with image URLs. If your application requires images to be
regularized, you would need to implement resizing them at upload, or lazily
at load time, or whatever works in your scenario. More work is needed in
this area, though if you check out http://kiwi.drasticcode.com you can see
that most image support is working (no resizing). You can also experiment
with the parser there as needed.
Hope that at least helps explain what we've done. Again, feedback and
particularly code contributions are appreciated!
Cheers,
Karl
Hi everyone. I just joined the list; this is my first post, but I've been
following the developments with interest.
I just learned about another promising MW parser, from the AboutUs guys, and
thought I'd share it here in the list:
Kiwi: A Fast, Formal WikiText Parser -
http://blog.aboutus.org/2011/01/31/kiwi-a-fast-formal-wikitext-parser/
Cheers,
Waldir