We had a whole bunch of folks who've had their hands in the world of MediaWiki parsing & rich text editing here at the Berlin Hackathon, and made some great progress on setting out some ideas for how to start actually working on it.
Tomorrow I'll distill our session notes into a clearer description of the core ideas & next steps (dare I say... a manifesto? :)
In the meantime, if you're brave you can peek at the raw session notes: http://etherpad.wikimedia.org/mwhack11Sat-Parser
We're reviving the wikitext-l mailing list for people interested in the project; it's gotten some traffic about interesting projects but we'll be making it an active working group. I'll also be making regular posts here on wikitech-l, on the Wikimedia tech blog, and on the wikis -- but I don't want to clutter wikitech-l *too* much with the nitty-gritty details. ;)
Project hub pages will go up tomorrow at http://www.mediawiki.org/wiki/Future
-- brion vibber (brion @ wikimedia.org / brion @ pobox.com)
On Sat, May 14, 2011 at 7:09 PM, Brion Vibber brion@wikimedia.org wrote:
In the meantime, if you're brave you can peek at the raw session notes: http://etherpad.wikimedia.org/mwhack11Sat-Parser
We're reviving the wikitext-l mailing list for people interested in the project; it's gotten some traffic about interesting projects but we'll be making it an active working group. I'll also be making regular posts here on wikitech-l, on the Wikimedia tech blog, and on the wikis -- but I don't want to clutter wikitech-l *too* much with the nitty-gritty details. ;)
Project hub pages will go up tomorrow at http://www.mediawiki.org/wiki/Future
I've stubbed out a couple sections on:
http://www.mediawiki.org/wiki/Future/Parser_plan
More specific things to follow based on the notes previously posted.
-- brion
Hi, I am sorry I did not make the meeting,. I have been working on swebel, and am in the middle or trying to convert the grammar from rats! to pegs format. there are very many tools that support pegs. I am aware of some of the difficulties in the grammar that are using some hacks to set the flags, will have to work on them later. mike
On Sun, May 15, 2011 at 2:13 PM, Brion Vibber brion@wikimedia.org wrote:
On Sat, May 14, 2011 at 7:09 PM, Brion Vibber brion@wikimedia.org wrote:
In the meantime, if you're brave you can peek at the raw session notes: http://etherpad.wikimedia.org/mwhack11Sat-Parser
We're reviving the wikitext-l mailing list for people interested in the project; it's gotten some traffic about interesting projects but we'll be making it an active working group. I'll also be making regular posts here on wikitech-l, on the Wikimedia tech blog, and on the wikis -- but I don't want to clutter wikitech-l *too* much with the nitty-gritty details. ;)
Project hub pages will go up tomorrow at http://www.mediawiki.org/wiki/Future
I've stubbed out a couple sections on:
http://www.mediawiki.org/wiki/Future/Parser_plan
More specific things to follow based on the notes previously posted.
-- brion
Wikitext-l mailing list Wikitext-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitext-l
On Sun, May 15, 2011 at 2:26 PM, Mike Dupont <jamesmikedupont@googlemail.com
wrote:
Hi, I am sorry I did not make the meeting,. I have been working on swebel, and am in the middle or trying to convert the grammar from rats! to pegs format. there are very many tools that support pegs. I am aware of some of the difficulties in the grammar that are using some hacks to set the flags, will have to work on them later.
Great! We seem to be initially working with PEG-style tools for now (and will probably stick with that for the formal portion of the grammar unless we have a major revelation; though there's some variation in PEG parser generator syntaxes, making clean definitions should help in making the implementations easier to start.)
Definitely looking forward to seeing updates & comparing some rule portions directly.
-- brion vibber (brion @ pobox.com / brion @ wikimedia.org)
well you can find my current work in progress here https://github.com/h4ck3rm1k3/sweble-wikitext/blob/master/swc-parser-lazy/sr... I am using this tool to compile it, peg-1.4 from http://piumarta.com/software/peg/peg-0.1.4/,
It is still work in progress, but I have a bunch of different peg tools I would like to try it out with. It seems that we will needs a standard api for declaring states and checking them... mike
On Mon, May 16, 2011 at 5:08 PM, Brion Vibber brion@wikimedia.org wrote:
On Sun, May 15, 2011 at 2:26 PM, Mike Dupont jamesmikedupont@googlemail.com wrote:
Hi, I am sorry I did not make the meeting,. I have been working on swebel, and am in the middle or trying to convert the grammar from rats! to pegs format. there are very many tools that support pegs. I am aware of some of the difficulties in the grammar that are using some hacks to set the flags, will have to work on them later.
Great! We seem to be initially working with PEG-style tools for now (and will probably stick with that for the formal portion of the grammar unless we have a major revelation; though there's some variation in PEG parser generator syntaxes, making clean definitions should help in making the implementations easier to start.)
Definitely looking forward to seeing updates & comparing some rule portions directly.
-- brion vibber (brion @ pobox.com / brion @ wikimedia.org)
I've written a PEG parser is Scheme, called genturfa'i:
http://wiki.call-cc.org/eggref/4/genturfahi
I would be more than happy to work with you on making a peg description that works with my parser. What would be the best way for me to help? Should I fork your github project? If I do that, will you articulate how best to include PEG parser-specific aspects of any tool that uses your PEG file?
-Alan
On Mon, May 16, 2011 at 09:22:08PM +0200, Mike Dupont wrote:
well you can find my current work in progress here https://github.com/h4ck3rm1k3/sweble-wikitext/blob/master/swc-parser-lazy/sr... I am using this tool to compile it, peg-1.4 from http://piumarta.com/software/peg/peg-0.1.4/,
It is still work in progress, but I have a bunch of different peg tools I would like to try it out with. It seems that we will needs a standard api for declaring states and checking them... mike
On Mon, May 16, 2011 at 5:08 PM, Brion Vibber brion@wikimedia.org wrote:
On Sun, May 15, 2011 at 2:26 PM, Mike Dupont jamesmikedupont@googlemail.com wrote:
Hi, I am sorry I did not make the meeting,. I have been working on swebel, and am in the middle or trying to convert the grammar from rats! to pegs format. there are very many tools that support pegs. I am aware of some of the difficulties in the grammar that are using some hacks to set the flags, will have to work on them later.
Great! We seem to be initially working with PEG-style tools for now (and will probably stick with that for the formal portion of the grammar unless we have a major revelation; though there's some variation in PEG parser generator syntaxes, making clean definitions should help in making the implementations easier to start.)
Definitely looking forward to seeing updates & comparing some rule portions directly.
-- brion vibber (brion @ pobox.com / brion @ wikimedia.org)
-- James Michael DuPont Member of Free Libre Open Source Software Kosova and Albania flossk.org flossal.org
Wikitext-l mailing list Wikitext-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitext-l
sure, fork away, my current work is to remove all duplicate rules. also the things like / &"\uE000" ParserEntity are not supported. I will try out your parser as well. ooopps, what is that? very strange,,,, ok, maybe I will leave it to you first. if you can post english instructions on how to use it...
mike
On Mon, May 16, 2011 at 9:38 PM, Alan Post alanpost@sunflowerriver.org wrote:
I've written a PEG parser is Scheme, called genturfa'i:
http://wiki.call-cc.org/eggref/4/genturfahi
I would be more than happy to work with you on making a peg description that works with my parser. What would be the best way for me to help? Should I fork your github project? If I do that, will you articulate how best to include PEG parser-specific aspects of any tool that uses your PEG file?
-Alan
On Mon, May 16, 2011 at 09:22:08PM +0200, Mike Dupont wrote:
well you can find my current work in progress here https://github.com/h4ck3rm1k3/sweble-wikitext/blob/master/swc-parser-lazy/sr... I am using this tool to compile it, peg-1.4 from http://piumarta.com/software/peg/peg-0.1.4/,
It is still work in progress, but I have a bunch of different peg tools I would like to try it out with. It seems that we will needs a standard api for declaring states and checking them... mike
On Mon, May 16, 2011 at 5:08 PM, Brion Vibber brion@wikimedia.org wrote:
On Sun, May 15, 2011 at 2:26 PM, Mike Dupont jamesmikedupont@googlemail.com wrote:
Hi, I am sorry I did not make the meeting,. I have been working on swebel, and am in the middle or trying to convert the grammar from rats! to pegs format. there are very many tools that support pegs. I am aware of some of the difficulties in the grammar that are using some hacks to set the flags, will have to work on them later.
Great! We seem to be initially working with PEG-style tools for now (and will probably stick with that for the formal portion of the grammar unless we have a major revelation; though there's some variation in PEG parser generator syntaxes, making clean definitions should help in making the implementations easier to start.)
Definitely looking forward to seeing updates & comparing some rule portions directly.
-- brion vibber (brion @ pobox.com / brion @ wikimedia.org)
-- James Michael DuPont Member of Free Libre Open Source Software Kosova and Albania flossk.org flossal.org
Wikitext-l mailing list Wikitext-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitext-l
-- .i ma'a lo bradi ku penmi gi'e du
Wikitext-l mailing list Wikitext-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitext-l
Sorry I couldn't make it to Berlin. Anyone interested is of course still welcome to check out the Kiwi grammar as it handles most hard cases now. It's LEG not PEG format but not that hard to convert.
Http://github.com/AboutUs/kiwi
Cheers, Karl On May 16, 2011 11:01 PM, "Mike Dupont" jamesmikedupont@googlemail.com wrote:
sure, fork away, my current work is to remove all duplicate rules. also the things like / &"\uE000" ParserEntity are not supported. I will try out your parser as well. ooopps, what is that? very strange,,,, ok, maybe I will leave it to you first. if you can post english instructions on how to use it...
mike
On Mon, May 16, 2011 at 9:38 PM, Alan Post alanpost@sunflowerriver.org
wrote:
I've written a PEG parser is Scheme, called genturfa'i:
http://wiki.call-cc.org/eggref/4/genturfahi
I would be more than happy to work with you on making a peg description that works with my parser. What would be the best way for me to help? Should I fork your github project? If I do that, will you articulate how best to include PEG parser-specific aspects of any tool that uses your PEG file?
-Alan
On Mon, May 16, 2011 at 09:22:08PM +0200, Mike Dupont wrote:
well you can find my current work in progress here
https://github.com/h4ck3rm1k3/sweble-wikitext/blob/master/swc-parser-lazy/sr...
I am using this tool to compile it, peg-1.4 from http://piumarta.com/software/peg/peg-0.1.4/,
It is still work in progress, but I have a bunch of different peg tools I would like to try it out with. It seems that we will needs a standard api for declaring states and checking them... mike
On Mon, May 16, 2011 at 5:08 PM, Brion Vibber brion@wikimedia.org
wrote:
On Sun, May 15, 2011 at 2:26 PM, Mike Dupont jamesmikedupont@googlemail.com wrote:
Hi, I am sorry I did not make the meeting,. I have been working on
swebel,
and am in the middle or trying to convert the grammar from rats! to pegs format. there are very many tools that support pegs. I am aware of some of
the
difficulties in the grammar that are using some hacks to set the flags, will have to work on them later.
Great! We seem to be initially working with PEG-style tools for now
(and
will probably stick with that for the formal portion of the grammar
unless
we have a major revelation; though there's some variation in PEG
parser
generator syntaxes, making clean definitions should help in making the implementations easier to start.)
Definitely looking forward to seeing updates & comparing some rule
portions
directly.
-- brion vibber (brion @ pobox.com / brion @ wikimedia.org)
-- James Michael DuPont Member of Free Libre Open Source Software Kosova and Albania flossk.org flossal.org
Wikitext-l mailing list Wikitext-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitext-l
-- .i ma'a lo bradi ku penmi gi'e du
Wikitext-l mailing list Wikitext-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitext-l
-- James Michael DuPont Member of Free Libre Open Source Software Kosova and Albania flossk.org flossal.org
Wikitext-l mailing list Wikitext-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitext-l
On Sun, May 15, 2011 at 2:13 PM, Brion Vibber brion@wikimedia.org wrote:
I've stubbed out a couple sections on:
http://www.mediawiki.org/wiki/Future/Parser_plan
More specific things to follow based on the notes previously posted.
Added a stub 'get involved' section on http://www.mediawiki.org/wiki/Future#Core_projects and the other pages; there'll be more exciting stuff to see within a couple weeks as Trevor & Neil select testing tools & we all get started on prelim grammar descriptions.
We can pretty well expect to discard these initial grammar steps that we're doing as we get farther along, but we're going to need something to work with for now. :)
Collecting parser test cases & helping to document the existing parse tree formats in use by some other parser variants will be very useful while we're getting those together -- please feel free to add some notes to the AST & test case pages (linked to from the above) or post notes directly on wikitext-l.
-- brion
On Mon, May 16, 2011 at 11:04 AM, Brion Vibber brion@wikimedia.org wrote:
On Sun, May 15, 2011 at 2:13 PM, Brion Vibber brion@wikimedia.org wrote:
I've stubbed out a couple sections on:
http://www.mediawiki.org/wiki/Future/Parser_plan
More specific things to follow based on the notes previously posted.
Added a stub 'get involved' section on http://www.mediawiki.org/wiki/Future#Core_projects and the other pages; there'll be more exciting stuff to see within a couple weeks as Trevor & Neil select testing tools & we all get started on prelim grammar descriptions.
We can pretty well expect to discard these initial grammar steps that we're doing as we get farther along, but we're going to need something to work with for now. :)
Collecting parser test cases & helping to document the existing parse tree formats in use by some other parser variants will be very useful while we're getting those together -- please feel free to add some notes to the AST & test case pages (linked to from the above) or post notes directly on wikitext-l.
-- brion
If you have found bugs in MediaWiki that should be Parser Test Cases[0] for the new parser, you can tag them with the "newparser" keyword in Bugzilla.
Also, the raw notes from our Berlin sessions are now on mediawiki.org.[1,2]
[0] http://www.mediawiki.org/wiki/Future/Parser_test_cases [1] http://www.mediawiki.org/wiki/Berlin_Hackathon_2011/Notes/Saturday#Parser_li... [2] http://www.mediawiki.org/wiki/Berlin_Hackathon_2011/Notes/Saturday/Parser
Sumana Harihareswara Volunteer Development Coordinator Wikimedia Foundation
wikitext-l@lists.wikimedia.org