Hackathon parser session initial notes

List overview All Threads
Download

newer

older

new parser test cases

Parser discussions from Berlin...

Brion Vibber

14 May 2011 14 May '11

5:09 p.m.

We had a whole bunch of folks who've had their hands in the world of MediaWiki parsing & rich text editing here at the Berlin Hackathon, and made some great progress on setting out some ideas for how to start actually working on it. Tomorrow I'll distill our session notes into a clearer description of the core ideas & next steps (dare I say... a manifesto? :) In the meantime, if you're brave you can peek at the raw session notes: http://etherpad.wikimedia.org/mwhack11Sat-Parser We're reviving the wikitext-l mailing list for people interested in the project; it's gotten some traffic about interesting projects but we'll be making it an active working group. I'll also be making regular posts here on wikitech-l, on the Wikimedia tech blog, and on the wikis -- but I don't want to clutter wikitech-l *too* much with the nitty-gritty details. ;) Project hub pages will go up tomorrow at http://www.mediawiki.org/wiki/Future -- brion vibber (brion @ wikimedia.org / brion @ pobox.com)

Attachments:

attachment.htm (text/html — 1.2 KB)

Show replies by date

Brion Vibber

15 May 15 May

12:13 p.m.

On Sat, May 14, 2011 at 7:09 PM, Brion Vibber <brion(a)wikimedia.org> wrote:

...

In the meantime, if you're brave you can peek at the raw session notes: http://etherpad.wikimedia.org/mwhack11Sat-Parser We're reviving the wikitext-l mailing list for people interested in the project; it's gotten some traffic about interesting projects but we'll be making it an active working group. I'll also be making regular posts here on wikitech-l, on the Wikimedia tech blog, and on the wikis -- but I don't want to clutter wikitech-l *too* much with the nitty-gritty details. ;) Project hub pages will go up tomorrow at http://www.mediawiki.org/wiki/Future

I've stubbed out a couple sections on: http://www.mediawiki.org/wiki/Future/Parser_plan More specific things to follow based on the notes previously posted. -- brion

Mike Dupont

12:26 p.m.

Hi, I am sorry I did not make the meeting,. I have been working on swebel, and am in the middle or trying to convert the grammar from rats! to pegs format. there are very many tools that support pegs. I am aware of some of the difficulties in the grammar that are using some hacks to set the flags, will have to work on them later. mike On Sun, May 15, 2011 at 2:13 PM, Brion Vibber <brion(a)wikimedia.org> wrote:

...

On Sat, May 14, 2011 at 7:09 PM, Brion Vibber <brion(a)wikimedia.org> wrote:

I've stubbed out a couple sections on: http://www.mediawiki.org/wiki/Future/Parser_plan More specific things to follow based on the notes previously posted. -- brion _______________________________________________ Wikitext-l mailing list Wikitext-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitext-l

-- James Michael DuPont Member of Free Libre Open Source Software Kosova and Albania flossk.org flossal.org

Brion Vibber

16 May 16 May

3:08 p.m.

On Sun, May 15, 2011 at 2:26 PM, Mike Dupont <jamesmikedupont(a)googlemail.com

...

wrote:

...

Great! We seem to be initially working with PEG-style tools for now (and will probably stick with that for the formal portion of the grammar unless we have a major revelation; though there's some variation in PEG parser generator syntaxes, making clean definitions should help in making the implementations easier to start.) Definitely looking forward to seeing updates & comparing some rule portions directly. -- brion vibber (brion @ pobox.com / brion @ wikimedia.org)

Mike Dupont

7:22 p.m.

well you can find my current work in progress here https://github.com/h4ck3rm1k3/sweble-wikitext/blob/master/swc-parser-lazy/s… I am using this tool to compile it, peg-1.4 from http://piumarta.com/software/peg/peg-0.1.4/, It is still work in progress, but I have a bunch of different peg tools I would like to try it out with. It seems that we will needs a standard api for declaring states and checking them... mike On Mon, May 16, 2011 at 5:08 PM, Brion Vibber <brion(a)wikimedia.org> wrote:

...

On Sun, May 15, 2011 at 2:26 PM, Mike Dupont <jamesmikedupont(a)googlemail.com> wrote:

-- James Michael DuPont Member of Free Libre Open Source Software Kosova and Albania flossk.org flossal.org

Alan Post

7:38 p.m.

I've written a PEG parser is Scheme, called genturfa'i: http://wiki.call-cc.org/eggref/4/genturfahi I would be more than happy to work with you on making a peg description that works with my parser. What would be the best way for me to help? Should I fork your github project? If I do that, will you articulate how best to include PEG parser-specific aspects of any tool that uses your PEG file? -Alan On Mon, May 16, 2011 at 09:22:08PM +0200, Mike Dupont wrote:

...

On Sun, May 15, 2011 at 2:26 PM, Mike Dupont <jamesmikedupont(a)googlemail.com> wrote:

-- James Michael DuPont Member of Free Libre Open Source Software Kosova and Albania flossk.org flossal.org _______________________________________________ Wikitext-l mailing list Wikitext-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitext-l

-- .i ma'a lo bradi ku penmi gi'e du

Mike Dupont

9 p.m.

sure, fork away, my current work is to remove all duplicate rules. also the things like / &"\uE000" ParserEntity are not supported. I will try out your parser as well. ooopps, what is that? very strange,,,, ok, maybe I will leave it to you first. if you can post english instructions on how to use it... mike On Mon, May 16, 2011 at 9:38 PM, Alan Post <alanpost(a)sunflowerriver.org> wrote:

...

On Sun, May 15, 2011 at 2:26 PM, Mike Dupont <jamesmikedupont(a)googlemail.com> wrote:

-- .i ma'a lo bradi ku penmi gi'e du _______________________________________________ Wikitext-l mailing list Wikitext-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitext-l

-- James Michael DuPont Member of Free Libre Open Source Software Kosova and Albania flossk.org flossal.org

Karl Matthias

18 May 18 May

1:33 p.m.

Sorry I couldn't make it to Berlin. Anyone interested is of course still welcome to check out the Kiwi grammar as it handles most hard cases now. It's LEG not PEG format but not that hard to convert. Http://github.com/AboutUs/kiwi Cheers, Karl On May 16, 2011 11:01 PM, "Mike Dupont" <jamesmikedupont(a)googlemail.com> wrote:

...

wrote:

...

> I've written a PEG parser is Scheme, called genturfa'i: > > http://wiki.call-cc.org/eggref/4/genturfahi > > I would be more than happy to work with you on making a peg > description that works with my parser. What would be the > best way for me to help? Should I fork your github project? > If I do that, will you articulate how best to include PEG > parser-specific aspects of any tool that uses your PEG file? > > -Alan > > On Mon, May 16, 2011 at 09:22:08PM +0200, Mike Dupont wrote: >> well you can find my current work in progress here >>

https://github.com/h4ck3rm1k3/sweble-wikitext/blob/master/swc-parser-lazy/s…

...

>> I am using this tool to compile it, peg-1.4 from >> http://piumarta.com/software/peg/peg-0.1.4/, >> >> It is still work in progress, but I have a bunch of different peg >> tools I would like to try it out with. >> It seems that we will needs a standard api for declaring states and >> checking them... >> mike >> >> On Mon, May 16, 2011 at 5:08 PM, Brion Vibber <brion(a)wikimedia.org>

wrote:

...

>> > On Sun, May 15, 2011 at 2:26 PM, Mike Dupont >> > <jamesmikedupont(a)googlemail.com> wrote: >> >> >> >> Hi, >> >> I am sorry I did not make the meeting,. I have been working on

swebel,

...

>> >> and am in the middle or trying to convert the grammar from rats! to >> >> pegs format. >> >> there are very many tools that support pegs. I am aware of some of

the

...

>> >> difficulties in the grammar that are using some hacks to set the >> >> flags, will have to work on them later. >> > >> > Great! We seem to be initially working with PEG-style tools for now

(and

...

>> > will probably stick with that for the formal portion of the grammar

unless

...

>> > we have a major revelation; though there's some variation in PEG

parser

...

>> > generator syntaxes, making clean definitions should help in making the >> > implementations easier to start.) >> > >> > Definitely looking forward to seeing updates & comparing some rule

portions

...

directly. -- brion vibber (brion @ pobox.com / brion @ wikimedia.org)

-- .i ma'a lo bradi ku penmi gi'e du _______________________________________________ Wikitext-l mailing list Wikitext-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitext-l

Brion Vibber

16 May 16 May

3:04 p.m.

On Sun, May 15, 2011 at 2:13 PM, Brion Vibber <brion(a)wikimedia.org> wrote:

...

I've stubbed out a couple sections on: http://www.mediawiki.org/wiki/Future/Parser_plan More specific things to follow based on the notes previously posted.

Added a stub 'get involved' section on http://www.mediawiki.org/wiki/Future#Core_projects and the other pages; there'll be more exciting stuff to see within a couple weeks as Trevor & Neil select testing tools & we all get started on prelim grammar descriptions. We can pretty well expect to discard these initial grammar steps that we're doing as we get farther along, but we're going to need something to work with for now. :) Collecting parser test cases & helping to document the existing parse tree formats in use by some other parser variants will be very useful while we're getting those together -- please feel free to add some notes to the AST & test case pages (linked to from the above) or post notes directly on wikitext-l. -- brion

Sumana Harihareswara

3 Jun 3 Jun

5:37 p.m.

New subject: [Wikitech-l] Hackathon parser session initial notes

On Mon, May 16, 2011 at 11:04 AM, Brion Vibber <brion(a)wikimedia.org> wrote:

...

On Sun, May 15, 2011 at 2:13 PM, Brion Vibber <brion(a)wikimedia.org> wrote:

I've stubbed out a couple sections on: http://www.mediawiki.org/wiki/Future/Parser_plan More specific things to follow based on the notes previously posted.

If you have found bugs in MediaWiki that should be Parser Test Cases[0] for the new parser, you can tag them with the "newparser" keyword in Bugzilla. Also, the raw notes from our Berlin sessions are now on mediawiki.org.[1,2] [0] http://www.mediawiki.org/wiki/Future/Parser_test_cases [1] http://www.mediawiki.org/wiki/Berlin_Hackathon_2011/Notes/Saturday#Parser_l… [2] http://www.mediawiki.org/wiki/Berlin_Hackathon_2011/Notes/Saturday/Parser Sumana Harihareswara Volunteer Development Coordinator Wikimedia Foundation

4736

days inactive

4756

days old

wikitext-l@lists.wikimedia.org

Manage subscription

9 comments

5 participants

tags (0)

participants (5)

Alan Post
Brion Vibber
Karl Matthias
Mike Dupont
Sumana Harihareswara