A chapter is being worked on for the next edition of the Open Source
Architecture ORA book, here:
https://www.mediawiki.org/wiki/MediaWiki_architecture_document/text
I suspect there's some more information about current parser project work
that should be in there, but isn't. I'm going to be doing an edit pass
on that page tonight or tomorrow, and if there's info you think should be
in there you or I should put it in?
Cheers,
-- jra
--
Jay R. Ashworth Baylink jra(a)baylink.com
Designer The Things I Think RFC 2100
Ashworth & Associates http://baylink.pitas.com 2000 Land Rover DII
St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274
Forwarding -- the rest of the thread is at
http://lists.wikimedia.org/pipermail/wikitech-l/2011-October/thread.html#56…
.
-------- Original Message --------
Subject: [Wikitech-l] #parser parser function - does this make any sense?
Date: Sat, 29 Oct 2011 02:01:15 +0200
From: Daniel Werner <DanWeEtz(a)web.de>
Reply-To: Wikimedia developers <wikitech-l(a)lists.wikimedia.org>
To: wikitech-l(a)lists.wikimedia.org
I am thinking about creating a very simple parser function #parse doing
nothing but returning parameter 1 with an "'noparse' => false" option.
Is there anything like this (or what could be abused for this) already
or is there any reason why this might be a bad idea?
The reason I want to have something like this is, I want to create a
template (for template and parser function black-box tests) accepting
something like {{((}}#somefunction:a{{!}}b{{!}}c{{))}} as parameter
value, showing {{#somefunction|a|b|c}} as output and at the same time
calling {{#parse: {{((}}#somefunction:a{{!}}b{{!}}c{{))}} }} so that
besides the definition also the result can be shown by the template output.
regards,
Daniel
_______________________________________________
Wikitech-l mailing list
Wikitech-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Etherpad'ed notes from today's editor/parser status meeting:
2011-10-26
Attending:
* Visual Editor team
** Trevor
** Inez
* Parser team
** Brion
** Gabriel Wicke (skype)
* Integration
** Neil
* Managy people
** Erik
** Alolita
== Status check ==
How are we going to
- Write a parser
- Integrate it with the visual editor
- Deploy the visual editor
Parser Status
- Basic parser in place
- Produces intermediate JSON object tree
- Doesn't yet produce Wikidom as it's been defined
- Considering if Wikidom should be changed
- Still working on markup support (mixed HTML and Wikitext)
- Written in JavaScript and is currently in the "ParserPlayground"
- Using PEG for part of the parsing process (primarily tokenizing)
VisualEditor Goals
- Get something working in the wiki environment so people can play with
it
- Ideas for places to integrate first
- New page creation - Potential Use Case
- http://www.mediawiki.org/wiki/Visual_editor/Task_management
== Decemberish test deployment? ==
Distinct stages that could be deployed on sites for testing:
* Visual editor in the wiki -> lets you edit but nothing else
** ''too early to advertise for anything but interaction testing''
** just need to actually plug it in
* Visual editor in the wiki and lets you save
** ''_could_ start to use as opt-in test for new pages''
** add API module to save, should be easy-ish
* Visual editor in the wiki and lets you load and save.
** ''this would be ideal place to start advertising for testers''
** combining the parser and reconciling the formats
== Visual Editor side ==
VE is in midst of core redo to make undo etc working (internal
representation changes)-- still stabilizing!
VE frontend todos for page saving:
* reintegrate serializers (in progress)
* undo/redo/copy/paste working (mostly there)
** '''Trevor and Inez are already working on this, but it needs to be
finished before we can integrate saving.'''*
Parser/VE integration todos (for page loading):
* parser & wikidom specs need to be resynchronized
** possibly needs a translator step
* support for more structures (lists etc)
** '''This is big work and needs a lot of coordination between VE & Parser
groups!'''*
** brion & gabriel: can start working with their existing spec -- let's at
least get that working
*** then we'll need to start figuring out if we need to do major changes
based on issues
*** parser & VE may need to evolve together for now!
Lots more front-end work needed in VE ecosystem; we expect to bring on
another hire in that area.
"WikiDom will be the fence between these two neighbors [VE & Parser groups]"
== Parser side ==
*Immediate parser work:*
* get it producing output compatible with VE's current wikidom input
** we do expect to have to evolve this over time! but let's get it together
with the VE first
* make sure there's a clean JS API for the parser so the VE can call it
*Next stages parser work:*
* start making sure tables, templates are all working as expected (fixup
stages?)
** at least properly nested templates..
* iterate as necessary
* make sure auto tests are running
** round-trip tests on wikipedia corpus already available
** need to test HTML rendering against MW's current internal parser
*Later stages parser work: (after December)*
* depending on what we've seen from auto test...
* ... redoing the PEG?
* ... redoing the wikidom?
* ... redoing how/when templates are expanded?
* make sure all structures really supported!
* add proper fix-ups similar to what tidy does right now
== Upcoming work ==
* Mediawiki extension for Visual Editor: Architecture is to be driven by
Trevor; Integration w Parser
* RL2 support
* Internationalization support
* Using underscore.js for js based parser functionality in the same
extension as VEd
** replace jQuery bits like $.each; underscore should be cleaner and is
easier to use in node for batch tests
* Unknown markup indicator
** at least black-box markers for extension hooks and such
Milestones!!!!!!!
* Trevor will check up with Howie
** various VE stuff on its way
* getting VE and parser into a common extension (separate code modules
loaded by it)
* parser integration may not make it by December, but we'll see what's ready
by then
Welcome, Gabriel. I'm looking forward to working with you, and to
seeing you posting on the parser/visual editor list about our progress
on that very important endeavor! cc'ing the parser list (wikitext-l).
Sumana Harihareswara
Volunteer Development Coordinator
Wikimedia Foundation
On Tue, Oct 25, 2011 at 1:22 AM, Alolita Sharma <asharma(a)wikimedia.org> wrote:
> Hi All,
>
> Please join me in welcoming Gabriel Wicke as a Software Developer in
> WMF’s Features Engineering team. Gabriel will be working on the
> Visual Editor - Parser project, one of Wikimedia’s high priority
> projects this year. He will be working closely with our guru Brion
> Vibber on extending the parser to support WikiDom interactions with
> the Visual Editor client being developed by lead engineer Trevor
> Parscal, Wikia developer Inez Korczyński and front-end developer Neil
> Kandalgaonkar.
>
> As many of you may already know, Gabriel has been member of the
> Wikipedia community for many years now. He discovered Wikipedia in
> 2003, when it was still running on two servers. Using his previous
> experience with Squid caching, he got involved in technical
> discussions and hacking. In 2004, Gabriel designed and implemented the
> initial Squid caching layer, and later wrote the MonoBook skin.
>
> After completing his Computer Science degree and doing research in
> transactional distributed systems and Haskell, Gabriel is looking
> forward to more practical challenges at Wikimedia.
>
> Gabriel is an avid sportsman and professional sailor. When he’s not in
> front of a computer coding away, he is often sailing on his own or
> with friends. He was a member of the German national team in the
> Olympic 49er class from 2001-2008, and is now racing an A-Class
> catamaran. Pretty awesome!
>
> Say hello to Gabriel online. He can be found on #mediawiki as gwicke.
>
> Welcome back Gabriel! Great to have you on the Wikimedia Features team :-)
>
> Alolita
> --
> Alolita Sharma
> Director, Features Engineering
> Wikimedia Foundation
>
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l(a)lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Dear Wikitext experts,
please, check out Sztakipedia, a new Wiki RTE at:
http://pedia.sztaki.hu/ (please check the video first, and then the tool itself)
which aims at implementing some of the Visions you described here:
http://www.mediawiki.org/wiki/Future/Parser_plan (the RTE part)
Some background:
Sztakipedia did not start out as an editor for Wikipedia. It was meant
to be a web-based editor for UIMA annotated rich content, supported
with natural language background processing.
The tool was functional by the end of 2010, and we wanted a popular
application to demonstrate its features, so went on applying it to
Wiki editing.
To do that, we have made some wiki-specific stuff:
-After checking out many parsers, we have created a new one in JavaCC
-Created lots of content helpers based on dbpedia, like the link
recommendation, infobox recommendation, infobox editor help
-Integrated external resources to help editing, like the Book
Recommendation or Yahoo-based category recommendation
Sztakipedia is right now in its alpha phase, with many show stoppers,
like handling cite references properly, or editing templates embedded
in templates,
etc...
I am aware that you are working on a new syntax, parser and RTE and
they will eventually become the official ones for Wiki editing
(Sztakipedia is in Java anyway).
However, I still think that there is much to learn from our project. We will
write a paper next month on the subject and I will be honored is some
of you read and comment it. The main contents will be:
-problematic stuff in the current wikitext syntax we struggled with
-usability tricks, like extracting the infobox pages to provide help
for the fields, showing the abstracts of the articles to be linked
-recommendations, machine learning to support the user+ background theory
Our plan right now is to create an API for our recommendation services
and helpers and a MediaWiki js plugin to get its results to the
current wiki editor. This way I hope the results of this research -
which started out as a rather theoretical one - will be used in a real
world scenario by at least a few people. I hope we will be able to
extend the your planned new RTE the same way in the future.
Please, share with me your thoughs/comments/doubts about Sztakipedia.
Also I wanted to ask some things:
-Which is the most wanted helper feature according to you:
infobox/category/link recommendation? External data import from the
Linked Open Data? (Like our Book Recommender right now which has
millions of book records in it?) Field _value_ recommendation for
infoboxes from the text? Other?
-How do you measure the performance of a parser? I saw hints to some
300 parser test cases somewhere...
-Which is the best way to mash up external services to support the Wiki editor
interface (because if you call an external REST service from JS in mediawiki, it
will be cross-site scripting I'm afraid)?
Thank you very much,
Best Regards
Mihály Héder
MTA Sztaki,
Budapest, Hungary
This past Wednesday, I held a triage focused on issues from the
Wikibooks an Wikisource projects
** https://bugzilla.wikimedia.org/18861 - Search should give transcluded
text for the page
First, a confession. I'm not an experienced Wikipedian. So when I
saw this bug, I didn't understand the benefit. "Wouldn't you
just need to add namespaces to the search index?" I thought.
Luckily, Roan was in this triage and offered a simple use case:
Wiktionary does crazy things like
{{buildATableOfAllInflections|word|inflectioncase}} — which will
produce a table of all inflections of "word", based on which case
applies to it. So then if you search for an inflection of "word",
say "words", you won't find it.
With the help of this explanation, I was able to understand the
usefulness and what was needed. MWSearch needs to index *expanded*
Wikitext rather than just raw Wikitext.
This would probably also fix that annoying bug where incategory:foo
queries don't work properly with categories from templates
This has been proposed as a Summer of Code idea for 2012.
** https://bugzilla.wikimedia.org/28277 - Error when importing pages from
English Wikipedia to Portuguese Wikibooks
This strange bug caused problems on the Portuguese Wikibooks project,
and we were able to reproduce it during triage. I put Helder.wiki's
steps for reproducing this on the bug and hope to find a developer to
work on it soon.
** https://bugzilla.wikimedia.org/189 - please find a solution for a
music module
After some discussion, I decided to close this bug (which has gotten
over 115 comments) and focus any new effort on action items derived
from it like "Make [[mw:Extension:LilyPond]] safe against DoS attacks"
(https://bugzilla.wikimedia.org/29630).
Sumana is already using this issue as a possible area for volunteers
to work on.
Side note: prior to this triage, the LilyPond Extension existed only
on a MediaWiki page. After the triage, I committed the LilyPond code
to SVN and, almost immediately, it began getting valuable reviews and
updates. http://hexm.de/80
I think this really shows the value of our code review process —
especially as we've improved it over the past year or so.
** https://bugzilla.wikimedia.org/27256 - Correcting content page count at
en.wikibooks and pt.wikibooks
It looks like we spent a bit of time discussing this bug without any
of us being aware of 1.18's new $wgArticleCountMethod. I've updated
the bug with the necessary information.
** https://bugzilla.wikimedia.org/22911 - Install
extension:SubpageSortkey on wikibooks
Bawolff has created an extension to solve Helder.wiki's original
request ("Default 'sort key' for namespaces should be more namespaces
with subpages should be customisable"). At this point it simply needs
to be reviewed and deployed.
** https://bugzilla.wikimedia.org/15071 - Wikibooks/Wikisource needs
means to associate separate pages with books.
** https://bugzilla.wikimedia.org/2308 - ability to watch "bundles" of
sub-pages
These two bugs and Raylton's Extension:BookManager revolved around
these projects' desire to treat books as entities that can be
manipulated in the same way as wiki pages. They'd like the ability to
watch, delete, or move books as well as a have pages like
Special:RandomBook.
Adding some of these features (watching, for example) to all pages in
a category, might help admins in other projects besides these.
** https://bugzilla.wikimedia.org/30666 - Show subpages on page deletion
As Bawolff said, this looks like a sane feature request in general, not
just for wikibooks. Adrignola gave a couple of gadgets that enable
subpage deletion, but the gadgets didn't provide a clean way to undelete
sub pages en-masse.
This sparked a discussion on some other enhancements that would be
good to have. For example the ability to watch all articles in a
category (https://bugzilla.wikimedia.org/1710)
** https://bugzilla.wikimedia.org/26881 - noinclude tag breaks Proofread
under Internet Explorer
This was a on the wishlist for wikibooks and included a patch. I
committed it (http://mediawiki.org/wiki/Special:Code/MediaWiki/98422).
** https://bugzilla.wikimedia.org/12130 - Edit form eats heading CRs
(leading blank newlines/whitespace) on save/preview
This bug keeps popping up and, while there are work-arounds, the
behavior is non-intuitive. Mediawiki erases the first (and only the
first) blank like each time you click submit or preview. After
everyone in the triage meeting confirmed this, I showed the problem to
Krinkle who agreed that this should be fixed and left a comment with
an idea of how to fix it.
Next Triage: October 5th -- focus on Fund-raising issues http://hexm.de/81
Bug Triage calendar: http://hexm.de/TriageCal
--
Mark A. Hershberger
Bugmeister
Wikimedia Foundation
mhershberger(a)wikimedia.org
717.271.1084
I'm starting to take some more time away from helping the general MediaWiki
1.18 development and get back to the JavaScript back-end parser
experiments[1] that'll power the initial testing of the visual editor's work
on existing pages.
[1] http://www.mediawiki.org/wiki/Extension:ParserPlayground
I've made some progress[2] on in-tree template expansion, modeled on parts
of PPFrame::expand[3], Parser::argSubstitution[4] and
Parser::braceSubstitution[5] from the current MediaWiki parser.
[2] http://www.mediawiki.org/wiki/Special:Code/MediaWiki/98393
[3]
http://svn.wikimedia.org/viewvc/mediawiki/trunk/phase3/includes/parser/Prep…
[4]
http://svn.wikimedia.org/viewvc/mediawiki/trunk/phase3/includes/parser/Pars…
[5]
http://svn.wikimedia.org/viewvc/mediawiki/trunk/phase3/includes/parser/Pars…
There are two key differences between this in-progress code and how the
stock parser does it:
1) The stock parser expands a preprocessor node tree as it flattens it into
another string (eg for later parsing steps on the compound markup) -- the
experimental parser produces another node tree.
Later-stage parsing will deal with re-assembling start and end tokens when
producing HTML or other output, but within the tree structure instead of a
flat stream of string tokens -- that'll let output know how to mark where
pieces of output came from in the input so they can be hooked up to editing
activation.
2) Operations like fetching a template from the network are inherently
asynchronous in a JavaScript environment, so the iteration loop uses a
closure function to 'suspend' itself, calling back into the function when
the async operation completes.
Currently this is fairly naive and suspends the entire iterator, meaning
that all work gets serialized and every network round-trip will add wait
time. A smarter next step could be to keep on iterating over other nodes,
then come back to the one that was waiting when it's ready -- this could be
a win when working with many templates and high network latency. (Another
win can be to pre-fetch things you know you're going to need in bulk!)
I'm actually a bit intrigued by the idea of more aggressive asynchronous
work on the PHP-side code now; while latency is usually low within the data
center, things still add up. Working with multiple data centers for failover
and load balancing may make it more likely for Wikimedia's own sites to
experience slow data fetches, and things like InstantCommons[6] can already
require waiting for potentially slow remote data fetches on third-party
sites using shared resources over the internet.
[6] http://www.mediawiki.org/wiki/InstantCommons
Alas we can't just take JavaScript-modeled async code and port it straight
to PHP, but PHP does let you do socket i/o and select() and whatnot and
could certainly do all sorts of networked requests in background during
other CPU operations.
This is still very much in-progress code (and in-progress data structure),
but do feel free to take a peek over and give feedback. :)
-- brion vibber (brion @ wikimedia.org / brion @ pobox.com)
Hi all,
Not long ago I have stumbled upon Brion's presentation at
http://upload.wikimedia.org/wikipedia/commons/d/d5/Parser_and_Editor_-_Haif…
and I think I have something interesting to tell you folks about.
Over a year ago me and my friend have started to create an intuitive,
multipurpose and flexible wiki markup syntax. Looking back now I think
we did a good job (and you can see it yourself, below) but that's not
the main point of this message.
I have written a PHP framework implementing descent recursive processing
of a (wiki) document into corresponding DOM. Produced tree can be
serialized into a binary format for storage/transmission or dumped
(rendered) into different representations such as HTML or plain text (or
FB2, XML, RTF, etc., etc.) - if each node supports the target format.
Even though the project haven't got enough members yet we have developed
several child projects that use the aforementioned framework - a fast
blog engine and a JavaScript editor (incomplete). The latter is
particularly interesting because it makes full use of per-node editing
(e.g. text node -> bold fragment -> paragraph -> entire document) - and
unlike solutions built on top of HTML/raw wiki code this one is 100%
accurate since it relies on the document tree itself.
What's also interesting about the framework is that in itself it and its
DOM are markup-independent so, say, if a MediaWiki markup module is
written it will allow the frontend (the wiki engine or something else)
seamlessly operate on documents of both old and new syntaxes. Moreover,
the compat module can support rendering into the new markup marking it
transparent to migrate to the new format, if necessary.
The project home page is: http://uverse.i-forge.net/wiki
There's a concise but complete description of our markup language and
our plans; you can experiment with the markup there, as well as find
links to related sites and more in-depth articles.
JavaScript editor page: http://uverse.i-forge.net/wiki/demo/Pages/Swacked
The global goal of the project is to create a uniform markup syntax for
all electronic texts and standardize it to allow 3rd party
implementations.
--
Regards,
Proger_XP
Thought you might want to see the EditSurface ToDo list that Trevor,
Inez, and Neil are working off:
http://etherpad.wikimedia.org/VisualEditorTodo
Also, Trevor reports from last week & early this week:
* Wrote lots of tests for new code structure and made some changes as needed
* Continuing work on cleaning up and porting old code into new structure
* Got new code structure to render paragraphs and lists
* Working on rendering other things, such as tables
* Redesigning selection system to better support a variety of block types
*Wrote a bunch of code that proved our hypothesis was possible, but then
there were design flaws. We learned a lot from that to redesign the
architecture, and now are building it right. Made changes to wikidom,
needs to be communicated to members of the team.*
Neil's been very busy with the upload wizard and timed media handler,
and Brion with code review & fixes ahead of the 1.18 deployment
(example: http://www.mediawiki.org/wiki/User:Brion_VIBBER/Math_fixes ).
So, no news on the Etherpad integration or the parser front.
Also, the Wikimedia Foundation is hiring for a visual editor software
developer, so please spread the word:
http://wikimediafoundation.org/wiki/Job_openings/Software_Developer_Rich_Te…
--
Sumana Harihareswara
Volunteer Development Coordinator
Wikimedia Foundation
We're trying to deploy MediaWiki 1.18 on Wikimedia Foundation sites by
mid-September, and get into a faster deployment pace for future
versions. That means we need help reviewing our backlog of unreviewed
code and fixing the FIXMEs. Could you help by reviewing some code, and
maybe even helping fix FIXMEs?
NEW parser-related commits that could use comments (You can click on a
revision number to get the code review page, where you can comment):
http://www.mediawiki.org/w/index.php?path=%2Ftrunk%2Fphase3%2Fincludes%2Fpa…
FIXMEs that are blocking the release of MediaWiki 1.18:
http://www.mediawiki.org/wiki/MediaWiki_roadmap/1.18/Revision_report
Thanks.
--
Sumana Harihareswara
Volunteer Development Coordinator
Wikimedia Foundation