Hi there,
I am investigating a breakage in my extension that has occurred in MW 1.34 but which didn't seem to be a problem on MW 1.29. (I have not tested interim versions to see where the issue first arises.)
One of the parser hooks in the extension needs to perform variable expansion. What is happening is a lot more complicated than this example, but effectively
<my_hook Foo="What the foo!">{{{Foo}}}</my_hook>
should end up generating the following output, using variable expansion:
What the foo!
The semantics of variable handling need to follow the MW semantics, including default values (possibly nested), parser functions, etc. therefore it needs to use the MW parser to perform the expansion.
Assuming the arguments that MW passes into the parser hook are named $Text, $Vars, $Parser and $Frame, the relevant code looks something like this (again, a bit more complicated in practice):
$NewFrame = new PPTemplateFrame_DOM($Frame->preprocessor, $Frame, array(), $Vars, $Frame->title); return $Parser->replaceVariables($Text, $NewFrame);
(I have included a more detailed listing of the code that I am using for doing the parse at the end of this message.)
My code was working fine on MW 1.29 and earlier, but when I upgrade to 1.34 I am finding that I get a fatal exception thrown when my tag is encountered:
/index.php?title=Main_Page MWException from line 373 of ~\includes\parser\PPFrame_DOM.php: PPFrame_DOM::expand: Invalid parameter type
I have generated a backtrace and the top of the stack is as follows:
#0 ~\includes\parser\Parser.php(3330): PPFrame_DOM->expand(PPNode_Hash_Tree, integer) #1 MyExtension.php (434): Parser->replaceVariables(string, PPTemplateFrame_DOM) #2 ~\includes\parser\Parser.php(4293): MyExtensionParserHook(string, array, Parser, PPTemplateFrame_Hash)
(The subsequent call stack entries are the parent functions you would expect to see in that situation.)
Can anyone see why the above code would no longer work as it did on previous versions? What is the current recommended method for manually expanding template variables from within a parser hook?
Kind regards,
- Mark Clements (HappyDog)
---------------------------------- Full example (with extension-specific code omitted): ----------------------------------
function MyExtensionParserHook($Text, $Vars, $Parser, $Frame) {
// 1) Manipulate $Text and $Vars
// (omitted)
// 2) Expand variables in the resulting text.
// Set up a new frame which mirrors the existing one but which also has the // field values as arguments. // If we are already in a template frame, merge the field arguments with the // existing template arguments first. if ($Frame instanceof PPTemplateFrame_DOM) { $NumberedArgs = $Frame->numberedArgs; $NamedArgs = array_merge($Frame->namedArgs, $Vars); } else { $NumberedArgs = array(); $NamedArgs = $Vars; } $NewFrame = new PPTemplateFrame_DOM($Frame->preprocessor, $Frame, $NumberedArgs, $NamedArgs, $Frame->title);
// Perform a recursive parse on the input, using our newly created frame. return $Parser->replaceVariables($Text, $NewFrame); }
Hi,
The DOM-based wikitext preprocessor (Preprocessor_DOM class and friends) was deprecated in MediaWiki 1.34 and removed in MediaWiki 1.35 as part of the Wikimedia Parsing Team’s work around Parsoid-PHP.[1] In the short/medium term, the easiest fix to keep your code working would be to use the other preprocessor implementation (class Preprocessor_Hash and friends) instead. Since your code already has access to a PPFrame instance, you can also try invoking its newChild() method to construct a new child frame with your arguments, instead of creating the instance directly.
In the long term, the legacy wikitext preprocessor will be removed, so you may want to reach out to the Parsing Team[2] to find out how you can make your code ready for Parsoid-PHP.
Hope this helps.
— [1] https://phabricator.wikimedia.org/T204945 [2] https://www.mediawiki.org/wiki/Parsing
Best, Máté Szabó Sr. Software Engineer he - him - his
Fandom Poland sp. z o.o. z siedzibą w Poznaniu, ul. Abp. A. Baraniaka 6 Sąd Rejonowy Poznań – Nowe Miasto i Wilda w Poznaniu, VIII Wydział Gospodarczy Krajowego Rejestru Sądowego, KRS 0000254365 NIP: 5252358778 Kapitał zakładowy: 50.000,00 złotych
On 29 Dec 2020, at 12:48, Mark Clements (HappyDog) gmane@kennel17.co.uk wrote:
Hi there,
I am investigating a breakage in my extension that has occurred in MW 1.34 but which didn't seem to be a problem on MW 1.29. (I have not tested interim versions to see where the issue first arises.)
One of the parser hooks in the extension needs to perform variable expansion. What is happening is a lot more complicated than this example, but effectively
<my_hook Foo="What the foo!">{{{Foo}}}</my_hook>
should end up generating the following output, using variable expansion:
What the foo!
The semantics of variable handling need to follow the MW semantics, including default values (possibly nested), parser functions, etc. therefore it needs to use the MW parser to perform the expansion.
Assuming the arguments that MW passes into the parser hook are named $Text, $Vars, $Parser and $Frame, the relevant code looks something like this (again, a bit more complicated in practice):
$NewFrame = new PPTemplateFrame_DOM($Frame->preprocessor, $Frame, array(), $Vars, $Frame->title); return $Parser->replaceVariables($Text, $NewFrame);
(I have included a more detailed listing of the code that I am using for doing the parse at the end of this message.)
My code was working fine on MW 1.29 and earlier, but when I upgrade to 1.34 I am finding that I get a fatal exception thrown when my tag is encountered:
/index.php?title=Main_Page MWException from line 373 of ~\includes\parser\PPFrame_DOM.php: PPFrame_DOM::expand: Invalid parameter type
I have generated a backtrace and the top of the stack is as follows:
#0 ~\includes\parser\Parser.php(3330): PPFrame_DOM->expand(PPNode_Hash_Tree, integer) #1 MyExtension.php (434): Parser->replaceVariables(string, PPTemplateFrame_DOM) #2 ~\includes\parser\Parser.php(4293): MyExtensionParserHook(string, array, Parser, PPTemplateFrame_Hash)
(The subsequent call stack entries are the parent functions you would expect to see in that situation.)
Can anyone see why the above code would no longer work as it did on previous versions? What is the current recommended method for manually expanding template variables from within a parser hook?
Kind regards,
- Mark Clements (HappyDog)
Full example (with extension-specific code omitted):
function MyExtensionParserHook($Text, $Vars, $Parser, $Frame) {
// 1) Manipulate $Text and $Vars
// (omitted)// 2) Expand variables in the resulting text.
// Set up a new frame which mirrors the existing one but which also has the // field values as arguments. // If we are already in a template frame, merge the field arguments with the // existing template arguments first. if ($Frame instanceof PPTemplateFrame_DOM) { $NumberedArgs = $Frame->numberedArgs; $NamedArgs = array_merge($Frame->namedArgs, $Vars); } else { $NumberedArgs = array(); $NamedArgs = $Vars; } $NewFrame = new PPTemplateFrame_DOM($Frame->preprocessor, $Frame, $NumberedArgs, $NamedArgs, $Frame->title);
// Perform a recursive parse on the input, using our newly created frame. return $Parser->replaceVariables($Text, $NewFrame); }
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://www.google.com/url?q=https://lists.wikimedia.org/mailman/listinfo/wi...
Thanks, Máté, That's helpful.
What is the most appropriate way to reach-out to the Parsing Team? I had assumed it would be via this list, but no-one else has responded to my question, so I guess not.
The method that I am using is based on what I was told to do for the previous parser implementation. It would be useful if there were some migration documentation to help extension developers migrate to the new implementation. Is that on the roadmap, somewhere?
Kind regards,
- Mark Clements (HappyDog)
"Máté Szabó" mszabo@wikia-inc.com wrote in message news:80E88BAC-AE9B-42AE-A0BA-834A39A7A292@wikia-inc.com...
Hi,
The DOM-based wikitext preprocessor (Preprocessor_DOM class and friends) was deprecated in MediaWiki 1.34 and removed in MediaWiki 1.35 as part of the Wikimedia Parsing Team's work around Parsoid-PHP.[1] In the short/medium term, the easiest fix to keep your code working would be to use the other preprocessor implementation (class Preprocessor_Hash and friends) instead. Since your code already has access to a PPFrame instance, you can also try invoking its newChild() method to construct a new child frame with your arguments, instead of creating the instance directly.
In the long term, the legacy wikitext preprocessor will be removed, so you may want to reach out to the Parsing Team[2] to find out how you can make your code ready for Parsoid-PHP.
Hope this helps.
[1] https://phabricator.wikimedia.org/T204945 [2] https://www.mediawiki.org/wiki/Parsing
Best, Máté Szabó Sr. Software Engineer he - him - his
Fandom Poland sp. z o.o. z siedziba w Poznaniu, ul. Abp. A. Baraniaka 6 Sad Rejonowy Poznan - Nowe Miasto i Wilda w Poznaniu, VIII Wydzial Gospodarczy Krajowego Rejestru Sadowego, KRS 0000254365 NIP: 5252358778 Kapital zakladowy: 50.000,00 zlotych
On 29 Dec 2020, at 12:48, Mark Clements (HappyDog) gmane@kennel17.co.uk wrote:
Hi there,
I am investigating a breakage in my extension that has occurred in MW 1.34 but which didn't seem to be a problem on MW 1.29. (I have not tested interim versions to see where the issue first arises.)
One of the parser hooks in the extension needs to perform variable expansion. What is happening is a lot more complicated than this example, but effectively
<my_hook Foo="What the foo!">{{{Foo}}}</my_hook>
should end up generating the following output, using variable expansion:
What the foo!
The semantics of variable handling need to follow the MW semantics, including default values (possibly nested), parser functions, etc. therefore it needs to use the MW parser to perform the expansion.
Assuming the arguments that MW passes into the parser hook are named $Text, $Vars, $Parser and $Frame, the relevant code looks something like this (again, a bit more complicated in practice):
$NewFrame = new PPTemplateFrame_DOM($Frame->preprocessor, $Frame, array(), $Vars, $Frame->title); return $Parser->replaceVariables($Text, $NewFrame);
(I have included a more detailed listing of the code that I am using for doing the parse at the end of this message.)
My code was working fine on MW 1.29 and earlier, but when I upgrade to 1.34 I am finding that I get a fatal exception thrown when my tag is encountered:
/index.php?title=Main_Page MWException from line 373 of ~\includes\parser\PPFrame_DOM.php: PPFrame_DOM::expand: Invalid parameter type
I have generated a backtrace and the top of the stack is as follows:
#0 ~\includes\parser\Parser.php(3330): PPFrame_DOM->expand(PPNode_Hash_Tree, integer) #1 MyExtension.php (434): Parser->replaceVariables(string, PPTemplateFrame_DOM) #2 ~\includes\parser\Parser.php(4293): MyExtensionParserHook(string, array, Parser, PPTemplateFrame_Hash)
(The subsequent call stack entries are the parent functions you would expect to see in that situation.)
Can anyone see why the above code would no longer work as it did on previous versions? What is the current recommended method for manually expanding template variables from within a parser hook?
Kind regards,
- Mark Clements (HappyDog)
Full example (with extension-specific code omitted):
function MyExtensionParserHook($Text, $Vars, $Parser, $Frame) {
// 1) Manipulate $Text and $Vars
// (omitted)// 2) Expand variables in the resulting text.
// Set up a new frame which mirrors the existing one but which also has the // field values as arguments. // If we are already in a template frame, merge the field arguments with the // existing template arguments first. if ($Frame instanceof PPTemplateFrame_DOM) { $NumberedArgs = $Frame->numberedArgs; $NamedArgs = array_merge($Frame->namedArgs, $Vars); } else { $NumberedArgs = array(); $NamedArgs = $Vars; } $NewFrame = new PPTemplateFrame_DOM($Frame->preprocessor, $Frame, $NumberedArgs, $NamedArgs, $Frame->title);
// Perform a recursive parse on the input, using our newly created frame. return $Parser->replaceVariables($Text, $NewFrame); }
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://www.google.com/url?q=https://lists.wikimedia.org/mailman/listinfo/wi...
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Mon, Jan 18, 2021 at 7:46 AM Mark Clements (HappyDog) gmane@kennel17.co.uk wrote:
Thanks, Máté, That's helpful.
What is the most appropriate way to reach-out to the Parsing Team? I had assumed it would be via this list, but no-one else has responded to my question, so I guess not.
The method that I am using is based on what I was told to do for the previous parser implementation. It would be useful if there were some migration documentation to help extension developers migrate to the new implementation. Is that on the roadmap, somewhere?
See https://www.mediawiki.org/wiki/Parsoid/Extension_API and https://www.mediawiki.org/wiki/Talk:Parsoid/Extension_API. There is also a bit of explanation about the changes at <https://lists.wikimedia.org/pipermail/wikitech-l/2020-September/093827.html along with a link to a tech talk that Subbu gave on the topic.
Bryan
"Máté Szabó" mszabo@wikia-inc.com wrote in message news:80E88BAC-AE9B-42AE-A0BA-834A39A7A292@wikia-inc.com...
Hi Máté,
I know it's been a while, but I've only now found some time to work on this in any depth.
The DOM-based wikitext preprocessor (Preprocessor_DOM class and friends) was deprecated in MediaWiki 1.34 and removed in MediaWiki 1.35 as part of the Wikimedia Parsing Team's work around Parsoid-PHP.[1]
I guess that explains why things changed in MW 1.34, specifically.
In the short/medium term, the easiest fix to keep your code working would be to use the other preprocessor implementation (class Preprocessor_Hash and friends) instead.
I think this is what I have now done. The solution I implemented was to replace the following line:
$NewFrame = new PPTemplateFrame_DOM($Frame->preprocessor, $Frame, array(), $Vars, $Frame->title);
With this:
if (is_a($Frame, "PPFrame_Hash")) $TemplateFrameClass = "PPTemplateFrame_Hash"; else $TemplateFrameClass = "PPTemplateFrame_DOM";
$NewFrame = new $TemplateFrameClass($Frame->preprocessor, $Frame, array(), $Vars, $Frame->title);
This seems to work on both MW versions I am testing on (1.29 and 1.34) and fits-in with your explanation, above.
Since your code already has access to a PPFrame instance, you can also try invoking its newChild() method to construct a new child frame with your arguments, instead of creating the instance directly.
I couldn't get this to work. I needed to pass additional arguments into the constructor, but got an error if I passed in an array of string => string pairs and there was no documentation about how to convert such an array into a format that the function would accept, so I gave up on this approach.
In the long term, the legacy wikitext preprocessor will be removed, so you may want to reach out to the Parsing Team[2] to find out how you can make your code ready for Parsoid-PHP.
Based on that comment, I suspect that further upgrade work will be required in due course, but at least I have solved the immediate problem.... for now!
Thanks for your help,
- Mark Clements (HappyDog)
Hi Mark,
Yeah, newChild() expects a preprocessor node instance (rather than the raw arguments array itself) for the arguments, which can be obtained by calling Preprocessor::newPartNodeArray() with the given set of arguments.
For Parsoid-PHP and the potential upgrade work it requires, it is probably something that will need to be addressed eventually, but from what I understand, there’s plenty of time until then, so it should not be an immediate concern. :)
Best, Máté Szabó Sr. Software Engineer he - him - his
Fandom Poland sp. z o.o. z siedzibą w Poznaniu, ul. Abp. A. Baraniaka 6 Sąd Rejonowy Poznań – Nowe Miasto i Wilda w Poznaniu, VIII Wydział Gospodarczy Krajowego Rejestru Sądowego, KRS 0000254365 NIP: 5252358778 Kapitał zakładowy: 50.000,00 złotych
On 8 Apr 2021, at 02:44, Mark Clements (HappyDog) gmane@kennel17.co.uk wrote:
"M�t� Szab�" mszabo@wikia-inc.com wrote in message news:80E88BAC-AE9B-42AE-A0BA-834A39A7A292@wikia-inc.com...
Hi M�t�,
I know it's been a while, but I've only now found some time to work on this in any depth.
The DOM-based wikitext preprocessor (Preprocessor_DOM class and friends) was deprecated in MediaWiki 1.34 and removed in MediaWiki 1.35 as part of the Wikimedia Parsing Team's work around Parsoid-PHP.[1]
I guess that explains why things changed in MW 1.34, specifically.
In the short/medium term, the easiest fix to keep your code working would be to use the other preprocessor implementation (class Preprocessor_Hash and friends) instead.
I think this is what I have now done. The solution I implemented was to replace the following line:
$NewFrame = new PPTemplateFrame_DOM($Frame->preprocessor, $Frame, array(), $Vars, $Frame->title);
With this:
if (is_a($Frame, "PPFrame_Hash")) $TemplateFrameClass = "PPTemplateFrame_Hash"; else $TemplateFrameClass = "PPTemplateFrame_DOM";
$NewFrame = new $TemplateFrameClass($Frame->preprocessor, $Frame, array(), $Vars, $Frame->title);
This seems to work on both MW versions I am testing on (1.29 and 1.34) and fits-in with your explanation, above.
Since your code already has access to a PPFrame instance, you can also try invoking its newChild() method to construct a new child frame with your arguments, instead of creating the instance directly.
I couldn't get this to work. I needed to pass additional arguments into the constructor, but got an error if I passed in an array of string => string pairs and there was no documentation about how to convert such an array into a format that the function would accept, so I gave up on this approach.
In the long term, the legacy wikitext preprocessor will be removed, so you may want to reach out to the Parsing Team[2] to find out how you can make your code ready for Parsoid-PHP.
Based on that comment, I suspect that further upgrade work will be required in due course, but at least I have solved the immediate problem.... for now!
Thanks for your help,
- Mark Clements
(HappyDog)
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
"Máté Szabó" mszabo@wikia-inc.com wrote in message news:633F17EB-07CB-42A8-B8F8-30D703D787E1@wikia-inc.com...
Hi Mark,
Yeah, newChild() expects a preprocessor node instance (rather than the raw arguments array itself) for the arguments, which can be obtained by calling Preprocessor::newPartNodeArray() with the given set of arguments.
That's useful info - I will take a look to see if that leads to a cleaner solution. Would be good if there were some documentation that covers all this, but I guess one can only dream! :-)
For Parsoid-PHP and the potential upgrade work it requires, it is probably something that will need to be addressed eventually, but from what I understand, there's plenty of time until then, so it should not be an immediate concern. :)
Yeah - I started looking into it, but there's still too many holes in the documentation and too much stuff that feels like it might change further. I'll wait until the release notes indicate impending doom!
Kind regards,
- Mark Clements (HappyDog)
-----------
Best, Máté Szabó Sr. Software Engineer he - him - his
Fandom Poland sp. z o.o. z siedziba w Poznaniu, ul. Abp. A. Baraniaka 6 Sad Rejonowy Poznan - Nowe Miasto i Wilda w Poznaniu, VIII Wydzial Gospodarczy Krajowego Rejestru Sadowego, KRS 0000254365 NIP: 5252358778 Kapital zakladowy: 50.000,00 zlotych
On 8 Apr 2021, at 02:44, Mark Clements (HappyDog) gmane@kennel17.co.uk wrote:
"M?t? Szab?" mszabo@wikia-inc.com wrote in message news:80E88BAC-AE9B-42AE-A0BA-834A39A7A292@wikia-inc.com...
Hi M?t?,
I know it's been a while, but I've only now found some time to work on this in any depth.
The DOM-based wikitext preprocessor (Preprocessor_DOM class and friends) was deprecated in MediaWiki 1.34 and removed in MediaWiki 1.35 as part of the Wikimedia Parsing Team's work around Parsoid-PHP.[1]
I guess that explains why things changed in MW 1.34, specifically.
In the short/medium term, the easiest fix to keep your code working would be to use the other preprocessor implementation (class Preprocessor_Hash and friends) instead.
I think this is what I have now done. The solution I implemented was to replace the following line:
$NewFrame = new PPTemplateFrame_DOM($Frame->preprocessor, $Frame, array(), $Vars, $Frame->title);
With this:
if (is_a($Frame, "PPFrame_Hash")) $TemplateFrameClass = "PPTemplateFrame_Hash"; else $TemplateFrameClass = "PPTemplateFrame_DOM";
$NewFrame = new $TemplateFrameClass($Frame->preprocessor, $Frame, array(), $Vars, $Frame->title);
This seems to work on both MW versions I am testing on (1.29 and 1.34) and fits-in with your explanation, above.
Since your code already has access to a PPFrame instance, you can also try invoking its newChild() method to construct a new child frame with your arguments, instead of creating the instance directly.
I couldn't get this to work. I needed to pass additional arguments into the constructor, but got an error if I passed in an array of string => string pairs and there was no documentation about how to convert such an array into a format that the function would accept, so I gave up on this approach.
In the long term, the legacy wikitext preprocessor will be removed, so you may want to reach out to the Parsing Team[2] to find out how you can make your code ready for Parsoid-PHP.
Based on that comment, I suspect that further upgrade work will be required in due course, but at least I have solved the immediate problem.... for now!
Thanks for your help,
- Mark Clements
(HappyDog)
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
wikitech-l@lists.wikimedia.org