Wikitech-l November 2010

wikitech-l@lists.wikimedia.org

88 participants
70 discussions

recursiveTagParse makes data vanish
by Cindy Cicalese 18 Nov '10

18 Nov '10

Hello. I'm developing a Semantic MediaWiki extension to partition an #ask query based upon the first character of one of the queried properties so that the user can page through partitions of what would otherwise be a very large query result. For example, I'd like to query all pages in Category:Author in my wiki, but there are almost 4,000 of those, so I want to partition the results dynamically by the first character of the author's last name. I have a working version, but I'm in the process of enhancing it to use Ajax so that it only needs to refresh the <div> with the new query results rather than the entire page as well as retaining previous partitions in hidden <div>'s so previous queries don't need to be repeated. I'm having two parsing issues in the Ajax callback. I do not believe that they are related to Ajax per se. The second issue may be caused by my inexpert fix to the first issue, although I ran across the same second issue previously in a different context. I have reproduced both issues in a vastly simplified, minimal test program that I will discuss below. I'm using MediaWiki 1.16.0 and SemanticMediaWiki 1.5.2. The first issue is that I need access to the parser in the Ajax callback so I can call recursiveTagParse on the results for the query. I tried using $wgParser, but it does not appear to be in a good state in the callback. I had a series of errors where the parser was calling functions on non-objects. I played around with it a bit and was able to get past all of these errors with this rather ugly code: global $wgParser; $wgParser->mOptions = new ParserOptions; $wgParser->initialiseVariables(); $wgParser->clearState(); $wgParser->setTitle(new Title($title)); where $title is the title retrieved using getTitle on the original (pre-Ajax) working parser and sent as a parameter to JavaScript and back. These calls allow me to get a parser that does not give me run-time errors, but I'm not at all confident that this is the correct approach. The second issue I'm having is that when I call recursiveTagParse on the data returned from the query in the Ajax callback, some of the data is "disappears". I should note that I'm invoking the same query and parsing code in both the initial display of the page and in the Ajax callback. It works initially but does not work in the callback using the questionable parser instance described above. However, I had also seen the same blanking behavior in a previous task where I did not have the first contributing issue. In that case, I changed my approach completely to avoid the recursiveTagParse, but I don't believe I have the option of doing that here. I found a bug report that seems similar, but I'm not sure if it is the same problem, and there doesn't seem to be any movement on that bug: https://bugzilla.wikimedia.org/show_bug.cgi?id=24556. I'm reproducing my simplified test case below. You can run it by placing the following wikitext on a page: {{#testQuery:[[Category:Author]] |limit=5 |searchlabel= |format=table }} replacing [[Category:Author]] with the query term of your choice. When you install the PHP extension code below, you'll see a page with a button and a table containing up to 5 query results. When you press the button, the same query is invoked, but the table is empty. If you comment out the line that contains the call to recursiveTagParse in the PHP code below, you'll see that the query results are indeed being returned in both cases, but they are getting obliterated in the Ajax callback by recursiveTagParse. It may very well be that solving the first issue will solve the second one, but since I saw the second issue in another context previously, I wonder if they really are two separate issues. Thank you very much for any assistance in working through these issues! Cindy PHP extension code: <?php /** * To activate the functionality of this extension include the following * in your LocalSettings.php file: * include_once("$IP/extensions/TestQuery/TestQuery.php"); */ if( !defined( 'MEDIAWIKI' ) ) die( "This is an extension to the MediaWiki package and cannot be run standalone." ); # credits $wgExtensionCredits['parserhook'][] = array ( 'name' => 'TestQuery', 'version' => '1.0', 'author' => "Cindy Cicalese", 'description' => "Bug test" ); $wgUseAjax = true; $wgAjaxExportList[] = 'testQueryPopulateDiv'; $wgHooks['LanguageGetMagic'][] = 'wfExtensionTestQuery_Magic'; $wgHooks['ParserFirstCallInit'][] = 'efTestQueryParserFunction_Setup'; function efTestQueryParserFunction_Setup (& $parser) { $parser->setFunctionHook('testQuery', 'testQuery'); return true; } function wfExtensionTestQuery_Magic(& $magicWords, $langCode) { $magicWords['testQuery'] = array (0, 'testQuery'); return true; } function testQuery($parser, $query) { $params = func_get_args(); array_shift($params); // first is $parser; strip it array_shift($params); // second is query string; strip it $testQuery = new TestQuery(); $output = $testQuery->firstVisit($parser, $query, $params); $parser->disableCache(); return array($parser->insertStripItem($output, $parser->mStripState), 'noparse' => false); } function testQueryPopulateDiv($query, $paramString, $title) { $params = explode("|", $paramString); $testQuery = new TestQuery(); $output = $testQuery->populateDiv($query, $params, $title); return $output; } class TestQuery { private $template = false; function firstVisit($parser, $query, $params) { $js = <<<EOT <script type="text/javascript"> function buttonClicked() { var query = document.forms['TestQuery'].Query.value; var params = document.forms['TestQuery'].Params.value; var title = document.forms['TestQuery'].Title.value; var div = document.getElementById('TestDiv'); sajax_do_call('testQueryPopulateDiv', [query, params, title], div); } </script> EOT; $parser->mOutput->addHeadItem($js); $this->parseParameters($params); $output = $this->buildForm($query, $params, $parser->getTitle()); $result = $this->getData($parser, $query, $params); $output .= $this->buildDiv($result); return $output; } function populateDiv($query, $params, $title) { global $wgParser; $wgParser->mOptions = new ParserOptions; $wgParser->initialiseVariables(); $wgParser->clearState(); $wgParser->setTitle(new Title($title)); $this->parseParameters($params); $currentPartitionData = $this->getData($wgParser, $query, $params); return $currentPartitionData; } private function parseParameters($params) { foreach ($params as $param) { if (preg_match("/^ *format *= *template *$/", $param) === 1) { $this->template = true; } } } private function buildForm($query, $params, $title) { $paramString = implode("|", $params); $out = <<<EOT <center> <button type='button' id='TestButton' onClick="buttonClicked()">Test Button </button> <form id='TestQuery' method='post' action=''> <input type='hidden' name='Query' value='$query'> <input type='hidden' name='Params' value='$paramString'> <input type='hidden' name='Title' value='$title'> </form> </center><br> EOT; return $out; } private function getData($parser, $query, $params) { $result = $this->doSMWAsk($parser, $query, $params); $result = $parser->recursiveTagParse($result); return $result; } private function doSMWAsk($parser, $query, $rawParams) { SMWQueryProcessor::processFunctionParams($rawParams, $qs, $params, $printouts); $output = SMWQueryProcessor::getResultFromQueryString($query, $params, $printouts, SMW_OUTPUT_WIKI); return $output; } private function buildDiv($result) { $output = "<div id='TestDiv' display='block'>"; $output .= $result; $output .= "</div>"; return $output; } } -- Dr. Cynthia Cicalese Lead Software Systems Engineer The MITRE Corporation

5 7

Job queue on Wiki Farms
by Daniel Friesen 17 Nov '10

17 Nov '10

There was a thought about the job queue that popped into my mind today. From what I understand, for a Wiki Farm, in order to use runJobs.php instead of using the in-request queue (which on high traffic sites is less desireable) the Wiki Farm has to run runJobs.php periodically for each and every wiki on the farm. So, for example. If a Wiki Farm has 10,000 wiki it's hosting, say the Wiki Host really wants to ensure that the queue is run at least hourly to keep the data on the wiki reasonably up to date, the wiki farm essentially needs to call runJobs.php 10,000 times an hour (ie: one time for each individual wiki), irrelevantly of whether a wiki has jobs or not. Either that or poll each database before hand, which in itself is 10,000 database calls an hour plus the runJobs execution which still isn't that desireable. What do people think of having another source class for the job queue like we have for file storage, text storage, etc... The idea being that Wiki Farms would have the ability to implement a new Job Queue source which instead derives jobs from a single shared database with the same structure as the normal job queue, but with a farm specific wiki id inside the table as well. Using this method a Wiki Farm would be able to set up a cron job (or perhaps a daemon to be even more effective at dispatching the job queue runs) which instead of making 10,000 calls to runJobs outright, it would fetch a random job row from the shared job queue table, look at the wiki id inside the row and execute a runJobs (perhaps with a limit=1000) script for that wiki to dispatch the queue and run some jobs for that wiki. It would of course continue looking at random jobs from the shared table and dispatching more runJobs executions serving the role of trying to keep the job queues running for all wiki on the farm, but without making wasteful runJobs calls for a pile of wikis which have no jobs to run. Any comments? -- ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]

4 6

Deferred, but still in need of review
by Jack Phoenix 16 Nov '10

16 Nov '10

Hi all, as you probably know, we mark commits related to extensions (and other things) not used by Wikimedia as "deferred" in CodeReview. I myself work mostly on extensions not used on Wikimedia (for example, SocialProfile) and thus most of my commits are marked as deferred by fellow developers. However, I think that this is evil and we shouldn't do this. I want to have my SocialProfile-related commits (as well as other extension commits) reviewed, but the deferred status doesn't mean "review me later", it means "nobody cares about this revision", or at least currently it means that. Nowadays when a revision is marked as deferred, it's highly unlikely that anyone ever bothers reviewing it. Ideally there should be a way to tag commits, such as commits to extensions not used by WMF, as non-mission critical yet in need of review. The deployment queue is already long enough and people who are reviewing code for Wikimedia deployment are having a hard time catching up; I don't want to make their work any more difficult than it already is, I just want to have my extensions reviewed to make sure that no glaring errors or security vulnerabilities slip in. Thanks and regards, -- Jack Phoenix MediaWiki developer

10 11

Open and transparency or not
by a b 16 Nov '10

16 Nov '10

Interesting..... users/developers comment on the inclusion of reCATCHPA within the fund-raising[1] code since other projects have been they can't utilize its services then ~two weeks later (which isn't all that long in wmf time) its project documentation is moved off wiki (mw wiki) into the office[2] wiki which is private to WMF staff members only.... I will let everybody think about that. [1]. http://www.mediawiki.org/wiki/Special:Code/MediaWiki/70919 [2]. http://www.mediawiki.org/w/index.php?title=Payment_Fraud_Prevention&action=…

4 3

WinCache support review ?
by Ashar Voultoiz 16 Nov '10

16 Nov '10

Hello, Chad added support for the Microsoft Windows Cache (WINCACHE) with r66364 and r66369, thus fixing bug 22844. This is obviously not going to be used for WikiMedia and I am not sure a lot of our developers are able to test it. Should we just mark both revisions as deferred or is there anyone willing to review this feature ? =) Links: http://www.mediawiki.org/wiki/Special:Code/MediaWiki/66364 http://www.mediawiki.org/wiki/Special:Code/MediaWiki/66369 https://bugzilla.wikimedia.org/show_bug.cgi?id=22844 -- Ashar Voultoiz

3 2

Bugzilla Weekly Report
by reporter 15 Nov '10

15 Nov '10

MediaWiki Bugzilla Report for November 08, 2010 - November 15, 2010 Status changes this week Bugs NEW : 131 Bugs ASSIGNED : 5 Bugs REOPENED : 16 Bugs RESOLVED : 113 Total bugs still open: 5140 Resolutions for the week: Bugs marked FIXED : 82 Bugs marked REMIND : 0 Bugs marked INVALID : 7 Bugs marked DUPLICATE : 13 Bugs marked WONTFIX : 7 Bugs marked WORKSFORME : 3 Bugs marked LATER : 1 Bugs marked MOVED : 0 Specific Product/Component Resolutions & User Metrics New Bugs Per Component Site requests 6 SemanticForms 4 AbuseFilter 4 WikiBhasha 4 Special pages 3 New Bugs Per Product MediaWiki 19 Wikimedia 12 MediaWiki extensions 28 mwEmbed 4 XML Snapshots 1 Top 5 Bug Resolvers jeluf [AT] gmx.de 25 ialex.wiki [AT] gmail.com 18 agarrett [AT] wikimedia.org 8 roan.kattouw [AT] gmail.com 6 rhalsell [AT] wikimedia.org 5

1 0

Call to arms - installer translations
by Max Semenik 15 Nov '10

15 Nov '10

As you all know, the installer was rewritten recently. Step-by-step wizard, JS magic, UI for configuration options that previously had to be manually changed in LocalSettings, in-place upgrades, you name it. However, there's one thing about it that needs YOUR help: translations. The new installer (at last!) has localisable interface, and it needs translations to be usable to more people around the world. However, from my experience with installer translations: * Most translators are non-techies, they have problems with long messages full of that weird IT lingo. * Things are bad when translators avoid translating a message because they don't understand it. Things get much, much worse if they think that they know how to translate them, but in reality they're wrong. * While simple users usually can take a look at wiki to have an idea how things work, this is not an option with the installer. Some messages may therefore be translated out of context. So those of you who are proficient in languages other than English - YOU can make a difference. Go to http://translatewiki.net and start translating. If there are already some translations in your language, double-check them - I assure you, there will be things to fix. Once everything is complete, try running the installer a few times, covering as many execution paths as you can - there will be things to fix. This way, we can ensure much better localisation. Oh, and forgive me my propagandist style - I couldn't help myself :) -- Max Semenik ([[User:MaxSem]])

2 1

Refresh of the Mediawiki logo
by Leo Koppelkamm 15 Nov '10

15 Nov '10

Hi, after seeing the new "powered by MediaWiki" button, I got motivated to rework the existing tournesol logo a bit. Mainly because the use of photos in logos is often discouraged and the path version looks a bit "friendlier" as well. What do you think of it? Fullsize: http://commons.wikimedia.org/wiki/File:Mediawiki_logo_reworked.svg 135px ($wgLogo size): http://upload.wikimedia.org/wikipedia/commons/thumb/3/30/Mediawiki_logo_rew… Mediawiki_logo_reworked.svg.png

13 14

CategoryFeed
by Max Semenik 14 Nov '10

14 Nov '10

This extension received no significant updates for the last 5 years and doesn't work with anything newer than 1.4. It doesn't even have a description page on mw.org. Nevertheless, numerous developers was^H^H^H spent their time on it while doing batch improvements to the whole extensions directory. Is someone interested in reviving it, or we can delete it right away? -- Max Semenik ([[User:MaxSem]])

1 0

Re: [Wikitech-l] Resource Loader bug 20962 ( {{PLURAL}} support )
by Dmitriy Sintsov 14 Nov '10

14 Nov '10

* Michael Dale <mdale(a)wikimedia.org> [Sat, 13 Nov 2010 17:08:50 +0100]: > On 11/10/2010 05:53 PM, Dmitriy Sintsov wrote: > > > > As Roan have pointed out, local messages don't need a complete parser > > anyway. I hope you have the time to merge it, if so can you announce > > that, please? > > Dmitriy > > The bug has been updated with a patch. Some integration notes are in the > bug comment. > Thank you a lot. I'll update from trunk and try to use it client-side in my extension. Dmitriy

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l November 2010