Wikitech-l January 2011

wikitech-l@lists.wikimedia.org

94 participants
78 discussions

Christmas server failure report
by Platonides 12 Jan '11

12 Jan '11

Earlier today, /a filled with binlogs in db27, which was s3 & s7 master. nagios had warned too early / nobody noticed. Slaves lagged, lots of locks, the wikis got to a halt. Revisions between 6:50 and 8:20 pm UTC were lost (although they can be manually reimported from db27). The new s3 and s7 master is db17, with only one slave: db25. After the master switch, we started having problems due to cached revision text in memcached, due to the duplication of old_id values, so we made them read-only until UTC midnight. We decided not to disable $wgRevisionCacheExpiry but to remove the faulty entries, thus I quickly prepared the script maintenance/purgeStaleMemcachedText.php to clean them. There were problems in hewiki, since data there didn't clean. On one instance doing $wgMemc->get persisted even after a $wgMemc->delete on that same key (???). Other than the hewiki issues, it seemed to run fine. There will be lots of wrong entries in diff and parser cache needing a manual action=purge but a purge will clean them. Flagged revs caches were not touched. Wikis using it may show the wrong content (with the additional fun of some users viewing the right one). There are also PPFrame_DOM->expand errors that started around the same time, even on wikis on a different cluster. They usually only happen once, and it succeeds just reloading. https://bugzilla.wikimedia.org/show_bug.cgi?id=26429

4 4

MediaWiki extension test plans for Cite/ ConfirmEdit extensions
by Nadeesha Weerasinghe 11 Jan '11

11 Jan '11

Test plans for the following MediaWiki extensions are available at, Cite : http://www.mediawiki.org/wiki/Cite_Extension_Test_Plan ConfirmEdit : http://www.mediawiki.org/wiki/ConfirmEdit_Test_Plan Test scenarios which can be automated are specified under 'Can Automate?' column and will be automated using selenium/PHP Unit. Regards, Nadeesha Weerasinghe

2 1

Chrome drops H.264 - to go Theora and VP8
by David Gerard 11 Jan '11

11 Jan '11

http://blog.chromium.org/2011/01/html-video-codec-support-in-chrome.html Chromium will support only Theora and VP8, Chrome to follow. Same as Firefox 4. (How close are we to allowing VP8 on WMF? I understand Greg Maxwell, as one of those actually working on making the VP8 code drop usable, had some qualms about the code at the time.) - d.

3 2

dataset1, xml dumps
by Ariel T. Glenn 11 Jan '11

11 Jan '11

For folks who have not been following the saga on http://wikitech.wikimedia.org/view/Dataset1 we were able to get the raid array back in service last night on the XML data dumps server, and we are now busily copying data off of it to another host. There's about 11T of dumps to copy over; once that's done we will start serving these dumps read-only to the public again. Because the state of the server hardware is still uncertain, we don't want to do anything that might put the data at risk until that copy has been made. The replacement server is on order and we are watching that closely. We have also been working on deploying a server to run one round of dumps in the interrim. Thanks for your patience (which is a way of saying, I know you are all out of patience, as am I, but hang on just a little longer). Ariel

14 31

Big problem to solve: good WYSIWYG on WMF wikis
by David Gerard 10 Jan '11

10 Jan '11

[crossposted to foundation-l and wikitech-l] "There has to be a vision though, of something better. Maybe something that is an actual wiki, quick and easy, rather than the template coding hell Wikipedia's turned into." - something Fred Bauder just said on wikien-l. Our current markup is one of our biggest barriers to participation. AIUI, edit rates are about half what they were in 2005, even as our fame has gone from "popular" through "famous" to "part of the structure of the world." I submit that this is not a good or healthy thing in any way and needs fixing. People who can handle wikitext really just do not understand how offputting the computer guacamole is to people who can cope with text they can see. We know this is a problem; WYSIWYG that works is something that's been wanted here forever. There are various hideous technical nightmares in its way, that make this a big and hairy problem, of the sort where the hair has hair. However, I submit that it's important enough we need to attack it with actual resources anyway. This is just one data point, where a Canadian government office got *EIGHT TIMES* the participation in their intranet wiki by putting in a (heavily locally patched) copy of FCKeditor: http://lists.wikimedia.org/pipermail/mediawiki-l/2010-May/034062.html "I have to disagree with you given my experience. In one government department where MediaWiki was installed we saw the active user base spike from about 1000 users to about 8000 users within a month of having enabled FCKeditor. FCKeditor definitely has it's warts, but it very closely matches the experience non-technical people have gotten used to while using Word or WordPerfect. Leveraging skills people already have cuts down on training costs and allows them to be productive almost immediately." http://lists.wikimedia.org/pipermail/mediawiki-l/2010-May/034071.html "Since a plethora of intelligent people with no desire to learn WikiCode can now add content, the quality of posts has been in line with the adoption of wiki use by these people. Thus one would say it has gone up. "In the beginning there were some hard core users that learned WikiCode, for the most part they have indicated that when the WYSIWYG fails, they are able to switch to WikiCode mode to address the problem. This usually occurs with complex table nesting which is something that few of the users do anyways. Most document layouts are kept simple. Additionally, we have a multilingual english/french wiki. As a result the browser spell-check is insufficient for the most part (not to mention it has issues with WikiCode). To address this a second spellcheck button was added to the interface so that both english and french spellcheck could be available within the same interface (via aspell backend)." So, the payoffs could be ridiculously huge: eight times the number of smart and knowledgeable people even being able to *fix typos* on material they care about. Here are some problems. (Off the top of my head; please do add more, all you can think of.) - The problem: * Fidelity with the existing body of wikitext. No conversion flag day. The current body exploits every possible edge case in the regular expression guacamole we call a "parser". Tim said a few years ago that any solution has to account for the existing body of text. * Two-way fidelity. Those who know wikitext will demand to keep it and will bitterly resist any attempt to take it away from them. * FCKeditor (now CKeditor) in MediaWiki is all but unmaintained. * There is no specification for wikitext. Well, there almost is - compiled as C, it runs a bit slower than the existing PHP compiler. But it's a start! http://lists.wikimedia.org/pipermail/wikitext-l/2010-August/000318.html - Attempting to solve it: * The best brains around Wikipedia, MediaWiki and WMF have dashed their foreheads against this problem for at least the past five years and have got *nowhere*. Tim has a whole section in the SVN repository for "new parser attempts". Sheer brilliance isn't going to solve this one. * Tim doesn't scale. Most of our other technical people don't scale. *We have no resources and still run on almost nothing*. ($14m might sound like enough money to run a popular website, but for comparison, I work as a sysadmin at a tiny, tiny publishing company with more money and staff just in our department than that to do *almost nothing* compared to what WMF achieves. WMF is an INCREDIBLY efficient organisation.) - Other attempts: * Starting from a clear field makes it ridiculously easy. The government example quoted above is one. Wikia wrote a good WYSIWYG that works really nicely on new wikis (I'm speaking here as an experienced wikitext user who happily fixes random typos on Wikia). Of course, I noted that we can't start from a clear field - we have an existing body of wikitext. So, specification of the problem: * We need good WYSIWYG. The government example suggests that a simple word-processor-like interface would be enough to give tremendous results. * It needs two-way fidelity with almost all existing wikitext. * We can't throw away existing wikitext, much as we'd love to. * It's going to cost money in programming the WYSIWYG. * It's going to cost money in rationalising existing wikitext so that the most unfeasible formations can be shunted off to legacy for chewing on. * It's going to cost money in usability testing and so on. * It's going to cost money for all sorts of things I haven't even thought of yet. This is a problem that would pay off hugely to solve, and that will take actual money thrown at it. How would you attack this problem, given actual resources for grunt work? - d.

44 135

Matching main namespace articles with associated talk page
by Diederik van Liere 09 Jan '11

09 Jan '11

Dear dev's, I am wondering whether the Mediawiki db contains a foreignkey relationship between a main namespace article and the associated talk page (if present). Having this information would greatly simplify analytic projects to monitor editor behaviour and understanding revert behaviour (among other topics). Currently, I am manually matching these two sets of pages by matching titles. I have two questions: 1) If this foreignkey does not exist, would it be worthwhile to create it? 2) If this foreignkey does exist, what would it take to expose this in the XML dumps? Best regards, Diederik

7 7

testing FlaggedRevs
by Amir E. Aharoni 08 Jan '11

08 Jan '11

I completed the translation of FlaggedRevs into Hebrew and i would like to test it. It is already installed on he.wikisource and the admins there agree that i'll play with it and gave me reviewer rights. My question is: Is there a proper manual test suite for FlaggedRevs, which guides the tester through all the possible permutations of revision levels and relevant messages? This must be manual, because the translation uses many PLURAL tricks and there many messages that are similar except small wording changes, but all of them must be checked. Thanks, -- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com "We're living in pieces, I want to live in peace." - T. Moore

1 0

Wikipedia dumps: nothing happening?
by Neil Harris 07 Jan '11

07 Jan '11

It looks like the Wikipedia database dump process is completely idle at the moment, with nothing having changed since the crash. Does anyone know when the dumps are likely to resume, or, if not, what the current status of the work on the dump system is? -- Neil

2 1

Update on 1.17
by Rob Lanphier 07 Jan '11

07 Jan '11

Hi everyone, Here's a quick update on the 1.17 release of Mediawiki. The code review group continues to make headway on the backlog of outstanding checkins in "new" status. We peaked at 1400 unreviewed checkins back in September, last month we were at 800, and now we're now under 300. We *hope* this means we can push a 1.17 version of MediaWiki to production sometime this month, with a tarball available some time after that. That still looks like a bit of a stretch goal, but more plausible now than the last time this came up. The roadmap is more-or-less up-to-date at this point (though some of the info from this mail should probably find its way onto the page: http://www.mediawiki.org/wiki/MediaWiki_roadmap/1.17 As of earlier today, there's a tracking bug for the few bugs that still need to be fixed (thanks bawolff!): https://bugzilla.wikimedia.org/show_bug.cgi?id=26611 The trendline for 1.17 is looks pretty good for us to get through the code review backlog for the most part: http://toolserver.org/~robla/crstats/crstats.117branch.html (focus on "new" hitting zero). However, there's still much to do. We really should get a good grip on testing this release. Roan set up a prototype wiki, and I did some mucking with it[1], so it's somewhat similar to production: http://prototype.wikimedia.org/deployment-en/Main_Page Other languages are set up too (ar, de, es, fr, it, ja, ko, nl, pl, ru, si, sr, zh). Links can be found on deployment-en's homepage. Note that only the barest of spot checking was done, and it appears at least one (ar) is broken as of this writing. One known issue on deployment-en is that Pending Changes/FlaggedRevs isn't configured properly (still looks like FlaggedRevs, so you'll see "unchecked" at the top of all of the articles). Once we get things set up a little better, it should be a good environment for testing. Rob [1] p.s. Roan: here's the mucking around that I did. I enabled FlaggedRevs as noted above, and ran update.php. I disabled LabeledSectionTransclusion because of an error in LabeledSectionTransclusion::setup

5 9

Need some input
by Chad 05 Jan '11

05 Jan '11

Ninja vs. Pirate. Discuss. -Chad

8 12

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l January 2011