Wikitech-l September 2012

wikitech-l@lists.wikimedia.org

128 participants
114 discussions

New extension: Diff
by Jeroen De Dauw 25 Oct '12

25 Oct '12

Hey, I'm happy to announce the first release of a new little extension I wrote called Diff. https://www.mediawiki.org/wiki/Extension:Diff It's a small utility library which might be of use to anyone creating a new extension :) Cheers -- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. --

13 37

scaled media (thumbs) as *temporary* files, not stored forever
by Ariel T. Glenn 24 Oct '12

24 Oct '12

So it's time to have this discussion again. At least, I think we're having it again, though I could not find previous threads on this list about the subject. In short, scaled media is currently generated on the fly for any size and for any user. The resulting files are kept around forever or until we run perilously short of space, at which point we make some guesses about what we can toss and then do a mass purge. Last time we did so, we had the rotation bug going at the same time, which made for a real fine mess. A little bit of crunching shows me that we have about 6 million images in use on the projects, and yet we manage to have around 130 million thumbnails. Just for fun I checked to see how many thumbs each image has, what sizes we are looking at, etc. Here's the results. Some "standard" sizes are most popular, with between 200K and 640K media files having thumbs scaled to each of these widths: 75, 120, 150, 180, 200, 220, 320, 640, 800, 1024, and 1280 pixels But there's plenty of "odd" sizes with lots of thumbs too. For example, over 65K files with width 181px, 20K with width 138px. As an experiment and before having this data, I purged from ms5 (no longer in use for thumbs) 1/16 of the thumbs that were greater than 100px wide but not one of these widths: 120px, 200px, 220px, 250px, 320px, 640px, 800px We got back over 300GB of space. The other thing about delivering any scaled version on demand is that we have some media files with several hundred different thumb sizes in there. Here's a few of the top offenders for your entertainment: 2514 wikipedia/commons/thumb/f/f9/Orange_and_cross_section.jpg 2285 wikipedia/commons/thumb/f/fb/Thrermal_grease.jpg 2218 wikipedia/commons/thumb/f/fc/Blue_sport.jpg 2071 wikipedia/commons/thumb/f/f3/Flag_of_Switzerland.svg 2062 wikipedia/commons/thumb/f/f2/Flag_of_Costa_Rica.svg 2034 wikipedia/commons/thumb/f/f8/Wiktionary-logo-en.svg 1915 wikipedia/commons/thumb/f/f6/VeulesLesRoses.JPG 1689 wikipedia/commons/thumb/f/fa/Wikibooks-logo.svg 1447 wikipedia/commons/thumb/f/fa/Wikiquote-logo.svg 1371 wikipedia/commons/thumb/f/f0/Mori_Uncanny_Valley.svg 1249 wikipedia/commons/thumb/f/f5/Grand_prismatic_spring.jpg 1246 wikipedia/commons/thumb/f/f3/Mature.jpg 1191 wikipedia/commons/thumb/f/f7/Kirchdorf_in_Tirol.JPG 1187 wikipedia/commons/thumb/f/f8/Camille_Cabral_pour_les_Trans.JPG 1143 wikipedia/commons/thumb/f/f7/Profanity.svg 1079 wikipedia/commons/thumb/f/f2/HSV_color_solid_cone.png 1040 wikipedia/commons/thumb/f/f2/Carmen_Electra.jpg 1032 wikipedia/commons/thumb/f/f1/Pink_eye.jpg 1001 wikipedia/commons/thumb/f/f6/USNS_Medgar_Evers_announcement.jpg I'd comment on some of those but I'd be too snarky. So there are some things we could change: 1. We could generate and keep only certain sizes, tossing the rest. 2. We could keep *nothing*, scaling all media as required. 3. We could have a cron job that was clever about tossing thumbs every day (not sure how easy it would be to be clever). 4. ?? In any of these cases, the squids will have copies of recently requested scaled media, so we won't be scaling the same file to the same size over and over in a short time frame. What do folks think about how to proceed? Ariel

26 50

Gerrit question: pushing to another branch
by Jeroen De Dauw 24 Oct '12

24 Oct '12

Hey, Is it possible to push to a branch other then the default one on gerrit using git review? This is needed when you want to have more then one branch on which you have reviewed code, or if you want different levels of review. For example if you want a novice committer play around with an extension a bit and push new functionality that gets reviewed but is not ready to go onto master until it really has stabilized and finalized. Cheers -- Jeroen De Dauw http://www.bn2vs.com Don't panic. Don't be evil. --

10 12

1.20 blocker bugs and code merges
by Mark A. Hershberger 22 Oct '12

22 Oct '12

While going through the bugs that were marked as 1.20 tarball blockers today, most seemed to have been left to rot -- that is, they didn't really deserve the tag since people were not motivated to fix them and the people who marked them 1.20 tarball blockers didn't put in the effort to find someone to fix them. Sometimes, that person was me, so this shouldn't be seen as overly harsh criticism. I did see a couple of exceptions: * https://bugzilla.wikimedia.org/38273 -- Tidy occasionally isn't executed Seems to be happening somewhat regularly since July on different WMF sites. Some effort to track it down has been made, but no solution has been found yet. I've a feeling this bug would bite third-party users of MW if we released a tarball with it. * https://bugzilla.wikimedia.org/35894 -- Reports of secret key generation "hanging" on windows` This is a bug that will really affect only new tarball users on Windows. While this probably isn't a lot of people, I would still like to see it fixed before a tarball is released. MaxSem had some ideas and I'll talk to him about how feasibly he can implement them. Otherwise, there are several patches mostly by Krinkle that need to be merged for a 1.20 tarball: https://bugzilla.wikimedia.org/31676 - ResourceLoader should work around IE stylesheet limit https://bugzilla.wikimedia.org/38158 - jquery.byteLimit sometimes causes an unexpected 0 maxLength being enforced https://bugzilla.wikimedia.org/40448 - Replace mediawiki.legacy.mwsuggest with SimpleSearch. https://bugzilla.wikimedia.org/40500 - ResourceLoader should not ignore media-type for urls in debug mode https://bugzilla.wikimedia.org/35923 - Followup to bug 11374 - tweaks to mediawiki.action.history.diff.css https://bugzilla.wikimedia.org/40498 - ResourceLoader should not output an empty "@media print { }" block. I still plan on being in #wikimedia-dev at UTC1100 on Tuesday, October 2 (See [0] for timezone conversion) for the triage, but I'll probably be spending that time putting in merge requests for the above issues (unless someone beats me to it). Also Tuesday, I'll make a new tarball with the code merged as my final release candidate for 1.20. Sam has said that because we're doing a review-first model with Gerrit people should feel more liberty to put merge requests in, so have at it! [1] Mark. [0-short] <http://hexm.de/lr> [0-long] <http://www.timeanddate.com/worldclock/meetingdetails.html?year=2012&month=1…> [1-short] <http://hexm.de/lw> [1-long] <https://gerrit.wikimedia.org/r/#/q/status:open+project:mediawiki/core+branc…> -- http://hexmode.com/ Necessitous men are not, truly speaking, free men. -- Vernon v Bethell (1762), quoted by FDR's Second Bill of Rights

3 5

MediaWiki community metrics
by Quim Gil 18 Oct '12

18 Oct '12

Hi there, Asking for a task to volunteer, Sumana encouraged me to look at the topic of community metrics. She pointed to https://wikitech.wikimedia.org/view/Pentaho as a starting point. After a first look at Pentaho and what some colleagues at the MeeGo project did with it [1], I searched (a bit) for any wiki pages or discussions about community metrics here, but couldn't find any. http://www.mediawiki.org/wiki/User:Qgil/MediaWiki_Community_Metrics Edits welcome! I'm looking for feedback, help, and a first prototype of an automatically refreshed report hopefully sooner than later. Something simple to build upon. Even if it's too tempting to define the first prototype thinking first on tools or data available, you are encouraged to start by proposing what questions do you want actually answered. What community trends do you want to know? See http://www.mediawiki.org/wiki/User:Qgil/MediaWiki_Community_Metrics#Trends_… - in few days we should have agreed on the first and most important trends we want to visualize. [1] http://wiki.meego.com/Metrics/Dashboard -- Quim

2 6

beta labs is now a fit test environment
by Chris McMahon 15 Oct '12

15 Oct '12

When I was hired as QA Lead almost seven months ago, WMF lacked a test environment where * code was routinely deployed ahead of production * the test environment emulated the production environment closely * aspects of the test environment (config, permissions, etc.) could be easily and reliably manipulated for testing purposes Today I am happy to announce that beta labs fulfills those needs. Beta labs is intended to host the upcoming release of Mediawiki, plus those extensions scheduled for deployment to production, for the purpose of testing and investigation. As of a little while ago, Mediawiki, AFTv5, New Pages Feed/Page Curation, and UploadWizard are being deployed to beta labs from git automatically and reliably. The configurations for those extensions are also being managed in git. The environment itself is managed via puppet, and emulates production to the greatest extent possible. Many many thanks to Antoine Musso for making this possible. As of this week, all these extensions are up, running, and configured to be useful. Note that they are not perfect, just useful. For example, right now on beta enwiki both AFTv4 and AFTv5 input forms appear on the same page in many cases, because I was experimenting with what happens when these extensions are not configured correctly. Some actions from the Page Curation toolbar never complete. As these glitches become important to testing, we will get them working correctly, and likely will find out some interesting things about the software along the way. The timing for this announcement is excellent, because new QA Engineers will be joining WMF soon (more on that next week), and beta labs will be a prime target for the browser-level end-to-end automated tests we will shortly be creating. Also, we have been wanting to retire the 'prototype' host for some time, and having AFTv5 etc. on beta labs should make that possible. In summary, beta labs is up and running with current code for Mediawiki and critical extensions, and at this point the best way to improve beta labs is to use it. http://en.wikipedia.beta.wmflabs.org/wiki/Special:ArticleFeedbackv5 http://en.wikipedia.beta.wmflabs.org/wiki/Special:NewPagesFeed http://commons.wikimedia.org/wiki/Special:UploadWizard

5 9

Skin pages on MW.org, and Skin repos in Gerrit
by Daniel Friesen 10 Oct '12

10 Oct '12

Skin pages on MW.org (and repos in Gerrit) are now ready. MediaWiki.org is now ready for modern skins (ones NOT using old QuickTemplate and skins/Foo.php patterns) to have pages about them just like extensions do. Relevant links: https://www.mediawiki.org/wiki/Category:All_skins https://www.mediawiki.org/wiki/Template:Skin Some examples that currently exist: https://www.mediawiki.org/wiki/Skin:Erudite https://www.mediawiki.org/wiki/Skin:Vector Gerrit can also handle your skin repos. Ask for a mediawiki/skins/{skinname} repo. So your skins can now have proper review and be updated with the various improvements that get made to MediaWiki's skinning system. When you do introduce a skin here please do follow the file layout patterns I used in my tutorials: http://blog.redwerks.org/2012/02/08/mediawiki-skinning-tutorial/ http://blog.redwerks.org/2012/02/28/mediawiki-subskin-tutorial/ I don't want to see any non-core skins using the outdated skins/TheSkin.php pattern in the new skins area. -- ~Daniel Friesen (Dantman, Nadir-Seen-Fire) [http://daniel.friesen.name]

6 8

best way to clean up the wiki from 3Gb of spam
by Yury Katkov 06 Oct '12

06 Oct '12

Hi everyone! I have found myself in the following situation several times: I created a wiki for some event or small project, everything works fine and after the event or project was done - nobody have seen this wiki for several months and does nothing on it. After several months somebody needs the wiki once again and realizes that the wiki database now have 3 Gb of text spam. Suppose that there is no back-up or rollback option in a wiki hosting. So here is the question: how to 1) remove all the spam 2) delete all the spam accounts 3) reduce the database size from 3Gb to the original size Cheers, Yury Katkov, WikiVote

8 24

translatewiki.net ending localisation support for svn.wikimedia.org MediaWiki extensions
by Siebrand Mazeland 03 Oct '12

03 Oct '12

Dear all, Starting October 1, 2012, translatewiki.net will drop support for all MediaWiki extensions it currently supports that still remain in the Subversion repository svn.wikimedia.org. Many, if not all of the extensions that by that time have not been migrated to git/Gerrit, are clearly not that well maintained and we want to prevent our translators from spending their time on it. At the moment, a maximum of 273 extensions are affected. In case this message makes you wonder what git and Gerrit is all about, and how you can revive your precious extension, please see the following links: * https://www.mediawiki.org/wiki/Git/Tutorial * https://www.mediawiki.org/wiki/Git/Conversion/Extensions_queue Cheers! Siebrand

2 1

Content handler feature merge (Wikidata branch) scheduled early next week
by Rob Lanphier 03 Oct '12

03 Oct '12

Hi everyone, Assuming no one finds any substantive issues, we plan on merging the ContentHandler feature (Wikidata branch) early next week, in time for 1.20wmf14 (assuming we're still calling this the "1.20" series then). The tracking bug for that is here: <https://bugzilla.wikimedia.org/show_bug.cgi?id=38622> The tree is here: <https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/core.git;a=tree;h=refs/he…> ...and recent commits are here: <https://gerrit.wikimedia.org/r/#/q/status:merged+project:mediawiki/core+bra…> We'd like to get it in right after a deployment branch point so that it has some time to settle in master before we foist it on everyone. Please take a look at this branch, and raise any issues with the branch or this plan on list. Thanks! Rob

6 11

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

Wikitech-l September 2012