[Wikimedia-l] [Wikimedia Announcements] Wikimedia engineering July 2013 report
Guillaume Paumier
gpaumier at wikimedia.org
Mon Aug 5 19:09:13 UTC 2013
Hi,
The report covering Wikimedia engineering activities in July 2013 is now
available.
Wiki version:
https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/July
Blog version:
https://blog.wikimedia.org/2013/08/05/engineering-july-2013-report/
We're also proposing a shorter, simpler and translatable version of this
report that does not assume specialized technical knowledge:
https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/July/summary
Below is the full HTML text of the report.
As always, feedback is appreciated on the usefulness of the report and its
summary, and on how to improve them.
------------------------------------------------------------------
Major news in July include:
- Giving<https://blog.wikimedia.org/2013/07/01/visualeditor-beta-rollout/>more
editors an easy-to-use editing interface (the VisualEditor) on several
Wikipedias
- Improving language support on our sites via summer interns'
projects<https://blog.wikimedia.org/2013/07/23/updates-from-the-language-engineering-google-summer-of-code-projects/>and
easier
configuration
options<https://blog.wikimedia.org/2013/07/01/universal-language-selector-uls-deployed-on-more-than-150-wikis/>,
and asking<https://blog.wikimedia.org/2013/07/31/translate-ui-wikipedias-visual-editor/>for
help translating the VisualEditor interface
- Enabling<https://blog.wikimedia.org/2013/07/25/edit-wikipedia-on-the-go/>users
to edit our sites from mobile devices, like phones and tablets, and
announcing<https://blog.wikimedia.org/2013/07/31/cooper-ux-bootcamp-wikipedia-mobile-editing/>a
future user experience bootcamp focusing on mobile editing
- Finishing<https://blog.wikimedia.org/2013/07/23/pywikipediabot-moving-to-git-on-july-26/>our
transition from keeping source code in Subversion to storing it in Git
- Launching<https://blog.wikimedia.org/2013/07/25/aircel-partnership-brings-wikipedia-zero-to-india/>a
Wikipedia Zero partnership with Aircel, giving mobile subscribers in
India the potential to access Wikipedia at no data cost
- Updating<https://blog.wikimedia.org/2013/08/01/future-https-wikimedia-projects/>the
Wikimedia movement on how we intend to protect our users' privacy with
HTTPS
- Signing<https://blog.wikimedia.org/2013/07/26/future-third-party-releases-mediawiki/>a
contract with longtime MediaWiki contributors to manage MediaWiki
releases for the open source community
- Explaining<https://blog.wikimedia.org/2013/07/03/wikimedia-foundation-fixes-software-bugs/>how
we find and gather software problems and deliver the fixes to users
*Note: We're also providing a shorter, simpler and translatable version of
this report<https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/July/summary>that
does not assume specialized technical knowledge.
*
Personnel Work with us <https://wikimediafoundation.org/wiki/Work_with_us>
Are you looking to work for Wikimedia? We have a lot of hiring coming up,
and we really love talking to active community members about these roles.
- Software Engineer -
Fundraising<http://hire.jobvite.com/Jobvite/Job.aspx?j=oawpXfwM>
- Software Engineer - Language
Engineering<http://hire.jobvite.com/Jobvite/Job.aspx?j=oH3gXfwH>
- Software Engineer - Multimedia
Systems<http://hire.jobvite.com/Jobvite/Job.aspx?j=oj40Wfw3>
- Senior Software Engineer -
Platform<http://hire.jobvite.com/Jobvite/Job.aspx?j=ouLnWfwi>
- Product Manager -
Platform<http://hire.jobvite.com/Jobvite/Job.aspx?j=o3vtXfwI>
- Dev-Ops Engineer -
SRE<http://hire.jobvite.com/Jobvite/Job.aspx?j=ocLCWfwf>
- Software Engineer - Editor Engagement
Experimentation<http://hire.jobvite.com/Jobvite/Job.aspx?j=o8NJXfwl>
- Software Engineer - Editor
Engagement<http://hire.jobvite.com/Jobvite/Job.aspx?j=o6NJXfwj>
- Director of Program -
Mobile<http://hire.jobvite.com/Jobvite/Job.aspx?j=oGqAXfwn>
- Front-end Developer -
Analytics<http://hire.jobvite.com/Jobvite/Job.aspx?j=olyGXfwg>
Announcements
- Bryan Davis joined the Platform Engineering team as a Senior Software
Engineer, working generally on backend software issues and starting off
supporting multimedia
(announcement<http://lists.wikimedia.org/pipermail/wikitech-l/2013-July/070923.html>
).
- C. Scott Ananian joined the Parsoid team as a Senior Features Engineer
(announcement<http://lists.wikimedia.org/pipermail/wikitech-l/2013-July/070316.html>
).
- Kenan Wang joined the Product team as Product Manager for Mobile (
announcement<https://twitter.com/wikimediaatwork/status/359473872527044609>
).
Technical Operations
*Site infrastructure*
Lots of Puppet<https://blog.wikimedia.org/2013/02/01/from-duct-tape-to-puppets/>refactoring
work got done this month, including considerable reorganization
of the puppet masters. Several manifests have been moved into modules, but
completing this project will take many months.
*Data Dumps <https://www.mediawiki.org/wiki/WMF_Projects/Data_Dumps>*
The English Wikipedia dumps ran out of our Ashburn data center this month,
and so did a number of other big wikis' dumps. There's an issue with the
abstract dumps that needs to be sorted out for those, but other than that
everything ran smoothly.Petr Onderka has been getting a lot of work done on
the incremental dumps. A first preview of the code was
announced<http://lists.wikimedia.org/pipermail/wikitech-l/2013-July/070971.html>as
well as a proposed binary file format which the program currently
uses.
For a preview of what's coming up, you can check the
timeline<https://www.mediawiki.org/wiki/User:Svick/Incremental_dumps#Timeline>.
Your comments and suggestions are welcome!
*Wikimedia Labs <https://www.mediawiki.org/wiki/Wikimedia_Labs>*
Though there were some features introduced this month, the majority of our
time was spent on documentation, tracking down bugs and improving
usability. We had a documentation sprint this month, targeted at improving
documentation for the Tool Labs <http://tools.wmflabs.org/> project. Work
continued on stabilizing the NFS server -- we believe we've tracked down
the stability issues to RAID controller problems. The compute nodes are
becoming increasingly low on disk space, but we've tracked this down in a
change in behavior of nova and have deployed a fix. nova-network was
starting to experience timeouts due to excessive load, leading to instance
creation failures. We've extended dhcp renewal times to reduce load. We
upgraded wikitech.wikimedia.org and wiktech-static to the 1.22wmf11 version
of MediaWiki. We also: deployed the AJAX-enabled delete instance feature;
deployed a change to display more informative instance statuses; fixed
issues in LdapAuthentication that broke blocking and renaming users; and
deployed a change to allow service groups to be added to service groups, to
make sharing code and data between tools easier. Features
Engineering<https://www.mediawiki.org/wiki/Wikimedia_Features_engineering>
Editor
retention: Editing tools
*VisualEditor <https://www.mediawiki.org/wiki/VisualEditor>*
In July, the VisualEditor team began switching the deployment from opt-in
alpha to opt-out beta, so becoming the default editor for users of the
various Wikipedias. The deployed version of the code was updated three
times (1.22-wmf10<https://www.mediawiki.org/wiki/MediaWiki_1.22/wmf10#VisualEditor>,
1.22-wmf11<https://www.mediawiki.org/wiki/MediaWiki_1.22/wmf11#VisualEditor>and
1.22-wmf12<https://www.mediawiki.org/wiki/MediaWiki_1.22/wmf12#VisualEditor>),
with several mid-deployment releases as the code was developed to patch
urgent issues. There were a number of user interface improvements, most
notably to the references insertion dialog, alongside fixes to a number of
bugs uncovered by the community.
*Parsoid <https://www.mediawiki.org/wiki/Parsoid>*
In July, the Parsoid team supported the deployment of VisualEditor as
default editor on eight Wikipedias, continuing to monitor bug reports,
feedback pages, and village pump and fixed a number of bugs to eliminate
instances of dirty diffs and other corruption that were reported. An
absence of performance issues let us focus our attention on functionality
and dirty-diff related bugs. This continued to be the primary focus of our
work this month. On the staffing side, C. Scott Ananian joined the Parsoid
team as a full-time employee -- he has been working with us since earlier
this year, first as a volunteer and then as a contractor. Marc Ordinas i
Llopis from Spain and Arlo Breault from Canada joined the Parsoid team as
contractors this month.
Editor engagement features
*Flow <https://www.mediawiki.org/wiki/Flow_Portal/Project_information>*
This month, we released two new
prototypes<https://www.mediawiki.org/wiki/Flow_Portal/Interactive_Prototype>to
showcase some ideas around Flow-enabled user-to-user discussion. We
continued to collect user feedback and prioritize use-cases for a potential
minimum viable product.
*Notifications <https://www.mediawiki.org/wiki/Echo_%28Notifications%29>*
In July, we released our final features for Notifications on the English
Wikipedia and mediawiki.org and meta.wikimedia.org. Benny Situ completed
development of HTML Email
notifications<https://www.mediawiki.org/wiki/Echo/Feature_requirements#HTML_single_email_notifications>,
as well as improved
notifications<https://www.mediawiki.org/wiki/Echo/Feature_requirements#Improved_Notification_Structure>,
based on designs by Vibha Bamba. Fabrice Florin managed the release of
these final features, and prepared this release
plan<https://www.mediawiki.org/wiki/Echo/Release_Plan_2013>to deploy
Notifications on more wiki projects, starting with French and
Polish Wikipedias in August. Dario Taraborelli and Matthias Mullie updated
our new metrics
dashboards<http://ee-dashboard.wmflabs.org/dashboards/enwiki-features>,
while Aaron Halfaker completed his report on our A/B test of new user
activity<https://meta.wikimedia.org/wiki/Research:Notifications/Experiment_1/Results>.
To learn more, visit the project
portal<https://en.wikipedia.org/wiki/Wikipedia:Notifications>,
read the FAQ page<https://en.wikipedia.org/wiki/Wikipedia:Notifications/FAQ>and
join the discussion on the talk
page <https://en.wikipedia.org/wiki/Wikipedia_talk:Notifications>.
*Article feedback <https://www.mediawiki.org/wiki/Article_feedback>*
In July, we deployed a few last features and bug fixes for the Article
Feedback Tool <https://www.mediawiki.org/wiki/Article_feedback/Version_5>(AFT5)
on the
English <https://en.wikipedia.org/wiki/Special:ArticleFeedbackv5> and
French<https://en.wikipedia.org/wiki/fr:Sp%C3%A9cial:ArticleFeedbackv5>Wikipedias.
Matthias Mullie released the
auto-archive<https://www.mediawiki.org/wiki/Article_feedback/Version_5/Feature_Requirements#Auto-archive_comments>feature,
as well as this list
of articles with feedback enabled on
enwiki<http://ee-prototype.wmflabs.org/metrics/aftv5_enabled_pages.php?wiki=enwiki>and
on
frwiki<http://ee-prototype.wmflabs.org/metrics/aftv5_enabled_pages.php?wiki=frwiki>.
At the request of the French Wikipedia community, he also developed
new feedback
notifications<https://www.mediawiki.org/wiki/Echo/Feature_requirements#Watched_Page_-_Feedback>to
let users know when feedback is marked as useful for a page they watch
(or for a comment they posted). The team plans to make the AFT5 tool
available to other wiki projects interested in testing this tool, provided
that no new development is required to support their needs, as outlined in
the release plan<https://www.mediawiki.org/wiki/Article_feedback/Version_5/Release_Plan_2013>
.
Editor engagement experiments
*We're hiring!* Are you a front-end developer? Do you know someone
who is? Apply
today <http://hire.jobvite.com/Jobvite/Job.aspx?j=o8NJXfwl&c=qSa9VfwQ>.
Editor engagement
experiments<https://www.mediawiki.org/wiki/Editor_engagement_experiments>
**
In July, the Editor Engagement Experiments (E3) team made progress on a
number of continuing projects. In terms of features, the team also
completed work to integrate the onboarding new
Wikipedians<https://www.mediawiki.org/wiki/Onboarding_new_Wikipedians>project
with new infrastructural changes and feature releases.
For the GettingStarted<https://www.mediawiki.org/wiki/Extension:GettingStarted>,
E3 collaborated with Platform engineering to ensure compatibility with the
new "SUL2 <https://www.mediawiki.org/wiki/SUL2>" cross-wiki authentication
architecture. For the GuidedTour
extension<https://www.mediawiki.org/wiki/Extension:GuidedTour>,
the team completed a first release of support for guided tours of the
VisualEditor <https://www.mediawiki.org/wiki/VisualEditor> interface,
alongside tours of the legacy wikitext editor, and developed a plan to
refactor<https://www.mediawiki.org/wiki/Extension:GuidedTour/Refactoring_brainstorming>the
GuidedTour extension as well as its API. E3 also planned for its sixth
A/B test of the GettingStarted workflow (see proposed specification and
mockups <https://www.mediawiki.org/wiki/Onboarding_new_Wikipedians#Proposed>).
As an addition to the team's redesign of account creation and login
(launched in May-June), we enhanced the design of the form for users who
fulfill account creation requests for others.
E3 team member Matthew Flaschen also worked with two Google Summer of
Code<https://www.mediawiki.org/wiki/GSOC_2013>students on their
projects. Richa
Jain <https://www.mediawiki.org/wiki/User:Rjain> is working on the Annotator
extension <https://www.mediawiki.org/wiki/Extension:Annotator>, which
allows adding inline comments to a wiki page. Rahul
Maliakkal<https://www.mediawiki.org/wiki/User:Rahul21>is working on
the Pronunciation
Recording extension<https://www.mediawiki.org/wiki/Extension:PronunciationRecording>,
for adding audio of pronunciations to Wiktionary.
On the experimental tools and data analysis front, E3 completed a
significant rewrite of the
Puppet<https://wikitech.wikimedia.org/wiki/Puppet>configuration for
EventLogging <https://www.mediawiki.org/wiki/EventLogging>, our data
collection pipeline, among other changes. For the
MediaWiki-Vagrant<https://www.mediawiki.org/wiki/MediaWiki-Vagrant>portable
desktop development environment, E3 added support for flexibly
provisioning and unit testing extensions such as GettingStarted,
GuidedTour, ParserFunctions <https://www.mediawiki.org/wiki/ParserFunctions>,
EventLogging, and others. Last but not least, the
micro-survey<https://meta.wikimedia.org/wiki/Research:Gender_micro-survey>of
gender of new account registrations was enabled on German, French,
Italian, and Polish Wikipedias, while data analysis on the English
Wikipedia results began.
Support
*2013 Wikimedia
fundraiser<https://www.mediawiki.org/wiki/2013_Wikimedia_fundraiser>
*
In July, the fundraising team did its first successful tests of our new
payments gateway: Adyen. The (as yet) US-only Credit Card backup gateway
performed similarly to our primary credit card processor in A/B testing,
and can be successfully used as a failover. We also ran, for the first
time, several short campaign tests targeted at mobile devices in the US. In
these tests, users were able to choose between Paypal or Amazon Payments.
Additional tests to determine peak times, appropriate localities, and
optimum messaging for mobile campaigns will continue throughout August, as
the campaigns are prepared.
Mobile <https://www.mediawiki.org/wiki/Wikimedia_Mobile_engineering>
*Wikipedia Zero <https://www.mediawiki.org/wiki/Wikipedia_Zero>*
This month, the team launched Wikipedia Zero with Aircel in
India<https://blog.wikimedia.org/2013/07/25/aircel-partnership-brings-wikipedia-zero-to-india/>,
a carrier with about 60 million cellphone subscribers. We also completed
our first cut of automation testing, started the implementation of the
Wikipedia Zero software re-architecture, and patched bugs. During July we
planned for the upcoming year in Wikipedia Zero. On the engineering front
we are focusing first on test automation and re-architecture concurrent
with SMS/USSD and J2ME releases, and afterward will be focusing efforts on
end user UX and carrier-oriented enhancements that will support the
continued growth of the program.
*Mobile web projects <https://www.mediawiki.org/wiki/Mobile_web_projects>*
This month, the mobile web team released a new contributory
nav<https://blog.wikimedia.org/2013/07/25/edit-wikipedia-on-the-go/>to
all Wikimedia mobile sites, including the existing upload and
watchlist
star features, as well as an edit button. This means that editing (in the
form of section-level markup editing) is now enabled on all mobile
Wikimedia sites for logged in users. In beta, we began work on mobile
notifications restyling, as well as guiders for first-time editors and
uploaders.
Platform Engineering<https://www.mediawiki.org/wiki/Wikimedia_Platform_Engineering>
MediaWiki
Core
*MediaWiki 1.22 <https://www.mediawiki.org/wiki/MediaWiki_1.22/Roadmap>*
In July 2013, MediaWiki 1.22wmf10 through 1.22wmf13 were successfully
deployed to Wikimedia project sites. We skipped the week of July 4th as
there was reduced capacity in both engineering and operations due to the US
holiday. We also named Markus Glaser and Mark
Hershberger<https://blog.wikimedia.org/2013/07/26/future-third-party-releases-mediawiki/>as
the new contractors maintaining the MediaWiki "tarball" for release to
other system administrators and organizations.
*Git conversion <https://www.mediawiki.org/wiki/Git/Conversion>*
With the migration of pywikipediabot from Subversion to
Git<https://blog.wikimedia.org/2013/07/23/pywikipediabot-moving-to-git-on-july-26/>,
we were able to switch svn.wikimedia.org to read-only mode, thus completing
this migration. We plan to keep the Subversion service around indefinitely
for archival purposes, and can still migrate any dormant project that
hasn't already been migrated on request.
*Multimedia <https://www.mediawiki.org/wiki/Multimedia>*
In July, we continued to expand our multimedia
team<https://www.mediawiki.org/wiki/Multimedia>:
Mark Holmquist joined as front-end software developer, working with product
manager Fabrice Florin and engineering director Rob Lanphier, as well as
contractors Brian Wolff and Jan Gerber. We prepared a first multimedia plan
for the coming year<https://www.mediawiki.org/wiki/Multimedia/2013-14_Goals>and
discussed our goals with community members in two separate events: a
multimedia roundtable<https://meta.wikimedia.org/wiki/Roundtables/Roundtable_2>and
an
IRC chat<https://meta.wikimedia.org/wiki/IRC_office_hours/Office_hours_2013-07-18b>.
Based on community feedback, we identified five main areas of activity
2013-2014: improving the viewing experience and upload pipeline in the
first half of the year, then focusing on file curation, discovery and
placement in articles for the second half of the year. Our overall goals
for this year are to increase both the number of contributions and files
used in Wikipedia articles. For now, we have started work on a new media
viewer <https://mingle.corp.wikimedia.org/projects/multimedia/cards/8> to
display images in larger size when you click on a thumbnail, as well as
display file information and a full-screen viewing option, right on the
same page. We plan to have a first version of that feature next month, and
will be testing it as part of a beta
experiment<https://mingle.corp.wikimedia.org/projects/multimedia/cards/2>on
a few pilot sites. We will also be hosting more community planning
discussions, such as this multimedia roundtable at Wikimania
2013<http://wikimania2013.wikimedia.org/wiki/Submissions/Multimedia_Roundtable>.
To participate in these discussions and keep up with our work, we invite
you to join this new multimedia mailing
list<https://lists.wikimedia.org/mailman/listinfo/multimedia>.
Last but not least, we are also recruiting for two more positions for our
team: a multimedia systems
engineer<http://hire.jobvite.com/Jobvite/Job.aspx?j=oj40Wfw3&c=qSa9VfwQ>and
a senior
software engineer<http://hire.jobvite.com/Jobvite/Job.aspx?j=ouLnWfwi&c=qSa9VfwQ>.
Please spread the word about this unique opportunity to create a richer
multimedia experience for Wikipedia and MediaWiki sites!
*Admin tools development<https://www.mediawiki.org/wiki/Admin_tools_development>
*
This activity was on hiatus in August.
*Search <https://www.mediawiki.org/wiki/Search>*
Nik Everett and Chad Horohoe have continued writing an extension to
implement ElasticSearch searching for MediaWiki, and we've finished most of
the required features. Next comes getting it deployed, scaled, and fixing
the inevitable bugs. We're aiming to deploy to the test site
beta.wmflabs.org before the end of the month. Peter Youngmeister and Asher
Feldman will be handling the operations tasks for the new setup.
*Auth systems <https://www.mediawiki.org/wiki/Auth_systems>*
Engineers worked towards for an
OAuth<https://www.mediawiki.org/wiki/Auth_systems/OAuth>deployment to
the
beta cluster <https://www.mediawiki.org/wiki/Beta_cluster> in early August,
and aim to roll OAuth out to the test wikis (e.g., test2.wikipedia.org)
after Wikimania.
*HipHop deployment <https://www.mediawiki.org/wiki/HipHop_deployment>*
HipHop work was mainly on hold in
July<http://lists.wikimedia.org/pipermail/hiphop/2013-July/000043.html>,
with the exception of some minor work on virtual machines.
*Security auditing and
response<https://www.mediawiki.org/wiki/Security_auditing_and_response>
*
The team continued to respond to reported security issues, and addressing
outstanding bugs.
Quality assurance
*Quality Assurance <https://www.mediawiki.org/wiki/Quality_Assurance>*
This month QA made contributions to the VisualEditor,
UniversalLanguageSelector and Mobile web projects, among others, finding
and reporting issues in a timely manner. Our intern with the Outreach
Program for Women <https://www.mediawiki.org/wiki/User:Rachel99> is working
on more automated browser tests. We continue to engage our community on the QA
mail list <https://lists.wikimedia.org/mailman/listinfo/qa> and in live
sessions, where we have several contributors (see Volunteer coordination
and outreach<https://www.mediawiki.org/wiki/Volunteer_coordination_and_outreach>
).
*Beta cluster <https://www.mediawiki.org/wiki/Beta_cluster>*
The Beta cluster continues to be a target for automated and manual testing.
It also finally has a syslog receiver on deployment-bastion, thus solving
bug 36748 <https://bugzilla.wikimedia.org/show_bug.cgi?id=36748> (no
syslog::server in beta). The logs can be accessed via either
/home/wikipedia/syslog or /data/project/logs/syslog/ . This is thanks to
Leslie Carr.
*Browser testing<https://www.mediawiki.org/wiki/Quality_Assurance/Browser_testing>
*
In July we added coverage for a number of features, including VisualEditor,
UniversalLanguageSelector, and Mobile Search. We are making extensive use
of beta labs as well as the test2wiki test environment. Our automated
browser tests continue to identify important issues during feature
development.
Analytics
We reviewed our planning document with the Sue and Erik and the Engineering
Directors. Reception was positive and we will be communicating next steps
more widely in August. The Analytics team focused on short term
deliverables, reliability and hiring in July. We identified two potential
candidates for front-end/Python work. We have been performing multiple
phone screens together with Recruiting, and the hiring pipelines are good.
*Analytics infrastructure<https://www.mediawiki.org/wiki/Analytics/Infrastructure>
*
*Kraken*:
- We kicked off a reliability project with Ops with the end goal of
stabilizing Hadoop and the logging infrastructure. Teams have been in
discussions on architecture and planning, and should have a path forward in
the next 2 weeks. We identified a consultant who will perform a system
audit to aid the project.
- We continue adding new
metrics<https://gerrit.wikimedia.org/r/#/c/75766/>and alerts to
monitor all the different parts of the webrequest dataflows
into Kraken. We expect to keep making improvements in the coming months
until we have a fully reliable data pipeline into Kraken.
- We puppetized Hue <https://gerrit.wikimedia.org/r/74388>,
Hive<https://gerrit.wikimedia.org/r/71569>,
and Oozie <https://gerrit.wikimedia.org/r/69804>. We also have a working
setup of the Hadoop cluster in Labs for testing
purposes<https://kraken.wmflabs.org:8888/>.
All Puppet work is open sourced.
*Logging Infrastructure*:
- We started this month with designing a canary event monitoring
system<https://mingle.corp.wikimedia.org/projects/analytics/cards/789>.
A canary event is an artificial event that is injected at the start of the
data workflow and which we will monitor to see it reaches its final
destination; that way we can ensure that the dataflows are functioning.
- We are investigating what data format to use for sending the
webrequest messages from Varnish to the Hadoop cluster. Formats that we are
scrutinizing are JSON, Protobuf and AVRO, but we are also looking at
compressions algorithms such as Snappy.
*Analytics Visualization, Reporting &
Applications<https://www.mediawiki.org/wiki/Analytics/Visualization,_Reporting_%26_Applications>
*
*Wikimetrics*: We successfully launched the initial version of Wikimetrics:
see metrics.wmflabs.org. This version has support for cohort upload and two
metrics: 1) bytes added and 2) namespace edits. We are working on adding
support for time-series and aggregators. In the coming sprints we will
focus on adding new
metrics<https://mingle.corp.wikimedia.org/projects/analytics/cards/779>
.
*Wikipedia Zero*: Dashboards have been moved off of Hadoop for the time
being and are now being populated again. We have identified some issues
with logrotation that are causing gaps in the graphs, and will look into
these problems. Also, we have been working on technical handoff as Evan
Rosen leaves the Foundation.
*Limn*: No development news.
*Wikistats*: No development news.
*Data Releases <https://www.mediawiki.org/wiki/Analytics/Data_Releases>*
- Erik Zachte published data and longitudinal analyses of edit and
revert trends for Wikimedia projects (read the
announcement<http://infodisiac.com/blog/2013/07/new-edit-and-revert-stats/>).
We provided data and ad-hoc analysis for the presentation *A State of
Decline? The State of Wikimedia Communities as of July 2013* at the July
2013 Monthly Metrics
Meeting<https://meta.wikimedia.org/wiki/Metrics_and_activities_meetings/2013-07-11>
.
- We published the analysis of a controlled experiment that we ran in
June to test the Impact of
notifications<https://meta.wikimedia.org/wiki/Research:Notifications/Experiment_1>on
new contributors and a pre-release A/B test of Visual
Editor <https://meta.wikimedia.org/wiki/Research:VE> on the English
Wikipedia. We performed an extensive audit of the quality of the data
collected during and after the VE test, taking into account browser
limitations and known bugs, and posted an
update<https://meta.wikimedia.org/wiki/Research_talk:VisualEditor#Updates_on_VE_data_analysis>on
the state of the analysis. We released via our open data repository
the
complete dataset <http://datahub.io/dataset/wikimedia-visualeditor> of
the sample of new registered users who participated in the split test to
ensure the replicability of the analysis.
- We released real-time dashboards on edit activity, new account
registrations and reverts for the 10 Wikipedias on which VE has been rolled
out. (en <http://ee-dashboard.wmflabs.org/dashboards/enwiki-metrics>
• de<http://ee-dashboard.wmflabs.org/dashboards/dewiki-metrics>•
es <http://ee-dashboard.wmflabs.org/dashboards/eswiki-metrics> •
fr<http://ee-dashboard.wmflabs.org/dashboards/frwiki-metrics>•
he <http://ee-dashboard.wmflabs.org/dashboards/hewiki-metrics> •
it<http://ee-dashboard.wmflabs.org/dashboards/itwiki-metrics>•
nl <http://ee-dashboard.wmflabs.org/dashboards/nlwiki-metrics> •
pl<http://ee-dashboard.wmflabs.org/dashboards/plwiki-metrics>•
ru <http://ee-dashboard.wmflabs.org/dashboards/ruwiki-metrics> •
sv<http://ee-dashboard.wmflabs.org/dashboards/svwiki-metrics>
)
Engineering community team
*Bug management <https://www.mediawiki.org/wiki/Bug_management>*
A PATCH_TO_REVIEW status was introduced in
Bugzilla<http://lists.wikimedia.org/pipermail/wikitech-l/2013-July/070759.html>which
is automatically set (by the Gerrit Notification Bot) on bug reports
when a commit message in Gerrit mentions a corresponding bug number.
Andre prepared
a patch for using the InlineHistory extension in
Bugzilla<https://bugzilla.wikimedia.org/show_bug.cgi?id=47256>and a
patch <https://bugzilla.wikimedia.org/show_bug.cgi?id=36762> to make
Bugzilla's guided bug entry
form<https://bugzilla.wikimedia.org/enter_bug.cgi?product=Wikimedia&component=Bugzilla&format=guided>for
new users usable for Wikimedia Bugzilla. Andre also continued his
weekly blogposts of Bugzilla
tips<https://www.mediawiki.org/wiki/Bug_management#Tricks_and_best_practices_in_Bugzilla>.
Thanks to Daniel Zahn, Bugzilla administrators now regularly receive an
email with a database dump of Bugzilla's "audit
log"<https://gerrit.wikimedia.org/r/#/c/56562/>which lists the most
recent taxonomy changes in Bugzilla (component or
keyword additions, etc.). In Bugzilla's taxonomy, the components in the
"Parsoid" product
<https://bugzilla.wikimedia.org/show_bug.cgi?id=50685>were reorganized
as requested by its main developer, and the remaining open
"OggHandler" tickets were closed as it has been superseded by
TimedMediaHandler.
*Mentorship programs <https://www.mediawiki.org/wiki/Mentorship_programs>*
Quim Gil <https://www.mediawiki.org/wiki/User:Qgil> organized meetings with
each Google Summer of
Code<https://www.mediawiki.org/wiki/Summer_of_Code_2013>and Outreach
Program for Women<https://www.mediawiki.org/wiki/Outreach_Program_for_Women>team,
one by one. Most projects were already at full speed, and for them,
the meeting was primarily social and nice to have. A few really benefited
from going through a checklist to highlight early problems easy to solve
now. All GSoC and OPW projects, 21 in total, are now on track.
*Technical communications<https://www.mediawiki.org/wiki/Technical_communications>
*
Like in June, Guillaume
Paumier<https://www.mediawiki.org/wiki/User:Guillom>was seconded to
the VisualEditor deployment effort, working on
communications, documentation and liaising with the French Wikipedia. Work
on technical communications mostly focused on perennial activities like
ongoing communications support to the engineering staff.
*Volunteer coordination and
outreach<https://www.mediawiki.org/wiki/Volunteer_coordination_and_outreach>
*
On Community metrics <https://www.mediawiki.org/wiki/Community_metrics>, Quim
Gil <https://www.mediawiki.org/wiki/User:Qgil> focused on the consolidation
of korma.wmflabs.org, the new dashboard for automated community metrics. We
have made good progress on this alpha, including basic metrics from Git,
Bugzilla and mailing lists being retrieved on a daily basis, and have filed
bugs and enhancement requests on GitHub
(mediawiki-dashboard<https://github.com/Bitergia/mediawiki-dashboard/issues>,
VizGrimoireJS <https://github.com/VizGrimoire/VizGrimoireJS/issues>). We
are deciding on the key metrics we need in order to make decisions, e.g.
average time to resolve on Gerrit changesets or bug reports. We also
planned and promoted a Browser Testing Automation workshop with
Cucumber<https://www.mediawiki.org/wiki/Meetings/2013-07-18>together
with the QA team, with 13 people participating online. You can
watch the session here
<https://www.mediawiki.org/wiki/Meetings/2013-07-18>(1h40). The
experience was useful, as we agreed on
MediaWiki-Vagrant <https://www.mediawiki.org/wiki/Vagrant> as the default
environment for automated testing and highlighted the list of easy
bugs<https://bugzilla.wikimedia.org/buglist.cgi?keywords=easy%2C%20&keywords_type=allwords&resolution=---&resolution=LATER&resolution=DUPLICATE&query_format=advanced&component=Quality%20Assurance&product=Wikimedia&list_id=219895>.
Also, the Engineering Community team held its quarterly
review<http://lists.wikimedia.org/pipermail/wikitech-l/2013-July/070632.html>
.
Language engineering<https://www.mediawiki.org/wiki/Wikimedia_Language_engineering>
The
language team deployed Universal Language Selector
(ULS)<https://www.mediawiki.org/wiki/Universal_Language_Selector>to
most Wikimedia wikis to provide easier
configuration options<https://blog.wikimedia.org/2013/07/01/universal-language-selector-uls-deployed-on-more-than-150-wikis/>to
readers and contributors. ULS provides a flexible way to configure and
deliver language settings like interface language, fonts, and input methods
(keyboard mappings). Also, ULS allows users to type text in different
languages not directly supported by their keyboard, read content in a
script for which fonts are not available locally, or customise the language
in which menus are displayed. For more information, please see the
FAQ<https://www.mediawiki.org/wiki/Universal_Language_Selector/FAQ>
. The Language engineering team also mentored summer interns'
projects<https://blog.wikimedia.org/2013/07/23/updates-from-the-language-engineering-google-summer-of-code-projects/>to
improve language support on our sites, and
asked<https://blog.wikimedia.org/2013/07/31/translate-ui-wikipedias-visual-editor/>for
volunteer help translating the VisualEditor interface.
Kiwix <http://www.kiwix.org>
*The Kiwix project is funded and executed by Wikimedia
CH<https://meta.wikimedia.org/wiki/Wikimedia_CH>
.*
We are preparing the first release of a new Wikipedia ZIM creation
solution<http://sourceforge.net/p/kiwix/other/ci/master/tree/mwhtmldumper/>for
August. We also have achieved a new release of Kiwix
for Android<https://play.google.com/store/apps/details?id=org.kiwix.kiwixmobile>;
this new version includes a few bug fixes and new features. Beside the
release of traditional Wikipedia ZIM files, we have also published two
interesting ZIM files: one which includes 2,500 ebooks (EPUB & PDF) of
French literature<http://download.kiwix.org/zim/other/bouquineux.zim.torrent>and
one with the new Wikipedia
for Schools selection<http://download.kiwix.org/zim/wikipedia_en_for_schools.zim.torrent>.
The ZIM incremental update GSoC
project<https://www.mediawiki.org/wiki/User:Kiran_mathew_1993/ZIM_incremental_updates_for_Kiwix>progresses
well too: first working versions of
zimpatch <https://github.com/kiranmathewkoshy/zimpatch> &
zimdiff<https://github.com/kiranmathewkoshy/zimdiff>console tools are
available, and integration with Kiwix has started. Kiwix
developers will be available at Wikimania<http://wikimania2013.wikimedia.org>,
during the hacking days and at the WikimediaCH both during Wikimania itself.
Wikidata <https://meta.wikimedia.org/wiki/Wikidata>
*The Wikidata project is funded and executed by Wikimedia
Deutschland<https://meta.wikimedia.org/wiki/Wikimedia_Deutschland/en>
.*
In July, we deployed Wikidata to all Wikivoyage sites in all languages, to
manage their language links. We updated the continued Roadmap for Wikidata
Development <https://www.mediawiki.org/wiki/Roadmap#Wikidata_deployment>.
Coveralls.io
support <https://coveralls.io/r/wikimedia> has been added to most of our
components. Since the first deployment of Phase1 to Wikipedia, about 240
million interwikilinks (5GB text) have been removed from articles
(2012<http://simia.net/languagelinks/index.html>vs
2013 <http://simia.net/languagelinks/2013.html> analysis). In other news,
the AAAI Feigenbaum Prize for Watson was donated to the Wikimedia
Foundation by IBM research to support work, especially on
Wikidata<https://blog.wikimedia.org/2013/07/16/ibm-research-watson-aaai-prize-wikimedia-foundation/>
. Denny Vrandečić
explains<http://lists.wikimedia.org/pipermail/wikidata-l/2013-July/002518.html>why
Wikidata items are identified with a Q.
Future The engineering management team continues to update the
*Deployments<https://wikitech.wikimedia.org/wiki/Deployments>
* page weekly, providing up-to-date information on the upcoming deployments
to Wikimedia sites, as well as the *engineering
roadmap<https://www.mediawiki.org/wiki/Roadmap>
*, listing ongoing and future Wikimedia engineering efforts. *Annual
goals*<https://www.mediawiki.org/wiki/Wikimedia_Engineering/2013-14_Goals>for
the 2013–2014 fiscal year are being drafted by some teams and have
been
finalized by others. *
This article was written collaboratively by Wikimedia engineers and
managers, and assembled by Sumana Harihareswara. See revision
history<https://www.mediawiki.org/w/index.php?title=Wikimedia_engineering_report/2013/July&action=history>and
associated status pages. A wiki
version<https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/July>is
also available.
*
--
Guillaume Paumier
Technical Communications Manager — Wikimedia Foundation
https://donate.wikimedia.org
-------------- next part --------------
_______________________________________________
Please note: all replies sent to this mailing list will be immediately directed to Wikimedia-l, the public mailing list of the Wikimedia community. For more information about Wikimedia-l:
https://lists.wikimedia.org/mailman/listinfo/wikimedia-l
_______________________________________________
WikimediaAnnounce-l mailing list
WikimediaAnnounce-l at lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimediaannounce-l
More information about the Wikimedia-l
mailing list