[Wikimedia-l] [Wikimedia Announcements] Wikimedia engineering August 2013 report

Guillaume Paumier gpaumier at wikimedia.org
Fri Sep 6 17:19:02 UTC 2013 

Hi,

The report covering Wikimedia engineering activities in August 2013 is now
available.

Wiki version:
https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/August
Blog version:
https://blog.wikimedia.org/2013/09/06/engineering-august-2013-report/

We're also proposing a shorter, simpler and translatable version of this
report that does not assume specialized technical knowledge:
https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/August/summary

Below is the full HTML text of the report.

As always, feedback is appreciated on the usefulness of the report and its
summary, and on how to improve them.

------------------------------------------------------------------

Major news in August include:

   - A discussion about using the secure HTTP
protocol<https://blog.wikimedia.org/2013/08/01/future-https-wikimedia-projects/>on
Wikimedia sites, followed by a switch
   to that protocol for all registered
users<https://blog.wikimedia.org/2013/08/28/https-default-logged-in-users-wikimedia-sites/>
   ;
   - The launch of the Notifications
feature<https://blog.wikimedia.org/2013/08/30/notifications-launch-on-mobile/>on
the mobile site;
   - A discussion about how security issues are
handled<https://blog.wikimedia.org/2013/08/30/team-wikimedia-security-has-a-spot-for-you/>in
our community;
   - The Wikimania conference, which was notably an opportunity for the
   Language engineering
team<https://blog.wikimedia.org/2013/08/02/join-the-language-engineer-team-at-wikimania-in-hong-kong-next-week/>to
meet with users and improve language support, particularly for
   the Javanese
language<https://blog.wikimedia.org/2013/08/23/restoring-the-forgotten-javanese-script-through-wikimedia/>
   ;
   - A much-anticipated upgrade of the software used by our volunteer
   e-mail response
team<https://blog.wikimedia.org/2013/08/09/wikimedias-email-response-upgrade/>,
   OTRS.

*Note: We're also providing a shorter, simpler and translatable version of
this report<https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/August/summary>that
does not assume specialized technical knowledge.
*

*
*
Personnel Work with us <https://wikimediafoundation.org/wiki/Work_with_us>

Are you looking to work for Wikimedia? We have a lot of hiring coming up,
and we really love talking to active community members about these roles.

   - Vice President of Engineering: Search Firm Engagement -
Engineering<http://hire.jobvite.com/Jobvite/Job.aspx?j=oDVOXfw3>
   - User Experience
Designer<http://hire.jobvite.com/Jobvite/Job.aspx?j=oO8OXfwr>
   - Software Engineer -
Fundraising<http://hire.jobvite.com/Jobvite/Job.aspx?j=oawpXfwM>
   - Software Engineer - Editor Engagement
Experimentation<http://hire.jobvite.com/Jobvite/Job.aspx?j=o8NJXfwl>
   - Software Engineer - Editor
Engagement<http://hire.jobvite.com/Jobvite/Job.aspx?j=o6NJXfwj>
   - Software Engineer - Language
Engineering<http://hire.jobvite.com/Jobvite/Job.aspx?j=oH3gXfwH>
   - Software Engineer - Multimedia
Systems<http://hire.jobvite.com/Jobvite/Job.aspx?j=oj40Wfw3>
   - Senior Software Engineer -
Multimedia<http://hire.jobvite.com/Jobvite/Job.aspx?j=oC9OXfwg>
   - QA Engineer - Manual Testing- Visual
Editor<http://hire.jobvite.com/Jobvite/Job.aspx?j=oNeQXfwy>
   - FE Developer -
Analytics<http://hire.jobvite.com/Jobvite/Job.aspx?j=olyGXfwg>
   - Product Manager -
Platform<http://hire.jobvite.com/Jobvite/Job.aspx?j=o3vtXfwI>
   - Dev-Ops Engineer -
SRE<http://hire.jobvite.com/Jobvite/Job.aspx?j=ocLCWfwf>

Technical Operations

*Data Dumps <https://www.mediawiki.org/wiki/WMF_Projects/Data_Dumps>*
All dumps ran from the data center in Ashburn this month; only the
miscellaneous and experimental services remain to be moved. GSOC student
Petr Onderka completed the first incremental dump-producing
code<http://www.mail-archive.com/xmldatadumps-l@lists.wikimedia.org/msg00513.html>,
along with a draft
specification<https://www.mediawiki.org/wiki/User:Svick/Incremental_dumps/File_format/Diff_specification>for
the new format. Test it out and let us know what you think!

*Wikimedia Labs <https://www.mediawiki.org/wiki/Wikimedia_Labs>*
Due to Wikimania and staff vacations, this month had a relatively low
number of infrastructure changes, but we had a relatively high influx of
users and tools. We ran three workshops during Wikimania and helped
Toolserver users migrate their tools to Labs. We did have a few
infrastructure changes, though: A change for the service group interface
was merged but not yet deployed. It removes the service group interface
from the project interface, reducing clutter. An API was pushed in for
project and service group information, to make the information available
from Wikitech, rather than LDAP. Other infrastructure changes were
bugfixes, which can be found through bugzilla.

*OTRS*
OTRS got a long overdue update to version 3.2.9 with the generous volunteer
support of Martin and Marcel of Znuny GmbH. As part of the upgrade, the
service was migrated from pmtpa to eqiad, and spam filtering was
overhauled. Features
Engineering <https://www.mediawiki.org/wiki/Wikimedia_Features_engineering>
Editor
retention: Editing tools

*VisualEditor <https://www.mediawiki.org/wiki/VisualEditor>*
In August, the VisualEditor team continued work, and presented and ran
workshops at Wikimania in Hong
Kong<https://meta.wikimedia.org/wiki/Wikimania_2013>to discuss how to
best improve the system. The deployed version of the code
was updated three times
(1.22-wmf13<https://www.mediawiki.org/wiki/MediaWiki_1.22/wmf13#VisualEditor>,
1.22-wmf14<https://www.mediawiki.org/wiki/MediaWiki_1.22/wmf14#VisualEditor>and
1.22-wmf15<https://www.mediawiki.org/wiki/MediaWiki_1.22/wmf15#VisualEditor>),
with several mid-deployment releases as the code was developed to patch
urgent issues. The focus in this work was in improving the stability and
performance of the system, fixing a number of bugs uncovered by the
community, and making some usability improvements.

*Parsoid <https://www.mediawiki.org/wiki/Parsoid>*
In August, the Parsoid team continued to polish compatibility with existing
wikitext. User feedback after the July VisualEditor release was
instrumental in the identification of issues and the development of support
for important use cases of creative templating.

The increased team size also allowed us to perform some long-standing code
cleanup, make Parsoid compatible with Node 0.10, and improve testing.
The round-trip
testing infrastructure <http://parsoid.wmflabs.org:8001/> received a
much-needed overhaul. The storage back-end switched from SQLite to MySQL,
which improved throughput a lot and is allowing us to test new code far
more quickly than before. Performance statistics are now recorded, which
will let us identify performance bottlenecks as well as catch performance
regressions.
During Wikimania, the Kiwix team used Parsoid output to create an offline
copy of Wikivoyage. With standard HTML libraries and the rich RDFa
information in the Parsoid DOM, downloading and modifying the HTML
representation was done in about 1000 lines of
JavaScript<http://sourceforge.net/p/kiwix/other/ci/master/tree/mwhtmldumper/mwhtmldumper.js>
.
Editor engagement features

*Notifications <https://www.mediawiki.org/wiki/Echo_%28Notifications%29>*
In August, we released Notifications on the French, Hungarian, Polish,
Portuguese and Swedish Wikipedias, after extensive testing on the English
Wikipedia, as well as mediawiki.org and Meta-Wiki. This engagement tool was
well received by our new communities, especially social features such as
Mentions and Thanks, which enable users to communicate more effectively
than before. Benny Situ led the engineering work for this deployment and
fixed a number of bugs, with the help of Erik Benhardson and Matthias
Mullie. Fabrice Florin managed community relations for these new releases,
updating this release
plan<https://www.mediawiki.org/wiki/Echo/Release_Plan_2013>and
reaching out to more projects, to prepare for worldwide deployments on
all wiki projects in coming months. To that end, we teamed up with Philippe
Beaudette, Maggie Dennis, Patrick Earley, Jan Eissfeld, Anna Koval, Keegan
Peterzell, and Sherry Snyder to coordinate these releases with the
communities they serve. Dario Taraborelli created new metrics dashboards for
French <http://ee-dashboard.wmflabs.org/dashboards/frwiki-features>,
Hungarian <http://ee-dashboard.wmflabs.org/dashboards/huwiki-features>,
Polish <http://ee-dashboard.wmflabs.org/dashboards/plwiki-features>,
Portuguese <http://ee-dashboard.wmflabs.org/dashboards/ptwiki-features> and
Swedish <http://ee-dashboard.wmflabs.org/dashboards/sewiki-features>Wikipedias.
Lastly, we presented our work on Notifications in two talks at
Wikimania 2013, with both a general
overview<https://wikimania2013.wikimedia.org/wiki/Submissions/Notifications>and
a
technical presentation<https://wikimania2013.wikimedia.org/wiki/Submissions/How_to_enhance_your_MediaWiki_extensions_with_Echo_notifications>(see
slides<https://www.mediawiki.org/wiki/File:Notifications_Slides_-_08-10-2013.pdf>).
We are very grateful to all our community champions for each language and
look forward to more collaborations in the future. Our next major
deployment to non-English Wikipedias will take place on Sep. 17, to be
followed by weekly releases throughout the fall, as outlined in our release
plan <http://www.mediawiki.org/wiki/Echo/Release_Plan_2013>. To learn more,
visit the project
portal<https://en.wikipedia.org/wiki/Wikipedia:Notifications>,
read the help page <https://www.mediawiki.org/wiki/Help:Notifications> and
join the discussion on the talk
page<https://en.wikipedia.org/wiki/Wikipedia_talk:Notifications>
.

*Flow <https://www.mediawiki.org/wiki/Flow_Portal/Project_information>*
In August, we continued development of the Flow prototype by implementing
revisioning, moderation<https://www.mediawiki.org/wiki/Flow_Portal/Functional_Specifications/Moderation,_Protection,_and_Refactoring>,
and display code, on top of the storage and block abstractions. We have
deployed this prototype to an internal labs instance to encourage the full
team's involvement in development. Additionally, we participated in an agile
workshop<https://www.mediawiki.org/wiki/Wikimedia_Engineering/Flow_Team_Development_Workshops>run
by Arthur and Tomasz from the mobile team. This workshop facilitated
planning the Flow MVP and setting goals for the team's first development
sprint, along with providing information about agile guidelines and
practices that have worked well for the mobile team.

*Article feedback <https://www.mediawiki.org/wiki/Article_feedback>*
In August, we made a few feature tweaks and bug fixes for the Article
Feedback Tool <https://www.mediawiki.org/wiki/Article_feedback/Version_5>(AFT5)
on the
English <https://en.wikipedia.org/wiki/Special:ArticleFeedbackv5> and
French<https://en.wikipedia.org/wiki/fr:Sp%C3%A9cial:ArticleFeedbackv5>Wikipedias.
Matthias Mullie released a few patches to improve the opt-in/opt-out
tool<https://www.mediawiki.org/wiki/Article_feedback/Version_5/Feature_Requirements#Enable.2Fdisable_feedback>,
and tested the new feedback
notifications<https://www.mediawiki.org/wiki/Echo/Feature_requirements#Watched_Page_-_Feedback>to
let users know when feedback is marked as useful for a page they watch
(or for a comment they posted). We also presented our work on AFT5 at
Wikimania 2013, with designer Pau Giner and our French and German champions
Benoît Evellin and Denis Barthel, in this
session<https://wikimania2013.wikimedia.org/wiki/Submissions/Article_Feedback>(see
slides<https://www.mediawiki.org/wiki/File:Article_Feedback_Slides_-_08-11-2013.pdf>).
The team plans to make the AFT5 tool available to other wiki projects
interested in testing this tool later this year, as outlined in the release
plan<https://www.mediawiki.org/wiki/Article_feedback/Version_5/Release_Plan_2013>
.
Editor engagement experiments

*Editor engagement
experiments<https://www.mediawiki.org/wiki/Editor_engagement_experiments>
*
In August, the Editor Engagement Experiments team (E3) primarily focused on
development for its next and final A/B test of the Getting
Started<https://www.mediawiki.org/wiki/Extension:GettingStarted>task
suggestion system, a part of a project aimed at onboarding
new Wikipedians <https://www.mediawiki.org/wiki/Onboarding_new_Wikipedians>.
The team also worked on enhancements and bug fixes for the GuidedTour
extension <https://www.mediawiki.org/wiki/Extension:GuidedTour>, such as
adding the ability to customize default tour actions and better integration
with VisualEditor <https://www.mediawiki.org/wiki/VisualEditor>. During
part of August, the majority of the E3 team was at Wikimania 2013 in Hong
Kong, and delivered three talks, including on: guided
tours<https://wikimania2013.wikimedia.org/wiki/Submissions/Make_your_user_experience_easy_to_learn:_a_guided_tour>,
the team's new editor onboarding
process<https://wikimania2013.wikimedia.org/wiki/Submissions/Forget_the_tutorials,_be_bold%21_How_one_feature_has_attracted_thousands_of_new_editors>,
and product management<https://wikimania2013.wikimedia.org/wiki/Submissions/What_is_a_product_manager,_and_why_does_Wikimedia_need_them%3F>at
the Wikimedia Foundation.
Mobile <https://www.mediawiki.org/wiki/Wikimedia_Mobile_engineering>

*Commons App <https://www.mediawiki.org/wiki/Wikimedia_Apps/Commons>*
This month, the Mobile Apps team pushed out additional releases of the
Commons photo uploader app for iOS and Android. The iOS version includes a
major UI revamp by Monte, while the Android version has received multiple
incremental updates by Yuvi and Brion. Yuvi has been working on modernizing
support for campaigns in UploadWizard, which will make it easier to
coordinate uploads for events like Wiki Loves Monuments. Viewer,
contributor, and admin user interfaces for campaigns will come to the web,
with campaign-tied uploading in the web and mobile app. The team also
started making plans for the next generation of the Wikipedia reader app,
which will be more closely integrated with the mobile web site to ensure
that new features are always available through a web view, even where we
don't create specific native support. More details will be put together in
the next couple months.

*Wikipedia Zero <https://www.mediawiki.org/wiki/Wikipedia_Zero>*
This month, the team completed version 1 of Wikipedia Zero automation
tests, continued programming the re-architecture of Wikipedia Zero,
implemented search engine non-indexing, and analyzed HTTPS requirements in
support of a push for greater usage of HTTPS across Wikimedia projects. The
Wikipedia Zero engineering team thanks Amit Kapoor from the Wikipedia Zero
partnerships team, who wrapped up work with Wikimedia Foundation this
month, for his hard word getting the program off the ground. And the team
is also pleased to welcome Carolynne Schloeder, who joins the Wikipedia
Zero program as Director of Mobile, Programs.

*Mobile web projects <https://www.mediawiki.org/wiki/Mobile_web_projects>*
This month we continued to improve the mobile editing feature, monitoring
and triaging bugs and expanding the feature show at the section level of
articles. We also released the first iteration of mobile notifications to
projects where Echo is enabled (English, French, Polish, Portuguese,
Hungarian, and Swedish Wikipedia, as well as and Meta). In beta, we built a
new notifications treatment to be released in later months and continued
working on mobile talk pages.
Language engineering<https://www.mediawiki.org/wiki/Wikimedia_Language_engineering>

*Language tools <https://www.mediawiki.org/wiki/Language_tools>*
The language team continued maintenance of the
UniversalLanguageSelector<https://www.mediawiki.org/wiki/UniversalLanguageSelector>,
in particular improving performance and integration
testing<https://www.mediawiki.org/wiki/Language_Testing_Plan>,
and completed its integration with EventLogging, which will provide
metrics<https://www.mediawiki.org/wiki/Language_tools/Impact_Measurement_and_Metrics>useful
e.g. to choose
the best default font for a
language<https://www.mediawiki.org/wiki/Universal_Language_Selector/FAQ#What_if_the_default_font_chosen_is_not_the_best_one_for_the_language.3F_Can_this_be_changed.3F>.
Counts from translatewiki.net<https://ganglia.wikimedia.org/latest/graph.php?r=week&z=xlarge&title=&vl=&x=&n=&hreg%5B%5D=client-side&mreg%5B%5D=%5E.*UniversalLanguageSelector.*&gtype=line&glegend=show&aggregate=1&embed=1&_=1378369345031>are
live and a deployment plan for Wikimedia projects is under analysis.
The team also released<http://lists.wikimedia.org/pipermail/wikitech-l/2013-August/071529.html>its
monthly version of the MediaWiki Language Extension Bundle (MLEB)
which
is used by third party developers and community members to add language
support for their MediaWiki applications.

The team continued mentoring four Google Summer of Code (GSoC) students.
Praveen Singh, mentored by Santhosh Thottingal, released a Chrome
extension<https://chrome.google.com/webstore/detail/wikimedia-input-tools/fjnfifedbeeeibikgpggddmfbaeccaoh>for
Wikimedia Input Tools and contributed to the Indic Font Specification,
a collaborative open source project. Team members also continued to work
with Red Hat on various language initiatives.
The team participated at Wikimania in Hong Kong, which was an opportunity
to meet face to face, as well as to interact with Wikipedians and community
members to solve a variety of issues, including dealing with Chinese
language variants<https://blog.wikimedia.org/2013/09/03/language-support-at-wikimania-2013-in-hong-kong/>and
adding
language assets for
Javanese<https://blog.wikimedia.org/2013/08/23/restoring-the-forgotten-javanese-script-through-wikimedia/>.
The team also presented various
talks<https://blog.wikimedia.org/2013/08/02/join-the-language-engineer-team-at-wikimania-in-hong-kong-next-week/>on
language engineering.
Platform Engineering<https://www.mediawiki.org/wiki/Wikimedia_Platform_Engineering>
MediaWiki
Core

*Multimedia <https://www.mediawiki.org/wiki/Multimedia>*
In August, we continued to expand our multimedia
team<https://www.mediawiki.org/wiki/Multimedia>:
Bryan Davis joined us as senior platform engineer, working with product
manager Fabrice Florin, front-end engineer Mark Holmquist and engineering
director Rob Lanphier. We discussed multimedia plans and new feature ideas
with community members in two separate events: a multimedia roundtable at
Wikimania 2013 <https://meta.wikimedia.org/wiki/Roundtables/Roundtable_3>and an
IRC chat<https://meta.wikimedia.org/wiki/IRC_office_hours/Office_hours_2013-08-22>,
and updated our multimedia plan for the coming
year<https://www.mediawiki.org/wiki/Multimedia/2013-14_Goals>based on
their feedback (see
slides <https://www.mediawiki.org/wiki/File:Multimedia_Project_Slides.pdf>).
Summer contractor Brian Wolff completed the development of new gallery tags
to support more appealing layouts for thumbnails, while Jan Gerber made
improvements to the Score and TimeMediaHandler extensions. Mark Holmquist
started development on the Media
Viewer<https://www.mediawiki.org/wiki/Multimedia/Media_Viewer>,
based on designs by May Tee-Galloway and Jared Zimmerman; this new tool
will display images in larger size when clicking on article thumbnails, as
well as display file information and a full-screen viewing option, right on
the same page. We aim to test a first version of this tool as part of a beta
experiment <https://mingle.corp.wikimedia.org/projects/multimedia/cards/2>on
a few pilot sites in coming weeks. To discuss these features and keep
up
with our work, we invite you to join this new multimedia mailing
list<https://lists.wikimedia.org/mailman/listinfo/multimedia>.
Last but not least, we are also recruiting for two more positions for our
team: a multimedia systems
engineer<http://hire.jobvite.com/Jobvite/Job.aspx?j=oj40Wfw3&c=qSa9VfwQ>and
a senior
software engineer<http://hire.jobvite.com/Jobvite/Job.aspx?j=ouLnWfwi&c=qSa9VfwQ>.
Please spread the word about this unique opportunity to create a richer
multimedia experience for Wikipedia and MediaWiki sites!

*Search <https://www.mediawiki.org/wiki/Search>*
In August we deployed CirrusSearch to test2.wikipedia.org and
mediawiki.organd we're testing there. We're actively looking for other
volunteers to
test out CirrusSearch. Right now, CirrusSearch is not the primary search
for mediawiki.org; you have to use a URL parameter to test it. We're hoping
to make it the primary in September.

*Auth systems <https://www.mediawiki.org/wiki/Auth_systems>*
The team deployed OAuth to mediawiki.org on Aug 20th, and are working on
enhancement requests before the extension is enabled on the rest of the WMF
wikis. Several small bugs were fixed in SUL.

*Security auditing and
response<https://www.mediawiki.org/wiki/Security_auditing_and_response>
*
The team responded to reported issues, and prepared for the next MediaWiki
release, scheduled on September 3. We worked with Operations to enable
HTTPS for user logins in most geographies.
Quality assurance

*Quality Assurance <https://www.mediawiki.org/wiki/Quality_Assurance>*
This month QA began collaborating closely with Release Engineering to
coordinate improvement of reporting, monitoring, and testing software
releases. Our goal is to make our frequent software releases even more
reliable than they already are, and to use the tools and systems in place
today such as the beta labs cluster to make those reliable releases even
more frequent.

*Browser testing<https://www.mediawiki.org/wiki/Quality_Assurance/Browser_testing>
*
This month saw a significant change to the structure and organization of
browser tests, with tests and builds for CirrusSearch,
UniversalLanguageSelector, and VisualEditor following the example of
MobileFrontend and now residing in the git repositories for those
extensions, rather than in the /qa/browsertests repository. This creates
opportunities for more frequent and more accurate Jenkins builds of the
tests, while also reducing the overhead required for analyzing test
failures.
Engineering community team

*Bug management <https://www.mediawiki.org/wiki/Bug_management>*
Andre gave presentations on Improving MediaWiki quality: How everybody can
help with bug report
triaging<https://wikimania2013.wikimedia.org/wiki/Submissions/Improving_MediaWiki_quality:_How_everybody_can_help_with_bug_report_triaging>and
Transparency
and collaboration in Wikimedia
engineering<https://wikimania2013.wikimedia.org/wiki/Submissions/Transparency_and_collaboration_in_Wikimedia_engineering>at
Wikimania
2013 <http://wikimania2013.wikimedia.org/>. He updated Bugzilla's technical
documentation <https://wikitech.wikimedia.org/wiki/Bugzilla.wikimedia.org>and
documented how
to test Bugzilla code changes on the Wikimedia Labs
instance<https://www.mediawiki.org/wiki/Bug_management/Bugzilla_Labs_instance>.
Bugzilla now consequently
links<https://bugzilla.wikimedia.org/buglist.cgi?bug_id=42524,45271,52764>to
canonical places explaining how to write a good bug report and
explaining Bugzilla's UI fields. Bugzilla also shows a new "Show other
bugs" link <https://bugzilla.wikimedia.org/show_bug.cgi?id=46413> next to
the "Component" area to make finding similar reports easier. Andre cleaned
up his Greasemonkey triage helper
scripts<https://git.wikimedia.org/tree/wikimedia%2Fbugzilla%2Ftriagescripts>by
providing a setting for each functionality at the beginning of the
file;
a blog post<http://blogs.gnome.org/aklapper/2013/08/23/bugzillatips-triage-helpertools-greasemonkey/>provides
more details. Bugzilla's
testing instance <http://boogs.wmflabs.org/> on Wikimedia Labs saw several
patches deployed for testing, which after some more testing should end up
in the live Bugzilla; changes include: showing the history of a bug report
inline between the
comments<https://bugzilla.wikimedia.org/show_bug.cgi?id=47256>and
configuring the
guided bug entry form
<https://bugzilla.wikimedia.org/show_bug.cgi?id=36762>for users that
are new to bug reporting.

*Mentorship programs <https://www.mediawiki.org/wiki/Mentorship_programs>*

The 20 Google Summer of
Code<https://www.mediawiki.org/wiki/Summer_of_Code_2013>projects
passed the official mid-term evaluation at the beginning of
August, and the Outreach Program for
Women<https://www.mediawiki.org/wiki/Outreach_Program_for_Women/Round_6>project
is on track as well. Katie
Filbert <https://meta.wikimedia.org/wiki/User:Aude> (Aude), David
Cuenca<https://www.mediawiki.org/wiki/User:Micru>(Micru) and Quim
Gil <https://www.mediawiki.org/wiki/User:Qgil> (Qgil) will participate at
GSoC Mentors Summit in Mountain View (CA, USA) on October 19-20.

Monthly reports from the projects:

   - Refactoring of ProofreadPage
extension<https://www.mediawiki.org/wiki/Extension:Proofread_Page/GSoC#August>
   - Section handling in Semantic
forms<https://www.mediawiki.org/wiki/User:Himeshi/GSoC_2013/Project#August>
   - jQuery.IME extensions for Firefox and
Chrome<https://www.mediawiki.org/wiki/User:Prageck/GSoC_2013_Application#Monthly_Report:_August_2013>
   - Android app for MediaWiki
translation<https://www.mediawiki.org/wiki/User:Orsagi/GSoC_2013_proposal#Monthly_Report:_August_2013>
   - Mobilizing
Wikidata<https://www.mediawiki.org/wiki/User:Pragunbhutani/GSoC_2013_Updates#Monthly_Report:_August_2013>
   - Improve support for book
structures<https://meta.wikimedia.org/wiki/Book_management/Progress#August>
   - Incremental data
dumps<https://www.mediawiki.org/wiki/User:Svick/Incremental_dumps#August_report>
   - Language Coverage Matrix
Dashboard<https://www.mediawiki.org/wiki/User:Harsh4101991/GSoC_2013#August>
   - Internationalization and Right-To-Left Support in
VisualEditor<https://www.mediawiki.org/wiki/User:Mooeypoo/GSOC_2013_Proposal:_RTL_Support_in_VisualEditor#Monthly_Report:_August_2013>
   - Browser test automation for Visual
Editor<https://www.mediawiki.org/wiki/User:Rachel99/proposal1#August>
   - VisualEditor plugin for source
code<https://www.mediawiki.org/wiki/User:Beanixster#August>
   - UploadWizard: Book upload
customization<https://www.mediawiki.org/wiki/User:Rasel160/GSoC2013/Status#2013-08-monthly>
   - Prototyping inline
comments<https://www.mediawiki.org/wiki/User:Rjain/Gsoc2013/Project_Updates#August>
   - Improvement of glossary
tools<https://www.mediawiki.org/wiki/GSOC_2013_Improvement_of_glossary_tools#August>
   - Incremental updates for
Kiwix<https://www.mediawiki.org/wiki/User:Kiran_mathew_1993/ZIM_incremental_updates_for_Kiwix#Reports>
   - Pronunciation Recording
Tool<https://www.mediawiki.org/wiki/User:Rahul21/Gsoc2013/Project_Updates#August>
   - Bayesian Spam
Filter<https://www.mediawiki.org/wiki/User:Anubhav_iitr/Bayesan_spam_filter/Project_updates>
   - Wikidata language fallback and
conversion<https://www.mediawiki.org/wiki/User:Liangent/wb-lang/updates#August>

 *Technical communications<https://www.mediawiki.org/wiki/Technical_communications>
*
Guillaume Paumier <https://www.mediawiki.org/wiki/User:Guillom> continued
to focus on the VisualEditor deployment effort, working on communications,
documentation and liaising with the French Wikipedia. Work on technical
communications mostly focused on perennial activities like Tech
news<https://meta.wikimedia.org/wiki/Tech/News>and ongoing
communications support to the engineering staff.

*Volunteer coordination and
outreach<https://www.mediawiki.org/wiki/Volunteer_coordination_and_outreach>
*
We had a team presentation at Wikimania: Transparency and collaboration in
Wikimedia engineering<http://wikimania2013.wikimedia.org/wiki/Talk:Submissions/Transparency_and_collaboration_in_Wikimedia_engineering>,
explaining how volunteers can make a difference. Following the work on
Community
metrics <https://www.mediawiki.org/wiki/Community_metrics>, the five key
performance indicators<https://www.mediawiki.org/wiki/Community_metrics#Key_performance_indicators>(KPIs)
were discussed and agreed upon. We are focusing on the first one: who
contributes code<https://www.mediawiki.org/wiki/Community_metrics#Who_contributes_code>.
A list of Key Wikimedia software
projects<https://wikitech.wikimedia.org/wiki/Key_Wikimedia_software_projects>has
been created to define the scope of these KPIs. Recruiting automated
browser testers keeps being our top priority. We are organizing the next
workshop in San Francisco and online on September 18: Epic fail: figuring
out Selenium test
results<http://www.meetup.com/wikimedia-tech/events/137381022/>
.
Analytics <https://www.mediawiki.org/wiki/Analytics>

*Analytics infrastructure<https://www.mediawiki.org/wiki/Analytics/Infrastructure>
*
We continue to pursue the initiatives listed in our planning document.
We've had one analyst accept a job offer (welcome
Aaron<https://en.wikipedia.org/wiki/User:EpochFail>!)
and are in discussions with a software engineer. We continue to have a
solid pipeline and are spending a lot of time interviewing. Wikimetrics is
on target for an early September release and we've made good progress
against our hadoop infrastructure goals. In co-operation with Ops, we've
completed our reinstall of the Hadoop cluster and run several days of
reliability testing over the labor day weekend. We are currently
investigating replacing the Oracle JDK with the Open JDK to be in line with
our goals of using open source whenever possible. Our project to replace
udp2log with Kafka is making steadily progress. Varnishkafka, which will
replace varnishncsa, has been
debianized<https://gerrit.wikimedia.org/r/#/c/78782/>and the first
performance tests of compressing the message sets are very
encouraging. We created a test environment in Labs to test Kafka failover
modes and we have been prototyping with
Camus<https://github.com/linkedin/camus>to consume the data from a
broker and write it to HDFS. We are right now
thinking about how to set up Kafka in a multi data-center environment. The
Zookeepers have been reinstalled through Puppet as well.

*Analytics Visualization, Reporting &
Applications<https://www.mediawiki.org/wiki/Analytics/Visualization,_Reporting_%26_Applications>
*
In close collaboration with Dario, Jaime and Jessie, we have worked on new
features for Wikimetrics. In particular, we are adding new metrics such as
survival, pages created <https://gerrit.wikimedia.org/r/#/c/80659/>,
aggregation
of metrics <https://gerrit.wikimedia.org/r/#/c/76742/>, metadata in the CSV
output <https://gerrit.wikimedia.org/r/#/c/81261/>, a support
page<https://gerrit.wikimedia.org/r/#/c/79457/>and we have now more
than 90%
test coverage <https://gerrit.wikimedia.org/r/#/c/81153/> of the codebase.
In preparation for the reinstallation of the Hadoop cluster, we moved all
Wikipedia Zero jobs off the cluster. We took this opportunity to add
additional monitoring to the creation of Wikipedia Zero dashboards. We have
worked with Wikipedia Zero to identify a problem with Geolocation of
requests that has created large jumps in total traffic. We spent quite some
time creating a more robust process for updating and monitoring
gp.wmflabs.org. This dashboard is used by various internal stakeholders and
receives its information from different datastreams using different
scripts. We have been working on running these scripts under the general
purpose stats user, adding additional monitoring to prevent stale data and
puppetized some of the jobs.

*Data Releases <https://www.mediawiki.org/wiki/Analytics/Data_Releases>*
In August, we attended WikiSym and Wikimania. Dario
Taraborelli<https://en.wikipedia.org/wiki/User:DarTar>gave a keynote
address on actionable
Wikipedia research<https://www.mediawiki.org/wiki/File:Wikipedia_research_keynote_WikiSym_2013.pdf>at
WikiSym, where several other Wikipedia
research papers<https://meta.wikimedia.org/wiki/Research:Newsletter/2013/August>were
presented. At Wikimania, we hosted two sessions focused on Wikimedia
data<https://wikimania2013.wikimedia.org/wiki/Submissions/Datafying_Wikimedia>and
analytics
tools <https://www.mediawiki.org/wiki/File:UserMetrics_Wikimania_2013.pdf>.
We also worked with Platform engineering this month on analyzing and
visualizing HTTPS failure rates by
country<https://docs.google.com/spreadsheet/ccc?key=0Ams-fyukCIlMdFRwMWJ3czFRQ0NEeVliNklYTDhfdHc#gid=3>,
in preparation for the switch to HTTPS as a default. We released new
dashboards for the launch of notifications on 5 other Wikipedias and
continued to provide ad-hoc support to teams in Editor Engagement. Last, we
continued screening and interviewing candidates for an open research
analyst position.
Kiwix <http://www.kiwix.org>

*The Kiwix project is funded and executed by Wikimedia
CH<https://meta.wikimedia.org/wiki/Wikimedia_CH>
.*
Release of the new Mediawiki
offliner<http://sourceforge.net/p/kiwix/other/ci/master/tree/mwhtmldumper/>was
a little bit delayed; we are still fixing stability bugs. This
solution
has already proven its efficiency, as we have released 20 new ZIM files
this month: a new throughput record. The ZIM incremental update GSoC
project<https://www.mediawiki.org/wiki/User:Kiran_mathew_1993/ZIM_incremental_updates_for_Kiwix>progresses
too, as the student works on the integration of zimdiff/zimpatch
in the Kiwix ecosystem. Kiwix developers have had a 6 days hackathon in
Hong-Kong to prepare the next Kiwix release <http://www.kiwix.org>, after
some final work on compilation.
Wikidata<https://meta.wikimedia.org/wiki/Wikidata>

*The Wikidata project is funded and executed by Wikimedia
Deutschland<https://meta.wikimedia.org/wiki/Wikimedia_Deutschland/en>
.*
In August, the Wikidata team was present at 3 events: COSCUP,
Wikimania<http://blog.wikimedia.de/2013/08/22/wikidata-and-other-technical-bits-at-wikimania/>and
a meetup about Wikidata and Incubator. A lot of work has been put into
improving the API and its documentation. The team also worked on the
ability to reorder the qualifiers and sources in a statement, improved the
speed of Wikidata slightly, and made progress on the ability to query for
statements with a specific property and value, as well as merging items. An
improved proposal <https://www.wikidata.org/wiki/Wikidata:Wiktionary> for
the support of Wiktionary has been published. They also started the paper
cuts <https://www.wikidata.org/wiki/Wikidata:Paper_cuts> initiative to find
and fix small bugs that have a large impact on how enjoyable it is to use
Wikidata. Denny and Adam gave a short overview of the state of Wikidata and
answered questions during an office
hour<https://meta.wikimedia.org/wiki/IRC_office_hours/Office_hours_2013-08-26>on
IRC. The biggest news for August though was the activation
of data access (Wikidata phase 2) on
Wikivoyage<https://www.wikidata.org/wiki/Wikidata:PC#Data_acccess_for_Wikivoyage_and_bugfixes_plus_testing_needed_for_the_URL_datatype>
. Future The engineering management team continues to update the *
Deployments <https://wikitech.wikimedia.org/wiki/Deployments>* page weekly,
providing up-to-date information on the upcoming deployments to Wikimedia
sites, as well as the *engineering
roadmap<https://www.mediawiki.org/wiki/Roadmap>
*, listing ongoing and future Wikimedia engineering efforts.
------------------------------

*This report was written collaboratively by Wikimedia engineers and
managers. See revision
history<https://www.mediawiki.org/w/index.php?title=Wikimedia_engineering_report/2013/August&action=history>and
associated status pages. A wiki
version<https://www.mediawiki.org/wiki/Wikimedia_engineering_report/2013/August>is
also available.
*

-- 
Guillaume Paumier
Technical Communications Manager — Wikimedia Foundation
https://donate.wikimedia.org
-------------- next part --------------
_______________________________________________
Please note: all replies sent to this mailing list will be immediately directed to Wikimedia-l, the public mailing list of the Wikimedia community. For more information about Wikimedia-l:
https://lists.wikimedia.org/mailman/listinfo/wikimedia-l
_______________________________________________
WikimediaAnnounce-l mailing list
WikimediaAnnounce-l at lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimediaannounce-l

More information about the Wikimedia-l mailing list