Almost a year ago, the Wikimedia Foundation migrated most of our
services from our old data center in Tampa to the new one in Ashburn
[1]. In the next couple of months Labs and Tool Labs will be following
suit -- we expect to have everything moved to Ashburn by mid-January at
the latest.
This move will provide some immediate benefits (lower latency with
production, quicker database replication) and many long-term benefits
(better stability, happier Operations staff). We don't yet have a
specific timeline for stages of the migration, but there are a few
things you can do now to help us prepare for the change and to bolster
your projects against possible disruption.
1) Subscribe to Labs-l, and read it. [2] Labs-l is low-volume, and
future migration announcements may not be sent to other lists.
2) Tool Labs users: As long as your tools are properly managed by the
grid engine and can survive stops and restarts, the migration will be
quite painless. If your tools aren't, or can't... fix them :)
3) Labs project admins: Clean up old projects and instances. If you
have instances that are no longer of interest, delete them. If you know
of entire projects that are no longer in use, please contact me directly
and I'll mop up.
4) Labs instance owners: Make sure that puppet is running properly on
your instances. If '$sudo puppetd -tv' produces any red lines, then fix
them or contact me for help with fixing. When instances move to the new
data center we'll be relying on puppet to update location-specific
settings, so instances without puppet may not survive the move. If your
instance uses self-hosting puppet (via puppetmaster::self or
role::puppet::self) then you will also need to update your local puppet
repo. [3]
5) All Labs users: if you have valuable data residing on local instance
storage, start backing it up to shared storage in /data/project. You
should be doing this anyway -- no instance is safe from catastrophe, ever.
6) If your project or tool generates log files, have a look at purging
old log data. The last time we did a data migration there was at least
one terabyte-sized logfile that really gummed up the works.
Updates about this change will be posted to this list as soon as we
know about them. Any potential downtime will be announced well in
advance. In the meantime, don't hesitate to ask questions about the
above steps on IRC or the mailing list.
-Andrew
[1]
https://blog.wikimedia.org/2013/01/19/wikimedia-sites-move-to-primary-data-…
[2] https://lists.wikimedia.org/mailman/listinfo/labs-l
[3] https://wikitech.wikimedia.org/wiki/Help:Self-hosted_puppetmaster#FAQ
Hi Everyone,
As an applicant of FOSS-OPW Round 7, I took a deep and profound interest
in one of the featured projects "Complete the MediaWiki development course
at Codeacademy". I chosen I am planning to focus and contribute to this
project.
***Project Synopsis***
The project is about completing the media wiki development course on Code
Academy and also enhancing the course.
- The Primary goal of the project is to develop instructional materials
to teach students to use Wikimedia API. Currently the course shows only
rudimentary API usage. What needs to be done is significantly more
sections, better explanations and tests/walkthroughs for students to
complete.
- The secondary goal of the project is that the tutorial does not need
to cover all options (there are too many of them), but should get students
started with the basic usage scenarios for most of the common tasks API
users expect.
-Mentor, Yuri Astrakhan
--
*Regards,*
*Diwanshi Pandey*
In the MW extension development I'm doing, I'm thinking of writing some
operations that use [[Comet_(programming)]] to deliver continuous
updates to the client, rather than the Ajax pattern of one request, one
response.
Has anyone messed with this? Any code I should crib from, or advice or
cautionary tales? Also, if it develops into something useful, I could
split it out for others to use.
Thanks,
LW
tl;dr: I’d appreciate thoughts from the Wikimedia technical community
at large whether the designation of individual technical contributors
as "architects" should be meaningful, and if so, how to expand it
beyond the original triumvirate (Brion, Tim & Mark), e.g. by
transitioning to a community-driven process for recognizing
architects.
Hi all,
in March 2011 and June 2011, Brion Vibber, Mark Bergsma and Tim
Starling were announced as Lead Software Architect, Lead Operations
Architect and Lead Platform Architect of the Wikimedia Foundation,
respectively. Together, these three individuals laid much of the
foundation of Wikimedia’s technical infrastructure, from MediaWiki
itself to our caching and load balancing setup. So it was a logical
step to recognize their immense contributions, and to entrust them
with continued high-level stewardship in Wikimedia’s technical
ecosystem.
Since then, WMF's engineering organization has grown pretty
dramatically. We've also seen increased engagement in the Wikimedia
technical community from other organizations. Wikimedia Germany is
probably most notable among them with the Wikidata project, and Wikia
has partnered directly on VisualEditor development and is generally
striving to increase visibility of its open source modifications to
MediaWiki.
At WMF, this has increasingly raised the question how the architecture
of Wikimedia’s technical infrastructure can be evolved at this new,
larger scale, and how we can bring more voices into that conversation.
I've shared this note with the architects ahead of time and taken some
initial feedback into account.
Rob Lanphier has taken a lead role in giving the RFC process a kick in
the pants as a solid, asynchronous, transparent process for organizing
and resolving technical proposals. Brion, Tim and Mark are explicitly
listed as the three individuals who can close an RFC (interpreting or
helping reach consensus, or making an informed decision where there’s
a lack of community participation), and have helped clear the RFC
backlog and evolve the architecture guidelines. In addition, Rob is
organizing the MediaWiki architecture summit in January, where we can
talk about some of the most pressing or contentious architectural
questions in person.
However, Brion, Tim and Mark are not infinitely scalable, nor are they
immortal (except in our hearts). They can’t be in every conversation,
know every part of Wikimedia’s technical ecosystem, review every RFC,
etc. We also have many other deeply talented technical contributors,
including some who have many years of experience in our technical
context specifically -- not just at WMF. Beyond just making technical
decisions, architectural leadership creates opportunities for
mentorship, modeling and nurturing the kind of behavior we want to
foster in our technical community.
So how should this role evolve going forward? Some possible paths (you
know I like to present options ;-):
Option A: We change nothing and don't promote any new people into
architect roles for a while. I truly don’t think this is an option for
much longer -- we need to find better ways to encourage some of our
other capable technical contributors to feel ownership over
Wikimedia’s technical direction, and fill gaps in architectural
leadership today. That said, it would be possible to make the RFC
process more egalitarian and to reduce the emphasis on formalized
technical leadership.
Option B: WMF handles it as it sees fit. This basically means WMF gets
to decide who to designate as "Architects" and at what point, which
would mostly leave this decision in the hands of managers. This is a
very WMF-centric view of the world, but it’s of course the way most
organizations operate.
Option C: We get rid of the special role of architects. I personally
don’t favor this option either, because I think recognizing the most
sane and experienced voices in our technical community and according
them some real leadership influence over Wikimedia’s technical
direction is important (and a useful counterbalance to pointy-haired
folks like yours truly ;-).
Option D: We come up with some kind of open process for
designating/confirming folks as architects, according to some
well-defined criteria (including minimum participation in the RFC
process, well-defined domain expertise in certain areas, a track
record of constructive engagement, etc.). Organizations like WMF can
choose to recognize this role as they see fit (likely according salary
increases to individuals who demonstrate successful architectural
leadership), but it’s a technical leadership role that’s awarded by
Wikimedia’s larger technical community, similar to +2 status.
Each of these would need to be unpacked and further developed. For
option D in particular, I think it’s important to recognize that the
level of impact beyond WMF for technical decisions varies
significantly. While parts of WMF’s overall operating infrastructure
are of interest to third parties, decisions that primarily relate to
keeping our cluster and network secure, performant and stable are
really ours to make and it’s unlikely that relevant architectural
decisions would e.g. be driven by a third party employee. At the same
time, we still aspire to making such work accessible and visible to
volunteers.
I do think it’s possible to make option D work in a way that
recognizes the need for different kinds of architectural leadership
(e.g. MediaWiki development vs. network architecture), and that
applies selection standards that are consistent with a role’s impact
on the community.
I’m sure there are other possible ways to proceed as well (aren't
there always :P) and would love to hear your thoughts.
All best,
Erik
[1] http://lists.wikimedia.org/pipermail/wikimediaannounce-l/2011-March/000130.…
[2] http://lists.wikimedia.org/pipermail/wikimediaannounce-l/2011-June/000186.h…
--
Erik Möller
VP of Engineering and Product Development, Wikimedia Foundation
Hi folks,
for a long time we've relied on the mwlib libraries by PediaPress to
generate PDFs on Wikimedia sites. These have served us well (we
generate >200K PDFs/day), but they architecturally pre-date a lot of
important developments in MediaWiki, and actually re-implement the
MediaWiki parser (!) in Python. The occasion of moving the entire PDF
service to a new data-center has given us reason to re-think the
architecture and come up with a minimally viable alternative that we
can support long term.
Most likely, we'll end up using Parsoid's HTML5 output, transform it
to add required bits like licensing info and prettify it, and then
render it to PDF via phantomjs, but we're still looking at various
rendering options.
Thanks to Matt Walker, C. Scott Ananian, Max Semenik, Brad Jorsch and
Jeff Green for joining the effort, and thanks to the PediaPress folks
for giving background as needed. Ideally we'd like to continue to
support printed book generation via PediaPress' web service, while
completely replacing the rendering tech stack on the WMF side of
things (still using the Collection extension to manage books). We may
need to deprecate some output formats - more on that as we go.
We've got the collection-alt-renderer project set up on Labs (thanks
Andrew) and can hopefully get a plan to our ops team soon as to how
the new setup could work.
If you want to peek - work channel is #mediawiki-pdfhack on FreeNode.
Live notes here:
http://etherpad.wikimedia.org/p/pdfhack
Stuff will be consolidated here:
https://www.mediawiki.org/wiki/PDF_rendering
Some early experiments with different rendering strategies here:
https://github.com/cscott/pdf-research
Some improvements to Collection extension underway:
https://gerrit.wikimedia.org/r/#/q/status:open+project:mediawiki/extensions…
More soon,
Erik
--
Erik Möller
VP of Engineering and Product Development, Wikimedia Foundation
We have merged a couple changes so that jQuery UI is not loaded by code
that doesn't need it. See
https://bugzilla.wikimedia.org/show_bug.cgi?id=55550 for details.
That means gadgets and user scripts that use jQuery UI should explicitly
load the appropriate module.
https://www.mediawiki.org/wiki/Extension:Gadgets has info on adding
gadget dependencies.
User scripts can use mw.loader.using
(https://www.mediawiki.org/wiki/RL/DM#mw.loader.using).
A list of jquery.ui modules is at
https://www.mediawiki.org/wiki/RL/DM#jQuery_UI .
There is one remaining issue that may affect a few wikis. If your wiki
uses jquery.ui buttons in wikitext, the styles will not be loaded unless
something explicitly loads or depends on jquery.ui.
If you notice styles missing from your wiki, you can add:
// Load jquery.ui.button so button styles work in wikitext
mw.loader.using( 'jquery.ui.button' );
to the site Common.js
Don't do this unless it's actually needed, since this does have a
performance impact (although it is not loading all of jquery.ui). If
you do add it, please include the comment so people know why it's being
loaded.
This will roll out to the test wikis, plus MW.org, 2013-10-31, the
remaining non-Wikipedia wikis the 4th, and the Wikipedias, the 7th.
Matt Flaschen
The Engineering Community team at the Wikimedia Foundation has opened a
position for a
Technical Writer - Contract - 3 Months (+)
The description is copied below. Technical writers with a contribution
history at mediawiki.org or other MediaWiki-based sites will be
especially considered. If you are interested, please apply using this
web form:
http://hire.jobvite.com/j/?aj=oMK5Xfwi&s=Community
Wikimedia's Engineering Community team is responsible for developing
clear documentation for MediaWiki. All of our documentation is written
collaboratively in wiki pages, involving all kinds of profiles, from WMF
professional developers to anonymous users. There are some areas of our
documentation that lack content or are outdated, while others have grown
organically and they need pruning and polishing. It is complex to
recruit volunteers for this type of work.
Scope of Work
The technical writer will report to Quim Gil, Technical Contributor
Coordinator. The main area of focus will be system architecture
documentation, though we may identify other related areas during the
course of the contract. Our main technical documentation exists on
https://www.mediawiki.org and https://wikitech.wikimedia.org.
Ideally, this person will start immediately, and should start no later
than December 1, 2013. The technical writer will attend the MediaWiki
Architecture Summit on January 23-24 in San Francisco, where s/he will
be in charge of consolidating the meeting notes taken and polishing the
related documentation.
This is the only on-site task planned. The technical writer can be in
any location during the rest of the contract period as long as it has
good Internet connectivity. We can offer an on-site location in our
offices in San Francisco, although this contract does not include any
relocation or visa support.
Outcome and Performance Standards
This work requires familiarity with wiki syntax and collaborative
workflows. The technical writer will be improving actual pages edit by
edit, with the possibility to find other edits being published by other
contributors as well as related discussions where s/he is supposed to
engage and respond. Quim Gil and other Engineering Community team
members will monitor regularly the work and will assist with the
community dialog if needed.
The technical writer will have a backlog of areas to work on, agreed
with the EC team. There will be weekly reviews or a similar procedure to
review and sign off the tasks completed.
Qualifications:
3 years of professional experience writing technical documentation
Experience volunteering in one or more free software projects as a
documentation writer is highly valuable. (Please provide links to your
user profile and main works.)
Knowledge of PHP / JavaScript (even better when supported with pet
projects, open source contributions or certified training)
--
Quim Gil
Technical Contributor Coordinator @ Wikimedia Foundation
http://www.mediawiki.org/wiki/User:Qgil
Hi.
Our Bugzilla installation at <https://bugs.wikimedia.org/> currently
restricts the capabilities of new users as a knee-jerk response to prior
Bugzilla-related vandalism. There are further details at
<https://bugzilla.wikimedia.org/40497>.
Increasingly new users are making manual requests to be assigned to bugs,
as they cannot edit others' bugs by default. This is problematic and
disruptive to development efforts.
My suggestion is to re-add the "editbugs" user right to new users by
default (revert the old settings adjustment). Otherwise, an acceptable
workaround needs to be found.
MZMcBride