Forwarded as this is of potentially wider interest. This may be a
breaking change for some older bots that haven't been maintained.
---------- Forwarded message ----------
From: Sam Reed <reedy(a)wikimedia.org>
Date: Wed, Sep 5, 2012 at 3:06 PM
Subject: [Wikitech-l] HTML5, it's a coming (again)!
To: wikitech-l(a)lists.wikimedia.org
It's been a long time coming (for the nth time..), but we're scheduling a
deployment of HTML5 across the Wikimedia cluster [1]. This is set for Monday
17th September at 18:00-20:00 UTC [2].
The intention is to set $wgHtml5 [3] to true everywhere. It's been running
on MediaWiki.org and our 2 test wikis for quite a while, and other sites
like translatewiki.net with no issues.
The intention is to leave it enabled unless it causes major problems. If
you're running an application that screen scrapes, shame on you; you've had
enough notice to get it fixed! ;)
Now is the time to fix up your scripts and programs (where necessary), tell
your friends!
Sam
[1] https://bugzilla.wikimedia.org/show_bug.cgi?id=27478
[2] http://wikitech.wikimedia.org/view/Software_deployments#Week_of_Sept_17
[3] https://www.mediawiki.org/wiki/Manual:$wgHtml5
_______________________________________________
Wikitech-l mailing list
Wikitech-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
---------- Forwarded message ----------
From: Tim Starling <tstarling(a)wikimedia.org>
Date: Fri, Sep 21, 2012 at 4:07 AM
Subject: [Wikitech-l] #switch limits
To: wikitech-l(a)lists.wikimedia.org
Over the last week, we have noticed very heavy apache memory usage on
the main Wikimedia cluster. In some cases, high memory usage resulted
in heavy swapping and site-wide performance issues.
After some analysis, we've identified the main cause of this high
memory usage to be geographical data ("données") templates on the
French Wikipedia, and to a lesser extent, the same data templates
copied to other wikis for use on articles about places in Europe.
Here is an example of a problematic template:
<https://fr.wikipedia.org/w/index.php?title=Mod%C3%A8le:Donn%C3%A9es_PyrF1-2…>
That template alone uses 47MB for 37000 #switch cases, and one article
used about 15 similarly sized templates.
The simplest solution to this problem is for the few Wikipedians
involved to stop doing what they are doing, and to remove the template
invocations which have already been introduced. Antoine Musso has
raised the issue on the French Wikipedia's "Bistro" and some of the
worst cases have already been fixed.
To protect site stability, I've introduced a new preprocessor
complexity limit called the "preprocessor generated node count", which
is incremented by about 6 for each #switch case. When the limit is
exceeded, an exception is thrown, preventing the page from being saved
or viewed.
The limit is currently 4 million (~667,000 #switch cases), and it will
soon be reduced to 1.5 million (~250,000 #switch cases). That's a
compromise which allows most of the existing geographical pages to
keep working, but still allows a memory usage of about 230MB.
At some point, we would like to patch PHP upstream to cause memory for
DOM XML trees to be allocated from the PHP request pool, instead of
with malloc(). But to deploy that, we would need to reduce the limit
to the point where the template DOM cache can easily fit in the PHP
memory limit of 128MB.
In the short term, we will be working with the template editors to
ensure that all articles can be viewed with a limit of 1.5 million.
That's not a very viable solution in the long term, so I'd also like
to introduce save-time warnings and tracking categories for pages
which use more than, say, 50% of the limit, to encourage authors to
fix articles without being directly prompted by WMF staff members.
At some point in the future, you may be able to put this kind of
geographical data in Wikidata. Please, template authors, wait
patiently, don't implement your own version of Wikidata using wikitext
templates.
-- Tim Starling
_______________________________________________
Wikitech-l mailing list
Wikitech-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
All,
This fortnight's update for VisualEditor[0] - I'm skipping the one
for Parsoid based on your feedback, but it is available here[1]:
The VisualEditor was updated as part of the wider MediaWiki 1.20wmf12
branch deployment[2] on Monday 17 September.
The last two weeks included the annual Wikimedia Engineering all-tech
meeting and the Wikimedia Foundation all-staff meeting, so this
iteration was in effect shorter than others. The team have continued
to spend most of the coding time over the two weeks since 1.20wmf11
working on how the code integrates together. This work will provide
cleaner interfaces between components of the VisualEditor, so new
developers can re-use and extend VisualEditor to support new 'node
types' like categories or tables when we work on these later.
Beyond the API work, there have been a few minor changes to the user
interface made this iteration. Firstly, the link inspector now checks
for invalid titles like "Foo{}bar" (bug 33094 [3]), and long titles in
the suggestions box are replaced with ellipses in the middle rather
than the end so they can be distinguished (bug 39591[4]). The save
dialog's length down-counter has been labelled to indicate it counts
the number of bytes rather than characters, as that is what the
database holds (bug 40035[5]); longer-term, we may wish to find a
better way to show this to users. Finally, we fixed a bug where the
table of contents was restored wrongly if a user edited and then
cancelled without saving (bug 39753[6]).
A complete list of individual code commits is available in the
1.20/wmf12 changelog[7], and all Bugzilla bugs closed in this period
on Bugzilla's list[8].
[0] - https://www.mediawiki.org/wiki/VisualEditor/status#2012-09-17_.28MW_1.20wmf…
[1] - https://www.mediawiki.org/wiki/Parsoid/status#2012-09-17
[2] - https://www.mediawiki.org/wiki/MediaWiki_1.20/wmf12
[3] - https://bugzilla.wikimedia.org/show_bug.cgi?id=33094
[4] - https://bugzilla.wikimedia.org/show_bug.cgi?id=39591
[5] - https://bugzilla.wikimedia.org/show_bug.cgi?id=40035
[6] - https://bugzilla.wikimedia.org/show_bug.cgi?id=39753
[7] - https://www.mediawiki.org/wiki/MediaWiki_1.20/wmf12#VisualEditor
[8] - https://bugzilla.wikimedia.org/buglist.cgi?query_format=advanced&bug_status…
Yours,
--
James D. Forrester
Product Manager for Visual Editor and Flagged Revisions
Wikimedia Foundation, Inc.
jforrester(a)wikimedia.org | @jdforrester | +1 415-839-6885 x6844
Hi everyone,
We've reverted MediaWiki from 1.20wmf11 to 1.20wmf10 in order to fix a
problem with watchlists failing:
https://bugzilla.wikimedia.org/show_bug.cgi?id=40103
The root cause of this problem is a compatibility issue with jQuery
1.7.2. We can't yet cleanly upgrade to jQuery 1.8. There are pending
changes that might make this possible.
We may also try reverting all wikis to 1.20wmf10, and then moving
forward with 1.20wmf12 on Monday. Still all TBD.
Rob
Forwarding from wikitech-l.
-------- Original Message --------
Subject: [Wikitech-l] HTML5, it's a coming (again)!
Date: Wed, 5 Sep 2012 23:06:15 +0100
From: Sam Reed <reedy(a)wikimedia.org>
Reply-To: Wikimedia developers <wikitech-l(a)lists.wikimedia.org>
To: <wikitech-l(a)lists.wikimedia.org>
It's been a long time coming (for the nth time..), but we're scheduling a
deployment of HTML5 across the Wikimedia cluster [1]. This is set for Monday
17th September at 18:00-20:00 UTC [2].
The intention is to set $wgHtml5 [3] to true everywhere. It's been running
on MediaWiki.org and our 2 test wikis for quite a while, and other sites
like translatewiki.net with no issues.
The intention is to leave it enabled unless it causes major problems. If
you're running an application that screen scrapes, shame on you; you've had
enough notice to get it fixed! ;)
Now is the time to fix up your scripts and programs (where necessary), tell
your friends!
Sam
[1] https://bugzilla.wikimedia.org/show_bug.cgi?id=27478
[2] http://wikitech.wikimedia.org/view/Software_deployments#Week_of_Sept_17
[3] https://www.mediawiki.org/wiki/Manual:$wgHtml5
_______________________________________________
Wikitech-l mailing list
Wikitech-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l
---------- Forwarded message ----------
From: Peter Youngmeister <py(a)wikimedia.org>
Date: Thu, Aug 30, 2012 at 3:15 PM
Subject: [Wikitech-l] upgrades to search cluster
To: Wikimedia developers <wikitech-l(a)lists.wikimedia.org>
Hi All,
I have moved all search traffic to hosts now running ubuntu 12.04
(precise). This should be a no-op, as the search software itself has not
changed, and all automated tests looked good, but I wanted to make you all
aware as I neither the time nor languages skills to manually test every
language.
If you notice any irregularities, please let me know!
--peter
Hello,
I am the Wikimedia Foundation's Product Manager for the
VisualEditor[0] project (and others).
I am conscious that the VisualEditor will be a major change that is
coming down the tracks, and that I need to do more to make sure that
all our communities (through you) know what is happening (and make
sure they have as much opportunity to tell us when we're wrong, as
well as help guide the priorities for development and improvement).
Right now (and for the past few months), both the core of MediaWiki
and all the extensions on Wikimedia's cluster are updated on the live
site every fortnight in a new branch. Though VisualEditor is currently
only deployed to mediawiki.org[1], in time (when it's ready) we'll be
making it available on all other Wikimedia wikis. I have started to
write up the technical "changelog" as a status update, giving some
(hopefully ;-)) more useful, human-readable "release notes".
I thought it might be useful to copy them below, both for the
VisualEditor tool and the new Parsoid service that is separate but
vital for VisualEditor to work.
However, I am worried that posting these here every two weeks might be
a bit spammy, so I'd love feedback not just on the content (for which
the best venue is the central feedback page[2]) but also as to whether
you would value me doing this every fortnight (or perhaps less
regularly?), or if there are better, or additional fora that might be
suited to getting this information for users on the Wikimedia
projects.
VisualEditor[3]
The VisualEditor was updated as part of the wider on Monday 20 August
MediaWiki 1.20wmf10 branch deployment[4].
The most visible new item in the two weeks since 1.20wmf9 is the
much-improved link inspector. This now guides users to create a link
to a suggested existing article, a redlink or an external link, and
replaces the previous basic functionality that did not suggest links
or inform you if the target of your link existed. We have also
improved the save dialogue, streamlining the interaction based on
feedback from the design team.
There have also been a number of bug fixes, such as preserving spaces
before and after the content in headings and other forms (so that
"<nowiki>== Foo ==</nowiki>" doesn't have spacing either side of it in
the editor display, but doesn't strip them either -- bug 37935 [5]),
using browsers' native deletion mechanisms which helps with support
for short-cuts and internationalisation (bug 38461 [6]), and handling
of "alien nodes" (pieces of content that the editor does not know how
to edit yet) so that they do not break the rest of the editor when
included. However, most of the changes have been improvements to the
code architecture to allow it to be re-used and extended to support
new 'node types' like categories or tables when we work on these
later.
A complete list of individual code commits is available in the
1.20/wmf10 changelog[7], and all Bugzilla bugs closed in this period
on Bugzilla's list[8].
Parsoid[9]
The Parsoid team[A] worked on the final tasks in the JavaScript
prototype, in preparation for the C++ port. The port will allow an
efficient integration with PHP and Lua, improve performance and allow
the parallelization of the parser in the longer term in preparation
for production use.
An important milestone we reached is the implementation and
verification of the template DOM range encapsulation algorithm, which
now identifies all template-affected parts of the DOM for
round-tripping and protection in the VisualEditor[0]. We are currently
implementing template round-tripping based on this. Other new features
include oldid support so that previous versions of pages can be
edited, rather than just the current one, and more complete error
reporting in the web service. Wikitext escaping in the serializer is
much improved, and now also handles interactions across multiple DOM
nodes. An ongoing task has been improving test coverage to enable us
to refactor code with more confidence and also help test the
correctness of the C++ port.
Most details of the C++ port were researched. A basic build system
including the selected libraries was set up, and design work on the
basic data structures has started, ahead of full porting which we
expect to start next iteration.
The full list of Parsoid bugs closed in the last two weeks is
available in Bugzilla[B].
Hope this is helpful! As I said, feedback gratefully received.
[0] - https://mediawiki.org/wiki/VisualEditor
[1] - https://mediawiki.org/wiki/VisualEditor:Welcome
[2] - https://mediawiki.org/wiki/VisualEditor/Feedback
[3] - https://mediawiki.org/wiki/VisualEditor/status#2012-08-20_.28MW_1.20wmf10.29
[4] - https://mediawiki.org/wiki/MediaWiki_1.20/wmf10
[5] - https://bugzilla.wikimedia.org/show_bug.cgi?id=37935
[6] - https://bugzilla.wikimedia.org/show_bug.cgi?id=38461
[7] - https://mediawiki.org/wiki/MediaWiki_1.20/wmf10#VisualEditor
[8] - https://bugzilla.wikimedia.org/buglist.cgi?query_format=advanced&bug_status…
[9] - https://mediawiki.org/wiki/Parsoid/status#2012-08-20_.28MW_1.20wmf10.29
[A] - https://mediawiki.org/wiki/Parsoid
[B] - https://bugzilla.wikimedia.org/buglist.cgi?list_id=140016&chfieldto=2012-08…
James.
--
James D. Forrester
Product Manager for Visual Editor and Flagged Revisions
Wikimedia Foundation, Inc.
jforrester(a)wikimedia.org | @jdforrester | +1 415-839-6885 x6844