On Fri, Nov 16, 2012 at 12:34 PM, Andre Klapper <aklapper(a)wikimedia.org> wrote:
This was fixed last night. It has nothing to do with bits, it was
caused by memcached corruption.
* Maybe also: Edit conflict sometimes does not show
the current
text:
https://bugzilla.wikimedia.org/show_bug.cgi?id=42163
I think it's very very very unlikely that this is because of a
front-end issue, it's almost certainly a backend issue.
Speculation:
Looking at the monthly stats on
http://ganglia.wikimedia.org/latest/graph_all_periods.php?c=Bits%20applicat…
(click on "Inspect") there is a drop of "Out" on November 5/6th.
From
http://wikitech.wikimedia.org/view/Server_admin_log#November_5 :
21:54 aaron synchronized wmf-config/mc.php 'Switched all wikis to
memcached-multiwrite.'
21:54 logmsgbot_: aaron synchronized wmf-config/InitialiseSettings.php
'Disabled spammy (with pecl memcached) memcached.log, the useful one is
memcached-serious.'
The drop is strange, but could be explained by the memcached changes
and the logging changes, if similar drops were observed in the other
Apache clusters.
Roan