On 12/09/2019 23:42, Antoine Musso wrote:
Hello,
Deployment of MediaWiki 1.34.0-wmf.22 on the Wikimedia cluster is
currently blocked. It is blocked on a php7.2 bug that has yet to be
tracked and determined.
Immediately after deploying to group1 (which has commons and wikidata),
I noticed an infamous error in the logs:
PHP Notice: Undefined index:"
Which sounds similar to known errors with php7.2 (T229433, T231089 among
others) but this time it hits LBFactoryMulti.php which is for the
database load balancing. So that sounds scary and I thus decided to
block the train.
The notice happens at:
$groupLoads[ILoadBalancer::GROUP_GENERIC]
The GROUP_GENERIC constant is an empty string. Brad "anomie" Jorsch
quickly found out the code always set that key!
https://phabricator.wikimedia.org/T232613#5485181
So what is left is a edge case issue in php 7.2, maybe in the opcache or
something even scarier.
Thanks to Tim Starling, we have a solution to issue a core dump of php
when that issue happens. There is already available on the cluster, but
it has to be investigated which is not that trivial.
Hello,
Eventually Tim Starling and Giuseppe Lavagetto continued the debugging
over the week-end (thank you!). The issue came from the PHP Memcached
extension:
https://phabricator.wikimedia.org/T232613#5494695
The fix has been deployed on the Wikimedia Cluster at 12:00 UTC and the
errors are gone.
I am thus processing promoting 1.34.0-wmf.22 to the rest of the wikis.
Thank you Tim and Giuseppe!
--
Antoine "hashar" Musso