On 12/09/2019 23:42, Antoine Musso wrote:

Hello,

Deployment of MediaWiki 1.34.0-wmf.22 on the Wikimedia cluster is currently blocked. It is blocked on a php7.2 bug that has yet to be tracked and determined.

Immediately after deploying to group1 (which has commons and wikidata), I noticed an infamous error in the logs:

PHP Notice: Undefined index:"

Which sounds similar to known errors with php7.2 (T229433, T231089 among others) but this time it hits LBFactoryMulti.php which is for the database load balancing. So that sounds scary and I thus decided to block the train.

The notice happens at:

$groupLoads[ILoadBalancer::GROUP_GENERIC]

The GROUP_GENERIC constant is an empty string. Brad "anomie" Jorsch quickly found out the code always set that key!

https://phabricator.wikimedia.org/T232613#5485181

So what is left is a edge case issue in php 7.2, maybe in the opcache or something even scarier.

Thanks to Tim Starling, we have a solution to issue a core dump of php when that issue happens. There is already available on the cluster, but it has to be investigated which is not that trivial.

Hello,

Eventually Tim Starling and Giuseppe Lavagetto continued the debugging over the week-end (thank you!). The issue came from the PHP Memcached extension: https://phabricator.wikimedia.org/T232613#5494695

The fix has been deployed on the Wikimedia Cluster at 12:00 UTC and the errors are gone.

I am thus processing promoting 1.34.0-wmf.22 to the rest of the wikis.

Thank you Tim and Giuseppe!