On 14/11/2018 15:29, Antoine Musso wrote:
I am holding deployment of 1.33.0-wmf.4 to the Wikimedia infrastructure. Some code path attempts to retrieve a memcached entry using an empty string as a key. It is most probably going to cause major issues if we keep pushing the train.
The task is: https://phabricator.wikimedia.org/T209429
memcached error: A BAD KEY WAS PROVIDED/CHARACTERS OUT OF RANGE
It originates from MemcachedPeclBagOStuff::checkResult() which logs an error albeit without any context. Probably adding an exception context would give us a stacktrace and thus find the faulting code.
Hello,
I started a patch to add a stacktrace to the log context, but eventually had a personal emergency. Luckily Brad Jorsch crafted a patch that we have deployed, he found the root cause and I have deployed the fix.
The task is resolved. I have resumed with the train and 1.33.0-wmf.4 is now deployed on group 1 as expected.
If there is anything looking suspicious on group0 or group1 wikis please fill sub tasks to the blocker task: https://phabricator.wikimedia.org/T206658