On 17/03/2020 20:14, Brennen Bearnes wrote:

The 1.35.0-wmf.23 version of MediaWiki is still blocked[0].

Neither this version nor the subsequent 1.35.0-wmf.24 can proceed until this issue is resolved:

• T247562: Warning: Memcached::setMulti(): failed to set key

global:segment:...

• https://phabricator.wikimedia.org/T247562

Thanks to the many folks who have contributed debugging on T247562 so far.  If anyone else has any insight, further input would be certainly be appreciated.

-- Your erratic train enabler

[0]. https://phabricator.wikimedia.org/T233871

Hello,

Eventually we have found the root cause of the issue. The local cache was being invalidated when different wikis tried to use it which caused MediaWiki processes to fetch a large key from the same memcached server and overloading it.

The hotfix is deployed. I will push 1.35.0-wmf.23 to all wikis this afternoon at or slightly after 13:00 UTC.