Hi!
Striving for operational excellence, here's a summary of recent issues in
production that can currently affect end-users.
## Statistics
* 124 fatals. (channel=fatal)
* 3,986 exceptions. (channel=exception)
* 34,559 PHP errors. (channel=error)
* 42,222 errors in total. (channel=*, level=ERROR)
From
logstash.wikimedia.org, last 7 days, having
type=mediawiki.
## Higher impact
These caused users (of web or api) to see errors. From
logstash.wikimedia.org, ordered by frequency:
* [Babel extension]
https://phabricator.wikimedia.org/T199941 - MWException
"Language::isValidBuiltInCode must be passed a string".
* [Flow extension]
https://phabricator.wikimedia.org/T70526 -
InvalidDataException "Flow workflow is for different page".
* [Page deletion]
https://phabricator.wikimedia.org/T195692 - Undelete for
certain pages aborted by IncompleteRevisionException.
* [Special pages]
https://phabricator.wikimedia.org/T199044 -
Special:ActiveUsers aborted by DBQueryTimeoutError.
* [MobileFrontend]
https://phabricator.wikimedia.org/T199066 -
Special:MobileContributions shows "Special:Badtitle" (Revision::ensureTitle
error).
* [AbuseFilter extension]
https://phabricator.wikimedia.org/T187153 -
Special:Abuselog throws BadMethodCallException on details/examine.
## Noise
These are caused by code not behaving unexpectedly, but with limited impact
due to graceful recovery by PHP or other code. These harm our ability to
detect and prevent higher impact issues (through in Scap).
They may also be symptoms of deeper issues.
* [CirrusSearch extension]
https://phabricator.wikimedia.org/T180807 -
Undefined index: _score and _id in CompSuggestQueryBuilder.
* [Collection extension]
https://phabricator.wikimedia.org/T189636 -
Undefined 'subtitle' and 'title' indexes in CollectionPageTemplate.
* [WMF-JobQueue]
https://phabricator.wikimedia.org/T199594 - Exception "Job
queue is read-only".
* [CentralAuth extension]
https://phabricator.wikimedia.org/T170971 - iconv
Detected an illegal character.
* [Cite extension]
https://phabricator.wikimedia.org/T182929 - Undefined
index: "" in Cite.php.
## Thank you
Thanks to the following people who recently helped investigate and/or
resolve similar issues.
* Jdforrester (James Forrester)
* Marostegui (Manuel Aróstegui)
* jcrespo (Jaime Crespo)
* SBisson (Stephane Bisson)
* Tpt (Tpt)
* Anomie (Brad Jorsch)
* Joe (Giuseppe Lavagetto)
* Legoktm (Kunal Mehta)
* Aaron (Aaron Schulz)
* mobrovac (Marko Obrovac)
And everyone else I forgot to mention, thank you!
-- Timo