Hi!

Striving for operational excellence, here's a summary of recent issues in production that can currently affect end-users.

## Statistics

* 124 fatals. (channel=fatal) * 3,986 exceptions. (channel=exception) * 34,559 PHP errors. (channel=error) * 42,222 errors in total. (channel=*, level=ERROR)

From logstash.wikimedia.org, last 7 days, having type=mediawiki.

## Higher impact

These caused users (of web or api) to see errors. From logstash.wikimedia.org, ordered by frequency:

* [Babel extension] https://phabricator.wikimedia.org/T199941 - MWException "Language::isValidBuiltInCode must be passed a string".

* [Flow extension] https://phabricator.wikimedia.org/T70526 - InvalidDataException "Flow workflow is for different page".

* [Page deletion] https://phabricator.wikimedia.org/T195692 - Undelete for certain pages aborted by IncompleteRevisionException.

* [Special pages] https://phabricator.wikimedia.org/T199044 - Special:ActiveUsers aborted by DBQueryTimeoutError.

* [MobileFrontend] https://phabricator.wikimedia.org/T199066 - Special:MobileContributions shows "Special:Badtitle" (Revision::ensureTitle error).

* [AbuseFilter extension] https://phabricator.wikimedia.org/T187153 - Special:Abuselog throws BadMethodCallException on details/examine.

## Noise

These are caused by code not behaving unexpectedly, but with limited impact due to graceful recovery by PHP or other code. These harm our ability to detect and prevent higher impact issues (through in Scap). They may also be symptoms of deeper issues.

* [CirrusSearch extension] https://phabricator.wikimedia.org/T180807 - Undefined index: _score and _id in CompSuggestQueryBuilder. * [Collection extension] https://phabricator.wikimedia.org/T189636 - Undefined 'subtitle' and 'title' indexes in CollectionPageTemplate. * [WMF-JobQueue] https://phabricator.wikimedia.org/T199594 - Exception "Job queue is read-only". * [CentralAuth extension] https://phabricator.wikimedia.org/T170971 - iconv Detected an illegal character. * [Cite extension] https://phabricator.wikimedia.org/T182929 - Undefined index: "" in Cite.php.

## Thank you

Thanks to the following people who recently helped investigate and/or resolve similar issues.

* Jdforrester (James Forrester) * Marostegui (Manuel Aróstegui) * jcrespo (Jaime Crespo) * SBisson (Stephane Bisson) * Tpt (Tpt) * Anomie (Brad Jorsch) * Joe (Giuseppe Lavagetto) * Legoktm (Kunal Mehta) * Aaron (Aaron Schulz) * mobrovac (Marko Obrovac)

And everyone else I forgot to mention, thank you!

-- Timo