Hi all!
Logspam makes it difficult to glance at error logs after a deployment and
reason about a deployment's impact [0]. Release Engineering is making a
conscious effort (in Scrum of Scrums, in Phabricator, and on mailing lists) to
connect logspam tasks with folks that can make an impact on these tasks (and,
consequentially, make an impact on deployments). Sometimes, as now, our effort
takes the form of a broad appeal to help investigate high impact logspam tasks.
This week in logspam:
* wfShellExec errors end up in HHVM log [1]
These errors are the noisiest of the noisy recently. They take various forms,
but many relate to PDF handling and start with "SyntaxError"
* Warning: Cannot modify header information - headers already sent [2]
The latest iteration of this error seems to have started with the
release of 1.29.0-wmf.10
* Warning: timed out after 0.2 seconds when connecting to rdb1001.eqiad.wmnet [110]:
Connection timed out [3]
saw a bit of movement last week, but there are some unanswered questions and
the message is still going strong.
* Couple of session related ones
** Session "{session}": Unverified user provided and no metadata to auth it [4]
** Session "{session}": Metadata merge failed: {exception} [5]
* Throttler: throttler data not found for {user} [6]
If you or anyone you know has information that can lead to the cessation of
these errors, please add that person or a comment on the tasks listed here.
Thus concludes another exciting week in logspam.
<3
-- RelEngers
[0] <https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering_Team/Logspam>
[1] <https://phabricator.wikimedia.org/T157646>
[2] <https://phabricator.wikimedia.org/T157392>
[3] <https://phabricator.wikimedia.org/T125735>
[4] <https://phabricator.wikimedia.org/T158367>
[5] <https://phabricator.wikimedia.org/T158365>
[6] <https://phabricator.wikimedia.org/T158364>