Hey folks,

This is the 16th weekly update from revision scoring team that we have sent to this mailing list.

New developments:

• We created dashboards for the ORES service in the Beta cluster[1] and created panes for tracking failed jobs[2].

• We extended the documentation for the ORES review tool[3,4]

Maintenance:

• We did some work to make the Beta cluster look more like production so that we can do better testing before the next deployment

• We set up a password on the Beta redis server[5]

• We configured the Beta ORES extension to actually use the Beta ORES service[6]

• We also prepared a set of puppet changes for the deployment of a refactored version of ORES to production[7]

Issues in WMFLabs

• We investigated a series of timeout errors that were appearing in the logs[8]

• We investigated a periodic redis-related error that shower up when scoring edits[9]

• We fixed our "05" web node that was periodically running out of memory[10]

Estimating future resource needs

• In preparation for buying new hardware, we measured our past memory usage and extrapolated forward two years to estimate what hardware requirements we'll have[11]

1. https://phabricator.wikimedia.org/T142294 - Dashboard or pane for ORES service in beta

2. https://phabricator.wikimedia.org/T142119 - Dashboard or pane for ORES failed jobs on beta

3. https://phabricator.wikimedia.org/T140150 - Make user-centered documentation for review tool

4. https://www.mediawiki.org/wiki/ORES_review_tool

5. https://phabricator.wikimedia.org/T141823 - Set up password on ORES Beta redis server

6. https://phabricator.wikimedia.org/T141825 - Config beta ORES extension to use the beta ORES service

7. https://phabricator.wikimedia.org/T141575 - Puppet config changes for ORES refactor

8. https://phabricator.wikimedia.org/T141368 - [Investigate] ORES time out errors in logs

9. https://phabricator.wikimedia.org/T141946 - [Investigate] Periodic redis related errors in wmflabs

10. https://phabricator.wikimedia.org/T141523 - [Investigate] web-05 downtime

11. https://phabricator.wikimedia.org/T142046 - Extrapolate memory usage per worker forward 2 years

Sincerely,

Aaron from the Revision Scoring team