📘 Read on Phabricator at
How’d we do in our strive for operational excellence last month? Read on to
## 📊 Month in numbers
* 8 documented incidents. 
* 27 new Wikimedia-prod-error reports. 
* 26 Wikimedia-prod-error reports closed. 
* 199 currently open Wikimedia-prod-error reports in total. 
With a median of 4–5 documented incidents per month (over the last three
years), there were a fairly large number of them this past month.
To read more about these incidents and pending actionables; check <
Explore Wikimedia incident stats at <https://codepen.io/Krinkle/full/wbYMZK>
## 📖 Unset vs array splice
Our error monitor (Logstash) received numerous reports about an “Undefined
offset” error from the OATHAuth extension. This extension powers the
Two-factor auth (2FA) login interface on Wikipedia (<
ItSpiderman and Reedy investigated the problem. The error message:
PHP Notice: Undefined offset: 8
This error means that the code was accessing item number 8 from a list (an
array), but the item does not exist. Normally, when a “2FA scratch token”
is used, we remove it from a list, and save the remaining list for next
The code used the `count()` function to compute the length of the list, and
used a for-loop to iterate through the list. When the code found the user’s
token, it used the `unset( $list[$num] )` operation to remove token $num
from the list, and then save $list for next time.
The problem with removing a list item in this way is that it leaves a
“gap”. Imagine a list with 4 items, like [ 1: …, 2: …, 3: … , 4: … ]. If we
unset item 2, then the remaining list will be [ 1: …, 3: …, 4: … ]. The
next time we check this list, the length of the list is now 3 (so far so
good!), but the for-loop will access the items as 1-2-3. The code would not
know that 3 comes after 1, causing an error because item 2 does not exist.
And, the code would not even look at item 4!
When a user used their first ever scratch token, everything worked fine.
But from their second token onwards, the tokens could be rejected as
“wrong” because the code was not able to find them.
To avoid this bug, we changed the code to use `array_splice( $list, $num, 1
)` instead of `unset( $list[$num] )`. The important thing about
array_splice is that it renumbers the items in the list, leaving no gaps.
## 📉 Outstanding reports
Take a look at the workboard and look for tasks that might need your help.
The workboard lists error reports, grouped by the month in which they were
Breakdown of recent months:
* March: 3 of 10 reports left (unchanged). ⚠️
* April: 4 of 14 left (unchanged).
* May: (All clear!)
* June: 4 of 11 left (unchanged).
* July: 8 of 18 left (unchanged).
* August: Two reports closed! 2 of 14 reports left.
* September: One report closed, 7 of 12 left.
* October: Two reports closed, 6 of 12 left.
* November: 5 of 5 left (unchanged).
* December: 6 of 9 left (unchanged).
* January: One report closed, 6 of 7 reports left.
* February: 7 new reports survived the month of February.
Last month’s total over recent months was 57 open reports. Of those, 6 got
closed, but with 7 new reports from February still open, the total is now
up at 58 open reports.
## 🎉 Thanks!
Thank you to everyone who helped by reporting, investigating, or resolving
problems in Wikimedia production.
Together we’re getting there!
Until next time,
– Timo Tijhof
 Incidents. –
 Tasks created. –
 Tasks closed. –
 Open tasks. –