Starting today, immediately after SWAT, thcipriani noticed a new error in fatalmonitor which appeared to coincide with running scap. The curious thing is that scap only affected localization, no code was changed.
At that point I took over for thcipriani and filed an UBN bug: https://phabricator.wikimedia.org/T168681
The error message:
Argument 1 passed to DataValues\UnboundedQuantityValue::newFromArray() must be an instance of array, string given in extensions/Wikidata/vendor/data-values/serialization/src/Deserializers/DataValueDeserializer.php on line 141
After monitoring the errors in kibana for a while I noticed a pattern: the error rate is changing in a very consistent way. The graph of errors makes an almost perfect sine wave, peaking at about 150 per minute and then bottoming out at zero. This pattern has repeated all afternoon. It seems like such a consistent pattern could only be caused by either a) a queued job or b) an api client. Since the bug is reproducible via the API, I suspect that it's b.
As of now it I believe this error to be unrelated to any recent code deployments. The timing might have just been coincidence. If that is the case then it should not block the train any further, however, Greg made the call to postpone further deployments until tomorrow.
If you have any input on this error, please comment on the phabricator task. You can also ping me on IRC, I will be around for much of the evening.
Thanks, that is all I've got for now. - Mukunda