Starting today, immediately after SWAT, thcipriani noticed a new error in fatalmonitor which appeared to coincide with running scap. The curious thing is that scap only affected localization, no code was changed.
At that point I took over for thcipriani and filed an UBN bug: https://phabricator.wikimedia.org/T168681
The error message:
Argument 1 passed to DataValues\UnboundedQuantityValue::newFromArray() must be an instance of array, string given in extensions/Wikidata/vendor/data-values/serialization/src/Deserializers/DataValueDeserializer.php on line 141
After monitoring the errors in kibana for a while I noticed a pattern: the error rate is changing in a very consistent way. The graph of errors makes an almost perfect sine wave, peaking at about 150 per minute and then bottoming out at zero. This pattern has repeated all afternoon. It seems like such a consistent pattern could only be caused by either a) a queued job or b) an api client. Since the bug is reproducible via the API, I suspect that it's b.
As of now it I believe this error to be unrelated to any recent code deployments. The timing might have just been coincidence. If that is the case then it should not block the train any further, however, Greg made the call to postpone further deployments until tomorrow.
If you have any input on this error, please comment on the phabricator task. You can also ping me on IRC, I will be around for much of the evening.
Thanks, that is all I've got for now. - Mukunda
Correction: the train will resume on Monday since there are no deployments on Fridays.
On Thu, Jun 22, 2017 at 6:49 PM, Mukunda Modell mmodell@wikimedia.org wrote:
Starting today, immediately after SWAT, thcipriani noticed a new error in fatalmonitor which appeared to coincide with running scap. The curious thing is that scap only affected localization, no code was changed.
At that point I took over for thcipriani and filed an UBN bug: https://phabricator.wikimedia.org/T168681
The error message:
Argument 1 passed to DataValues\UnboundedQuantityValue::newFromArray() must be an instance of array, string given in extensions/Wikidata/vendor/ data-values/serialization/src/Deserializers/DataValueDeserializer.php on line 141
After monitoring the errors in kibana for a while I noticed a pattern: the error rate is changing in a very consistent way. The graph of errors makes an almost perfect sine wave, peaking at about 150 per minute and then bottoming out at zero. This pattern has repeated all afternoon. It seems like such a consistent pattern could only be caused by either a) a queued job or b) an api client. Since the bug is reproducible via the API, I suspect that it's b.
As of now it I believe this error to be unrelated to any recent code deployments. The timing might have just been coincidence. If that is the case then it should not block the train any further, however, Greg made the call to postpone further deployments until tomorrow.
If you have any input on this error, please comment on the phabricator task. You can also ping me on IRC, I will be around for much of the evening.
Thanks, that is all I've got for now.
- Mukunda
I am looking into the issue...
The errors appear to have stopped. We didn't deploy new wikidata code this week so don't think it's related to deployment
My guess is it's a bot passing bad data to the api, and we are missing some check of the user input in the code thus causing this to be a fatal error. (We can probably improve handling for this in the code)
I think it's ok to proceed with deployment on monday.
Cheers, Katie
On Jun 22, 2017 20:28, "Mukunda Modell" mmodell@wikimedia.org wrote:
Correction: the train will resume on Monday since there are no deployments on Fridays.
On Thu, Jun 22, 2017 at 6:49 PM, Mukunda Modell mmodell@wikimedia.org wrote:
Starting today, immediately after SWAT, thcipriani noticed a new error in fatalmonitor which appeared to coincide with running scap. The curious thing is that scap only affected localization, no code was changed.
At that point I took over for thcipriani and filed an UBN bug: https://phabricator.wikimedia.org/T168681
The error message:
Argument 1 passed to DataValues\UnboundedQuantityValue::newFromArray() must be an instance of array, string given in extensions/Wikidata/vendor/dat a-values/serialization/src/Deserializers/DataValueDeserializer.php on line 141
After monitoring the errors in kibana for a while I noticed a pattern: the error rate is changing in a very consistent way. The graph of errors makes an almost perfect sine wave, peaking at about 150 per minute and then bottoming out at zero. This pattern has repeated all afternoon. It seems like such a consistent pattern could only be caused by either a) a queued job or b) an api client. Since the bug is reproducible via the API, I suspect that it's b.
As of now it I believe this error to be unrelated to any recent code deployments. The timing might have just been coincidence. If that is the case then it should not block the train any further, however, Greg made the call to postpone further deployments until tomorrow.
If you have any input on this error, please comment on the phabricator task. You can also ping me on IRC, I will be around for much of the evening.
Thanks, that is all I've got for now.
- Mukunda
Ops mailing list Ops@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/ops
wikitech-l@lists.wikimedia.org