Hi!
This morning we have switched the polling mechanism for Wikidata Query Service from using Recent Changes API to using Kafka events (https://wikitech.wikimedia.org/wiki/EventBus). This should improve reliability of the updates and reduce load on Wikidata servers. Given that we are putting more data into Wikidata and more servers for SPARQL services, having a scalable update mechanism is important, and that's what we are doing with Kafka updater.
If everything works well, you should not notice any change at all.
So far, we have inspected it in action for a few days and it seems to be working fine. However, since the functionality is new, it is possible that some bugs exist. The bugs might manifest themselves in certain new edits not being reflected in Wikidata Query Service data. If you notice this happening, please tell me (preferably via Phabricator ticket specifying the details of the problem, but if it doesn't work for you you can email me or ping me on IRC or on Wiki: Smalyshev_(WMF)).
Thanks,
Thanks Stas. How does this affect non-WMF clones of Wikidata QS?
On Tue, Mar 6, 2018 at 2:51 PM, Stas Malyshev smalyshev@wikimedia.org wrote:
Hi!
This morning we have switched the polling mechanism for Wikidata Query Service from using Recent Changes API to using Kafka events (https://wikitech.wikimedia.org/wiki/EventBus). This should improve reliability of the updates and reduce load on Wikidata servers. Given that we are putting more data into Wikidata and more servers for SPARQL services, having a scalable update mechanism is important, and that's what we are doing with Kafka updater.
If everything works well, you should not notice any change at all.
So far, we have inspected it in action for a few days and it seems to be working fine. However, since the functionality is new, it is possible that some bugs exist. The bugs might manifest themselves in certain new edits not being reflected in Wikidata Query Service data. If you notice this happening, please tell me (preferably via Phabricator ticket specifying the details of the problem, but if it doesn't work for you you can email me or ping me on IRC or on Wiki: Smalyshev_(WMF)).
Thanks,
Stas Malyshev smalyshev@wikimedia.org
Wikidata mailing list Wikidata@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata
On Wed, Mar 7, 2018 at 6:47 PM, Yuri Astrakhan yuriastrakhan@gmail.com wrote:
Thanks Stas. How does this affect non-WMF clones of Wikidata QS?
It does not affect non-WMF clones. The previous recent change poller is still part of the code. And we still use it on our test servers. We have no plans at this point to remove it. So you should be all good!
On Tue, Mar 6, 2018 at 2:51 PM, Stas Malyshev smalyshev@wikimedia.org wrote:
Hi!
This morning we have switched the polling mechanism for Wikidata Query Service from using Recent Changes API to using Kafka events (https://wikitech.wikimedia.org/wiki/EventBus). This should improve reliability of the updates and reduce load on Wikidata servers. Given that we are putting more data into Wikidata and more servers for SPARQL services, having a scalable update mechanism is important, and that's what we are doing with Kafka updater.
If everything works well, you should not notice any change at all.
So far, we have inspected it in action for a few days and it seems to be working fine. However, since the functionality is new, it is possible that some bugs exist. The bugs might manifest themselves in certain new edits not being reflected in Wikidata Query Service data. If you notice this happening, please tell me (preferably via Phabricator ticket specifying the details of the problem, but if it doesn't work for you you can email me or ping me on IRC or on Wiki: Smalyshev_(WMF)).
Thanks,
Stas Malyshev smalyshev@wikimedia.org
Wikidata mailing list Wikidata@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata
Wikidata mailing list Wikidata@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata