Re: [Wikidata] WDQS updater switched to Kafka

Hi! This morning we have switched the polling mechanism for Wikidata Query Service from using Recent Changes API to using Kafka events (https://wikitech.wikimedia.org/wiki/EventBus). This should improve reliability of the updates and reduce load on Wikidata servers. Given that we are putting more data into Wikidata and more servers for SPARQL services, having a scalable update mechanism is important, and that's what we are doing with Kafka updater. If everything works well, you should not notice any change at all. So far, we have inspected it in action for a few days and it seems to be working fine. However, since the functionality is new, it is possible that some bugs exist. The bugs might manifest themselves in certain new edits not being reflected in Wikidata Query Service data. If you notice this happening, please tell me (preferably via Phabricator ticket specifying the details of the problem, but if it doesn't work for you you can email me or ping me on IRC or on Wiki: Smalyshev_(WMF)). Thanks, -- Stas Malyshev smalyshev(a)wikimedia.org _______________________________________________ Wikidata mailing list Wikidata(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata