Hi all,

I’m excited to announce that the WMF Search team has just shipped the new Streaming Updater for Wikidata Query Service (WDQS), with the final server’s data transfer completing earlier today (19 Oct) — a little ahead of (the revised) schedule!

You may know WDQS as a way of querying information from Wikidata. In order to this, WDQS ingests data, particularly edit updates, from Wikidata to construct and maintain a massive knowledge graph. Wikidata has grown over the years in size and usage, and WDQS had started becoming a bottleneck, which created update lag.

The new Streaming Updater allows WDQS to go from an average of 10 edits/second to an average of 88 edits/second – almost a 900% increase in our ability to make sure that we can provide a more up to date knowledge graph, as well as a more stable and reliable update process.

For more information about some of the technical changes that could break existing workflows and usage, see this earlier announcement.

Big thanks and congratulations to the Search team, WMDE, and everyone else involved for making this happen!

“It’s absolutely insane how fast the new streaming updater catches up on lag
very exciting” – Ryan Kemper

Best,
Mike