Hi all,
I’m excited to announce that the WMF Search team has just shipped the new
Streaming Updater for Wikidata Query Service (WDQS), with the final
server’s data transfer completing earlier today (19 Oct) — a little ahead
of (the revised) schedule!
You may know WDQS as a way of querying information from Wikidata. In order
to this, WDQS ingests data, particularly edit updates, from Wikidata to
construct and maintain a massive knowledge graph. Wikidata has grown over
the years in size and usage, and WDQS had started becoming a bottleneck,
which created update lag.
The new Streaming Updater allows WDQS to go from an average of 10
edits/second to an average of 88 edits/second – almost a 900% increase in
our ability to make sure that we can provide a more up to date knowledge
graph, as well as a more stable and reliable update process.
For more information about some of the technical changes that could break
existing workflows and usage, see this earlier announcement
<https://www.wikidata.org/wiki/Wikidata:Project_chat/Archive/2021/03#New_WDQS_Streaming_Updater_now_available_on_pre-production_test_server_for_feedback>
.
Big thanks and congratulations to the Search team, WMDE, and everyone else
involved for making this happen!
“It’s absolutely insane how fast the new streaming updater catches up on lag
very exciting” – Ryan Kemper
Best,
Mike
—
*Mike Pham* (he/him)
Sr Product Manager, Search
Wikimedia Foundation <https://wikimediafoundation.org/>