Wikidata Query Service scaling update Aug 2021 - Wikidata

18 Aug 2021

Wikidata community members,

Thank you for all of your work helping Wikidata grow and improve
over the years. In the spirit of better communication, we would like
to take this opportunity to share some of the current challenges
Wikidata Query Service (WDQS) is facing, and some strategies we have
for dealing with them.

WDQS currently risks failing to provide acceptable service quality
due to the following reasons:

 1.

    Blazegraph scaling

     1.

        Graph size. WDQS uses Blazegraph as our graph backend. While
        Blazegraph can theoretically support 50 billion edges
        <https://blazegraph.com/>, in reality Wikidata is the
        largest graph we know of running on Blazegraph (~13 billion
        triples
        <https://grafana.wikimedia.org/d/000000489/wikidata-query-service?viewPanel=7&orgId=1&refresh=1m>),
        and there is a risk that we will reach a size
        <https://www.w3.org/wiki/LargeTripleStores#Bigdata.28R.29_.2812.7B.29>limit
        of what it can realistically support
        <https://phabricator.wikimedia.org/T213210>. Once Blazegraph
        is maxed out, WDQS can no longer be updated. This will also
        break Wikidata tools that rely on WDQS.

     2.

        Software support. Blazegraph is end of life software, which
        is no longer actively maintained, making it an unsustainable
        backend to continue moving forward with long term.

Blazegraph maxing out in size poses the greatest risk for
catastrophic failure, as it would effectively prevent WDQS from
being updated further, and inevitably fall out of date. Our long
term strategy to address this is to move to a new graph backend that
best meets our WDQS needs and is actively maintained, and begin the
migration off of Blazegraph as soon as a viable alternative is
identified <https://phabricator.wikimedia.org/T206560>.

In the interim period, we are exploring disaster mitigation options
for reducing Wikidata’s graph size in the case that we hit this
upper graph size limit: (i) identify and delete lower priority data
(e.g. labels, descriptions, aliases, non-normalized values, etc);
(ii) separate out certain subgraphs (such as Lexemes and/or
scholarly articles). This would be a last resort scenario to keep
Wikidata and WDQS running with reduced functionality while we are
able to deploy a more long-term solution.

 2.

    Update and access scaling

     1.

        Throughput. WDQS is currently trying to provide fast
        updates, and fast unlimited queries for all users. As the
        number of SPARQL queries grows over time
        <https://www.mediawiki.org/wiki/User:MPopov_(WMF)/Wikimania_2021_Hackathon>alongside
        graph updates, WDQS is struggling to sufficiently keep up
        <https://grafana.wikimedia.org/d/000000489/wikidata-query-service?viewPanel=8&from=now-6M&to=now&refresh=1d>in
        each dimension of service quality without compromising
        anywhere.  For users, this often leads to timed out queries.

     2.

        Equitable service. We are currently unable to adjust system
        behavior per user/agent. As such, it is not possible to
        provide equitable service to users: for example, a heavy
        user could swamp WDQS enough to hinder usability by
        community users.

In addition to being a querying service for Wikidata, WDQS is also
part of the edit pipeline of Wikidata (every edit on Wikidata is
pushed to WDQS to update the data there). While deploying the new
Flink-based Streaming Updater
<https://phabricator.wikimedia.org/T244590>will help with increasing
throughput of Wikidata updates, there is a substantial risk that
WDQS will be unable to keep up with the combination of increased
querying and updating, resulting in more tradeoffs between update
lag and querying latency/timeouts.

In the near term, we would like to work more closely with you to
determine what acceptable trade-offs would be for preserving WDQS
functionality while we scale up Wikidata querying. In the long term,
we will be conducting more user research to better understand your
needs so we can (i) optimize querying via SPARQL and/or other
methods, (ii) explore better user management that will allow us to
prevent heavy use of WDQS that does not align with the goals of our
movement and projects, and (iii) make it easier for users to set up
and run their own query services.

Though this information about the current state of WDQS may not be a
total surprise to many of you, we want to be as transparent with you
as possible to ensure that there are as few surprises as possible in
the case of any potential service disruptions/catastrophic failures,
and that we can accommodate your work as best as we can in the
future evolution of WDQS. We plan on doing a session on WDQS scaling
challenges during WikidataCon this year at the end of October.

Thanks for your understanding with these scaling challenges, and for
any feedback you have already been providing. If you have new
concerns, comments and questions, you can best reach us at this talk
page
<https://www.wikidata.org/wiki/Wikidata_talk:Query_Service_scaling_update_Aug_2021>.
Additionally, if you have not had a chance to fill out our survey
<https://docs.google.com/forms/d/e/1FAIpQLSe1H_OXQFDCiGlp0QRwP6-Z2CGCgm96MWBBmiqsMLu0a6bhLg/viewform?usp=sf_link>yet,
please tell us how you use the Wikidata Query Service (see privacy
statement
<https://foundation.wikimedia.org/wiki/WDQS_User_Survey_2021_Privacy_Statement>)!
Whether you are an occasional user or create tools, your feedback is
needed to decide our future development.

Best,

WMF Search + WMDE

_______________________________________________
Wikidata mailing list -- wikidata@lists.wikimedia.org
<mailto:wikidata@lists.wikimedia.org>
To unsubscribe send an email to wikidata-leave@lists.wikimedia.org
<mailto:wikidata-leave@lists.wikimedia.org>