On Wed, Jan 4, 2017 at 1:21 PM, Addshore <addshorewiki(a)gmail.com> wrote:
The query listed selecting from wikidatawiki.wb_terms
was me /
wmde-analytics and runs daily.
I agree that some sort of query limits would make sense.
What limits are currently in place?
The wikidata queries I have seen are probably the most concerning, they are
full scan+very large sorts of the wb_terms table > 700 times a day (which
means server query pileups), each one time taking over 2 hour (the former
query limit), on what I assume is an append-mostly table. With a few easy
tweaks (some triggers or a query rewrite), the same data could be obtained
instantly instead of taking so many resources.
Please ask for a hand (you can do it privately)- it is actually faster and
easier, and I am here to help, and you will actually make my life easier if
servers continue being up :-).
I will talk about this at the Wikimedia developers conference
https://phabricator.wikimedia.org/T149624 , a session I am inviting you all
you to go to if you are attending the conference. I will talk about the new
analytics labsdb service (with new servers!) and provide some feedback that
applies to the analytics boxes, too.