On Wed, Jan 4, 2017 at 1:21 PM, Addshore <addshorewiki@gmail.com> wrote:
The query listed selecting from wikidatawiki.wb_terms was me / wmde-analytics and runs daily.

I agree that some sort of query limits would make sense.
What limits are currently in place?

The wikidata queries I have seen are probably the most concerning, they are full scan+very large sorts of the wb_terms table > 700 times a day (which means server query pileups), each one time taking over 2 hour (the former query limit), on what I assume is an append-mostly table. With a few easy tweaks (some triggers or a query rewrite), the same data could be obtained instantly instead of taking so many resources.

Please ask for a hand (you can do it privately)- it is actually faster and easier, and I am here to help, and you will actually make my life easier if servers continue being up :-).

I will talk about this at the Wikimedia developers conference https://phabricator.wikimedia.org/T149624 , a session I am inviting you all you to go to if you are attending the conference. I will talk about the new analytics labsdb service (with new servers!) and provide some feedback that applies to the analytics boxes, too.