Hi!
> 3) Move the service to labs, not providing any firm guarantee of service
> level ?
I don't see any reason to move it to labs, and I don't think labs has
infrastructure to handle it. It can not run on virtual machines - in
fact, the only reason why it can't be as stable as we wish is because we
don't have hardware to run fully redundant configuration with capacity
to serve all cases. I don't see how using labs would help there - does
labs have more hardware capacity than production that is available for
wdqs use?
> I think there was one exception, which is services that needed a lot of
> resources so they could not run on vms, but don't we have a prototype of
> "labs on real hardware"?
I really wouldn't want to run this service on prototype that wasn't
tried before and introduce both additional point of failure and
unnecessary administration burden. I fail to see any existing issue that
this would solve, but it certainly has potential to introduce some.
> that you are describing- running easily out of resources (DOS). Even
> quarry, which I have publicly complained about in the past, for what you
> say, has a better resource management than wqs (30-minute limit
> execution, concurrency control, etc.).
We have 30-second limit now. People complain about it all the time
because it's too short, but see above about the hardware.
> icinga. But running an unstable service (wdqs) on top of another
> unstable service (wikidata data handling) will never be stable.
> Everytime a bot starts writing to wikidata 600 times per second, s5 dbs
> shake (that is why we are creating s8) and wqs goes down. :-)
> I would suggest using wqs on labs (or anywhere, non-production) with
> regular imports rather than real-time updates. Less headaches. I am
> literally aiming for that for labsdbs, too.
I don't think this is a good scenario. Delayed updates means severely
degraded user experience and won't save much performance as the data
needs to be moved over anyway. We could save something if we had proper
change tracking service that doesn't use recentchages API directly on
wikidata (so we get several uncached rc API calls per each edit) but has
some faster and more efficient intermediary, but AFAIK we don't have
that still. That would be a direction to look if updating is too
resource-consuming.
--
Stas Malyshev
smalyshev(a)wikimedia.org
