On 17.02.2016 10:34, Magnus Manske wrote:
On Wed, Feb 17, 2016 at 7:16 AM Stas Malyshev <smalyshev@wikimedia.org mailto:smalyshev@wikimedia.org> wrote:
Well, again the problem is that one use case that I think absolutely needs caching - namely, exporting data to graphs, maps, etc. deployed on wiki pages - is also the one not implemented yet because we don't have cache (not only, but one of the things we need) so we've got chicken and egg problem here :) Of course, we can just choose something now based on educated guess and change it later if it works badly. That's probably what we'll do.Wouldn't those usecases be wrapped in an extension or WMF-controlled JavaScript? In that case, queries could always indicate that use, and they could be cached, for hours if need be. No reason to put everything behind a long cache by default, just because of those controllable cases.
Also, what about creating more independent blazegraph instances? One (or more) could be for wiki extension queries, with long cache; others could be for Labs use (internal network only?), a "general" external-facing server, etc.
If the problem (and it's not even certain we have one) can be mitigated or solved with throwing a few more VMs at it, I'm all for it :-)
+1 for adding some servers before building complicated caching solutions
I think long caching periods for wiki queries could lead to user frustration for the reasons I gave in my other post. But maybe one can simply give the user a way to say "please recompute this query now" to avoid this.
Another thing one could do for wiki-based (and other) queries is to use caches as fallbacks in case of timeouts: "we will try our best to give you a fresh result, but if current load is too high, we will at least give you some older result." This makes most sense for wiki-based queries which repeat reliably over time (so it makes sense to keep one result, however old it is, for all queries still used on some wiki page). Would be more work to implement, so probably not the first thing to do without any real wiki usage experiences yet.
Markus