[Labs-l] Replication broken

[Magnus Manske]

It's a big ol' honking tool all right, which is why I made [1], which is
/much/ faster. However, some complex queries from catscan2, especially when
used in combination, are rather hard to do is a single query, and holding
everything in PHP memory is probably less efficient than temporary MySQL
tables.

One issue is that I can't really predict at runtime which query will take 5
seconds and which will take 12 hours. I'd be happy to set a generic limit
on the tool, but I don't know how to do that for the MySQL query.


[1] http://tools.wmflabs.org/catscan2/quick_intersection.php


On Tue, Dec 10, 2013 at 2:14 PM, Tim Landscheidt <tim at tim-landscheidt.de>wrote:

> Marc-André Pelletier <mpelletier at wikimedia.org> wrote:
>
> >> The same thing happened yesterday; Coren determined that a certain
> project
> >> that shall remain nameless, unless you consider "catscan2" to be a
> name, had
> >> been holding a lock on the database for over 12 hours.
>
> > And indeed that's the case again; I need to sit down with the catscan
> > people and find a better way for them to do whatever it is they are
> > trying to do in a way that does not result in:
>
> > ---TRANSACTION 9337D917, ACTIVE 52358 sec fetching rows
> > mysql tables in use 3, locked 3
> > 566321 lock struct(s), heap size 56850872, 10281493 row lock(s), undo
> > log entries 150718
>
> > I really want to avoid having to install a query killer and place hard
> > time limit on running time - no matter where the line it it will be an
> > annoyance to /someone/.  There are occasional legitimate uses for very
> > long queries; but they should be infrequent and not last *that* long.
>
> IIRC it's not necessarily long queries that lock down the
> DB, but queries that requests locks :-).  I see temporary
> tables and inserts in the catscan2 code, and I think that
> might be the culprit.
>
> That was one of the many surprises when moving from
> Toolserver to Labs: On Toolserver, a tool could request a
> lock for a minute (or whatever the threshold for the query
> killer was), the replication lag would go up, but the query
> that actually got killed was the read-only one that had been
> running peacefully for hours and not increased replication
> lag in any way.
>
> So, *please*, if we think about enabling some sort of query
> killer on Tools, let's make *very* sure that it aims accu-
> rately.
>
> Tim
>
>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
>



-- 
undefined
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.wikimedia.org/pipermail/labs-l/attachments/20131210/c5de5d6f/attachment-0001.html>