Replication was halted on a lock timeout, likely caused by a query writing to a userdb while reading from a dewiki table. I had intended to set innodb_locks_unsafe_for_binlog on all labsdb servers, which disables gap and next key locking, while also only guaranteeing read_committed isolation. This is completely safe in labs since nothing but the slave sql thread writes to the replicated tables, and it should prevent this issue. I ended up setting it for everything except s4 and s5; probably related to the mad scrambling for the Amsterdam hackathon :) S4 and s5 now need a mariadb restart for the setting to go live, which I'll work with Marc to schedule. <br>
<br>I've also set slave_transaction_retries to a very large value on all of the labsdb's, a change which does not require a restart. That should also prevent a recurrence in the interim before the the server restarts occur. And at least the s5 replica caught up very quickly. <br>
<br>-Asher<br><br><div class="gmail_quote">On Thu, Jun 13, 2013 at 1:22 AM, Magnus Manske <span dir="ltr"><<a href="mailto:magnusmanske@googlemail.com" target="_blank">magnusmanske@googlemail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Update: According to "select max(rev_timestamp) from revision", the replag is ~15h. Any particular reason?</div>
<div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jun 13, 2013 at 8:53 AM, Magnus Manske <span dir="ltr"><<a href="mailto:magnusmanske@googlemail.com" target="_blank">magnusmanske@googlemail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">At least wikidatawiki_p seems to have a replag in minutes or more. Is there a way to see this, either a web page or via shell?</div>
</blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br>
<br></blockquote></div><br>