<div dir="ltr">Thanks for the update, and for the work of course!<br></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Nov 23, 2015 at 4:54 PM Jaime Crespo <<a href="mailto:jcrespo@wikimedia.org">jcrespo@wikimedia.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Labsdb databases are still suffering from corruption. I am trying to repair the tables affected. Updates will be at: <<a href="https://phabricator.wikimedia.org/T119315" target="_blank">https://phabricator.wikimedia.org/T119315</a>> Expect some downtime in order to fix the issues.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Nov 23, 2015 at 2:01 PM, Magnus Manske <span dir="ltr"><<a href="mailto:magnusmanske@googlemail.com" target="_blank">magnusmanske@googlemail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Ah, didn't see this, created a bug report:<br><a href="https://phabricator.wikimedia.org/T119382" target="_blank">https://phabricator.wikimedia.org/T119382</a><br></div><div><div><br><div class="gmail_quote"><div dir="ltr">On Mon, Nov 23, 2015 at 12:30 PM Yetkin Sakal <<a href="mailto:superyetkin@yahoo.com" target="_blank">superyetkin@yahoo.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="color:#000;background-color:#fff;font-family:HelveticaNeue,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif;font-size:13px"><div>That is good news for us all, jcrespo.</div><div><br></div><div>Cheers!<br></div></div></div><div><div style="color:#000;background-color:#fff;font-family:HelveticaNeue,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif;font-size:13px"><div><span></span></div> <br><div><br><br></div><div style="display:block"> <div style="font-family:HelveticaNeue,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif;font-size:13px"> <div style="font-family:HelveticaNeue,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif;font-size:16px"> <div dir="ltr"><font face="Arial" size="2"> On Monday, November 23, 2015 2:00 PM, Jaime Crespo <<a href="mailto:jcrespo@wikimedia.org" target="_blank">jcrespo@wikimedia.org</a>> wrote:<br></font></div> <br><br> <div><div><div><div dir="ltr"><div>It seems that 2 out of 3 labs replica servers had crashed, creating corruption in the process. Replica lag is now going down.<br clear="none"><br clear="none"></div>Lag measuring (with 1 second resolution) is coming soon, will be available as a table, but I hit a bug and it is not being displayed correctly, so it is not yet public.<br clear="none">I will announce it as soon as it is fixed.<br clear="none"><div><br clear="none"></div></div><div><br clear="none"><div>On Sun, Nov 22, 2015 at 10:06 PM, Yetkin Sakal <span dir="ltr"><<a rel="nofollow" shape="rect" href="mailto:superyetkin@yahoo.com" target="_blank">superyetkin@yahoo.com</a>></span> wrote:<br clear="none"><blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div style="color:#000;background-color:#fff;font-family:HelveticaNeue,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif;font-size:13px">Thanks.<div><div><div><span></span></div> <br clear="none"><div><br clear="none"><br clear="none"></div><div style="display:block"> <div style="font-family:HelveticaNeue,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif;font-size:13px"> <div style="font-family:HelveticaNeue,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif;font-size:16px"> <div dir="ltr"><font face="Arial" size="2"> On Sunday, November 22, 2015 10:52 PM, John <<a rel="nofollow" shape="rect" href="mailto:phoenixoverride@gmail.com" target="_blank">phoenixoverride@gmail.com</a>> wrote:<br clear="none"></font></div> <br clear="none"><br clear="none"> <div><div><div><div dir="ltr">I have <a rel="nofollow" shape="rect" href="http://tools.wmflabs.org/betacommand-dev/cgi-bin/replag" target="_blank">http://tools.wmflabs.org/betacommand-dev/cgi-bin/replag</a> which gives a rough estimate, plus we have <a rel="nofollow" shape="rect" href="https://phabricator.wikimedia.org/T119315" target="_blank">https://phabricator.wikimedia.org/T119315</a><br clear="none"></div><div><br clear="none"><div>On Sun, Nov 22, 2015 at 3:26 PM, Yetkin Sakal <span dir="ltr"><<a rel="nofollow" shape="rect" href="mailto:superyetkin@yahoo.com" target="_blank">superyetkin@yahoo.com</a>></span> wrote:<br clear="none"><blockquote style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div style="color:#000;background-color:#fff;font-family:HelveticaNeue,Helvetica Neue,Helvetica,Arial,Lucida Grande,sans-serif;font-size:13px"><div dir="ltr">Is there anywhere I can keep track of the status pertaining to the replication lag on s3? My tools are showing data from nearly a day ago.<br clear="none"></div></div></div></div><br clear="none">_______________________________________________<br clear="none">
Labs-l mailing list<br clear="none">
<a rel="nofollow" shape="rect" href="mailto:Labs-l@lists.wikimedia.org" target="_blank">Labs-l@lists.wikimedia.org</a><br clear="none">
<a rel="nofollow" shape="rect" href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br clear="none">
<br clear="none"></blockquote></div><br clear="none"></div></div></div><br clear="none"><br clear="none"></div> </div> </div> </div></div></div></div></div></div><br clear="none">_______________________________________________<br clear="none">
Labs-l mailing list<br clear="none">
<a rel="nofollow" shape="rect" href="mailto:Labs-l@lists.wikimedia.org" target="_blank">Labs-l@lists.wikimedia.org</a><br clear="none">
<a rel="nofollow" shape="rect" href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br clear="none">
<br clear="none"></blockquote></div><br clear="none"><br clear="all"><br clear="none">-- <br clear="none"><div><div dir="ltr"><div>Jaime Crespo<br clear="none"></div><<a rel="nofollow" shape="rect" href="http://wikimedia.org/" target="_blank">http://wikimedia.org</a>><br clear="none"></div></div>
</div></div></div><br><br></div> </div> </div> </div></div></div>_______________________________________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org" target="_blank">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" rel="noreferrer" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br>
</blockquote></div>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div><div dir="ltr"><div>Jaime Crespo<br></div><<a href="http://wikimedia.org" target="_blank">http://wikimedia.org</a>><br></div></div>
</div>
</blockquote></div>