Wikimedia Labs not properly supporting infrastructure for volunteers - sad but mostly non-critical.<br><br><span style="line-height:19.7999992370605px">Wikimedia Labs not properly supporting </span><span style="line-height:19.7999992370605px">infrastructure for </span><span style="line-height:19.7999992370605px">other Wikimedia divisions, for months apparently - unacceptable.</span><br><br><br><div class="gmail_quote">On Wed Nov 26 2014 at 16:14:54 Dan Andreescu <<a href="mailto:dandreescu@wikimedia.org">dandreescu@wikimedia.org</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">We (analytics team) run wikimetrics on top of the labs data. For the last couple of months, we've seen missing data (data in prod that's not in labs) and a big slowdown on some queries that used to be fast. I was told about the cold indices theory but if it takes this long to warm them up maybe we should turn up the thermostat. Or... however that works in db world :)<div><br></div><div>Note: we absolutely adore our DBA and labs support folks, we think there may just be too much work that needs to get done and these problems are definitely affecting our products. Right now, we have to tell people to basically ignore the data for the last couple of months until this get sorted out.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Nov 26, 2014 at 9:57 AM, Maximilian Doerr <span dir="ltr"><<a href="mailto:maximilian.doerr@gmail.com" target="_blank">maximilian.doerr@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Could this in anyway, no matter how remote, be a link to the slow query times enwiki has been spitting out? Anomie suggests they are cold files in the database, but I don’t understand how that should be happening with indexes in that area.<div><br><div>
<div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">Cyberpower678<br>English Wikipedia Account Creation Team<br>Mailing List Moderator</div><div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><br></div><br>
</div><div><div>
<br><div><blockquote type="cite"><div>On Nov 26, 2014, at 09:54, Magnus Manske <<a href="mailto:magnusmanske@googlemail.com" target="_blank">magnusmanske@googlemail.com</a>> wrote:</div><br><div>I already submitted<br><br><a href="https://phabricator.wikimedia.org/T75980" target="_blank">https://phabricator.wikimedia.org/T75980</a><br><div><br></div><div>and</div><div><br></div><div><a href="https://phabricator.wikimedia.org/T75511" target="_blank">https://phabricator.wikimedia.org/T75511</a><br></div><br><div class="gmail_quote">On Wed Nov 26 2014 at 13:44:39 John <<a href="mailto:phoenixoverride@gmail.com" target="_blank">phoenixoverride@gmail.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Actually it looks like things in the databases are completely screwy. idwiki.labsdb shows zero tables, but enwiki.labsdb "use idwiki_p" shows what it should. Looks like there may be a bigger issue<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Nov 26, 2014 at 7:58 AM, Krinkle <span dir="ltr"><<a href="mailto:krinklemail@gmail.com" target="_blank">krinklemail@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div>Seems to be at least 62 days behind on commonswiki</div><div><br></div><div>tin.eqiad.wmnet$ sql commonswiki</div><div>rc_id: 142360816</div><div>rc_timestamp: 2014-11-26 12:55:31</div><div><div><br></div><div>tools-login$ sql commonswiki</div></div><div>rc_id: 136903866</div><div>rc_timestamp: 2014-09-24 12:51:42</div><div><br></div><div>See also <a href="https://tools.wmflabs.org/unpatrollededitstats/commonswiki/" target="_blank">https://tools.wmflabs.org/unpatrollededitstats/commonswiki/</a><br><br></div><div>
<div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">— Krinkle</div>
</div>
<br><div><div>On 23 Nov 2014, at 12:56, John <<a href="mailto:phoenixoverride@gmail.com" target="_blank">phoenixoverride@gmail.com</a>> wrote:</div><br><blockquote type="cite"><div dir="ltr">enwiki's replication lag is at 114256 and growing.<br></div>
_______________________________________________<br>Labs-l mailing list<br><a href="mailto:Labs-l@lists.wikimedia.org" target="_blank">Labs-l@lists.wikimedia.org</a><br><a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br></blockquote></div><br></div><br>_______________________________________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org" target="_blank">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br>
<br></blockquote></div><br></div>
______________________________<u></u>_________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org" target="_blank">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/<u></u>mailman/listinfo/labs-l</a><br>
</blockquote></div>
_______________________________________________<br>Labs-l mailing list<br><a href="mailto:Labs-l@lists.wikimedia.org" target="_blank">Labs-l@lists.wikimedia.org</a><br><a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br></div></blockquote></div><br></div></div></div></div><br>_______________________________________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org" target="_blank">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br>
<br></blockquote></div><br></div>
______________________________<u></u>_________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org" target="_blank">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/<u></u>mailman/listinfo/labs-l</a><br>
</blockquote></div>