<div dir="ltr">Hoi,<div>Well actually, if the product is causing production problems and, it is. Something needs to be done. The question is not if the people are amiable (they are) or if they are well intentioned (they are) but if the service that is provided is good enough (it is not). </div><div><br></div><div>The question is therefore why does it take so long of suffering for so many people. Why is this not noticed and why is this not escalated to the people ultimately responsible. It things are of a professional quality, escalation happened. I am therefore really interested in the plans that are in the process of implementation that will remedy this situation. If there are no such plans, it is relevant for stakeholders to address the existing situation. Maybe they should consider this anyway.,</div><div>Thanks,</div><div> GerardM</div></div><div class="gmail_extra"><br><div class="gmail_quote">On 26 November 2014 at 17:14, Dan Andreescu <span dir="ltr"><<a href="mailto:dandreescu@wikimedia.org" target="_blank">dandreescu@wikimedia.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">We (analytics team) run wikimetrics on top of the labs data. For the last couple of months, we've seen missing data (data in prod that's not in labs) and a big slowdown on some queries that used to be fast. I was told about the cold indices theory but if it takes this long to warm them up maybe we should turn up the thermostat. Or... however that works in db world :)<div><br></div><div>Note: we absolutely adore our DBA and labs support folks, we think there may just be too much work that needs to get done and these problems are definitely affecting our products. Right now, we have to tell people to basically ignore the data for the last couple of months until this get sorted out.</div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Nov 26, 2014 at 9:57 AM, Maximilian Doerr <span dir="ltr"><<a href="mailto:maximilian.doerr@gmail.com" target="_blank">maximilian.doerr@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">Could this in anyway, no matter how remote, be a link to the slow query times enwiki has been spitting out? Anomie suggests they are cold files in the database, but I don’t understand how that should be happening with indexes in that area.<div><br><div>
<div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">Cyberpower678<br>English Wikipedia Account Creation Team<br>Mailing List Moderator</div><div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><br></div><br>
</div><div><div>
<br><div><blockquote type="cite"><div>On Nov 26, 2014, at 09:54, Magnus Manske <<a href="mailto:magnusmanske@googlemail.com" target="_blank">magnusmanske@googlemail.com</a>> wrote:</div><br><div>I already submitted<br><br><a href="https://phabricator.wikimedia.org/T75980" target="_blank">https://phabricator.wikimedia.org/T75980</a><br><div><br></div><div>and</div><div><br></div><div><a href="https://phabricator.wikimedia.org/T75511" target="_blank">https://phabricator.wikimedia.org/T75511</a><br></div><br><div class="gmail_quote">On Wed Nov 26 2014 at 13:44:39 John <<a href="mailto:phoenixoverride@gmail.com" target="_blank">phoenixoverride@gmail.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Actually it looks like things in the databases are completely screwy. idwiki.labsdb shows zero tables, but enwiki.labsdb "use idwiki_p" shows what it should. Looks like there may be a bigger issue<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Nov 26, 2014 at 7:58 AM, Krinkle <span dir="ltr"><<a href="mailto:krinklemail@gmail.com" target="_blank">krinklemail@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div>Seems to be at least 62 days behind on commonswiki</div><div><br></div><div>tin.eqiad.wmnet$ sql commonswiki</div><div>rc_id: 142360816</div><div>rc_timestamp: 2014-11-26 12:55:31</div><div><div><br></div><div>tools-login$ sql commonswiki</div></div><div>rc_id: 136903866</div><div>rc_timestamp: 2014-09-24 12:51:42</div><div><br></div><div>See also <a href="https://tools.wmflabs.org/unpatrollededitstats/commonswiki/" target="_blank">https://tools.wmflabs.org/unpatrollededitstats/commonswiki/</a><br><br></div><div>
<div style="letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">— Krinkle</div>
</div>
<br><div><div>On 23 Nov 2014, at 12:56, John <<a href="mailto:phoenixoverride@gmail.com" target="_blank">phoenixoverride@gmail.com</a>> wrote:</div><br><blockquote type="cite"><div dir="ltr">enwiki's replication lag is at 114256 and growing.<br></div>
_______________________________________________<br>Labs-l mailing list<br><a href="mailto:Labs-l@lists.wikimedia.org" target="_blank">Labs-l@lists.wikimedia.org</a><br><a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br></blockquote></div><br></div><br>_______________________________________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org" target="_blank">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br>
<br></blockquote></div><br></div>
______________________________<u></u>_________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org" target="_blank">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/<u></u>mailman/listinfo/labs-l</a><br>
</blockquote></div>
_______________________________________________<br>Labs-l mailing list<br><a href="mailto:Labs-l@lists.wikimedia.org" target="_blank">Labs-l@lists.wikimedia.org</a><br><a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br></div></blockquote></div><br></div></div></div></div><br>_______________________________________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org" target="_blank">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br>
<br></blockquote></div><br></div>