<div dir="ltr">For onlookers, we caught up on IRC and things are good now.<br><div><br>Thank you for the honest feedback Maximillian. This was an unfortunate and unplanned outcome that we struggled to recover from. A handful of users we were not able to restore /home for completely and it's been an uphill battle. <br><div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jan 23, 2017 at 10:42 AM, Maximilian Doerr <span dir="ltr"><<a href="mailto:maximilian.doerr@gmail.com" target="_blank">maximilian.doerr@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word">The migration went horribly wrong on my project, which deleted roughly 60% of the project, including the .my.cnf file. All the permissions and file ownerships were screwed up on the remaining files, and completely broke the bot tasks running on it. I have restored the permissions, ownership and files, but I’m still missing my db cnf file to access the DB with. It was not fun having to fix this.<div><br><div>
<div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word"><div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">Cyberpower678<br>English Wikipedia Account Creation Team<br>English Wikipedia Administrator</div><div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">Global User Renamer</div></div></div></div>
</div>
<br><div><blockquote type="cite"><div>On Jan 19, 2017, at 13:03, Madhumitha Viswanathan <<a href="mailto:mviswanathan@wikimedia.org" target="_blank">mviswanathan@wikimedia.org</a>> wrote:</div><br class="m_8789013518512447566Apple-interchange-newline"><div><div dir="ltr" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">Update: 2 instances are still being restored - taxonbot.dwl and pole.wikidata-query, the rest on the previous list are done.</div><div class="gmail_extra" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><br><div class="gmail_quote">On Wed, Jan 18, 2017 at 11:20 PM, Madhumitha Viswanathan<span class="m_8789013518512447566Apple-converted-space"> </span><span dir="ltr"><<a href="mailto:mviswanathan@wikimedia.org" target="_blank">mviswanathan@<wbr>wikimedia.org</a>></span><span class="m_8789013518512447566Apple-converted-space"> </span>wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Hi all,<div><br></div><div>Update - most of the home directories have been restored at this point - we're down to the last few, that seem to have had a lot of data hence taking quite some time.</div><div><br></div><div>These are the pending instances(currently being restored):</div><div><br></div><div>data.eqiad.wmflabs</div><div>ldfclient.eqiad.wmflabs<br></div><div>mathosphere.eqiad.wmflabs</div><div>mwoffliner4.eqiad.wmflabs</div><div>nomad.eqiad.wmflabs</div><div>reading.eqiad.wmflabs</div><div>recommendations.eqiad.wmflabs<br></div><div>captcha-apiproxy-03.eqiad.wmfl<wbr>abs</div><div>dashboardchat.eqiad.wmflabs<br></div><div>maps-cruncher.eqiad.wmflabs</div><div>pdf.eqiad.wmflabs<br></div><div>mwoffliner[2-3].eqiad.wmflabs</div><div>phlogiston-2.eqiad.wmflabs</div><div>pole.eqiad.wmflabs<br></div><div><br></div><div>Other than this, it should all look good. Do let us know if something seems amiss here or on #wikimedia-labs. Thanks for being patient!</div><div><br></div></div><div class="gmail_extra"><div><div class="m_8789013518512447566h5"><br><div class="gmail_quote">On Wed, Jan 18, 2017 at 3:45 PM, Madhumitha Viswanathan<span class="m_8789013518512447566Apple-converted-space"> </span><span dir="ltr"><<a href="mailto:mviswanathan@wikimedia.org" target="_blank">mviswanathan@<wbr>wikimedia.org</a>></span><span class="m_8789013518512447566Apple-converted-space"> </span>wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Update: NFS for labs projects is back to read write now. <div><br></div><div>There was an incident in the process of the migration that caused /home in a significant number of labs instances (where /home was not on NFS) to be replaced by an erroneous symlink. We are currently working on restoring all the home directories - it looks like it may take at-least a couple hours. I'll keep the list updated on progress.</div><div><br></div></div><div class="gmail_extra"><div><div class="m_8789013518512447566m_8842336298895956995h5"><br><div class="gmail_quote">On Wed, Jan 18, 2017 at 11:34 AM, Madhumitha Viswanathan<span class="m_8789013518512447566Apple-converted-space"> </span><span dir="ltr"><<a href="mailto:mviswanathan@wikimedia.org" target="_blank">mviswanathan@<wbr>wikimedia.org</a>></span><span class="m_8789013518512447566Apple-converted-space"> </span>wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>Update: this is still in progress. There's some ongoing issues with home directories on non-nfs /home across Labs (not tools or maps) - we are working on it, and will update soon. Do feel free to reach out #wikimedia-labs if you have any questions or concerns.<br></div></div><div class="gmail_extra"><div><div class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702h5"><br><div class="gmail_quote">On Wed, Jan 18, 2017 at 8:38 AM, Madhumitha Viswanathan<span class="m_8789013518512447566Apple-converted-space"> </span><span dir="ltr"><<a href="mailto:mviswanathan@wikimedia.org" target="_blank">mviswanathan@<wbr>wikimedia.org</a>></span><span class="m_8789013518512447566Apple-converted-space"> </span>wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Reminder: This is starting soon, in ~22 minutes.</div><div class="gmail_extra"><div><div class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912h5"><br><div class="gmail_quote">On Tue, Jan 17, 2017 at 12:07 PM, Madhumitha Viswanathan<span class="m_8789013518512447566Apple-converted-space"> </span><span dir="ltr"><<a href="mailto:mviswanathan@wikimedia.org" target="_blank">mviswanathan@<wbr>wikimedia.org</a>></span><span class="m_8789013518512447566Apple-converted-space"> </span>wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Reminder: This is happening tomorrow, <span style="font-family:arial,helvetica,sans-serif;font-size:12.8px">starting 09:00 PST(16:00 UTC).</span></div><div class="gmail_extra"><div><div class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753h5"><br><div class="gmail_quote">On Wed, Jan 4, 2017 at 1:43 PM, Madhumitha Viswanathan<span class="m_8789013518512447566Apple-converted-space"> </span><span dir="ltr"><<a href="mailto:mviswanathan@wikimedia.org" target="_blank">mviswanathan@<wbr>wikimedia.org</a>></span><span class="m_8789013518512447566Apple-converted-space"> </span>wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid257" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-ace-line" style="margin:0px;padding:0px"><span style="background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px">Hello,</span></font></span></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid257" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-ace-line" style="margin:0px;padding:0px"><span style="background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px"><br></span></font></span></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid257" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-ace-line" style="margin:0px;padding:0px"><span style="background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px">Continuing the storage redundancy and reliability efforts for Labs (</span></font></span><font face="arial, helvetica, sans-serif"><a href="https://phabricator.wikimedia.org/T126083" target="_blank">https://phabricator.wikimedia<wbr>.org/T126083</a>)</font><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="font-family:arial,helvetica,sans-serif;margin:0px;padding:1px 0px">, the final migration to the new NFS storage cluster for Labs projects with NFS enabled is upcoming. The migration is planned to happen 2017-01-</span><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-author-a-z90zb7z67zz90zz77zuez122zbz77zz85zz74zbmn" style="font-family:arial,helvetica,sans-serif;margin:0px;padding:1px 0px">18</span><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="font-family:arial,helvetica,sans-serif;margin:0px;padding:1px 0px"> starting 09:00 PST(16:00 UTC). This *does not* affect tools, maps or any other projects that don't have /home or /data/project mounted. The migration window is expected to be fairly short (<3 hours) - but could last up to 6 hours.</span></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid187" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><span style="background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif"><br style="margin:0px;padding:0px"></font></span></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid188" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif">During the migration, no new data will be written to NFS (/home or /data/project), but existing data will be accessible in Read-only mode for the most part. Post migration, any services or jobs that were running on top of NFS (/home or /data/project) will require manual restarts. Jobs running on top of /scratch or /public/dumps will be unaffected. I will keep the lists and #wikimedia-labs updated on progress during and after the migration.</font></span></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid189" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><span style="background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif"><br style="margin:0px;padding:0px"></font></span></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid190" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif">The list of labs projects that will be affected in this migration are:</font></span></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid191" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif"> </font></span></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid192" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><ul class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-list-bullet1" style="margin:0px 0px 0px 1.5em;padding:0px"><li style="margin:0px;padding:0px"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif">catgraph</font></span></li></ul></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid193" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><ul class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-list-bullet1" style="margin:0px 0px 0px 1.5em;padding:0px"><li style="margin:0px;padding:0px"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif">account-creation-assistance</font></span></li></ul></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid194" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><ul class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-list-bullet1" style="margin:0px 0px 0px 1.5em;padding:0px"><li style="margin:0px;padding:0px"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif">contributors</font></span></li></ul></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid195" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><ul class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-list-bullet1" style="margin:0px 0px 0px 1.5em;padding:0px"><li style="margin:0px;padding:0px"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif">wikidata-topicmaps</font></span></li></ul></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid196" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><ul class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-list-bullet1" style="margin:0px 0px 0px 1.5em;padding:0px"><li style="margin:0px;padding:0px"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif">sugarcrm</font></span></li></ul></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid197" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><ul class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-list-bullet1" style="margin:0px 0px 0px 1.5em;padding:0px"><li style="margin:0px;padding:0px"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif">wikidumpparse</font></span></li></ul></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid198" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><ul class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-list-bullet1" style="margin:0px 0px 0px 1.5em;padding:0px"><li style="margin:0px;padding:0px"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif">video</font></span></li></ul></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid199" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><ul class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-list-bullet1" style="margin:0px 0px 0px 1.5em;padding:0px"><li style="margin:0px;padding:0px"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif">openstack</font></span></li></ul></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid200" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><ul class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-list-bullet1" style="margin:0px 0px 0px 1.5em;padding:0px"><li style="margin:0px;padding:0px"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif">testlabs</font></span></li></ul></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid201" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><ul class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-list-bullet1" style="margin:0px 0px 0px 1.5em;padding:0px"><li style="margin:0px;padding:0px"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif">wikidata-dev</font></span></li></ul></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid202" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><ul class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-list-bullet1" style="margin:0px 0px 0px 1.5em;padding:0px"><li style="margin:0px;padding:0px"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif">quarry</font></span></li></ul></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid203" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><ul class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-list-bullet1" style="margin:0px 0px 0px 1.5em;padding:0px"><li style="margin:0px;padding:0px"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif">huggle</font></span></li></ul></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid204" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><ul class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-list-bullet1" style="margin:0px 0px 0px 1.5em;padding:0px"><li style="margin:0px;padding:0px"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif">editor-engagement</font></span></li></ul></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid205" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><ul class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-list-bullet1" style="margin:0px 0px 0px 1.5em;padding:0px"><li style="margin:0px;padding:0px"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif">utrs</font></span></li></ul></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid206" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><ul class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-list-bullet1" style="margin:0px 0px 0px 1.5em;padding:0px"><li style="margin:0px;padding:0px"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif">wmt</font></span></li></ul></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid207" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><ul class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-list-bullet1" style="margin:0px 0px 0px 1.5em;padding:0px"><li style="margin:0px;padding:0px"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif">cvn</font></span></li></ul></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid208" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><ul class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-list-bullet1" style="margin:0px 0px 0px 1.5em;padding:0px"><li style="margin:0px;padding:0px"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif">fastcci</font></span></li></ul></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid209" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><ul class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-list-bullet1" style="margin:0px 0px 0px 1.5em;padding:0px"><li style="margin:0px;padding:0px"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif">toolsbeta</font></span></li></ul></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid210" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><ul class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-list-bullet1" style="margin:0px 0px 0px 1.5em;padding:0px"><li style="margin:0px;padding:0px"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif">project-proxy</font></span></li></ul></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid211" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><ul class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-list-bullet1" style="margin:0px 0px 0px 1.5em;padding:0px"><li style="margin:0px;padding:0px"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif">dumps</font></span></li></ul></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid212" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><ul class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-list-bullet1" style="margin:0px 0px 0px 1.5em;padding:0px"><li style="margin:0px;padding:0px"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif">bots</font></span></li></ul></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid213" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><ul class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-list-bullet1" style="margin:0px 0px 0px 1.5em;padding:0px"><li style="margin:0px;padding:0px"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif">snuggle</font></span></li></ul></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid214" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><ul class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-list-bullet1" style="margin:0px 0px 0px 1.5em;padding:0px"><li style="margin:0px;padding:0px"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif">math</font></span></li></ul></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid215" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><ul class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-list-bullet1" style="margin:0px 0px 0px 1.5em;padding:0px"><li style="margin:0px;padding:0px"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px;background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif">wikisource-tools</font></span></li></ul></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid216" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><span style="background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif"><br style="margin:0px;padding:0px"></font></span></div><div id="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-magicdomid217" class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:0px"><span style="background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif"><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px">The tracking task on phabricator is here - </span><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-url" style="margin:0px;padding:1px 0px"><a href="https://phabricator.wikimedia.org/T154336" style="margin:0px;padding:0px;white-space:pre-wrap" target="_blank">https://phabricator.wikimedi<wbr>a.org/T154336</a></span><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543m_3733973175350273914gmail-" style="margin:0px;padding:1px 0px">. Let us know if you have any questions or concerns on the list or on #wikimedia-labs.</span></font></span></div><div><span style="background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif"><br></font></span></div><font face="arial, helvetica, sans-serif">--</font><div><span style="background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif">Madhu Viswanathan</font></span></div><div><span style="background-color:rgb(255,255,255)"><font face="arial, helvetica, sans-serif">Operations Engineer, Wikimedia Labs</font></span></div></div></blockquote></div><br><br clear="all"><div><br></div></div></div><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753HOEnZb"><font color="#888888">--<span class="m_8789013518512447566Apple-converted-space"> </span><br><div class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753m_-3436871030114564543gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">--Madhu :)</div></div></font></span></div></blockquote></div><br><br clear="all"><div><br></div></div></div><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912HOEnZb"><font color="#888888">--<span class="m_8789013518512447566Apple-converted-space"> </span><br><div class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912m_3166649191047550753gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">--Madhu :)</div></div></font></span></div></blockquote></div><br><br clear="all"><div><br></div></div></div><span class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702HOEnZb"><font color="#888888">--<span class="m_8789013518512447566Apple-converted-space"> </span><br><div class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702m_-602657175111930912gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">--Madhu :)</div></div></font></span></div></blockquote></div><br><br clear="all"><div><br></div></div></div><span class="m_8789013518512447566m_8842336298895956995HOEnZb"><font color="#888888">--<span class="m_8789013518512447566Apple-converted-space"> </span><br><div class="m_8789013518512447566m_8842336298895956995m_-8397079878998996702gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">--Madhu :)</div></div></font></span></div></blockquote></div><br><br clear="all"><span class="HOEnZb"><font color="#888888"><div><br></div></font></span></div></div><span class="HOEnZb"><font color="#888888"><span class="m_8789013518512447566HOEnZb"><font color="#888888">--<span class="m_8789013518512447566Apple-converted-space"> </span><br><div class="m_8789013518512447566m_8842336298895956995gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">--Madhu :)</div></div></font></span></font></span></div></blockquote></div><span class="HOEnZb"><font color="#888888"><br><br clear="all"><div><br></div>--<span class="m_8789013518512447566Apple-converted-space"> </span><br><div class="m_8789013518512447566gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">--Madhu :)</div></div></font></span></div><span class="HOEnZb"><font color="#888888"><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important">______________________________<wbr>_________________</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><span style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important">Labs-l mailing list</span><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><a href="mailto:Labs-l@lists.wikimedia.org" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank">Labs-l@lists.wikimedia.org</a><br style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px"><a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" style="font-family:Helvetica;font-size:12px;font-style:normal;font-variant-caps:normal;font-weight:normal;letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank">https://lists.wikimedia.org/<wbr>mailman/listinfo/labs-l</a></font></span></div></blockquote></div><br></div></div><br>______________________________<wbr>_________________<br>
Labs-l mailing list<br>
<a href="mailto:Labs-l@lists.wikimedia.org">Labs-l@lists.wikimedia.org</a><br>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" rel="noreferrer" target="_blank">https://lists.wikimedia.org/<wbr>mailman/listinfo/labs-l</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>Chase Pettet<br></div>Engineering Manager -- Labs<br></div><div>chasemp on <a href="https://phabricator.wikimedia.org/p/chasemp/" target="_blank">phabricator</a> and IRC<br></div></div></div></div></div></div>
</div></div></div></div>