<html><head></head><body><div style="font-family: Verdana;font-size: 12.0px;"><div>
<div>"(so tools that rely solely on the database / API *should* be fine, if they aren't using their home directories for anything write) "</div>
<div> </div>
<div>Do you mean with that, that I can use an own tool server instance and I can work with API, database and /shared/tcl/bin/tclsh8.7 ?</div>
<div>So can I get a tool server from you?</div>
<div> </div>
<div>Martin ...</div>
<div> </div>
<div>
<div name="quote" style="margin:10px 5px 5px 10px; padding: 10px 0 10px 10px; border-left:2px solid #C3D9E5; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div style="margin:0 0 10px 0;"><b>Gesendet:</b> Freitag, 21. Oktober 2016 um 20:56 Uhr<br/>
<b>Von:</b> "Madhumitha Viswanathan" <mviswanathan@wikimedia.org><br/>
<b>An:</b> "Wikimedia Labs" <labs-l@lists.wikimedia.org><br/>
<b>Betreff:</b> Re: [Labs-l] Disruptive Tools NFS maintenance on 11/2/2016</div>
<div name="quoted-content">
<div>Hi,
<div class="gmail_extra">
<div class="gmail_quote">On Fri, Oct 21, 2016 at 11:29 AM, Martin Domdey <span><<a href="mailto:animalia@gmx.net" onclick="parent.window.location.href='animalia@gmx.net'; return false;" target="_blank">animalia@gmx.net</a>></span> wrote:
<blockquote class="gmail_quote" style="margin: 0.0px 0.0px 0.0px 0.8ex;border-left: 1.0px solid rgb(204,204,204);padding-left: 1.0ex;">
<div>
<div style="font-family: verdana;font-size: 12.0px;">
<div>
<div>Why do you need 48 hours for that?</div>
<div> </div>
<div>I'm submitting very many cron jobs the day to deliver much stuff and services to a lot of users in dewiki and other wikis. An outage window of 48 hours (!) is simply not possible.</div>
<div>Please suggest a solution how I can work on during the outage window or at least a crontab that can handle the data and files on tools.taxonbot. You maybe can install a NFS redundancy for at least that time.</div>
<div> </div>
</div>
</div>
</div>
</blockquote>
<div>
<div class="gmail-ace-line" id="gmail-magicdomid316" style="margin: 0.0px;padding: 0.0px;">
<div class="gmail-ace-line" id="gmail-magicdomid316" style="margin: 0.0px;padding: 0.0px;">
<div class="gmail-ace-line" id="gmail-magicdomid316" style="margin: 0.0px;padding: 0.0px;">Like mentioned, it may take upto 48 hours for the data migration to be complete - hopefully lesser, but we are dealing with a complex system with a nontrivial amount of data. The transition *is* to a redundant NFS server setup - we need a long maintenance window to make that happen. A full copy of tools data to a new server takes many days(~4-20!) depending on various factors, and we're doing successively smaller syncs to make the final migration period as small as possible. However, it's still not something we can entirely control - the maps project was migrated earlier this week, and the final sync still took about a day (even though maps has less data). So the 48h is a conservative estimate that allows us to do the migration in an orderly fashion.</div>
<div class="gmail-ace-line" id="gmail-magicdomid316" style="margin: 0.0px;padding: 0.0px;"> </div>
<div class="gmail-ace-line" id="gmail-magicdomid316" style="margin: 0.0px;padding: 0.0px;">To be more explicit, here is a (non exhaustive) list of things we expect to not work for the duration of the transition (which is up to 48h, but hopefully lesser):</div>
<div class="gmail-ace-line" id="gmail-magicdomid316" style="margin: 0.0px;padding: 0.0px;"> </div>
<div class="gmail-ace-line" id="gmail-magicdomid316" style="margin: 0.0px;padding: 0.0px;"> 1. Submitting new jobs to the grid</div>
<div class="gmail-ace-line" id="gmail-magicdomid316" style="margin: 0.0px;padding: 0.0px;"> 2. Restarting failing jobs on the grid</div>
<div class="gmail-ace-line" id="gmail-magicdomid316" style="margin: 0.0px;padding: 0.0px;"> 3. Deploying new code / writing anything on your tool / home directories</div>
<div class="gmail-ace-line" id="gmail-magicdomid316" style="margin: 0.0px;padding: 0.0px;"> 4. Any bots / webservices that require write access on their home directories to work (so tools that rely solely on the database / API *should* be fine, if they aren't using their home directories for anything write)</div>
<div class="gmail-ace-line" id="gmail-magicdomid316" style="margin: 0.0px;padding: 0.0px;"> 5. New cron jobs (because of #1)</div>
<div class="gmail-ace-line" id="gmail-magicdomid316" style="margin: 0.0px;padding: 0.0px;"> 6. New tool creation</div>
<div class="gmail-ace-line" id="gmail-magicdomid316" style="margin: 0.0px;padding: 0.0px;"> </div>
<div class="gmail-ace-line" id="gmail-magicdomid316" style="margin: 0.0px;padding: 0.0px;">Any previously submitted jobs that aren't writing to NFS (provided they don't die), will continue to run. Crons submit jobs to the grid, and without read-write NFS, job scheduling will not work. We apologize for the service interruption, but it is required to have a long term stable & reliable tools.</div>
<div class="gmail-ace-line" id="gmail-magicdomid316" style="margin: 0.0px;padding: 0.0px;"> </div>
<div class="gmail-ace-line" id="gmail-magicdomid316" style="margin: 0.0px;padding: 0.0px;">We're working on a detailed checklist for the transition, and will email it to the list once we have it available.</div>
</div>
</div>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin: 0.0px 0.0px 0.0px 0.8ex;border-left: 1.0px solid rgb(204,204,204);padding-left: 1.0ex;">
<div>
<div style="font-family: verdana;font-size: 12.0px;">
<div>
<div>Thank you</div>
<div>Martin ...</div>
<div> </div>
<div> </div>
<div>
<div style="margin: 10.0px 5.0px 5.0px 10.0px;padding: 10.0px 0.0px 10.0px 10.0px;border-left: 2.0px solid rgb(195,217,229);">
<div style="margin: 0.0px 0.0px 10.0px;"><b>Gesendet:</b> Freitag, 21. Oktober 2016 um 20:00 Uhr<br/>
<b>Von:</b> "Madhumitha Viswanathan" <<a href="mailto:mviswanathan@wikimedia.org" onclick="parent.window.location.href='mviswanathan@wikimedia.org'; return false;" target="_blank">mviswanathan@wikimedia.org</a>><br/>
<b>An:</b> "Wikimedia Labs" <<a href="mailto:labs-l@lists.wikimedia.org" onclick="parent.window.location.href='labs-l@lists.wikimedia.org'; return false;" target="_blank">labs-l@lists.wikimedia.org</a>>, <a href="mailto:labs-announce@lists.wikimedia.org" onclick="parent.window.location.href='labs-announce@lists.wikimedia.org'; return false;" target="_blank">labs-announce@lists.wikimedia.org</a><br/>
<b>Betreff:</b> [Labs-l] Disruptive Tools NFS maintenance on 11/2/2016</div>
<div>
<div>
<div class="gmail-h5">
<div>
<div>As the next step in our storage redundancy and reliability efforts for Labs, we have a significant migration coming up on 11/2 starting 08:00 PST(15:00 UTC) involving the tools NFS share. The maintenance window can be up to 48h long, and will affect most running tools. At the end of the migration, everything (except transient jobs) should ideally be working the same way as they were before the migration, but better.</div>
<div> </div>
<div>Here's what to expect during the maintenance window:</div>
<div> </div>
<div>* The tools NFS share (/data/project and /home) will be read-only for the duration of the maintenance, so no new data or logs will get written to it.</div>
<div>* New jobs cannot be submitted for the whole maintenance window - this means submitting jobs through cron or tools-mail will not function, although tools-mail can continue to send emails. </div>
<div>* Current jobs might keep running, but won't get rescheduled if they die. If they do not die and aren't writing to NFS they should be fine. </div>
<div>* All exec nodes will get depooled, rebooted and repooled and jobs that don't get rescheduled automatically will have died and need manual restarts.</div>
<div> </div>
<div>Do let us know if you have any questions or concerns on the lists or on #wikimedia-labs.</div>
<div> </div>
--
<div>Madhumitha Viswanathan</div>
<div>Operations Engineer, Wikimedia Labs</div>
</div>
</div>
</div>
<span class="gmail-">_______________________________________________ Labs-l mailing list <a href="mailto:Labs-l@lists.wikimedia.org" onclick="parent.window.location.href='Labs-l@lists.wikimedia.org'; return false;" target="_blank">Labs-l@lists.wikimedia.org</a> <a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a></span></div>
</div>
</div>
</div>
</div>
</div>
<br/>
_______________________________________________<br/>
Labs-l mailing list<br/>
<a href="mailto:Labs-l@lists.wikimedia.org" onclick="parent.window.location.href='Labs-l@lists.wikimedia.org'; return false;" target="_blank">Labs-l@lists.wikimedia.org</a><br/>
<a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a><br/>
</blockquote>
</div>
<div> </div>
--
<div class="gmail_signature">
<div>--Madhu :)</div>
</div>
</div>
</div>
_______________________________________________ Labs-l mailing list Labs-l@lists.wikimedia.org <a href="https://lists.wikimedia.org/mailman/listinfo/labs-l" target="_blank">https://lists.wikimedia.org/mailman/listinfo/labs-l</a></div>
</div>
</div>
</div></div></body></html>