[Foundation-l] New two-part schedule for 1.17 deployment

[George Herbert]

Thank you for this.

Will ops staff be monitoring wikitech-l for email reports of observed
problems from those of us who are IRC-impaired?  Is there an another
preferred non-IRC channel for reports?

Thanks.


On Thu, Feb 10, 2011 at 2:59 PM, Rob Lanphier <robla at wikimedia.org> wrote:
> Hi everyone,
>
> In case you missed it on the techblog, here's an update on the revised
> deployment plan for 1.17, part 1 of which starts in 7 hours:
> http://techblog.wikimedia.org/2011/02/1-17deployment-attempt2/
>
> Also copied below.
>
> Rob
> ------
>
> As covered on this blog this week, we had a few problems with our
> initial deployment of 1.17 to the Wikimedia cluster of servers.  We’ve
> investigated the problems, and believe we have fixed many of the
> issues.  Some of the unsolved issues are complicated enough that the
> only timely and reasonable way to investigate them is to deploy and
> react, so we’ve come up with a plan that lets us do it in a safe way
> by deploying on just a few wikis at a time (as opposed to all at once,
> as we tried earlier).
>
> We’re scheduling two deployment windows:
>
> First window – This wave will be deployed between Friday, February 11,
> 6:00 UTC – 12:00 UTC (10pm PST Thursday, February 10 in San
> Francisco).  This first wave will be to a limited set of wikis (see
> below).
> Second window – Wednesday February 16 (between 6:00 UTC – 12:00 UTC) –
> full deployment (tentative)
> Repeating what is new about 1.17:  There are many, many little fixes
> and improvements (see the draft release notes for an exhaustive list),
> as well as one larger improvement: Resource Loader.  Read more in the
> previous 1.17 deployment announcement.
>
>
> First window
> This first deployment window will be to a limited set of wikis:
>
> http://simple.wikipedia.org/ (simplewiki)
> http://simple.wiktionary.org/ (simplewiktionary)
> http://usability.wikimedia.org/ (usabilitywiki)
> http://strategy.wikimedia.org/ (strategywiki)
> http://meta.wikimedia.org/ (metawiki)
> http://eo.wikipedia.org/ (eowiki)
> http://en.wikiquote.org/ (enwikiquote)
> http://en.wikinews.org/ (enwikinews)
> http://en.wikibooks.org/ (enwikibooks)
> http://beta.wikiversity.org (betawikiversity)
> http://nl.wikipedia.org (nlwiki)
> Note that the point of this first round of wikis being switched over
> is to be able to observe the problem or problems without overloading
> the site and bringing it down.  This deployment should be small enough
> in scope that even if there are moderate performance problems, no one
> should notice without watching our monitoring tools.  We may not roll
> out to every wiki listed above during the first wave, but we plan to
> roll out to enough of them that we can gather enough debugging
> information to make the second wave (full deployment) go smoothly.
>
> Second window
> We will continue to roll this out to the rest of the wikis during this
> window.  Depending on our confidence level, we may deploy to the
> remaining wikis, or we may decide to deploy to a portion of the
> remaining wikis.  If necessary, we will schedule another window to
> finish the deployment.
>
> Technical details
> Here’s some more technical detail: one problem with the original
> Tuesday deploy was that the cache miss rate went up quite
> substantially.  We believe the problem was a problem with the
> configuration of the $wgCacheEpoch variable, which caused more
> aggressive culling of our cache than the servers could handle.  We
> have made adjustments, and so this shouldn’t be a problem during our
> next deployment attempt.
>
> The $wgCacheEpoch problem explains some of the problems we had, but
> not all of them.  Since we don’t have a clear explanation for all of
> the problems, we plan to modify the way we deploy this software so
> that we aren’t rolling this out to every wiki simultaneously.  As our
> software is currently built, this isn’t easy to do in a general way,
> but it turns out this release is suited to an incremental deployment.
> (Note: we also plan to develop a more general capacity to roll out
> incrementally for future releases).
>
> Thank you for your patience!  We hope that this time around we can
> deploy this in a way that you won’t notice anything other than the
> improvements.
>
> _______________________________________________
> foundation-l mailing list
> foundation-l at lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/foundation-l
>



-- 
-george william herbert
george.herbert at gmail.com