Tomorrow (June 11th) at 1600 UTC, we will be failing over the primary
NFS server to do maintenance and upgrades on it. The secondary partner
in the cluster is already upgraded and ready, and recent changes
*should* make it a fairly straightforward failover with a brief period
of high load. If it doesn't proceed smoothly, it will be a slightly
longer period of high load and NFS lockup as failover completes (10-20
min or so). After maintenance it will be failed back, which will also,
hopefully, be quick and painless.
--
Brooke Storm
SRE
Wikimedia Cloud Services
bstorm(a)wikimedia.org
IRC: bstorm_
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
Hello!
Next week we'll be rebuilding and upgrading the hardware that provides
DNS service to cloud-vps and toolforge. These rebuilds will start at
14:00 UTC and the whole process may take 2-3 hours. It's likely that DNS
lookups will be somewhat slower as clients fail over between the
in-progress and the working server. In theory there should be few other
user-facing effects from these upgrades.
In practice, though, this isn't something that we've done for quite a
while, and touching DNS is always risky since it underlies pretty much
everything. Here are some things to be ready for:
- As a precaution we'll be disabling Horizon during the window to
prevent new VMs or DNS changes landing in an inconsistent state.
- Some badly-behaved DNS clients won't fail over properly and will
report errors when their primary DNS server is down.
- Puppet will almost certainly experience transient failures, since
Puppet is known to be one of those badly-behaved clients.
- If things go very badly there may be periods of total DNS outage which
will result in many WMCS-hosted services failing. There's no particular
reason that this /should/ happen, but this is the worst-case scenario.
For additional context, the phabricator task for this work is
https://phabricator.wikimedia.org/T253780
- Andrew + the WMCS team
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
Thanks for all the work you folks do on Pywikibot, it's really a gem in the
ecosystem.
On Tue, Jun 9, 2020 at 11:45 AM Martin Urbanec <martin(a)urbanec.cz> wrote:
> As the last release of Python 2 is finally out, the July release of
> Pywikibot is going to be the **last release that supports Python 2**.
> Support of Python 3.4 and MediaWiki older than 1.19 is also going to be
> dropped. After this release, Pywikibot is not going to receive any further
> patches and bug fixes related to those Python and MediaWiki versions.
> Functions and other stuff specific to Python 3.4, Python 2.x or MediaWiki
> older than 1.19 will be removed.
>
> For your convenience, this release is marked with a "python2"
> git tag and it is also the last 3.0.x release. In case you really need it,
> the Pywikibot team created /shared/pywikibot/core_python2 repository in
> Toolforge and a python2-pywikibot package in software repositories of some
> operating systems.
>
> The Pywikibot team strongly recommends that you migrate your scripts from
> Python 2 to Python 3. The migration steps were described in the previous
> message, which can be found here:
> https://lists.wikimedia.org/pipermail/pywikibot/2020-January/009976.html
> Detailed plan of Python 2 deprecation with dates is described here:
> https://www.mediawiki.org/wiki/Manual:Pywikibot/Compatibility
>
> If you encounter any problems with the migration, you can always ask us
> here: https://phabricator.wikimedia.org/T242120
>
> Best regards,
>
> Pywikibot team
> _______________________________________________
> Wikimedia Cloud Services announce mailing list
> Cloud-announce(a)lists.wikimedia.org (formerly
> labs-announce(a)lists.wikimedia.org)
> https://lists.wikimedia.org/mailman/listinfo/cloud-announce
>
Hi All,
Happy Weekend!
Back in January, it was suggested that this list should be indexed (
https://phabricator.wikimedia.org/T242520).
No feedback has happened since and the mailman update to allow searching is
unblocked.
If this is something still or not desired, please indicate on the task.
Thanks,
Samuel
--
Thanks,
Samuel
At 2020-06-04T11:12 UTC a change was merged to the
operations/puppet.git repository which resulted in data loss for Cloud
VPS projects using a local Puppetmaster
(role::puppetmaster::standalone). The specific data loss is removal of
any local to the Puppetmaster instance commits overlaid on the
upstream labs/private.git repository. These patches would have
contained passwords, ssh keys, TLS certificates, and similar
authentication information for Puppet managed configuration.
The majority of Cloud VPS projects are not affected by this
configuration data loss. Several highly used and visible projects,
including Toolforge (tools) and Beta Cluster (deployment-prep), have
some impact. We have disabled Puppet across all Cloud VPS instances
that were reachable by our central command and control service (cumin)
and are currently evaluating impact and recovering data from
/var/logs/puppet.log change logs where available.
More information will be collected at
<https://phabricator.wikimedia.org/T254491> and an incident report
will also be prepared once the initial response is complete.
Bryan
--
Bryan Davis Technical Engagement Wikimedia Foundation
Principal Software Engineer Boise, ID USA
[[m:User:BDavis_(WMF)]] irc: bd808
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
On Thursday we will be upgrading the network infrastructure that routes
network traffic for all cloud-vps and toolforge networking. This will
involve at least one failover between hosts, which will interrupt
existing network connections.
There should not be a prolonged network outage, but some connections
will be reset. If your tools or services are not resilient to
unexpected disconnections, you may need to manually restart services
after the updates are complete.
The changeover will happen sometime between 14:00 and 15:00 UTC this
coming Thursday. Details about this can be found at
https://phabricator.wikimedia.org/T253124
-Andrew + the WMCS team
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
Hi there!
We just deployed tesseract-ocr v4.1.1 in the Toolforge grid.
The context of this update is the phabricator task T247422 [0].
Please report any issue you may find.
regards!
[0] https://phabricator.wikimedia.org/T247422
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation