At 6pm UTC, Thursday 2018-09-27, we’ll be taking the primary tools/project NFS system offline for a brief moment to attempt some upgrades, which should cause load to climb a bit and then settle. This should be a relatively low-impact change and has been thoroughly tested in a separate environment, but if anything goes wrong with the process, corrective action may cause a few minutes of problems.
Brooke Storm
Operations Engineer
Wikimedia Cloud Services
bstorm(a)wikimedia.org
IRC: bstorm_
At 6pm UTC, we will be briefly resetting the main NFS server for tools and projects to upgrade the tooling around NFS there. This should cause a brief climb in processes waiting for IO and then that will settle out.
If things don’t operate as expected, rolling back the change could take a few minutes, but that seems unlikely. Please let us know if there are prolonged issues in the #wikipedia-cloud channel.
Brooke Storm
Operations Engineer
Wikimedia Cloud Services
bstorm(a)wikimedia.org
IRC: bstorm_
There are currently 57 unclaimed projects on
https://wikitech.wikimedia.org/wiki/Cloud_VPS_2018_Purge. I will start
shutting down unclaimed projects at the beginning of next month, and
those projects will be left behind in the future network migration[1]
and, eventually, deleted.
If you see anything in this list that you care about, please indicate
that the project is in use on the Wikitech page. Here are the 57
projects that are currently unmarked:
aicaptcha
bots
bstorm-test
chicotestproject
ci-staging
collection-alt-renderer
contributors
deep-learning-services
discourse-wam
download
etytree
fastcci
glampipe
globaleducation
hat-imagescalers
hound
iiifls1
ircd
jupyter
maps
maps-team
matrix
mcr-dev
mediahandler-tests
mediawiki-docker
multimedia
mw-api-testing
mw-extension-ids
newsletter
nonfreewiki
openocr
orig
pagemigration
piwik
project-smtp
rdfiodev
reading-lists
redirects
reportcard
sentry
services
services-testbed
t136871
test-twemproxy
tor
traffic
wcdo
wdq-mm
wikibrain
wikidata-page-banner
wikidata-primary-sources-tool
wikidumpparse
wikifactmine
wikimetrics
wikisource-tools
wikistream
wlmjudging
[1]
https://phabricator.wikimedia.org/phame/post/view/112/neutron_is_finally_co…
All,
As you may have read about in other venues, the Wikimedia Foundation will
be conducting a switchover of datacenters on September 11 and 12, with the
switch back to occur on October 10 and 11. During the switchover and
switch-back events, the wikis will be in read-only mode.
Cloud VPS and Toolforge are not affected by these switchovers. However, if
you run bots or scripts that make edits to the wikis, they may be unable to
make edits during this time. This should appear to your bots as the
standard read-only mode which occurs unplanned on the production wikis from
time to time.
Please let me know if you have any questions.
----
James Hare
Associate Product Manager
Wikimedia Foundation
https://wikimediafoundation.org
We will be performing some updates (including reboots) on the tools NFS servers for Toolforge and CloudVPS instances that use project NFS (with the exception of dumps) at 1500 UTC on Wed Aug 29. The maintenance window will be two hours and more than one NFS server failover is expected during that time. This could cause some temporary impact to performance and load on the various connected servers during failovers.
Brooke Storm
Operations Engineer
Wikimedia Cloud Services
bstorm(a)wikimedia.org <mailto:bstorm@wikimedia.org>
IRC: bstorm_
In an attempt to identify abandoned VPS projects, I've created a wiki
page that lists all existing projects, here:
https://wikitech.wikimedia.org/wiki/Cloud_VPS_2018_Purge
Currently 85 projects[2] on that list are unclaimed. If you are a VPS
user, please visit that page and mark any projects that you use as
{{Used}}. Note that it's not necessary for you to be a project admin to
mark something -- if you know that you're currently using a resource and
want to keep using it, go ahead and mark it accordingly. If you /are/ a
project admin, please take a moment to mark which VMs are or aren't used
in your projects.
When October arrives, I will shut down and begin the process of
reclaiming unused projects.
If you think you use a VPS project but aren't sure which, I encourage
you to poke around on https://tools.wmflabs.org/openstack-browser/ to
see what looks familiar. You can also log in to
http://horizon.wikimedia.org which will provide you with a handy menu of
projects that you are currently a member of.
Thank you!
-Andrew and the WMCS team
[1]
https://phabricator.wikimedia.org/phame/post/view/112/neutron_is_finally_co…
[2] Here, for good measure, is that list. Every one of these projects
is currently a candidate for deletion:
aicaptcha
analytics
bots
chicotestproject
ci-staging
codereview
collection-alt-renderer
commonsarchive
community-labs-monitoring
contributors
dashiki
deep-learning-services
discourse
discourse-wam
download
etytree
fastcci
getstarted
glampipe
globaleducation
hat-imagescalers
hound
iiab
iiifls1
ircd
jupyter
kubernetes-testing
maps
maps-team
math
matrix
mcr-dev
mediahandler-tests
mediawiki-docker
multimedia
mw-api-testing
mw-extension-ids
mwfuzz
mwoffliner
newsletter
nonfreewiki
openocr
ores
orig
otrs
pagemigration
paws
phlogiston
piwik
project-smtp
rcm
rdfiodev
reading-lists
reading-web-staging
recommendation-api
redirects
reportcard
sciencesource
sentry
services
services-testbed
t136871
test-twemproxy
thumbor
tor
traffic
twl
utrs
video
wcdo
wdq-mm
wikibase-nearest-neighbors
wikibrain
wikidata-federation
wikidata-page-banner
wikidata-primary-sources-tool
wikidumpparse
wikifactmine
wikimetrics
wikisource-tools
wikistream
wikitolearn-dev
wildcat
wlmjudging
wmf-research-tools
We're drawing close to a painful migration event[1], during which we
will (probably) have to copy VMs between hosts one project at a time,
largely by hand. For that reason, I'm feeling even stingier than usual
about preserving unused and/or abandoned projects and instances.
It's been a couple of years since we last did this, and I'm sure there
are some newly forgotten projects in the cloud. So, I've created a wiki
page the lists all existing projects, here:
https://wikitech.wikimedia.org/wiki/Cloud_VPS_2018_Purge
If you are a VPS user, please visit that page and mark any projects that
you use as {{Used}}. Note that it's not necessary for you to be a
project admin to mark something -- if you know that you're currently
using a resource and want to keep using it, go ahead and mark it
accordingly. If you /are/ a project admin, please take a moment to mark
which VMs are or aren't used in your projects.
When October arrives, I will shut down and begin the process of
reclaiming unused projects.
If you think you use a VPS project but aren't sure which, I encourage
you to poke around on https://tools.wmflabs.org/openstack-browser/ to
see what looks familiar. Worst case, just email
cloud(a)lists.wikimedia.org with a description of your use case and we'll
sort it out there.
Exclusive toolforge users are free to ignore this task.
Thank you!
-Andrew and WMCS team
[1] The eqiad->eqiad1 migration, also known as the 'nova-network to
neutron' migration which I hope to write more about soon. It won't
happen by surprise, I promise.
Hi!
Next monday 13th we will be doing some maintenance on the main Cloud VPS
deployment to merge the keystone service of both main and eqiad1
deployments (the new one that we will eventually put into production).
Toolforge users will not be affected by this outage.
Day: Monday 13th August
Start time: 14:00 UTC
Finish time: 16:00 UTC or ASAP
Keystone is a central point in openstack, so most horizon operations
like login, creating/deleting VMs could be affected. On the other hand,
VMs will keep working and we don't expect any network outage.
This operation will allow us to have a smooth transition in the future
when we move all projects and instances to the new eqiad1 deployment and
is a previous step to having multi-region support in our Cloud VPS service.
Please let us know any question or suggestions you may have.
best regards.
I am happy to announce that Addshore and Legoktm have been granted
admin (root) level privileges in the Toolforge project. Both are long
time users of Toolforge and prolific contributors to Wikimedia's
technical spaces. One of the projects they are hoping to help us with
is bringing newer versions of PHP to the Toolforge Kubernetes cluster
[0].
[0]: https://phabricator.wikimedia.org/T195689
Bryan
--
Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org>
[[m:User:BDavis_(WMF)]] Manager, Technical Engagement Boise, ID USA
irc: bd808 v:415.839.6885 x6855
On July 10th and 11th, database servers available to Cloud Services users are undergoing maintenance. This should be short in duration with minimal impact at 2pm UTC.
The servers and services subject to the maintenance are:
s*.analytics.db.svc.eqiad.wmflabs — no impact expected (other than needing to reconnect)
toolsdb — Brief unavailability (more information in further notices)
wikilabels — Brief unavailability (more information in further notices)
OpenStreetMap — No impact expected
Brooke Storm
Operations Engineer
Wikimedia Cloud Services
bstorm(a)wikimedia.org
IRC: bstorm_