From 12:00 to 15:00 UTC on 2023-03-14 PAWS is cutting over its nfs storage.
As a result anything saved during this time frame will likely be lost.
Please do not save any files (or have bots save any files) during this time
as they likely will not make it through the cutover.
We'll send out an email to cloud-announce noting when the cutover is done
and it is safe to save files again.
More information can be found in the following tickets:
https://phabricator.wikimedia.org/T331056https://phabricator.wikimedia.org/T303663https://phabricator.wikimedia.org/T301280
Thank you,
--
*Vivian Rook (They/Them)*
Site Reliability Engineer
Wikimedia Foundation <https://wikimediafoundation.org/>
I am in the process of standardizing[0] the role names in WMCS cloud-vps
to conform with upstream conventions[1]. That requires me to rename two
existing user roles, 'user' and 'projectadmin':
- The role previously called 'user' will now be called 'reader'
- The role previously called 'projectadmin' will now be called 'member'
Despite the (IMO) less obvious names, a 'reader' can still log into
project VMs, and a 'member' can still create and delete VMs. Taavi has
thoughtfully upgraded the documentation about what roles can do what;
the complete docs can be found at
https://wikitech.wikimedia.org/wiki/Help:Cloud_services_user_roles_and_righ…
This renaming is phase one; phase two will involve switching to the
default upstream access rules for these two new roles.
Right now the old and new roles are co-existing in our system, but soon
I will entirely delete the old 'user' and 'projectadmin' roles. In the
meantime, please let me know if you find stray references to the old
role names, or if you find yourself unable to perform Horizon actions[1]
that you were previously able to do. Or, more seriously, able to do
things that you were not previously able to do!
Sorry for any inconvenience caused!
-Andrew
[0] Our OpenStack deployment has a very long history; it is older than
most deployments. That means that many conventions established in our
cloud now differ from the consensus standards created among newer
clouds. Periodically I try to update our cloud to conform to these new
standards; it reduces tech debt and also increases the chances that
official OpenStack documentation will be useful to our users.
[1] https://phabricator.wikimedia.org/T330759
[2] There is one edge case in Horizon that may require you to switch
projects in order to refresh the role permissions.
Hi there!
Today 2023-03-06, in a few minutes, we will restart the Toolforge internal
network, A brief interruption of network communications is expected during the
maintenance.
This is because we're re-deploying calico to the kubernetes cluster [0].
No action required on your side.
regards.
[0] https://phabricator.wikimedia.org/T328539
--
Arturo Borrero Gonzalez
Senior SRE / Wikimedia Cloud Services
Wikimedia Foundation
As part of the ToolsDB migration work [1], in about 1 hour from now I
will stop ToolsDB for a very short time (I expect the downtime to last
approximately 2 minutes).
You can follow along and report any issues in the #wikimedia-cloud IRC channel.
Thanks,
Francesco
[1] https://phabricator.wikimedia.org/T301949
--
Francesco Negri (he/him) -- IRC: dhinus
Site Reliability Engineer, Cloud Services team
Wikimedia Foundation
We are having some very concerning instability with the cloud-vps file
system. Out of an abundance of caution I have shut off EVERYTHING in
cloud-vps to prevent rampant data corruption.
I don't expect this outage to last long but will notify when things
start up again. Very sorry for the downtime!
-Andrew
Thanks largely to dschwen's hard work, we are about to move the
long-neglected postgres osmdb to a volunteer-managed project. Most
workloads have already moved to the new service. As far as anyone can
tell there is only a single tool still hitting osmdb.eqiad.wmnet.
Later in the week, that tool will break when I finally shut down the
eqiad.wmnet domain. If your tool is using that service, please refer to
https://phabricator.wikimedia.org/T323159 to coordinate migration.
- Andrew + the WMCS team
it would appear that pwb 8 has corrected some things and the `pwb.py`
script will no longer be included in paws when we upgrade, nicely the `pwb`
command itself will start working. If you were using `pwb.py` you will
likely have to start using `pwb` to run your code.
--
*Vivian Rook (They/Them)*
Site Reliability Engineer
Wikimedia Foundation <https://wikimediafoundation.org/>
The hardware that hosts osmdb.eqiad.wmnet is long past its end of life
and will be shut down on February 12th. WMF staff do not plan to support
that database after that date, and the domain will be shut down.
I am pretty sure that we have already made arrangements with all current
users of the service, but I'm sending this email out of an abundance of
caution. If you think you are using it, please chime in on the
associated phabricator ticket[0] so that we know you exist! There is a
volunteer-maintained replacement that you should be able to switch to
with a minimum of effort.
Thanks for reading!
-Andrew + the WMCS team
[0] https://phabricator.wikimedia.org/T323159