We will be failing over the Toolforge and Project NFS in 10 minutes to move the main interface to 10Gb Ethernet. The previous work should make this fairly non-disruptive, but that was believed in the past as well.
Brooke Storm
Cloud Service Team
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
Is there a problem with SSH on {dev,login}.tools.wmflabs.org <http://tools.wmflabs.org/>? I can ping both of those hosts, but SSH is not responding for me.
Question: Is it permitted to use WMCS (be it Cloud VPS or Toolforge) resources for non-Wikimedia, Mediawiki-run website purposes? (Let's assume sites like FANDOM(Wikia), Miraheze, or https://librewiki.net )
Background: I was discussing some non-wikimedia (but mediawiki-run site) stuff at some chatroom. Someone there thought they could use Wikimedia Cloud resources for non-WMF websites (I don't know if it relates to MW development) purposes. I am not sure if this is allowed (my interpretation is that it is not allowed), so I want some clarification on that.
나의 iPhone에서 보냄
The the following (absurdly long) URL:
> https://spi-tools-dev.toolforge.org/spi/timeline/MariaJaydHicky?users=Love2… <https://spi-tools-dev.toolforge.org/spi/timeline/MariaJaydHicky?users=Love2…>
gives me:
> <html>
> <head><title>502 Bad Gateway</title></head>
> <body>
> <center><h1>502 Bad Gateway</h1></center>
> <hr><center>openresty/1.15.8.1</center>
> </body>
> </html>
> <!-- a padding to disable MSIE and Chrome friendly error page -->
> <!-- a padding to disable MSIE and Chrome friendly error page -->
> <!-- a padding to disable MSIE and Chrome friendly error page -->
> <!-- a padding to disable MSIE and Chrome friendly error page -->
> <!-- a padding to disable MSIE and Chrome friendly error page -->
> <!-- a padding to disable MSIE and Chrome friendly error page -->
Which isn't terribly surprising. As far as I can tell, the request never gets as far as my app. Is there a specific hard limit on how long a URL can be handled by the WSGI / routing layers?
In yet another effort to restore replication and preserve the redundancy of the data in ToolsDB (user writable database in Toolforge), we need to take the database (tools.db.svc.eqiad.wmflabs) completely offline at 1700 UTC on 16 Dec. Apps that depend on the ToolsDB service will fail during the outage (which will take at least an hour, and we aren’t entirely sure exactly how long—expect multiple hours). This will be much faster than the last outage because we are doing a straight copy of the binary database files between the servers. Details of this mess and efforts to restore the replication service can be found at https://phabricator.wikimedia.org/T266587 <https://phabricator.wikimedia.org/T266587>
If we succeed in producing a viable copy of the database on another system, we will also perform an upgrade on the hypervisor it is on before closing the maintenance period. That should be an additional hour or so.
We appreciate your patience with this process. It is very important that we establish a second copy of this database, especially in light of recent crashes (https://phabricator.wikimedia.org/T253738 <https://phabricator.wikimedia.org/T253738>).
Brooke Storm
Staff SRE
Wikimedia Cloud Services
bstorm(a)wikimedia.org
IRC: bstorm_
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
TLDR: Wiki Replicas' architecture is being redesigned for stability and
performance. Cross database JOINs will not be available and a host
connection will only allow querying its associated DB. See [1]
<https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign> for
more details.
Hi!
In the interest of making and keeping Wiki Replicas a stable and performant
service, a new backend architecture is needed. There is some impact in the
features and usage patterns.
What should I do? To avoid breaking changes, you can start making the
following changes *now*:
- Update existing tools to ensure queries are executed against the proper
database connection
- Eg: If you want to query the `eswiki_p` DB, you must connect to the
`eswiki.analytics.db.svc.eqiad.wmflabs` host and `eswiki_p` DB, and not to
enwiki or other hosts
- Check your existing tools and services queries for cross database JOINs,
rewrite the joins in application code
- Eg: If you are doing a join across databases, for example joining
`enwiki_p` and `eswiki_p`, you will need to query them separately, and
filter the results of the separate queries in the code
Timeline:
- November - December: Early adopter testing
- January 2021: Existing and new systems online, transition period starts
- February 2021: Old hardware is decommissioned
We need your help
- If you would like to beta test the new architecture, please let us know
and we will reach out to you soon
- Sharing examples / descriptions of how a tool or service was updated,
writing a common solution or some example code others can utilize and
reference, helping others on IRC and the mailing lists
If you have questions or need help adapting your code or queries, please
contact us [2]
<https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication>, or
write on the talk page [3]
<https://wikitech.wikimedia.org/wiki/Talk:News/Wiki_Replicas_2020_Redesign>.
We will be sending reminders, and more specific examples of the changes via
email and on the wiki page. For more information see [1]
<https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign>.
[1]: https://wikitech.wikimedia.org/wiki/News/Wiki_Replicas_2020_Redesign
[2]: https://wikitech.wikimedia.org/wiki/Help:Cloud_Services_communication
[3]:
https://wikitech.wikimedia.org/wiki/Talk:News/Wiki_Replicas_2020_Redesign
--
Joaquin Oltra Hernandez
Developer Advocate - Wikimedia Foundation
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
Hi there!
Today 2020-12-10 @ 15:30 UTC we will perform an upgrade of the Toolforge
kubernetes cluster [0].
We don't expect any major disruption of the service, but we detected in past
upgrades that some components might be restarted, causing brief interruptions of
network flows.
Given the amount of worker nodes we have, more than 50, the operation will take
us at least a couple of hours.
Tools maintainers: you don't have to do anything during this operation, but if
you detect anything weird please contact us either in the phabricator task [0],
in the IRC channel #wikimedia-cloud or in the cloud(a)lists.wikimedia.org [1]
mailing list.
regards.
[0] https://phabricator.wikimedia.org/T263284
[1] https://lists.wikimedia.org/mailman/listinfo/cloud
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation