Hi all,
I propose we consolidate Toolforge dynamicproxy documentation into this new
wikitech page:
https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Dynamicproxy
The source for the information in this new page comes mostly from 3 sources:
* https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin#WebProxy
* https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Webservice
* new content added
I've added 'consolidation proposal warning' to the affected pages.
This is part of my effort to better document the Toolforge networking, specially
the k8s part. In the new docs [0], when I was trying to add a link to
dynamicproxy I couldn't find a single page with all the content, and that's why
I'm proposing the factorization.
Of course you can modify the index and content organization in the new proposed
page. There is a lot of stuff going on with dynamicproxy and it deserves its
own wikipage I think!
If nobody raises strong concerns about this, I plan to do this doc consolidation
soon. CC @bstorm @bd808.
regards.
[0]
https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Networking_and_i…
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
Hi there!
Next Monday 2019-10-28 @ 14:30 UTC we will do a maintenance operation on
Toolforge which consists in rebuilding the main front proxy [0] used to serve
webservices. We expect this to be done within a 30 minutes window.
The operation consists on replacing the old virtual machines supporting the
proxy (currently running Debian Jessie) with more modern instances running
Debian Buster. Both Grid/Kubernetes backends are affected by this change. We
don't expect a lot of service downtime, but there is a key point in the
operation which is migrating data stored in Redis which can be tricky. The o
Examples of things affected by this change:
* Browsing Toolforge webservices
* Browsing to https://tools.wmflabs.org/<toolname>
* Browsing to https://tools.wmflabs.org/admin/ (Toolforge landing page)
* Browsing PAWS (to some extent, since it shares part of the toolforge proxy)
Example of things not affected by this change:
* webservices backend operations
* SSH bastions
* grid queues, grid jobs
* wiki-replicas, toolsdb
* other CloudVPS projects
regards.
[0] https://phabricator.wikimedia.org/T235627
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
On 10/21/19 9:49 PM, Brooke Storm wrote:
> With a redundant power supply upgrade going on this week in the datacenter that
> could affect the VM that Toolsdb runs on, we anticipate a brief outage Thursday
> 10/24 @11am UTC of the mysql service to protect data in case anything goes
> wrong. This may require a restart of a tool to reconnect to the database. We do
> not anticipate any worse disruptions, but if there is any disruption beyond what
> is planned, a failover may be necessary, which will not include the
> non-replicated tables mentioned
> here https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#ToolsDB_Backups…
>
> The maintenance requiring this notice and action is detailed
> here https://phabricator.wikimedia.org/T227540. The VM resides on the
> cloudvirt1019 hypervisor, which is why it is in scope.
>
> We sincerely apologize for the short notice.
>
Reminder, this is happening in a few minutes!
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation
Hi,
I have a proposal.
The new k8s haproxy is in front of the api-server and the ingress [0].
In toolsbeta we have been using the following:
toolsbeta-k8s-master.toolsbeta.wmflabs.org:6443 (api-server)
toolsbeta-k8s-master.toolsbeta.wmflabs.org:30000 (ingress)
This haproxy knows which k8s nodes/controllers are UP and proxy the queries for
them. Right now, this FQDN is not using a floating IP, is a simple A record
pointing to the haproxy VM. This record is in the 'toolsbeta' CloudVPS project.
I've been wondering which FQDN would be nice to have in the final deployment.
We have 'toolforge.org', but `whatever.toolforge.org` is intended to be a tool
webservice, so I've been re-reading our DNS domains plans [1] and my proposal is
to introduce a new FQDN like this:
k8s.toolforge.wmcloud.org
Then we can use it this way:
k8s.toolforge.wmcloud.org:6443 (api-server)
k8s.toolforge.wmcloud.org:30000 (ingress)
This is because 'wmcloud.org' is set to become the replacement for 'wmflabs.org'
which is what we are currently using for
'toolsbeta-k8s-master.toolsbeta.wmflabs.org'.
We could also create k8s.toolsbeta.wmcloud.org (or whatever) in case we want to
retain the toolsbeta setup online.
I hope this proposal is not increasing our naming confusion and complexity.
Ideally we would use something like `k8s.toolforge.org` but that seems even more
confusing in the long term.
I already requested the wmcloud.org domains to be pointed to designate [2].
Let me know!
[0]
https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Networking_and_i…
[1]
https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/Enhanceme…
[2] https://phabricator.wikimedia.org/T235630
--
Arturo Borrero Gonzalez
SRE / Wikimedia Cloud Services
Wikimedia Foundation