Cloud-admin October 2019

cloud-admin@lists.wikimedia.org

4 participants
6 discussions

Toolforge dynamicproxy documentation improvement
by Arturo Borrero Gonzalez 30 Oct '19

30 Oct '19

Hi all, I propose we consolidate Toolforge dynamicproxy documentation into this new wikitech page: https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Dynamicproxy The source for the information in this new page comes mostly from 3 sources: * https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin#WebProxy * https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Webservice * new content added I've added 'consolidation proposal warning' to the affected pages. This is part of my effort to better document the Toolforge networking, specially the k8s part. In the new docs [0], when I was trying to add a link to dynamicproxy I couldn't find a single page with all the content, and that's why I'm proposing the factorization. Of course you can modify the index and content organization in the new proposed page. There is a lot of stuff going on with dynamicproxy and it deserves its own wikipage I think! If nobody raises strong concerns about this, I plan to do this doc consolidation soon. CC @bstorm @bd808. regards. [0] https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Networking_and_i… -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 1

[Toolforge] Proxy maintenance operation next Monday 2019-10-28 @ 14:30 UTC
by Arturo Borrero Gonzalez 28 Oct '19

28 Oct '19

Hi there! Next Monday 2019-10-28 @ 14:30 UTC we will do a maintenance operation on Toolforge which consists in rebuilding the main front proxy [0] used to serve webservices. We expect this to be done within a 30 minutes window. The operation consists on replacing the old virtual machines supporting the proxy (currently running Debian Jessie) with more modern instances running Debian Buster. Both Grid/Kubernetes backends are affected by this change. We don't expect a lot of service downtime, but there is a key point in the operation which is migrating data stored in Redis which can be tricky. The o Examples of things affected by this change: * Browsing Toolforge webservices * Browsing to https://tools.wmflabs.org/<toolname> * Browsing to https://tools.wmflabs.org/admin/ (Toolforge landing page) * Browsing PAWS (to some extent, since it shares part of the toolforge proxy) Example of things not affected by this change: * webservices backend operations * SSH bastions * grid queues, grid jobs * wiki-replicas, toolsdb * other CloudVPS projects regards. [0] https://phabricator.wikimedia.org/T235627 -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 1

Re: [Cloud-admin] [Cloud-announce] Brief ToolsDB Outage - Thursday 10/24 @11am UTC
by Arturo Borrero Gonzalez 24 Oct '19

24 Oct '19

On 10/21/19 9:49 PM, Brooke Storm wrote: > With a redundant power supply upgrade going on this week in the datacenter that > could affect the VM that Toolsdb runs on, we anticipate a brief outage Thursday > 10/24 @11am UTC of the mysql service to protect data in case anything goes > wrong. This may require a restart of a tool to reconnect to the database. We do > not anticipate any worse disruptions, but if there is any disruption beyond what > is planned, a failover may be necessary, which will not include the > non-replicated tables mentioned > here https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#ToolsDB_Backups… > > The maintenance requiring this notice and action is detailed > here https://phabricator.wikimedia.org/T227540. The VM resides on the > cloudvirt1019 hypervisor, which is why it is in scope. > > We sincerely apologize for the short notice. > Reminder, this is happening in a few minutes! -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 0

CloudVPS maintenance on Wednesday 2019-10-23 (round 3 of cloudvirt reboots)
by Arturo Borrero Gonzalez 23 Oct '19

23 Oct '19

Hello! Next Wednesday 2019-10-23 at 09:00 UTC we will be doing another maintenance operation on some of our cloudvirts servers (the hypervisor servers) that involves rebooting both the physical servers and the virtual machines running on them. The reasons is that we ned to update the running linux kernel version they have. In this window we will reboot 3 hypervisors: * cloudvirt1014 * cloudvirt1025 * cloudvirt1026 The procedure will be to reboot a server, wait for it to come back online (could take up to 5 minutes) and wait for all the VMs to come back online. Then move to the next server. Toolforge users may see their tools and webservices briefly disrupted due to several components of the Toolforge infrastructure being rebooted in this operation. If nothing changes (reallocated or new virtual machine, etc) this is the list of affected VM instances in each hypervisor: * cloudvirt1014: VM: globalcu-app-01 PROJECT: globalcu VM: xtools-dev05 PROJECT: xtools VM: ores-web-05 PROJECT: ores VM: roebling PROJECT: wikispore VM: ores-web-04 PROJECT: ores VM: canary1014-01 PROJECT: testlabs VM: cloudinfra-db02 PROJECT: cloudinfra VM: Krypton PROJECT: codereview VM: dashiki-staging-01 PROJECT: dashiki VM: discovery-production-02 PROJECT: shiny-r VM: logparse01 PROJECT: security-tools VM: tools-sgegrid-shadow PROJECT: tools VM: maps-tiles1 PROJECT: maps VM: wikimetrics-01 PROJECT: wikimetrics VM: tools-sgecron-01 PROJECT: tools VM: wikitextexp-base-1002 PROJECT: wikitextexp VM: kask-client PROJECT: services VM: clm-web-01 PROJECT: community-labs-monitoring VM: accounts-appserver4 PROJECT: account-creation-assistance VM: taxonbota PROJECT: dwl VM: tofawiki02 PROJECT: fa-wp VM: federated-commons PROJECT: wikidata-federation VM: puppetmaster PROJECT: thumbor VM: lizenzhinweisgenerator PROJECT: lizenzhinweisgenerator VM: hound-puppet-02 PROJECT: hound VM: packagist-mirror1 PROJECT: packagist-mirror VM: deployment-elastic06 PROJECT: deployment-prep VM: deployment-changeprop PROJECT: deployment-prep VM: deployment-restbase02 PROJECT: deployment-prep VM: deployment-imagescaler01 PROJECT: deployment-prep VM: deployment-zookeeper02 PROJECT: deployment-prep VM: deployment-kafka-jumbo-1 PROJECT: deployment-prep VM: deployment-memc07 PROJECT: deployment-prep VM: deployment-eventlog05 PROJECT: deployment-prep VM: deployment-cpjobqueue PROJECT: deployment-prep VM: deployment-mediawiki-07 PROJECT: deployment-prep VM: deployment-chromium01 PROJECT: deployment-prep VM: deployment-puppetdb02 PROJECT: deployment-prep VM: deployment-puppetmaster03 PROJECT: deployment-prep VM: deployment-cache-text05 PROJECT: deployment-prep VM: gfg01 PROJECT: video VM: encoding01 PROJECT: video VM: whgi PROJECT: wikidumpparse VM: wikilabels PROJECT: wmf-research-tools VM: gitservices PROJECT: getstarted VM: design-research-methods PROJECT: design VM: wikilabels-backups PROJECT: wikilabels VM: wikilabels-02 PROJECT: wikilabels VM: dumpgrepper PROJECT: visualeditor VM: compiler1001 PROJECT: puppet-diffs VM: af-puppetdb02 PROJECT: automation-framework VM: clickmodel PROJECT: search VM: tool PROJECT: recommendation-api VM: missing-sections PROJECT: recommendation-api VM: ores-lb-03 PROJECT: ores VM: puppet-ema-2 PROJECT: puppet VM: matrix-synapse-01 PROJECT: matrix VM: wikibrain-embeddings-01 PROJECT: wikibrain VM: qube-node2 PROJECT: k8splay VM: captcha-tf-43 PROJECT: privpol-captcha VM: k4-2 PROJECT: analytics * cloudvirt1025: VM: integration-agent-docker-1006 PROJECT: integration VM: striker-deploy04 PROJECT: striker VM: rec-wiki-2 PROJECT: recommendation-api VM: deployment-ms-fe03 PROJECT: deployment-prep VM: deployment-poolcounter05 PROJECT: deployment-prep VM: deployment-ms-be05 PROJECT: deployment-prep VM: readers-web-stephen PROJECT: reading-web-staging VM: traffic-upload-stretch PROJECT: traffic VM: traffic-recdns-anycast PROJECT: traffic VM: deployment-maps05 PROJECT: deployment-prep VM: gerrit-sizzle PROJECT: security-tools VM: tools-sgewebgrid-generic-0901 PROJECT: tools VM: shinken-puppetmaster-01 PROJECT: shinken VM: osmit-due PROJECT: osmit VM: deployment-acme-chief03 PROJECT: deployment-prep VM: meza-cindy PROJECT: pluggableauth VM: accounts-db4 PROJECT: account-creation-assistance VM: krenair-clientpackages-py3-jessie PROJECT: testlabs VM: deployment-sessionstore01 PROJECT: deployment-prep VM: paws-worker-04 PROJECT: paws VM: paws-ext-lb-02 PROJECT: paws VM: paws-int-lb-01 PROJECT: paws VM: paws-master-03 PROJECT: paws VM: paws-master-01 PROJECT: paws VM: language-readership PROJECT: language VM: wmde-wikidiff2-patched-stretch PROJECT: wikidiff2-wmde-dev VM: tools-sgebastion-08 PROJECT: tools VM: compiler1002 PROJECT: puppet-diffs VM: phragile-db PROJECT: phragile VM: cloud-puppetmaster-01 PROJECT: cloudinfra VM: chicotest-cappy01 PROJECT: chicotestproject VM: visualeditor-prototype2 PROJECT: visualeditor VM: programs-and-events-dashboard PROJECT: globaleducation VM: osmit-uno PROJECT: osmit VM: tools-sgewebgrid-lighttpd-0904 PROJECT: tools VM: canary1025-01 PROJECT: testlabs VM: mathosphere PROJECT: math VM: social-tools3 PROJECT: social-tools VM: togetherjs PROJECT: visualeditor VM: language-mleb-legacy PROJECT: language VM: women-in-red PROJECT: globaleducation VM: ntp-01 PROJECT: cloudinfra VM: mc-clusterA-1 PROJECT: test-twemproxy VM: wikifarm PROJECT: pluggableauth VM: login-test PROJECT: catgraph VM: puppenmeister PROJECT: planet * cloudvirt1026: VM: integration-agent-docker-1016 PROJECT: integration VM: wikidata-new-wbterm PROJECT: wikidata-dev VM: incubator-test PROJECT: incubator VM: cloudinfra-internal-puppetmaster01 PROJECT: cloudinfra VM: cloudinfra-db01 PROJECT: cloudinfra VM: tools-checker-03 PROJECT: tools VM: tools-static-13 PROJECT: tools VM: wp1 PROJECT: mwoffliner VM: pk8s PROJECT: planet VM: arturo-k8s-test-4-1 PROJECT: openstack VM: banner PROJECT: wikidumpparse VM: packager01 PROJECT: packaging VM: tools-package-builder-02 PROJECT: tools VM: canary1026-02 PROJECT: testlabs VM: security-checker1 PROJECT: packagist-mirror VM: logstack02 PROJECT: security-tools VM: logstack01 PROJECT: security-tools VM: mediawiki2latex-large PROJECT: collection-alt-renderer VM: tools-sge-services-03 PROJECT: tools VM: tools-sgewebgrid-lighttpd-0928 PROJECT: tools VM: tools-sgewebgrid-lighttpd-0921 PROJECT: tools VM: tools-sgewebgrid-generic-0903 PROJECT: tools VM: tools-sgeexec-0938 PROJECT: tools VM: tools-sgeexec-0936 PROJECT: tools VM: tools-sgeexec-0935 PROJECT: tools VM: tools-sgeexec-0919 PROJECT: tools VM: tools-sgeexec-0917 PROJECT: tools VM: tools-sgeexec-0916 PROJECT: tools VM: tools-sgeexec-0915 PROJECT: tools VM: tools-sgeexec-0914 PROJECT: tools VM: tools-paws-worker-1010 PROJECT: tools VM: tools-paws-worker-1019 PROJECT: tools VM: openstack-puppetmaster-01 PROJECT: openstack VM: web1 PROJECT: graphql VM: etytree-b PROJECT: etytree VM: canary1026-01 PROJECT: testlabs VM: db-instance PROJECT: videowiki VM: tools-sgeexec-0906 PROJECT: tools VM: mwoffliner5 PROJECT: mwoffliner regards -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 2

DNS name for the new k8s haproxy
by Arturo Borrero Gonzalez 21 Oct '19

21 Oct '19

Hi, I have a proposal. The new k8s haproxy is in front of the api-server and the ingress [0]. In toolsbeta we have been using the following: toolsbeta-k8s-master.toolsbeta.wmflabs.org:6443 (api-server) toolsbeta-k8s-master.toolsbeta.wmflabs.org:30000 (ingress) This haproxy knows which k8s nodes/controllers are UP and proxy the queries for them. Right now, this FQDN is not using a floating IP, is a simple A record pointing to the haproxy VM. This record is in the 'toolsbeta' CloudVPS project. I've been wondering which FQDN would be nice to have in the final deployment. We have 'toolforge.org', but `whatever.toolforge.org` is intended to be a tool webservice, so I've been re-reading our DNS domains plans [1] and my proposal is to introduce a new FQDN like this: k8s.toolforge.wmcloud.org Then we can use it this way: k8s.toolforge.wmcloud.org:6443 (api-server) k8s.toolforge.wmcloud.org:30000 (ingress) This is because 'wmcloud.org' is set to become the replacement for 'wmflabs.org' which is what we are currently using for 'toolsbeta-k8s-master.toolsbeta.wmflabs.org'. We could also create k8s.toolsbeta.wmcloud.org (or whatever) in case we want to retain the toolsbeta setup online. I hope this proposal is not increasing our naming confusion and complexity. Ideally we would use something like `k8s.toolforge.org` but that seems even more confusing in the long term. I already requested the wmcloud.org domains to be pointed to designate [2]. Let me know! [0] https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Networking_and_i… [1] https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/Enhanceme… [2] https://phabricator.wikimedia.org/T235630 -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

4 13

CloudVPS maintenance on Wednesday 2019-10-16 (round 2 of cloudvirt reboots)
by Arturo Borrero Gonzalez 16 Oct '19

16 Oct '19

Hello! Next Wednesday 2019-10-16 at 09:00 UTC we will be doing another maintenance operation on some of our cloudvirts servers (the hypervisor servers) that involves rebooting both the physical servers and the virtual machines running on them. The reasons is that we ned to update the running linux kernel version they have. In this window we will reboot 4 hypervisors: * cloudvirt1028 * cloudvirt1029 * cloudvirt1030 The procedure will be to reboot a server, wait for it to come back online (could take up to 5 minutes) and wait for all the VMs to come back online. Then move to the next server. Toolforge users may see their tools and webservices briefly disrupted due to several components of the Toolforge infrastructure being rebooted in this operation. If nothing changes (reallocated or new virtual machine, etc) this is the list of affected VM instances in each hypervisor: * cloudvirt1028: VM: ores-worker-05 PROJECT: ores VM: phamhi-puppetclient PROJECT: testlabs VM: integration-agent-jessie-docker-1001 PROJECT: integration VM: language-cx2 PROJECT: language VM: apertium PROJECT: language VM: janitor01 PROJECT: puppet VM: integration-agent-docker-1008 PROJECT: integration VM: integration-agent-docker-1005 PROJECT: integration VM: xtools-prod06 PROJECT: xtools VM: eventmetrics-prod02 PROJECT: eventmetrics VM: cloud-cumin-02 PROJECT: cloudinfra VM: appslabs PROJECT: mobile VM: meza-2 PROJECT: meza VM: meza-1 PROJECT: meza VM: labs-t224000-alex-osdev PROJECT: openstack VM: clouddb-wikilabels-01 PROJECT: clouddb-services VM: stretch PROJECT: thumbor VM: tools-puppetmaster-01 PROJECT: tools VM: commtech-nsfw PROJECT: commtech VM: builder-envoy PROJECT: packaging VM: cloudstore-dev-01 PROJECT: cloudstore VM: deployment-db05 PROJECT: deployment-prep VM: cloud-puppetmaster-02 PROJECT: cloudinfra VM: osmit-umap PROJECT: osmit VM: util-abogott-stretch PROJECT: testlabs VM: wdhqs-1 PROJECT: wikidata-history-query-service VM: tools-docker-registry-04 PROJECT: tools VM: adhoc-utils01 PROJECT: security-tools VM: tools-proxy-04 PROJECT: tools VM: tools-docker-builder-06 PROJECT: tools VM: tools-sgeexec-0921 PROJECT: tools VM: proxy-02 PROJECT: project-proxy VM: canary1028-01 PROJECT: testlabs VM: vconverter-instance PROJECT: videowiki VM: snuggle-enwiki-01 PROJECT: snuggle VM: saucelabs-02 PROJECT: integration VM: clm-test-01 PROJECT: community-labs-monitoring VM: deployment-memc05 PROJECT: deployment-prep VM: deployment-sca01 PROJECT: deployment-prep VM: af-puppetdb01 PROJECT: automation-framework VM: api PROJECT: openocr VM: a11y PROJECT: reading-web-staging VM: cyberbot-exec-iabot-01 PROJECT: cyberbot VM: fridolin PROJECT: catgraph * cloudvirt1029: VM: clouddb-wikilabels-02 PROJECT: clouddb-services VM: buster PROJECT: thumbor VM: gerrit-test5 PROJECT: git VM: toolsbeta-paws-worker-1001 PROJECT: toolsbeta VM: toolsbeta-flannel-etcd-01 PROJECT: toolsbeta VM: toolsbeta-services-01 PROJECT: toolsbeta VM: tools-worker-1021 PROJECT: tools VM: tools-worker-1015 PROJECT: tools VM: traffic-rpki PROJECT: traffic VM: paws-worker-03 PROJECT: paws VM: paws-worker-02 PROJECT: paws VM: cloud-bootstrapvz-buster-bootstrap PROJECT: openstack VM: structurednavigation PROJECT: structurednavigation VM: tools-worker-1027 PROJECT: tools VM: tools-sgebastion-07 PROJECT: tools VM: wmde-dashboards PROJECT: wmde-dashboards VM: wikidata-shex PROJECT: wikidata-dev VM: tools-proxy-03 PROJECT: tools VM: tools-sge-services-04 PROJECT: tools VM: abogott-proxy-canary PROJECT: testlabs VM: tools-sgewebgrid-lighttpd-0919 PROJECT: tools VM: tools-sgewebgrid-lighttpd-0918 PROJECT: tools VM: tools-sgewebgrid-lighttpd-0916 PROJECT: tools VM: tools-sgewebgrid-lighttpd-0914 PROJECT: tools VM: tools-sgewebgrid-lighttpd-0913 PROJECT: tools VM: tools-sgewebgrid-lighttpd-0912 PROJECT: tools VM: tools-sgewebgrid-lighttpd-0911 PROJECT: tools VM: tools-sgewebgrid-lighttpd-0905 PROJECT: tools VM: tools-sgeexec-0934 PROJECT: tools VM: tools-sgeexec-0932 PROJECT: tools VM: tools-sgeexec-0931 PROJECT: tools VM: tools-sgeexec-0930 PROJECT: tools VM: tools-sgeexec-0928 PROJECT: tools VM: tools-sgeexec-0913 PROJECT: tools VM: tools-sgeexec-0912 PROJECT: tools VM: tools-sgeexec-0907 PROJECT: tools VM: tools-paws-worker-1007 PROJECT: tools VM: canary1029-01 PROJECT: testlabs VM: maps-puppetmaster PROJECT: maps VM: deployment-imagescaler03 PROJECT: deployment-prep VM: integration-slave-jessie-1004 PROJECT: integration * cloudvirt1030: VM: taxonbota-b PROJECT: dwl VM: irc-buster PROJECT: dwl VM: janus1-1 PROJECT: analytics VM: integration-agent-docker-1012 PROJECT: integration VM: integration-agent-docker-1011 PROJECT: integration VM: tools-sgeexec-0929 PROJECT: tools VM: canary1030-01 PROJECT: testlabs VM: t166878 PROJECT: otrs VM: phragile-pro PROJECT: phragile VM: signwriting-swserver PROJECT: signwriting VM: signwriting-swis PROJECT: signwriting VM: commonsarchive-prod PROJECT: commonsarchive VM: math-docker PROJECT: math VM: ldfclient-new PROJECT: wikidata-query VM: wikidata-misc PROJECT: wikidata-dev VM: packaging PROJECT: thumbor VM: neon PROJECT: rcm VM: oxygen PROJECT: rcm VM: hafnium PROJECT: rcm VM: hound-app-01 PROJECT: hound VM: mediawiki2latex PROJECT: collection-alt-renderer VM: deployment-sca02 PROJECT: deployment-prep VM: deployment-memc04 PROJECT: deployment-prep VM: deployment-fluorine02 PROJECT: deployment-prep VM: deployment-mcs01 PROJECT: deployment-prep VM: deployment-parsoid09 PROJECT: deployment-prep VM: deployment-sca04 PROJECT: deployment-prep VM: deployment-kafka-jumbo-2 PROJECT: deployment-prep VM: deployment-kafka-main-1 PROJECT: deployment-prep VM: deployment-mediawiki-09 PROJECT: deployment-prep VM: deployment-webperf12 PROJECT: deployment-prep VM: deployment-deploy02 PROJECT: deployment-prep VM: deployment-deploy01 PROJECT: deployment-prep VM: deployment-maps04 PROJECT: deployment-prep VM: twlight-tracker PROJECT: twl VM: encoding02 PROJECT: video VM: encoding03 PROJECT: video VM: wikispeech-tts-dev PROJECT: wikispeech VM: pub2 PROJECT: wikiapiary VM: integration-slave-jessie-1001 PROJECT: integration VM: ores-staging-01 PROJECT: ores-staging VM: ve-font PROJECT: design VM: visualeditor-test2 PROJECT: visualeditor VM: ores-redis-02 PROJECT: ores VM: quarry-worker-01 PROJECT: quarry VM: fastcci-new-master PROJECT: fastcci VM: cvn-app8 PROJECT: cvn regards. -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 1

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-admin October 2019