- Cloud-admin - lists.wikimedia.org

Toolforge: new domain toolforge.org
by Arturo Borrero Gonzalez 07 Jul '20

07 Jul '20

Hi! We are happy to announce the new domain 'toolforge.org' is now ready to be adopted by our Toolforge community. There is a lot of information related to this change in a wikitech page we have for this: https://wikitech.wikimedia.org/wiki/News/Toolforge.org The most important change you will see happening is a new domain/scheme for Toolforge-hosted webservices: * from https://tools.wmflabs.org/<toolname>/ * to https://<toolname>.toolforge.org/ A live example of this change can be found in our internal openstack-browser webservice tool: * legacy URL: https://tools.wmflabs.org/openstack-browser/ * new URL: https://openstack-browser.toolforge.org This domain change is something we have been working on for months previous to this announcement. Part of our work has been to ensure we have a smooth transition from the old domain (and URL scheme) to the new canonical one. However, we acknowledge the ride might be bumpy for some folks, due to technical challenges or cases we didn't consider when planning this migration. Please reach out intermediately if you find any limitation or failure anywhere related to this change. The wikitech page also contains a section with information for common problems. You can check now if your webservice needs any specific change by creating a temporal redirection to the new canonical URL: $ webservice --canonical --backend=kubernetes start [..] $ webservice --canonical --backend=gridengine start [..] The --canonical switch will create a temporal redirect that you can turn on/off. Please use this to check how your webservice behaves with the new domain/URL scheme. If you start the webservice without --canonical, the temporal redirect will be removed. We aim to introduce permanent redirects for the legacy URLs on 2020-06-15. We expect to keep serving legacy URLs forever, by means of redirections to the new URLs. More information on the redirections can also be found in the wikitech page. The toolforge.org domain is finally here! <3 -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

4 8

Toolforge email server now enforcing ratelimiting
by Arturo Borrero Gonzalez 24 Jun '20

24 Jun '20

Hi, we just enabled email ratelimiting in our MTA server [0] in Toolforge. Please, report any problem or issue you may find related to this. The current limit is 100 messages per hour per sender address. We may tune the value as we observe the behavior of the system and the users. regards. [0] https://en.wikipedia.org/wiki/Message_transfer_agent -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 0

Toolforge grid now using tesseract-ocr 4.1.1
by Arturo Borrero Gonzalez 20 May '20

20 May '20

Hi there! We just deployed tesseract-ocr v4.1.1 in the Toolforge grid. The context of this update is the phabricator task T247422 [0]. Please report any issue you may find. regards! [0] https://phabricator.wikimedia.org/T247422 -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 0

Changes to CloudVPS web proxy (XFF) on 2020-04-15
by Arturo Borrero Gonzalez 15 Apr '20

15 Apr '20

Hi there! If you use a CloudVPS web proxy, this email is for you. Toolforge developers/users can ignore this email. We are introducing a change to eliminate the 'X-Forwarded-For' HTTP header that the CloudVPS web proxy adds when forwarding the HTTP request to your instance. This header contains the original IP address of the internet client that sent the request. This is private information that we would like to reduce in our environment [0]. You use the web proxy if you have a public web endpoint hosted in CloudVPS under the wmflabs.org domain. These are generally configured using Horizon in the DNS > Web Proxies section. Examples of web proxy names: * accounts.wmflabs.org * glampipe.wmflabs.org * incubator.wmflabs.org Full list can be seen in the Openstack Browser tool [1]. We are ready to introduce this change [2], but wanted to give some heads up for projects that do require this information for whatever reason. We would like to hear from you in the next couple of weeks. Please contact us in the phabricator task [0] and include some rationale why you need the XFF header. This is the timeline this change will follow: * 2020-04-01: this email, start collecting list of things that require XFF * 2020-04-07: start evaluating list of things that require XFF * 2020-04-15: introduce the change, with proper case whitelisting When the change is introduced, in two weeks from now, proxy backends that were not whitelisted will stop receiving the XFF header. Please reach out for any questions or comments. regards. [0] https://phabricator.wikimedia.org/T135046 [1] https://openstack-browser.toolforge.org/project/project-proxy [2] https://gerrit.wikimedia.org/r/c/operations/puppet/+/583098 -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 1

CloudVPS network change (routing source IP) on 2020-04-13
by Arturo Borrero Gonzalez 09 Apr '20

09 Apr '20

Hi there! In a few days from now (2020-04-13), the CloudVPS network will see a change happening that will likely go unnoticed, but it is important enough to share it with you beforehand. We will be changing the IPv4 address that we use as the main source NAT for egress connections (initiated in the VM instances). This change won't affect VM instances using floating IPs. Old IP address: 185.15.56.1 New IP address: 208.80.155.92 If you know of anywhere (a firewall, ACL or any other mechanism) that had this address hardcoded, you will need to update it. See this wikitech page for more details: https://wikitech.wikimedia.org/wiki/News/CloudVPS_NAT_change Please reach out if you have any doubts, questions, or any other issue. regards. -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

1 1

Nomination of Sam Reed (Reedy) for cloudroot/cloudadmin rights
by Bryan Davis 09 Apr '20

09 Apr '20

I created <https://phabricator.wikimedia.org/T249774> to nominate Reedy for basically the same rights as I hold in the Cloud Services environment. This nomination comes out of some conversations with Reedy and my selfish desire to be able to nerd snipe him into helping with more things. Per <https://wikitech.wikimedia.org/wiki/Help:Access_policies> I am announcing the nomination here. I will also follow up with the policy mandated inquiry to Trust & Safety. The archives of this list are public, so remember that if you choose to respond directly to this message with your endorsement or concerns. Private email to a WMCS team member, such as Brooke Storm since I am the nominating party and may be seen as biased, is potentially the best method of ensuring that concerns are heard during the 7 day comment period. Bryan -- Bryan Davis Technical Engagement Wikimedia Foundation Principal Software Engineer Boise, ID USA [[m:User:BDavis_(WMF)]] irc: bd808

1 0

Re: [Cloud-admin] Update 1 labsdb host to buster and 10.4
by Manuel Arostegui 02 Apr '20

02 Apr '20

Thanks Bryan - sorry for not answering faster, but looks like you only replied to cloud-admin and I am not there :-) Today in our 1:1 this subject came up and Jaime forwarded me the mail, as he is in cloud-admin hehe. Answers in line! On Mon, Mar 30, 2020 at 1:02 PM Jaime Crespo <jcrespo(a)wikimedia.org> wrote: > ---------- Forwarded message --------- > From: Bryan Davis <bd808(a)wikimedia.org> > Date: Tue, Mar 24, 2020 at 3:56 PM > Subject: Re: [Cloud-admin] Update 1 labsdb host to buster and 10.4 > To: Cloud Services administration and infrastructure discussion > <cloud-admin(a)lists.wikimedia.org> > > > On Tue, Mar 24, 2020 at 2:36 AM Manuel Arostegui > <marostegui(a)wikimedia.org> wrote: > > > > So far we have had normal 1 instance hosts upgraded, multi-instance (2 > mysqld processes) upgraded, and we need to have a multisource (labsdb) host > upgrade, to make sure 10.4 works fine or to know what might need work > (mysqld-exporter https://phabricator.wikimedia.org/T247290 or whatever), > better to know in advance. > > > > 10.4 also fixes some bugs that are hitting labsdb hosts specifically: > > - Grants race condition: https://jira.mariadb.org/browse/MDEV-14732 > > - GTID works on multisource: https://jira.mariadb.org/browse/MDEV-12012 > this is one of the early bugs we filed with MariaDB almost 3 years ago and > looks like it is now working even though - this requires some work on the > master's side, but my last tests are looking good and if we can enable GTID > on labsdb hosts that'd we be a BIG improvement towards avoiding corruption > during a crash. > > These all sound like good things. And thank you very much, seriously, > for the effort you have been putting into thinking about and caring > for the wiki replicas. > You are welcome! :-) > > > So, any objections to reimage labsdb1011 as Buster and 10.4 (/srv won't > be formatted, so we don't have to rebuild that host). > > Any idea what the roll back plan would look like if it turns out that > something about 10.4 and multisource do not work well together? Would > it be less risky to do labsdb1012 first and see how it works there? > The rollback plan is basically, reimage back to Stretch and reclone from labsdb1012. The idea to use labsdb1011 is to actually test 10.4 in this very unique environment (lots of heavy queries). Labsdb1012 is barely used, and only a few days during the month, so it wouldn't be representative. Also, the rollback is easier as we can use labsdb1012, as it is normally fine to stop it for 24h (as long as it is not during the few days it is used at the start of each month), so no user impact there. Whereas, stopping a wiki replica, means we have to put more pressure on the other 2 hosts for the time it is stopped. Does this make sense and answer your question? Thank you! Manuel.

2 2

Update 1 labsdb host to buster and 10.4
by Manuel Arostegui 25 Mar '20

25 Mar '20

Hello! As I have commented on some tickets, MariaDB 10.1 will reach its end-of-life 17th October 2020 ( https://mariadb.com/wp-content/uploads/2019/07/mariadb-engineering-policies… ). Debian Buster doesn't ship 10.1, but 10.4. We have been testing 10.4 in production for quite some time already ( https://phabricator.wikimedia.org/T242702 https://phabricator.wikimedia.org/T246604) We have filed some bugs, but nothing super worrying have been observed in production (bugs: https://jira.mariadb.org/browse/MDEV-21794 https://jira.mariadb.org/browse/MDEV-21813 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=953040) A few weeks ago, during some Wikidata overloads, we observed that the host with Buster and 10.4 performed better, CPU wise, than the ones running 10.1 and Stretch with InnoDB compression, having the same hardware. We haven't found 100% why, but we believe that 10.4 might have some optimizations. Given that there've been some Quarry reports about being slow ( https://phabricator.wikimedia.org/T246970 https://phabricator.wikimedia.org/T247978) Considering the fact that Quarry points to labsdb1011, and in order to discard a specific issue with that host we thought about replacing labsdb1011 with another host ( https://phabricator.wikimedia.org/T247978#5980018) but maybe we can just try to upgrade labsdb1011 to buster and 10.4 So far we have had normal 1 instance hosts upgraded, multi-instance (2 mysqld processes) upgraded, and we need to have a multisource (labsdb) host upgrade, to make sure 10.4 works fine or to know what might need work (mysqld-exporter https://phabricator.wikimedia.org/T247290 or whatever), better to know in advance. 10.4 also fixes some bugs that are hitting labsdb hosts specifically: - Grants race condition: https://jira.mariadb.org/browse/MDEV-14732 - GTID works on multisource: https://jira.mariadb.org/browse/MDEV-12012 this is one of the early bugs we filed with MariaDB almost 3 years ago and looks like it is now working even though - this requires some work on the master's side, but my last tests are looking good and if we can enable GTID on labsdb hosts that'd we be a BIG improvement towards avoiding corruption during a crash. So, any objections to reimage labsdb1011 as Buster and 10.4 (/srv won't be formatted, so we don't have to rebuild that host). Cheers Manuel.

2 1

Nova database in m5
by Manuel Arostegui 24 Mar '20

24 Mar '20

Hello! While working on m5 master I realised that the nova database has not had any writes for a year or so: root@db1133:/srv/sqldata/nova# ls -lhrt *.ibd | tail -n5 -rw-rw---- 1 mysql mysql 48M Apr 16 2019 instance_metadata.ibd -rw-rw---- 1 mysql mysql 2.9G Apr 16 2019 instance_system_metadata.ibd -rw-rw---- 1 mysql mysql 484M Apr 16 2019 block_device_mapping.ibd -rw-rw---- 1 mysql mysql 304K Apr 17 2019 compute_nodes.ibd -rw-rw---- 1 mysql mysql 128K Apr 26 2019 services.ibd Same for nova_api root@db1133:/srv/sqldata# ls -lhrt nova_api/*.ibd | tail -n5 -rw-rw---- 1 mysql mysql 128K May 23 2018 nova_api/flavor_extra_specs.ibd -rw-rw---- 1 mysql mysql 128K May 23 2018 nova_api/cell_mappings.ibd -rw-rw---- 1 mysql mysql 96K May 23 2018 nova_api/migrate_version.ibd -rw-rw---- 1 mysql mysql 176K Apr 16 2019 nova_api/build_requests.ibd -rw-rw---- 1 mysql mysql 364M Apr 16 2019 nova_api/request_specs.ibd Can those be removed? Thanks Manuel

2 2

IPv6 allocation for codfw1dev
by Arturo Borrero Gonzalez 09 Mar '20

09 Mar '20

Hi Arzhel, now that our BGP thing is mostly working, I think we can move forward with IPv6 for codfw1dev, which was my original intention before doing anything related to BGP :-P We would need to: * allocate a small IPv6 prefix for the transport network (between cr-codfw and the neutron virtual router). I'm open to suggestion to the size of this subnet. * allocate a big IPv6 prefix that we could use internally inside the cloud environment (for VMs). This should probably be a /48? I'm not sure yet if I would be doing subnetting of this big prefix and allocate smaller ones (/64) per project, or if we will be using the big one directly without sub-delegation. (Something to research anyway). I will follow-up next week. For the record, doing this research and design is one my goals/OKRs for this quarter (3 week left!). My plan is to use this wikitech page as my deliverable: https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/Enhanceme… regards. -- Arturo Borrero Gonzalez SRE / Wikimedia Cloud Services Wikimedia Foundation

2 2

2024

2023

2022

2021

2020

2019

2018

2017

Cloud-admin