[Cloud-admin] Toolsdb and linkwatcher

Hi there, Apparently linkwatcher is a key tool in fighting vandalism in the wikis, but we cannot afford hosting it on ToolsDB any longer. I requests your comments / suggestions. Timeline: ==== 8< ==== == 2019-05-22 == Brooke detects high storage usage problem by the linkwatcher tool in ToolsDB [0]. [0] https://phabricator.wikimedia.org/T224154 == 2019-06-26 == Bryan warns [1] about the linkwatcher tool potentially becoming a disaster condition for ToolsDB. [1] https://phabricator.wikimedia.org/T224154#5284580 == 2019-07-19 == After some actions are taken to move linkwatcher to its own Cloud VPS project, activity stops and the database data remains in ToolsDB [2]. [2] https://phabricator.wikimedia.org/T227377 == 2022-02-17 == ToolsDB stops replicating to its secondary due to depleted storage [3]. It is discovered [4] that linkwatcher uses 1/3 of total storage of ToolsDB, this is ~1TB out of ~3TB. Data reported in other metrics [5] may not be accurate. While it may not be the primary cause of the problem, the "disaster condition" mentioned by Bryan 3 years earlier is becoming more apparent. [3] https://phabricator.wikimedia.org/T301951 [4] https://phabricator.wikimedia.org/T301967 [5] https://tool-db-usage.toolforge.org/ ==== 8< ==== regards -- Arturo Borrero Gonzalez Site Reliability Engineer Wikimedia Cloud Services Wikimedia Foundation