I just wanted to give folks a heads up that in response to a few traffic storms in the Beta Cluster (deployment-prep CLoud VPS project) we have started using the very coarse protection of blocking IP ranges. These blocks are being applied at the Beta Cluster CDN edge where we have Varnish configuration that can discard traffic based on a list of CIDR ranges.

The ranges blocked at any point in time should be visible in the deployment-prep project's Hiera configuration that is logged in the cloud/instance-puppet.git repo. [0]

The hardly scientific process of choosing what to block so far has been done with processes like the one documented at https://phabricator.wikimedia.org/T392003. Hashar came up with a shell one-liner to count requests by IP address or IP address prefix depending on the regex provided. We then take the top addresses produced by that log filtering and perform a `whois` lookup to find the associated IP address allocation. The CIDR blocks associated with the allocation are then put into hiera config, a Puppet run is forced, and Varnish is restarted. Repeat as necessary to get to a reasonable rate of requests passing through Varnish to the backing MediaWiki instances where we are examining the logs.

If you feel that you have legitimate traffic for the Beta Cluster to handle that has gotten swept up in one of these blocks, please reach out by filing task on the #beta-cluster-infrastructure Phabricator board. [1]

If you think working to make this process of blocking easier or unnecessary sounds like a fun project I would love to chat more. Hit me up via email, libera.chat irc, or on-wiki with your ideas.

[0]: https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/refs/... [1]: https://phabricator.wikimedia.org/tag/beta-cluster-infrastructure/

Bryan