Hello,
some spammer was (or still is) sending spam with faked @wikipedia.org sender addresses. The bounces for this spam were sent back to our mail gateways, overloading them yesterday and today.
We changed the setup of the backup MX. It now knows about the existing mailboxes and rejects mail to unknown recipients directly. In the past, it accepted any mail for wikipedia.org or wikimedia.org. With this change, the load on the primary mail server has gone down dramatically.
At noon, our primary MX goeje was handling 200 concurrent mail connections, this was it's hard limit. After setting the limit to 500, 500 connections were established, but the box started heavy swapping.
The secondary MX has a relay_recipient_maps list configured, which is updated every 15 minutes from goeje. If a new mailbox or mailing list is set up, it takes up to 15 minutes until this mailbox is accessible via the backup MX.
Regards,
jens