Hi Steve,
But why?
Because we need to identify malicious behavior.
(This just broke one of my bots.) Are the details of this policy discussed anywhere?
I don't know. Probably. We always told people to specify User-Agent, just the check was broken.
Is it permissible to send
User-Agent: x
thus providing precisely the same amount of information as if not supplying the header at all?
No, you clearly miss very simple idea that with such user-agent you clearly identify yourself as malicious, whereas when you don't specify, you're either malicious or ignorant.
Do note, we're good at detecting spoofed user-agents too, so if your bots disguise as MSIE or Firefox or any other regular browser, your behavior is seen as malicious.
We do not like malicious behavior.
Domas