On Thu, Feb 4, 2010 at 8:36 PM, Daniel Friesen lists@nadir-seen-fire.com wrote:
I read a note before that someone caught Microsoft's Bing making a number of requests to their site with a UA that makes it look like ie6 and inflates stats. http://opengameart.org/forumtopic/dear-microsoft
Someone from Microsoft told this guy that they spoof the UA because some sites won't give content to UAs they don't recognise, including the msnbot:
http://eng.genius.com/blog/2010/01/12/user-agent-strings/
All the examples of spoofed IE 6 strings I could find included Windows NT 5.2 (ie, Windows Server 2003) in the UA; this OS is currently 0.64% of requests:
http://stats.wikimedia.org/wikimedia/squids/SquidReportOperatingSystems.htm
Here's someone talking about a Bing bot transcoding mobile sites that also spoofs IE 6:
http://wapreview.com/blog/?p=4127
Note that along with the UA header is a Via header with "1.1 WTRSCP19911". Presumably that's the shorthand for the transcoder service. I wonder if msnbot also gives a Via header along with the spoofed UA?