Society's version of WP:V, basically. Our culture accepts distinguishing groups of people based on physically verifiable characteristics; but psychological characteristics are much harder to verify conclusively, so we as a society tend to reject acknowledgement of those differences between groups of people.
I'm not sure how true that really is. Sure, it's easier to observe someone's height than it is to observe their tendency to go off-topic (incidentally, I'm not convinced that is a particularly female tendency), but the difficult part with getting accurate data on either characteristic is getting a large enough sample. People often base generalisations on anecdotal evidence, which is generally not representative. I would imagine generalisations about easily observed characteristics are more likely to based on anecdotal evidence than ones about more difficult to observe characteristics, which are likely to be based on formal studies, since that's the only way of establishing it (of course, this generalisation is, itself, a counter-example...).