Hi James, thanks a lot for your interest in our work! The problem of crowdworkers being biased is a problem definitely not to be neglected. Majority vote can help to sort out single extremist views of workers but if many workers are strongly biased then I agree that this might not be enough. We are actually already thinking about methods to improve future crowdsourced bias datasets. One way to improve the quality is to have a very well defined task that leaves only little room for subjective interpretation. For example, instead of letting the workers decide whether a statement is biased or not, we asked more specifically whether the statement reflects an opinion or contains bias words. Of course, the decision if a statement reflects a fact or an opinion is still subjective in many cases. Given your example it is hard to make a decision (even when being unbiased) without having the proper background knowledge. That is why our work mostly focuses on language bias, i.e. bias that is introduced through the use of judgemental language. Since there are many cases of bias without using judgemental language, we are definitely interested to come up with good approaches that cover these cases as well. Ideas and suggestions are always welcome! One other thing that we are planning to do for future crowdsourcing jobs is to ask workers for their political opinions and to take this background information into account when creating ground truth data. Best regards, Christoph Am 4/18/2018 um 2:22 PM schrieb James Salsman: