There's also a great presentation by a Google Relevance guy: https://builtvisible.com/how-google-works/
Judgement platform is halfway down.
Video of the talk: https://www.youtube.com/watch?v=iJPu4vHETXw (video is good; and a great way to spend 30min of a Friday afternoon)
--justin
On Fri, Apr 29, 2016 at 2:22 PM, Justin Ormont justin.ormont@gmail.com wrote:
If you want to get deep in to user instructions for Discernatron, check out Google's: https://static.googleusercontent.com/media/www.google.com/en//insidesearch/h...
It's likely a bit excessive, but there are some interesting examples and their rating scale beginning on page 76.
Rating scale & short descriptions of the scale:
Instructions for how to rate ambiguous queries:
Their judgement UI (or one of):
The map at the top shows where the querying user was located. WMF could do the same to try to better capture the intent of the user. For example a user searching for q={national gallery}, is looking for a different wikipage https://en.wikipedia.org/wiki/National_Gallery_(disambiguation) depending on where they are located.
Feedback button:
Search engine land also has a writeup on Bing's equivalent: http://searchengineland.com/bing-search-quality-rating-guidelines-130592
--justin
On Thu, Apr 21, 2016 at 4:13 PM, Justin Ormont justin.ormont@gmail.com
wrote: There's an example of user instructions for a judgement platform on pages 27-27[sic] of these slides:
http://resources.mpi-inf.mpg.de/d5/teaching/ws14_15/atir/slides/2014-atir-ch...
Judgment instructions generally need a couple of rounds to get right as they are quite task specific. For example, not defining what to do with a disambiguation page will cause some judges to rate a page a "1", and some rate a "3".
--justin