If you want to get deep in to user instructions for Discernatron, check out Google's: https://static.googleusercontent.com/media/www.google.com/en//insidesearch/howsearchworks/assets/searchqualityevaluatorguidelines.pdf

It's likely a bit excessive, but there are some interesting examples and their rating scale beginning on page 76.

Rating scale & short descriptions of the scale:

​
Instructions for how to rate ambiguous queries:

Their judgement UI (or one of): 

The map at the top shows where the querying user was located. WMF could do the same to try to better capture the intent of the user. For example a user searching for q={national gallery}, is looking for a different wikipage depending on where they are located. 
​

​Feedback button:

​
Search engine land also has a writeup on Bing's equivalent: http://searchengineland.com/bing-search-quality-rating-guidelines-130592

--justin

On Thu, Apr 21, 2016 at 4:13 PM, Justin Ormont <justin.ormont@gmail.com> wrote:

There's an example of user instructions for a judgement platform on pages 27-27[sic] of these slides: 
http://resources.mpi-inf.mpg.de/d5/teaching/ws14_15/atir/slides/2014-atir-ch09-evaluation.pdf

Judgment instructions generally need a couple of rounds to get right as they are quite task specific. For example, not defining what to do with a disambiguation page will cause some judges to rate a page a "1", and some rate a "3".

--justin