Forwarding this email, which accidentally went to the internal Discovery list. We decided on "Discernatron". :-)
Dan
---------- Forwarded message ---------- From: Erik Bernhardson ebernhardson@wikimedia.org Date: 21 April 2016 at 14:18 Subject: Re: [discovery-private] Fwd: [discovery] Lets play Name That Thing! To: Internal communications for WMF search and discovery team < discovery-private@lists.wikimedia.org>
Seems like Discernatron is the winner! I've create the repo at https://gerrit.wikimedia.org/r/wikimedia/discovery/discernatron and pushed the current state of the code there. I'll update any references in the code base and get a new version (with lots of other small updates i made yesterday as well) up sometime today.
We also have a filtered set of queries that me and Trey agreeded on, and OAuth credentials to use with meta.mediawiki.org for logins. One of the last sticking points for getting this pushed out is coming up with good instructions for users so they give us good information. Still debating :S I'll initially put this up today or tomorrow with a dozen queries we use for testing and ask people to try it out and let me know what can be fixed for a roll out + small announcement next week.
There's an example of user instructions for a judgement platform on pages 27-27[sic] of these slides: http://resources.mpi-inf.mpg.de/d5/teaching/ws14_15/atir/slides/2014-atir-ch...
Judgment instructions generally need a couple of rounds to get right as they are quite task specific. For example, not defining what to do with a disambiguation page will cause some judges to rate a page a "1", and some rate a "3".
--justin
On Thu, Apr 21, 2016 at 3:36 PM, Dan Garry dgarry@wikimedia.org wrote:
d on, and OAuth credentials to use with meta.mediawiki.org for logins. One of the last sticking points for getting this pushed out is coming u
If you want to get deep in to user instructions for Discernatron, check out Google's: https://static.googleusercontent.com/media/www.google.com/en//insidesearch/h...
It's likely a bit excessive, but there are some interesting examples and their rating scale beginning on page 76.
Rating scale & short descriptions of the scale:
Instructions for how to rate ambiguous queries:
Their judgement UI (or one of):
Map at the top shows where the querying user was located. WMF could do the same to try to better capture the intent of the user. For example a user searching for q={national gallery}, is looking for a different wikipage https://en.wikipedia.org/wiki/National_Gallery_(disambiguation) depending on where they are located.
Feedback button:
Search engine land also has a writeup on Bing's: http://searchengineland.com/bing-search-quality-rating-guidelines-130592
--justin
On Thu, Apr 21, 2016 at 4:13 PM, Justin Ormont justin.ormont@gmail.com wrote:
There's an example of user instructions for a judgement platform on pages 27-27[sic] of these slides:
http://resources.mpi-inf.mpg.de/d5/teaching/ws14_15/atir/slides/2014-atir-ch...
Judgment instructions generally need a couple of rounds to get right as they are quite task specific. For example, not defining what to do with a disambiguation page will cause some judges to rate a page a "1", and some rate a "3".
--justin
On Thu, Apr 21, 2016 at 3:36 PM, Dan Garry dgarry@wikimedia.org wrote:
d on, and OAuth credentials to use with meta.mediawiki.org for logins. One of the last sticking points for getting this pushed out is coming u
If you want to get deep in to user instructions for Discernatron, check out Google's: https://static.googleusercontent.com/media/www.google.com/en//insidesearch/h...
It's likely a bit excessive, but there are some interesting examples and their rating scale beginning on page 76.
Rating scale & short descriptions of the scale:
Instructions for how to rate ambiguous queries:
Their judgement UI (or one of):
The map at the top shows where the querying user was located. WMF could do the same to try to better capture the intent of the user. For example a user searching for q={national gallery}, is looking for a different wikipage https://en.wikipedia.org/wiki/National_Gallery_(disambiguation) depending on where they are located.
Feedback button:
Search engine land also has a writeup on Bing's equivalent: http://searchengineland.com/bing-search-quality-rating-guidelines-130592
--justin
On Thu, Apr 21, 2016 at 4:13 PM, Justin Ormont justin.ormont@gmail.com wrote: There's an example of user instructions for a judgement platform on pages 27-27[sic] of these slides: http://resources.mpi-inf.mpg.de/d5/teaching/ws14_15/atir/slides/2014-atir-ch...
Judgment instructions generally need a couple of rounds to get right as they are quite task specific. For example, not defining what to do with a disambiguation page will cause some judges to rate a page a "1", and some rate a "3".
--justin
There's also a great presentation by a Google Relevance guy: https://builtvisible.com/how-google-works/
Judgement platform is halfway down.
Video of the talk: https://www.youtube.com/watch?v=iJPu4vHETXw (video is good; and a great way to spend 30min of a Friday afternoon)
--justin
On Fri, Apr 29, 2016 at 2:22 PM, Justin Ormont justin.ormont@gmail.com wrote:
If you want to get deep in to user instructions for Discernatron, check out Google's: https://static.googleusercontent.com/media/www.google.com/en//insidesearch/h...
It's likely a bit excessive, but there are some interesting examples and their rating scale beginning on page 76.
Rating scale & short descriptions of the scale:
Instructions for how to rate ambiguous queries:
Their judgement UI (or one of):
The map at the top shows where the querying user was located. WMF could do the same to try to better capture the intent of the user. For example a user searching for q={national gallery}, is looking for a different wikipage https://en.wikipedia.org/wiki/National_Gallery_(disambiguation) depending on where they are located.
Feedback button:
Search engine land also has a writeup on Bing's equivalent: http://searchengineland.com/bing-search-quality-rating-guidelines-130592
--justin
On Thu, Apr 21, 2016 at 4:13 PM, Justin Ormont justin.ormont@gmail.com
wrote: There's an example of user instructions for a judgement platform on pages 27-27[sic] of these slides:
http://resources.mpi-inf.mpg.de/d5/teaching/ws14_15/atir/slides/2014-atir-ch...
Judgment instructions generally need a couple of rounds to get right as they are quite task specific. For example, not defining what to do with a disambiguation page will cause some judges to rate a page a "1", and some rate a "3".
--justin
