Hi Everybody,
I just ran an experiment that surprised me and I thought folks on this list
would find interesting.
*tl;dr* We found that navigation vector embeddings for articles (as
produced by Ellery Wulcyzn
<https://meta.wikimedia.org/wiki/Research:Wikipedia_Navigation_Vectors>)
outperform
content-based vector embeddings (word2vec on article text) by 62% vs 37%
accuracy in a task-based user study. I've volunteered to help with the
engineering to productionize navigation embedding
<https://phabricator.wikimedia.org/T158972> and this study reinforces my
eagerness to get navigation vectors out in the world!
*More detail: *The maps we use in Cartograph (cartograph.info) are almost
entirely built on "embedding" vectors for articles. We experimented with
two word2vec-based embeddings: *content vectors* mined from article text
and link structure, and* navigation vectors* mined from user browsing
sessions. For the latter, we used Ellery Wulczyn's navigation vectors
<https://meta.wikimedia.org/wiki/Research:Wikipedia_Navigation_Vectors>. By
staring at maps, our intuition told us that the navigation vectors seemed
better in "preference spaces" where the human taste space wasn't
necessarily easily encoded into Wikipedia text.
Last weekend we ran a Mechanical Turk experiment to test this intuition. We
created two Cartograph maps of movies: one built on navigation vectors and
one built on content vectors. We identified 40 relatively popular movies
that were not close neighbors in either map (i.e. cities that were not too
close to each other) and ran a Mechanical Turk study using the maps.
For each Turker, we randomly selected 5 seen movies (out of the 30), and
asked them to evaluate maps for each movie. For each movie city, we showed
the map region around the city, but hid the city and asked them to guess
the city from a list of 12 movies they had seen (screenshot below). We
added in trivial validation questions using sequels to ensure Turkers were
working in good faith (show a map for "Rocky II" that had "Rocky" at
the
center).
Result: Turkers exhibited 62% accuracy with the navigation vectors and 37%
accuracy with content vectors. We want to conduct several follow-up studies
to understand different subject areas and parameter settings and user
tasks, but the difference in performance was striking.
Our study shows the value of navigation vectors and makes me super excited
to contribute to the engineering needed to get them out to the world on a
regular basis. Imagine if every researcher and practitioner who uses
word2vec now on Wikipedia content switches to navigation vectors. That's a
huge audience!
Feedback and questions welcome!
-Shilad
--
Shilad W. Sen
Associate Professor
Mathematics, Statistics, and Computer Science Dept.
Macalester College
Senior Research Fellow, Target Corporation
ssen(a)macalester.edu
http://www.shilad.com
https://www.linkedin.com/in/shilad
651-696-6273
