On Tue, Aug 25, 2015 at 7:58 AM, Oliver Keyes <okeyes(a)wikimedia.org> wrote:
So it's a comparison of two search systems,
neither of which we use?
Well, sure... but they describe an interesting search paradigm that I don't
think we've even been considering (in the available paper). It's not the
type of query-by-example I'm used to seeing.
They intercept requests for wiki pages and convert infoboxes into
structured query forms that allow some basic boolean syntax. It converts
these queries into SPARQL and hits DBpedia to get results. Sounds
reasonable.
They do mention briefly in section 3.1 (last paragraph) that they basically
need a custom ("page-dependent") mapping from any given infobox to
appropriate internal representations for mapping to SPARQL. There are some
obvious machine learning approaches to try there. Since they don't mention
any machine learning, I assume they have done them manually, which may or
may not scale, depending on how many queries of the sort they are
interested in are covered by *n* manually mapped infobox types. Either way,
it's potentially brittle, since the Wikipedians tending the infoboxes won't
know about SWIPE.
As for the comparison to Xser (which I'm not familiar with, though it's
described here:
http://ceur-ws.org/Vol-1180/CLEF2014wn-QA-XuEt2014.pdf )
and plain keyword searches in Wikipedia, I'd really need to see the full
paper to comment properly, but I have some questions (which they may well
answer in the paper).
Plain keyword searches in Wikipedia are a fine baseline, though I wonder if
they preprocessed the natural language queries, or just tossed the whole
question into search (which it is not meant to handle, though it often
works anyway). And I don't know what counts as success—one of the first *n*
results contains the answer? How hard would a human have to look on a page
for the answer?
It seems that the SWIPE system requires a human to translate the query into
the infobox template (and know which template to use!). So, for the query
"who has Tom Cruise been married to?" (from the Xser paper), it seems the
user has to convert "married to" into the "spouse(s)" field of the
person
infobox—which is pushing the NLP processing into the human (of which I am a
fan, though it is not automatic).
I'm not liking that they claim 96% recall "among all answered
questions"—you don't get to ignore the ones you failed to answer when
calculating recall! 100% precision is nice.
Xser seems more like the NLP system I would have first imagined—parse a
query, convert it into a structured format, and hit the RDF store for
answers. SWIPE seems to get the human to do the hard parts (parsing and
converting to a structured format, with the help of existing infoboxes), so
of course it does better than Xser.
So what do we get out of this? If you haven't already thought of WDQS, then
you weren't paying attention! We could make things easier (for us, for
SWIPE, for anyone), if we could develop a standard way to map infobox
template fields to WDQS properties and contents to entities (someone
must've thought of this already).
Parsing the content of those fields (if you know what they are supposed to
contain) is easier than parsing random queries or other chunks of text.
That info could be used to automatically or semi-automatically populate
WDQS, or to refer WDQS results back to relevant Wiki pages, or turn
templates into query forms as SWIPE does.
Whether any of this gets onto our roadmap this century is a different
question, but there are some interesting things to think about here.
So, can anyone get me a copy of the full paper?
Thanks for the pointer, Tilman!
—Trey
Trey Jones
Software Engineer, Discovery
Wikimedia Foundation
On 25 August 2015 at 10:54, Tilman Bayer
<tbayer(a)wikimedia.org> wrote:
FYI just in case it's of interest and
hasn't shown up on the team's
radar yet:
http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=7194368 -
paywalled, unfortunately.
Quote from the abstract:
"This paper discusses expressivity and accuracy of the By-Example
Structured (BESt) Query paradigm implemented on the SWiPE system
through the Wikipedia interface. We define an experimental setting
based on the natural language questions made available by the QALD-4
challenge, in which we compare SWiPE against Xser, a state-of-the-art
Question Answering system, and plain keyword search provided by the
Wikipedia Search Engine. The experiments show that SWiPE outperforms
the results provided by Wikipedia, and it also performs sensibly
better than Xser, obtaining an overall 85% of totally correct answers
vs. 68% of Xser."
(For context, there's an earlier paper where they describe an earlier
version of that SWiPE - "Search Wikipedia by example" - project:
http://web.cs.ucla.edu/~zaniolo/papers/AtzoriZ12 )
--
Tilman Bayer
Senior Analyst
Wikimedia Foundation
IRC (Freenode): HaeB
_______________________________________________
Wikimedia-search mailing list
Wikimedia-search(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimedia-search
--
Oliver Keyes
Count Logula
Wikimedia Foundation
_______________________________________________
Wikimedia-search mailing list
Wikimedia-search(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimedia-search