Hi Leila,
I did something similar before. I was trying to create "top-level" category
labels for the articles, like history, society, technology, etc. I parsed
the wikitext in dump data to extract all the sub category labels of the
article. Also, by parsing pages of namespace 14, I created a
category-relation graph for all the category labels, where ideally, each
sub category can reach some "top-level" category. Then, for each article,
you can take the sub category label into the graph for the top-level
categories. More detail can be found in 3.3.2 Independent Variables -
Identity-based Attachment subsection in the paper. Hope it helps!
On Mon, Jul 10, 2017 at 8:45 PM, Stuart A. Yeates <syeates(a)gmail.com> wrote:
The category system on en.wiki is not an IS-A system
and there have been
several discussions about making it it based on mathematical principals
which have come to nothing because the consensus of editors is against it.
The best way to think about categories is as a locally-faceted related
links system.
Having said that, Category:Wikipedia maintenance is an important root
probably useful for separating the wheat from the chaff. Most of these are
also hidden categories. I'm not sure whether this flag appears in the SQL,
but see
https://en.wikipedia.org/wiki/Wikipedia:Categorization#Hiding_categories
cheers
stuart
--
...let us be heard from red core to black sky
On 11 July 2017 at 13:20, Leila Zia <leila(a)wikimedia.org> wrote:
Hi all,
[If you are not interested in discussions related to the category system
(on English Wikipedia)
, you can stop here. :)]
We have run into a problem that some of you may have thought about or
addressed before. We are trying to clean up the category system on
English
Wikipedia by turning the category structure to an
IS-A hierarchy. (The
output of this work can be useful for the research on template
recommendation [1], for example, but the use-cases won't stop there). One
issue that we are facing is the following:
We are currently
using
SQL dumps to extract categories associated with every article on English
Wikipedia (main namespace). [2]
Using this approach, we get 5 categories associated with Flow cytometry
bioinformatics article [3]:
Flow_cytometry
Bioinformatics
Wikipedia_articles_published_in_peer-reviewed_literature
Wikipedia_articles_published_in_PLOS_Computational_Biology
CS1_maint:_Multiple_names:_authors_list
The problem is that only the first two categories are the ones we are
interested in. We have one cleaning step through which we only keep
categories that belong to category Article and that step removes the last
category above, but the other two Wikipedia_... remain there. We need to
somehow prune the data and clean it from those two categories.
One way we could do the above would be to parse wikitext instead of the
SQL
dumps and focus on extracting categories marked
by pattern
[[Category:XX]],
but in that case, we would lose a good category
such as
Guided_missiles_of_Norway
because that's generated by a template.
Any ideas on how we can start with a "cleaner" dataset of categories
related to the topic of the articles as opposed to maintenance related or
other types of categories?
Thanks,
Leila
[1]
https://meta.wikimedia.org/wiki/Research:Expanding_Wikipedia
_stubs_across_languages
[2] The exact code we use is
SELECT p.page_id id, p.page_title title, cl.cl_to category
FROM categorylinks cl
JOIN page p
on cl.cl_from = p.page_id
where cl_type = 'page'
and page_namespace = 0
and page_is_redirect = 0
and the edges of the category graph are extracted with
*SELECT p.page_title category, cl.cl_to parent *
*FROM categorylinks cl *
*JOIN page p *
*ON p.page_id = cl.cl_from *
*where p.page_namespace = 14*
[3]
https://en.wikipedia.org/wiki/Flow_cytometry_bioinformatics
_______________________________________________
Wiki-research-l mailing list
Wiki-research-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
_______________________________________________
Wiki-research-l mailing list
Wiki-research-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wiki-research-l