I've started a thread on our "Revision scoring as a service" talk page
regarding labeled conversation datasets & modeling work we could do.
See
On Sun, Nov 15, 2015 at 12:41 PM, Risker <risker.wp(a)gmail.com> wrote:
I am going to quote Joseph Reagle, who responded to a
similarly titled
threat on Wiki-en-L:
date:13 November 2015 at 13:48
It's been great that Riot games has had someone like Lin (an experimental
psychologist) to think about issues of community and abuse. And I
appreciate that Lin has been previously been so forthcoming about their
experiences and findings.
But the much trumpeted League of Legends Tribunal has been down "for
maintenance" for months, even before this article was published, with much
discussion by the community of how it was broken. On this, Riot and Lin
have said nothing.
Copying Joseph in case he wants to respond to some of the discussions here.
Risker/Anne
On 15 November 2015 at 10:36, Pharos <pharosofalexandria(a)gmail.com> wrote:
The figure quoted is quite interesting, but do we
have a comparable
metric
for the Wikimedia projects? :
"... incidences of homophobia, sexism and racism ... have fallen to a
combined
2 percent of all games"
2% sounds "low", but do we indeed know if this is better or worse than
us?
Would our comparable metric be the % of bigoted
comments per article, per
talk page discussion, per time that an editor spends at the project? I
would think that encountering bigoted comments on 1 in 50 discussions
would
still be pretty significant.
Thanks,
Pharos
On Sun, Nov 15, 2015 at 1:21 PM, Ziko van Dijk <zvandijk(a)gmail.com>
wrote:
> Hello,
>
> Just yesterday I had a long talk with a researcher about how to define
> and detect trolls on Wikipedia. E.g., whether "unintentional trolling"
> should be included or not.
>
> In my opinion, it is not possible to detect by machine trollism,
> unkindness, harassment, mobbing etc., maybe with the exception of
> swear words. A lot of turntaking, deviation from the topic and other
> phenomena can be experienced by the participants as positive or as
> negative. You might need to ask them, and even then they might not be
> aware of a problem that works through in subtlety. Also, third persons
> not involved in the conversation can be effected negatively (look at
> ... page X... and you know why you don't like to contribute there).
>
> Kind regards
> Ziko
>
>
> 2015-11-15 17:40 GMT+01:00 Katherine Casey <
fluffernutter.wiki(a)gmail.com
>:
> > I'd be happy to offer my admin/oversighter experience and knowledge
to
> help
> > you develop the labeling and such, Aaron! I just commented on
Andreas's
> > proposal on the Community Wishlist, but
to summarize here: I see a
lot
of
> > potential pitfalls in trying to handle/generalize this with machine
> > learning, but I also see a lot of potential value, and I think it's
> > something we should be investigating.
> >
> > -Fluffernutter
> >
> > On Sun, Nov 15, 2015 at 11:32 AM, Aaron Halfaker <
> ahalfaker(a)wikimedia.org>
> > wrote:
> >
> >> >
> >> > The League of Legends team collaborated with outside scientists to
> >> > analyse their dataset. I would love to see the Wikimedia
Foundation
> engage
> > in a similar research project.
>
>
> Oh! We are! :) When we have time. :\ One of the projects that I'd
like
to
>> see done, but I've struggled to find the time for is a common talk
page
> >> parser[1] that could produce a dataset of talk page interactions.
I'd
like
>> this dataset to be easy to join to editor outcome measures. E.g.
there
> >> might be "aggressive" talk that we don't know is problematic
until
we
> see
> >> the kind of effect that it has on other conversation participants.
> >>
> >> Anyway, I want some powerful utilities and datasets out there to
help
>>
academics look into this problem more easily. For revscoring, I'd
like
> to
> >> be able to take a set of talk page diffs, have them classified in
Wiki
> >> labels[2] as "aggressive"
and the build a model for ORES[3] to be
used
>>
however people see fit. You could then use ORES to do offline
analysis
> of
> >> discussions for research. You could use ORES to interrupt the a
user
> >> before saving a change. I'm
sure there are other clever ideas that
> people
> >> have for what to do with such a model that I'm happy to enable it
via
the
>> service. The hard part is getting a good dataset labeled.
>>
>> If someone wants to invest some time and energy into this, I'm happy
to
> work
with you. We'll need more than programming help. We'll need a
lot of
> help to figure out what dimensions we'll
label talk page postings by
and to
> do the actual labeling.
>
> 1.
https://github.com/Ironholds/talk-parser
> 2.
https://meta.wikimedia.org/wiki/Wiki_labels
> 3.
https://meta.wikimedia.org/wiki/ORES
>
> On Sun, Nov 15, 2015 at 6:56 AM, Andreas Kolbe <jayen466(a)gmail.com>
wrote:
>>
>> > On Sat, Nov 14, 2015 at 9:13 PM, Benjamin Lees <
emufarmers(a)gmail.com>
> >> > wrote:
> >> >
> >> > > This article highlights the happier side of things, but it
appears
> >> > > that Lin's approach
also involved completely removing bad
actors:
>>
> > "Some players have also asked why we've taken such an aggressive
>> > > stance when we've been focused on reform; well, the key here is
that
> >> > > for most players, reform approaches are quite effective. But,
for
a
>> > > number of players, reform
attempts have been very unsuccessful
which
>> > > forces us to remove some of
these players from League
entirely."[0]
>> > >
>> >
>> >
>> > Thanks for the added context, Benjamin. Of course, banning bad
actors
> >> that
> >> > they consider unreformable is something Wikipedia admins have
always
done
>> > as well.
>> >
>> > The League of Legends team began by building a dataset of
interactions
> >> that
> >> > the community considered unacceptable, and then applied
> machine-learning
> >> to
> >> > that dataset.
> >> >
> >> > It occurs to me that the English Wikipedia has ready access to
such
a
> >> > dataset: it's the totality of revision-deleted and oversighted
talk
> page
> >> > posts. The League of Legends team collaborated with outside
> scientists to
> >> > analyse their dataset. I would love to see the Wikimedia
Foundation
> engage
> > in a similar research project.
> >
> > I've added this point to the community wishlist survey:
> >
> >
> >
>
https://meta.wikimedia.org/wiki/2015_Community_Wishlist_Survey#Machine-lear…
> >> >
> >> >
> >> >
> >> > > P.S. As Rupert noted, over 90% of LoL players are male (how much
> over
> >> > > 90%?).[1] It would be interesting to know whether this
percentage
has
>> > > changed along with the improvements described in the article.
>> > >
>> >
>> >
>> > Indeed.
>> > _______________________________________________
>> > Wikimedia-l mailing list, guidelines at:
>> >
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
>> > Wikimedia-l(a)lists.wikimedia.org
>> > Unsubscribe:
https://lists.wikimedia.org/mailman/listinfo/wikimedia-l
,
>> > <mailto:wikimedia-l-request@lists.wikimedia.org
?subject=unsubscribe>
> >> >
> >> _______________________________________________
> >> Wikimedia-l mailing list, guidelines at:
> >>
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
> >> Wikimedia-l(a)lists.wikimedia.org
> >> Unsubscribe:
https://lists.wikimedia.org/mailman/listinfo/wikimedia-l
,
> >> <mailto:wikimedia-l-request@lists.wikimedia.org
?subject=unsubscribe>
> >>
> >
> >
> >
> > --
> > Karen Brown
> > user:Fluffernutter
> >
> > *Unless otherwise specified, any email sent from this address is in
my
> > volunteer capacity and does not
represent the views or wishes of the
> > Wikimedia Foundation*
> > _______________________________________________
> > Wikimedia-l mailing list, guidelines at:
>
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
> > Wikimedia-l(a)lists.wikimedia.org
> > Unsubscribe:
https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
<mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe>
_______________________________________________
Wikimedia-l mailing list, guidelines at:
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l(a)lists.wikimedia.org
Unsubscribe:
https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
<mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe>
_______________________________________________
Wikimedia-l mailing list, guidelines at:
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l(a)lists.wikimedia.org
Unsubscribe:
https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
<mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe>
_______________________________________________
Wikimedia-l mailing list, guidelines at:
https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines
Wikimedia-l(a)lists.wikimedia.org
Unsubscribe:
https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
<mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe>