Modelling user behaviour on Wikipedia

List overview All Threads
Download

newer

older

U.S. open access policy comment...

Tech Talk livestream chat...

Kiril Simeonovski

24 Feb 2020 24 Feb '20

5:59 p.m.

Hi all, I am currently working on a research concerned with modelling user behaviour on Wikipedia. The idea is to design a field experiment over a random sample of Wikipedians in order to examine their risk preferences and define (dis)utilities that will be used in a utility-maximisation model. I have already submitted an abstract that got accepted for the biennial Foundations of Utility and Risk Conference 2020 <https://www.furconference.org/> and my future plans include presentation of the concept at other research conferences (including Wikimania 2020). You can visit the project page <https://meta.wikimedia.org/wiki/Research:Modelling_Behaviour_in_a_Peer_Production_Economy_upon_Evidence_from_Wikipedia> of this research on Meta. Your questions and comments are welcome at any time. Thank you! Best regards, Kiril

Show replies by thread

Stuart A. Yeates

24 Feb 24 Feb

7:35 p.m.

When you say "participant", "user" and "editor" do you actually mean account? I routinely notice what appear to be people attending real-file events using one account but then editing afterwards with a different account. cheers stuart -- ...let us be heard from red core to black sky On Tue, 25 Feb 2020 at 07:00, Kiril Simeonovski <kiril.simeonovski(a)gmail.com> wrote:

...

Kiril Simeonovski

7:41 p.m.

Hi Stuart, Yes, all those terms refer to a Wikipedia account. My plan is to avoid sampling accounts with very low activity (probably less then a minimum threshold of edits) because of the impracticality to draw any conclusion from them. Best, Kiril On Mon, Feb 24, 2020 at 8:36 PM Stuart A. Yeates <syeates(a)gmail.com> wrote:

...

and

define (dis)utilities that will be used in a utility-maximisation model. I have already submitted an abstract that got accepted for the biennial Foundations of Utility and Risk Conference 2020 <https://www.furconference.org/>

and my

future plans include presentation of the concept at other research conferences (including Wikimania 2020). You can visit the project page <

https://meta.wikimedia.org/wiki/Research:Modelling_Behaviour_in_a_Peer_Prod…

of this research on Meta. Your questions and comments are welcome at any time. Thank you! Best regards, Kiril _______________________________________________ Wiki-research-l mailing list Wiki-research-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

_______________________________________________ Wiki-research-l mailing list Wiki-research-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

Pine W

8:42 p.m.

Hi Kiril, Thank you for sharing your proposal. I am concerned about the possibility of Wikipedia being used as a laboratory for experiments that consume volunteers' time and/or personal data, and don't benefit Wikipedia or its participants. Does your research benefit the community, and if so, how? It sounds like your research intends to develop a model of decision trees for individual Wikipedians, and at first read I don't understand how the individual research subjects or the community would benefit. Sorry if this sounds defensive, but I hope that you understand why I'm asking. Pine ( https://meta.wikimedia.org/wiki/User:Pine ) On Mon, Feb 24, 2020 at 6:00 PM Kiril Simeonovski <kiril.simeonovski(a)gmail.com> wrote:

...

Stuart A. Yeates

8:51 p.m.

Removing accounts with low numbers of edits will, of course, blind your analysis to users who use throw-away accounts, even when they can clearly be attributed to the same individual. cheers stuart -- ...let us be heard from red core to black sky On Tue, 25 Feb 2020 at 08:42, Kiril Simeonovski <kiril.simeonovski(a)gmail.com> wrote:

...

and

and my

future plans include presentation of the concept at other research conferences (including Wikimania 2020). You can visit the project page <

https://meta.wikimedia.org/wiki/Research:Modelling_Behaviour_in_a_Peer_Prod…

_______________________________________________ Wiki-research-l mailing list Wiki-research-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

Kiril Simeonovski

9:42 p.m.

Hi Pine, The findings from the research will be articulated to draw clear conclusions about what causes utility and disutility from participation, and how this is perceived by different editors. For instance, it is natural to assume that editors come to contribute by adding content that will remain visible, while blocks and reverted edits are risk factors that drive them away, although different editors have different levels of risk aversion. Similarly to any other research, the benefit for the community and individual editors is going to be indirect but yet not insignificant to be accepted in the future process of decision-making (if the research demonstrates the existence of high level of risk aversion towards something, then it automatically signals that doing that thing is harmful for the environment). I know that it's impossible to predict the extent to which this research would make impact because the body of literature is very poor on volunteer-driven environments in a dynamic setting but it's definitely worth to start off something that might attract the attention of researchers in this direction. At the end, the research is not meant to carve rules in stone that any single editor should respect but rather to suggest something that individuals and communities might find useful (the means of doing this will definitely not turn Wikipedia into a laboratory or put someone's privacy in danger). Best, Kiril On Mon, Feb 24, 2020 at 9:43 PM Pine W <wiki.pine(a)gmail.com> wrote:

...

and

and my

future plans include presentation of the concept at other research conferences (including Wikimania 2020). You can visit the project page <

https://meta.wikimedia.org/wiki/Research:Modelling_Behaviour_in_a_Peer_Prod…

_______________________________________________ Wiki-research-l mailing list Wiki-research-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

Stuart A. Yeates

11:37 p.m.

Hi Kiril Let's just say that history has taught us to be risk-averse to drive-by researchers. Can you point us to other research output using this methodology? Do you (or any of your team) have significant editing experience? Are you familiar with the firestorm that is paid editing and sock puppetry?? cheers stuart -- ...let us be heard from red core to black sky On Tue, 25 Feb 2020 at 10:43, Kiril Simeonovski <kiril.simeonovski(a)gmail.com> wrote:

...

and

and my

future plans include presentation of the concept at other research conferences (including Wikimania 2020). You can visit the project page <

https://meta.wikimedia.org/wiki/Research:Modelling_Behaviour_in_a_Peer_Prod…

_______________________________________________ Wiki-research-l mailing list Wiki-research-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

Kiril Simeonovski

25 Feb 25 Feb

7:22 a.m.

Hi Stuart, Thank you for your thoughts so far. I really like how the discussion is progressing. The methodology will, of course, yield other results about editor dynamics and growth paths. Paid editing and sock puppetry as systemic risk factors could be included in the model exogenously but it might be possible to endogenise them in any future research. At this stage, the most important thing is to lay the grounds for developing a sensible model that can be later upgraded with new assumptions. As for the editing experience, I've been around since 2008 (this is my edit log <https://meta.wikimedia.org/wiki/Special:CentralAuth/Kiril_Simeonovski> ). Best, Kiril On Tue, Feb 25, 2020 at 12:37 AM Stuart A. Yeates <syeates(a)gmail.com> wrote:

...

natural

to assume that editors come to contribute by adding content that will remain visible, while blocks and reverted edits are risk factors that

drive

them away, although different editors have different levels of risk aversion. Similarly to any other research, the benefit for the community and individual editors is going to be indirect but yet not insignificant

be accepted in the future process of decision-making (if the research demonstrates the existence of high level of risk aversion towards something, then it automatically signals that doing that thing is harmful for the environment). I know that it's impossible to predict the extent to which this research would make impact because the body of literature is very poor on volunteer-driven environments in a dynamic setting but it's definitely worth to start off something that might attract the attention of researchers in this direction. At the end, the research is not meant to carve rules in stone that any single editor should respect but rather to suggest something that individuals and communities might find useful (the means of doing this will definitely not turn Wikipedia into a laboratory

put someone's privacy in danger). Best, Kiril On Mon, Feb 24, 2020 at 9:43 PM Pine W <wiki.pine(a)gmail.com> wrote: > Hi Kiril, > > Thank you for sharing your proposal. > > I am concerned about the possibility of Wikipedia being used as a > laboratory for experiments that consume volunteers' time and/or > personal data, and don't benefit Wikipedia or its participants. Does > your research benefit the community, and if so, how? It sounds like > your research intends to develop a model of decision trees for > individual Wikipedians, and at first read I don't understand how the > individual research subjects or the community would benefit. > > Sorry if this sounds defensive, but I hope that you understand why I'm > asking. > > Pine > ( https://meta.wikimedia.org/wiki/User:Pine ) > > On Mon, Feb 24, 2020 at 6:00 PM Kiril Simeonovski > <kiril.simeonovski(a)gmail.com> wrote: > > > > Hi all, > > > > I am currently working on a research concerned with modelling user > > behaviour on Wikipedia. The idea is to design a field experiment

over a

> > random sample of Wikipedians in order to examine their risk

preferences

> and > > define (dis)utilities that will be used in a utility-maximisation

model.

> > > > I have already submitted an abstract that got accepted for the > > biennial Foundations > > of Utility and Risk Conference 2020 <https://www.furconference.org/> > and my > > future plans include presentation of the concept at other research > > conferences (including Wikimania 2020). > > > > You can visit the project page > > < >

https://meta.wikimedia.org/wiki/Research:Modelling_Behaviour_in_a_Peer_Prod…

> > > > of this research on Meta. Your questions and comments are welcome at

any

time. Thank you! Best regards, Kiril _______________________________________________ Wiki-research-l mailing list Wiki-research-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

_______________________________________________ Wiki-research-l mailing list Wiki-research-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

Jonathan Morgan

4:05 p.m.

Taking a quick step back from all the very enthusiastic questioning of the researcher's motives... Kiril, Regarding your methods, Your proposal states <https://meta.wikimedia.org/wiki/Research:Modelling_Behaviour_in_a_Peer_Production_Economy_upon_Evidence_from_Wikipedia> that for this study "The editors will be sampled from the pool of contributors to all language editions over Wikipedia's entire history and will be classified into groups based on their longevity on the project." But it says little more than that. When I read this description, it does not sound to me like you will necessarily be contacting editors for this study, or intervening in any way into Wikipedia. Stuart and Pine's questions seem to assume that you will be in some way recruiting editors as participants, asking them to change their behavior, asking them questions, etc. *Will you be performing any of the above activities?* If not, the questions asked so far may be beside the point. Anyone is free to perform analysis on publicly available and free-licensed data. If you do plan to intervene in Wikipedia in some way, or work with editors as research participants or co-researchers, and you would like the members of this mailing list to provide you with feedback or other support, please describe the support or feedback you would like to receive in more detail. If your study is non-interventionist but you still want feedback, we can provide that too. Perhaps you can be more clear about the kind of feedback you want; that will keep the conversation going in an interesting and productive direction that everyone on the list can benefit from. Finally, we the members of this list (whether volunteers or WMF staff) are not peer reviewers, do not speak for the Wikipedia community, and are not empowered to approve or deny research requests. Best, Jonathan On Mon, Feb 24, 2020 at 11:23 PM Kiril Simeonovski < kiril.simeonovski(a)gmail.com> wrote:

...

). Best, Kiril On Tue, Feb 25, 2020 at 12:37 AM Stuart A. Yeates <syeates(a)gmail.com> wrote: > Hi Kiril

> Let's just say that history has taught us to be risk-averse to > drive-by researchers.

> Can you point us to other research output using this methodology? Do > you (or any of your team) have significant editing experience? Are you > familiar with the firestorm that is paid editing and sock puppetry??

> cheers > stuart

> -- > ...let us be heard from red core to black sky

> On Tue, 25 Feb 2020 at 10:43, Kiril Simeonovski > <kiril.simeonovski(a)gmail.com> wrote: >

> > Hi Pine, >

> > The findings from the research will be articulated to draw clear > > conclusions about what causes utility and disutility from participation,

and how this is perceived by different editors. For instance, it is

natural

to assume that editors come to contribute by adding content that will remain visible, while blocks and reverted edits are risk factors that

drive > them away, although different editors have different levels of risk > aversion. Similarly to any other research, the benefit for the

community

> and individual editors is going to be indirect but yet not

insignificant

to > be accepted in the future process of decision-making (if the research > demonstrates the existence of high level of risk aversion towards > something, then it automatically signals that doing that thing is

harmful > > for the environment). >

> > I know that it's impossible to predict the extent to which this research

> would make impact because the body of literature is very poor on > volunteer-driven environments in a dynamic setting but it's definitely > worth to start off something that might attract the attention of > researchers in this direction. At the end, the research is not meant to > carve rules in stone that any single editor should respect but rather

> suggest something that individuals and communities might find useful

(the

> means of doing this will definitely not turn Wikipedia into a

laboratory > or > > put someone's privacy in danger). >

> > Best, > > Kiril >

> > On Mon, Feb 24, 2020 at 9:43 PM Pine W <wiki.pine(a)gmail.com> wrote: >

> > > Hi Kiril, > >

> > > Thank you for sharing your proposal. > >

> > > I am concerned about the possibility of Wikipedia being used as a > > > laboratory for experiments that consume volunteers' time and/or > > > personal data, and don't benefit Wikipedia or its participants. Does > > > your research benefit the community, and if so, how? It sounds like > > > your research intends to develop a model of decision trees for > > > individual Wikipedians, and at first read I don't understand how the > > > individual research subjects or the community would benefit. > >

> > > Sorry if this sounds defensive, but I hope that you understand why I'm > > > asking. > >

> > > Pine > > > ( https://meta.wikimedia.org/wiki/User:Pine ) > >

> > > On Mon, Feb 24, 2020 at 6:00 PM Kiril Simeonovski > > > <kiril.simeonovski(a)gmail.com> wrote: > > >

> > > > Hi all, > > >

> > > > I am currently working on a research concerned with modelling user > > > > behaviour on Wikipedia. The idea is to design a field experiment > over a > > > > random sample of Wikipedians in order to examine their risk > preferences > > > and > > > > define (dis)utilities that will be used in a utility-maximisation > model. > > >

> > > > I have already submitted an abstract that got accepted for the > > > > biennial Foundations > > > > of Utility and Risk Conference 2020 < https://www.furconference.org/> > > > and my > > > > future plans include presentation of the concept at other research > > > > conferences (including Wikimania 2020). > > >

> > > > You can visit the project page > > > > < > >

https://meta.wikimedia.org/wiki/Research:Modelling_Behaviour_in_a_Peer_Prod… > > >

> > > > of this research on Meta. Your questions and comments are welcome at > any > > > > time. Thank you! > > >

> > > > Best regards, > > > > Kiril > > > > _______________________________________________ > > > > Wiki-research-l mailing list > > > > Wiki-research-l(a)lists.wikimedia.org > > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l > >

> > > _______________________________________________ > > > Wiki-research-l mailing list > > > Wiki-research-l(a)lists.wikimedia.org > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l > >

> > _______________________________________________ > > Wiki-research-l mailing list > > Wiki-research-l(a)lists.wikimedia.org > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

> _______________________________________________ > Wiki-research-l mailing list > Wiki-research-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

_______________________________________________ Wiki-research-l mailing list Wiki-research-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

Pine W

7:30 p.m.

Hi Kiril, I'd like to ask for further clarification on a couple of points. Some people may be risk averse to performing certain actions, such as moving pages, because they don't know what the rules are regarding those actions, but page moves aren't necessarily harmful for the environment. Does your research account for situations like this? I'm still having difficulty understanding how the community may benefit from the proposed research. Suppose that your research is highly successful according to your definition of success. What will the community gain from this? Based on the use of the word "experiment", I'm assuming that you're proposing an active intervention of some type in the community, and as Jonathan said, it would be good to have more information regarding what you have in mind. Hi Jonathan, My questions are largely based on Kiril's use of the word "experiment", but I appreciate the request for clarification. Regarding "Finally, we the members of this list (whether volunteers or WMF staff) are not peer reviewers, do not speak for the Wikipedia community, and are not empowered to approve or deny research requests.", I partly agree and partly disagree. This list isn't a peer review committee in the standard Western academic sense, but I think that people who are planning to do interventions in the course of their research in the Wikiverse would do well to consult this list for advice. Also, the community may, by consensus, place various restrictions on research projects, both broadly by policy and more narrowly regarding specific experiments or specific researchers; we probably would do that on wiki, but by seeking advice here, researchers are likely to get useful advice regarding what to do and what not to do. Also, some researchers may want to make requests for WMF funding, WMF staff time, and/or access to users' private data, and the people on this list might be good to consult before WMF grants any of those, especially in novel or borderline situations. Thanks, Pine ( https://meta.wikimedia.org/wiki/User:Pine )

Kiril Simeonovski

7:42 p.m.

Hi Jonathan, You have correctly deduced from the description that there will not be any interaction with editors and all the data for the research will be drawn from the publicly available edit histories. The most confusing part that gives impression of intervening is perhaps the "experiment", which conceptually differs in the social sciences from its more common meaning in a laboratory environment. That said, this research is not going to consume editor's time for surveys nor it is going to convert Wikipedia to a laboratory or ask people to change their behaviour. I came here with the announcement after creating the proposal on Meta and following the guidelines regarding research projects with the goal of getting some useful input from other researchers subscribed to the mailing list and learning how to administratively proceed with the proposal on Meta (What should be done next on Meta? Will there be an appointed WMF researcher to contact regarding this research?). My request for help from research community regarding this research will be mostly technical (e.g. smart random sampling of editors, existing tools for research purposes, etc.) Best, Kiril On Tue 25. Feb 2020 at 17:06, Jonathan Morgan <jmorgan(a)wikimedia.org> wrote:

...

that for this study "The editors will be sampled from the pool of contributors to all language editions over Wikipedia's entire history and will be classified into groups based on their longevity on the project." But it says little more than that. When I read this description, it does not sound to me like you will necessarily be contacting editors for this study, or intervening in any way into Wikipedia. Stuart and Pine's questions seem to assume that you will be in some way recruiting editors as participants, asking them to change their behavior, asking them questions, etc. *Will you be performing any of the above activities?* If not, the questions asked so far may be beside the point. Anyone is free to perform analysis on publicly available and free-licensed data. If you do plan to intervene in Wikipedia in some way, or work with editors as research participants or co-researchers, and you would like the members of this mailing list to provide you with feedback or other support, please describe the support or feedback you would like to receive in more detail. If your study is non-interventionist but you still want feedback, we can provide that too. Perhaps you can be more clear about the kind of feedback you want; that will keep the conversation going in an interesting and productive direction that everyone on the list can benefit from. Finally, we the members of this list (whether volunteers or WMF staff) are not peer reviewers, do not speak for the Wikipedia community, and are not empowered to approve or deny research requests. Best, Jonathan On Mon, Feb 24, 2020 at 11:23 PM Kiril Simeonovski < kiril.simeonovski(a)gmail.com> wrote: > Hi Stuart,

> Thank you for your thoughts so far. I really like how the discussion is > progressing.

> The methodology will, of course, yield other results about editor dynamics > and growth paths. Paid editing and sock puppetry as systemic risk factors > could be included in the model exogenously but it might be possible to > endogenise them in any future research. At this stage, the most important > thing is to lay the grounds for developing a sensible model that can be > later upgraded with new assumptions.

> As for the editing experience, I've been around since 2008 (this is my edit

log <

https://meta.wikimedia.org/wiki/Special:CentralAuth/Kiril_Simeonovski >

> ).

> Best, > Kiril

> On Tue, Feb 25, 2020 at 12:37 AM Stuart A. Yeates <syeates(a)gmail.com> > wrote:

> > Hi Kiril >

> > Let's just say that history has taught us to be risk-averse to > > drive-by researchers. >

> > Can you point us to other research output using this methodology? Do > > you (or any of your team) have significant editing experience? Are you > > familiar with the firestorm that is paid editing and sock puppetry?? >

> > cheers > > stuart >

> > -- > > ...let us be heard from red core to black sky >

> > On Tue, 25 Feb 2020 at 10:43, Kiril Simeonovski > > <kiril.simeonovski(a)gmail.com> wrote: > >

> > > Hi Pine, > >

> > > The findings from the research will be articulated to draw clear > > > conclusions about what causes utility and disutility from > participation, > > > and how this is perceived by different editors. For instance, it is > > natural > > > to assume that editors come to contribute by adding content that will > > > remain visible, while blocks and reverted edits are risk factors that > > drive > > > them away, although different editors have different levels of risk > > > aversion. Similarly to any other research, the benefit for the > community > > > and individual editors is going to be indirect but yet not > insignificant > > to > > > be accepted in the future process of decision-making (if the research > > > demonstrates the existence of high level of risk aversion towards > > > something, then it automatically signals that doing that thing is > harmful > > > for the environment). > >

> > > I know that it's impossible to predict the extent to which this > research > > > would make impact because the body of literature is very poor on > > > volunteer-driven environments in a dynamic setting but it's definitely

> > worth to start off something that might attract the attention of > > researchers in this direction. At the end, the research is not meant

to > > > carve rules in stone that any single editor should respect but rather > to > > > suggest something that individuals and communities might find useful > (the > > > means of doing this will definitely not turn Wikipedia into a > laboratory > > or > > > put someone's privacy in danger). > >

> > > Best, > > > Kiril > >

> > > On Mon, Feb 24, 2020 at 9:43 PM Pine W <wiki.pine(a)gmail.com> wrote: > >

> > > > Hi Kiril, > > >

> > > > Thank you for sharing your proposal. > > >

> > > > I am concerned about the possibility of Wikipedia being used as a > > > > laboratory for experiments that consume volunteers' time and/or > > > > personal data, and don't benefit Wikipedia or its participants. Does

> > > your research benefit the community, and if so, how? It sounds like > > > your research intends to develop a model of decision trees for > > > individual Wikipedians, and at first read I don't understand how

the > > > > individual research subjects or the community would benefit. > > >

> > > > Sorry if this sounds defensive, but I hope that you understand why > I'm > > > > asking. > > >

> > > > Pine > > > > ( https://meta.wikimedia.org/wiki/User:Pine ) > > >

> > > > On Mon, Feb 24, 2020 at 6:00 PM Kiril Simeonovski > > > > <kiril.simeonovski(a)gmail.com> wrote: > > > >

> > > > > Hi all, > > > >

> > > > > I am currently working on a research concerned with modelling user > > > > > behaviour on Wikipedia. The idea is to design a field experiment > > over a > > > > > random sample of Wikipedians in order to examine their risk > > preferences > > > > and > > > > > define (dis)utilities that will be used in a utility-maximisation > > model. > > > >

> > > > > I have already submitted an abstract that got accepted for the > > > > > biennial Foundations > > > > > of Utility and Risk Conference 2020 < > https://www.furconference.org/> > > > > and my > > > > > future plans include presentation of the concept at other research > > > > > conferences (including Wikimania 2020). > > > >

> > > > > You can visit the project page > > > > > < > > >

https://meta.wikimedia.org/wiki/Research:Modelling_Behaviour_in_a_Peer_Prod… > > > >

> > > > > of this research on Meta. Your questions and comments are welcome > at > > any > > > > > time. Thank you! > > > >

> > > > > Best regards, > > > > > Kiril > > > > > _______________________________________________ > > > > > Wiki-research-l mailing list > > > > > Wiki-research-l(a)lists.wikimedia.org > > > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l > > >

> > > > _______________________________________________ > > > > Wiki-research-l mailing list > > > > Wiki-research-l(a)lists.wikimedia.org > > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l > > >

> > _______________________________________________ > > Wiki-research-l mailing list > > Wiki-research-l(a)lists.wikimedia.org > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l >

> _______________________________________________ > Wiki-research-l mailing list > Wiki-research-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

-- Jonathan T. Morgan Senior Design Researcher Wikimedia Foundation User:Jmorgan (WMF) <https://meta.wikimedia.org/wiki/User:Jmorgan_(WMF)> (Uses He/Him) *Please note that I do not expect a response from you on evenings or weekends* _______________________________________________ Wiki-research-l mailing list Wiki-research-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

Kiril Simeonovski

10:03 p.m.

Hi Pine, You can see my reply to Jonathan about the nature of the research and what does "experiment" refer to in the context of this research. I think the answer to your question on how this research will benefit the community is highly dependent on your expectations and the scope of the research. While there are indicative research projects with the aim to tell whether the community is ready for introducing a novelty or implementing a major change, there are also projects that focus on the social aspects of the collaborative environments and how people behave in different situations. The latter group of projects seems not to offer direct benefit for the community but it doesn't mean they are useless and shouldn't be done at all. I know it's natural that people with different backgrounds have difficulties to comprehend the complexity of economic modelling and there should be an interdisciplinary way to present the findings so that they could practically reach to a wider group of people. This is something that will be addressed with this research. Yet, it's still possible to give you a direction of where the benefit for the community would come from. Similar researches in the past offered explanations to many relevant real-world questions such as how much time should people spend at work, why companies don't produce as much as they can what people want, why people tend to be more risk-averse towards unknown risks or how changes in the environment affect the decision-making by individuals. Best, Kiril On Tue, Feb 25, 2020 at 8:42 PM Kiril Simeonovski < kiril.simeonovski(a)gmail.com> wrote:

...

> Thank you for your thoughts so far. I really like how the discussion is > progressing.

> The methodology will, of course, yield other results about editor dynamics

and growth paths. Paid editing and sock puppetry as systemic risk

factors

could be included in the model exogenously but it might be possible to endogenise them in any future research. At this stage, the most

important > thing is to lay the grounds for developing a sensible model that can be > later upgraded with new assumptions.

> As for the editing experience, I've been around since 2008 (this is my edit

log <

https://meta.wikimedia.org/wiki/Special:CentralAuth/Kiril_Simeonovski >

> ).

> Best, > Kiril

> On Tue, Feb 25, 2020 at 12:37 AM Stuart A. Yeates <syeates(a)gmail.com> > wrote:

> > Hi Kiril >

> > Let's just say that history has taught us to be risk-averse to > > drive-by researchers. >

> > cheers > > stuart >

> > -- > > ...let us be heard from red core to black sky >

> > On Tue, 25 Feb 2020 at 10:43, Kiril Simeonovski > > <kiril.simeonovski(a)gmail.com> wrote: > >

> > > Hi Pine, > >

> > remain visible, while blocks and reverted edits are risk factors

that

drive > them away, although different editors have different levels of risk > aversion. Similarly to any other research, the benefit for the

community

> and individual editors is going to be indirect but yet not

insignificant > to > > be accepted in the future process of decision-making (if the

research > > > demonstrates the existence of high level of risk aversion towards > > > something, then it automatically signals that doing that thing is > harmful > > > for the environment). > >

> > worth to start off something that might attract the attention of > > researchers in this direction. At the end, the research is not

meant to

> > carve rules in stone that any single editor should respect but

rather > to > > > suggest something that individuals and communities might find useful > (the > > > means of doing this will definitely not turn Wikipedia into a > laboratory > > or > > > put someone's privacy in danger). > >

> > > Best, > > > Kiril > >

> > > On Mon, Feb 24, 2020 at 9:43 PM Pine W <wiki.pine(a)gmail.com> wrote: > >

> > > > Hi Kiril, > > >

> > > > Thank you for sharing your proposal. > > >

> > > your research benefit the community, and if so, how? It sounds

> > > your research intends to develop a model of decision trees for > > > individual Wikipedians, and at first read I don't understand how

the > > > > individual research subjects or the community would benefit. > > >

> > > > Sorry if this sounds defensive, but I hope that you understand why > I'm > > > > asking. > > >

> > > > Pine > > > > ( https://meta.wikimedia.org/wiki/User:Pine ) > > >

> > > > On Mon, Feb 24, 2020 at 6:00 PM Kiril Simeonovski > > > > <kiril.simeonovski(a)gmail.com> wrote: > > > >

> > > > > Hi all, > > > >

> > > > > I am currently working on a research concerned with modelling user

> > > > behaviour on Wikipedia. The idea is to design a field experiment > over a > > > > random sample of Wikipedians in order to examine their risk > preferences > > > and > > > > define (dis)utilities that will be used in a

utility-maximisation > > model. > > > >

> > > > > You can visit the project page > > > > > < > > >

https://meta.wikimedia.org/wiki/Research:Modelling_Behaviour_in_a_Peer_Prod… > > > >

> > > > > of this research on Meta. Your questions and comments are welcome > at > > any > > > > > time. Thank you! > > > >

> > _______________________________________________ > > Wiki-research-l mailing list > > Wiki-research-l(a)lists.wikimedia.org > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l >

> _______________________________________________ > Wiki-research-l mailing list > Wiki-research-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

Jonathan Morgan

11:55 p.m.

Hi Kiril! Thanks for all the patient and thoughtful clarifications and elaborations :) I left a couple of comments inline, below. Good luck with your project! As you can tell, we are a curious and thoughtful group here on wikiresearch-l. If you have methodological questions in the future, please don't hesitate to ask them here. On Tue, Feb 25, 2020 at 11:43 AM Kiril Simeonovski < kiril.simeonovski(a)gmail.com> wrote:

...

Putting your research proposal on Meta is best practice for all research projects related to Wikimedia. It is not a required step, but it's useful for increasing awareness of your project among the broader Wikimedia communities (researchers and everyone else). There is nothing else you need to do at this point, although we appreciate it if you would keep your project page up to date as you perform your research. When you're done, we always appreciate it if you link to any preprints, demos, code repos, slide decks, etc from that page as well. Your project won't automatically be assigned a WMF contact. The Wikimedia Foundation itself does not officially monitor or screen new research projects that are published on Meta, or review them for support. However, if you believe your research furthers the goals of the Wikimedia Movement, you might consider applying for a grant (example <https://meta.wikimedia.org/wiki/Grants:Project/Misinformation_And_Its_Discontents:_Narrative_Recommendations_on_Wikipedia%27s_Vulnerabilities_and_Resilience>). Individual WMF teams (including my team, Research) do occasionally partner with external researchers <https://research.wikimedia.org/collaborators.html> and those partnerships can include access to non-public data (under a Non-Disclosure Agreement). Some partnerships do involve funding, but this is not common. All partnerships are at the discretion of the team manager. In the case of my team, that manager is Leila Zia, Head of Research. Quick question: when you say "the guidelines regarding research projects" above, what document are you referring to? There are a lot of these pages in the Research namespace and they are not always up to date, unfortunately.

...

My request for help from research community regarding this research will be mostly technical (e.g. smart random sampling of editors, existing tools for research purposes, etc.)

We can definitely help you with these questions! You can also post questions related to data access and data infrastructure to the analytics mailing list <https://lists.wikimedia.org/mailman/listinfo/analytics>.

...

Best, Kiril On Tue 25. Feb 2020 at 17:06, Jonathan Morgan <jmorgan(a)wikimedia.org> wrote:

Taking a quick step back from all the very enthusiastic questioning of

the

researcher's motives... Kiril, Regarding your methods, Your proposal states <

https://meta.wikimedia.org/wiki/Research:Modelling_Behaviour_in_a_Peer_Prod…

way

into Wikipedia. Stuart and Pine's questions seem to assume that you will

in some way recruiting editors as participants, asking them to change

their

behavior, asking them questions, etc. *Will you be performing any of the above activities?* If not, the

questions

asked so far may be beside the point. Anyone is free to perform analysis

publicly available and free-licensed data. If you do plan to intervene in Wikipedia in some way, or work with

editors

as research participants or co-researchers, and you would like the

members

of this mailing list to provide you with feedback or other support,

please

describe the support or feedback you would like to receive in more

detail.

If your study is non-interventionist but you still want feedback, we can provide that too. Perhaps you can be more clear about the kind of

feedback

you want; that will keep the conversation going in an interesting and productive direction that everyone on the list can benefit from. Finally, we the members of this list (whether volunteers or WMF staff)

are

not peer reviewers, do not speak for the Wikipedia community, and are not empowered to approve or deny research requests. Best, Jonathan On Mon, Feb 24, 2020 at 11:23 PM Kiril Simeonovski < kiril.simeonovski(a)gmail.com> wrote:

Hi Stuart, Thank you for your thoughts so far. I really like how the discussion is progressing. The methodology will, of course, yield other results about editor

dynamics > and growth paths. Paid editing and sock puppetry as systemic risk

factors

> could be included in the model exogenously but it might be possible to > endogenise them in any future research. At this stage, the most

important

thing is to lay the grounds for developing a sensible model that can be later upgraded with new assumptions. As for the editing experience, I've been around since 2008 (this is my

edit

log <

https://meta.wikimedia.org/wiki/Special:CentralAuth/Kiril_Simeonovski > > > ). > > Best, > Kiril > > On Tue, Feb 25, 2020 at 12:37 AM Stuart A. Yeates <syeates(a)gmail.com> > wrote: > > > Hi Kiril > > > > Let's just say that history has taught us to be risk-averse to > > drive-by researchers. > > > > Can you point us to other research output using this methodology? Do > > you (or any of your team) have significant editing experience? Are

you

> > familiar with the firestorm that is paid editing and sock puppetry?? > > > > cheers > > stuart > > > > -- > > ...let us be heard from red core to black sky > > > > On Tue, 25 Feb 2020 at 10:43, Kiril Simeonovski > > <kiril.simeonovski(a)gmail.com> wrote: > > > > > > Hi Pine, > > > > > > The findings from the research will be articulated to draw clear > > > conclusions about what causes utility and disutility from > participation, > > > and how this is perceived by different editors. For instance, it is > > natural > > > to assume that editors come to contribute by adding content that

will

> > > remain visible, while blocks and reverted edits are risk factors

that

> > drive > > > them away, although different editors have different levels of risk > > > aversion. Similarly to any other research, the benefit for the > community > > > and individual editors is going to be indirect but yet not > insignificant > > to > > > be accepted in the future process of decision-making (if the

research

> demonstrates the existence of high level of risk aversion towards > something, then it automatically signals that doing that thing is

harmful

> for the environment). > > I know that it's impossible to predict the extent to which this

research > > would make impact because the body of literature is very poor on > > volunteer-driven environments in a dynamic setting but it's

definitely > > > worth to start off something that might attract the attention of > > > researchers in this direction. At the end, the research is not

meant

to > > > carve rules in stone that any single editor should respect but

rather

> to > > > suggest something that individuals and communities might find

useful

> (the > > > means of doing this will definitely not turn Wikipedia into a > laboratory > > or > > > put someone's privacy in danger). > > > > > > Best, > > > Kiril > > > > > > On Mon, Feb 24, 2020 at 9:43 PM Pine W <wiki.pine(a)gmail.com>

wrote:

> > > > > Hi Kiril, > > > > > > Thank you for sharing your proposal. > > > > > > I am concerned about the possibility of Wikipedia being used as a > > > laboratory for experiments that consume volunteers' time and/or > > > personal data, and don't benefit Wikipedia or its participants.

Does > > > > your research benefit the community, and if so, how? It sounds

> > > your research intends to develop a model of decision trees for > > > individual Wikipedians, and at first read I don't understand how

the > > > > individual research subjects or the community would benefit. > > > > > > > > Sorry if this sounds defensive, but I hope that you understand

why

I'm > > > asking. > > > > > > Pine > > > ( https://meta.wikimedia.org/wiki/User:Pine ) > > > > > > On Mon, Feb 24, 2020 at 6:00 PM Kiril Simeonovski > > > <kiril.simeonovski(a)gmail.com> wrote: > > > > > > > > Hi all, > > > > > > > > I am currently working on a research concerned with modelling

user > > > > > behaviour on Wikipedia. The idea is to design a field

experiment

> > over a > > > > > random sample of Wikipedians in order to examine their risk > > preferences > > > > and > > > > > define (dis)utilities that will be used in a

utility-maximisation

model. > > > > > > I have already submitted an abstract that got accepted for the > > > biennial Foundations > > > of Utility and Risk Conference 2020 <

https://www.furconference.org/> > > > and my > > > > future plans include presentation of the concept at other

research

> > > conferences (including Wikimania 2020). > > > > > > You can visit the project page > > > < > >

https://meta.wikimedia.org/wiki/Research:Modelling_Behaviour_in_a_Peer_Prod…

> > > > > > > > > > of this research on Meta. Your questions and comments are

welcome

any > > > time. Thank you! > > > > > > Best regards, > > > Kiril > > > _______________________________________________ > > > Wiki-research-l mailing list > > > Wiki-research-l(a)lists.wikimedia.org > > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l > > > > _______________________________________________ > > Wiki-research-l mailing list > > Wiki-research-l(a)lists.wikimedia.org > > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l > > > _______________________________________________ > Wiki-research-l mailing list > Wiki-research-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wiki-research-l _______________________________________________ Wiki-research-l mailing list Wiki-research-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

_______________________________________________ Wiki-research-l mailing list Wiki-research-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l

Kiril Simeonovski

26 Feb 26 Feb

7:39 p.m.

Hi Jonathan, I spent some time to carefully read the FAQ about research and the guidelines linked from there. As for the progress of the project, I will provide regular updates on Meta. Best, Kiril On Wed, Feb 26, 2020 at 12:56 AM Jonathan Morgan <jmorgan(a)wikimedia.org> wrote:

...

Hi Jonathan, You have correctly deduced from the description that there will not be

any

interaction with editors and all the data for the research will be drawn from the publicly available edit histories. The most confusing part that gives impression of intervening is perhaps the "experiment", which conceptually differs in the social sciences from its more common meaning

a laboratory environment. That said, this research is not going to

consume

editor's time for surveys nor it is going to convert Wikipedia to a laboratory or ask people to change their behaviour. I came here with the announcement after creating the proposal on Meta and following the guidelines regarding research projects with the goal of getting some useful input from other researchers subscribed to the

mailing

list and learning how to administratively proceed with the proposal on