Hi Kerry,
Thanks for the feedback! I'll reply to a couple of your points below:
One observation I would make is that like many education experiments, it
does not control for (what I call) the "highly motivated researcher effect". What I've learned from a lifetime of "new ways to teach" is that the standard experiment is to parachute in a highly motivated researcher into the classroom to introduce the new method, collect data showing improved learning, and then advocate for the new method to be rolled out more widely. However, rolling out more widely involves taking regular teachers (good, bad, and in-between) to learn and apply a new approach, and techniques often fail in the face of teacher lack of enthusiasm to learn anything new, complaints it makes more demands on teachers to use the new method, etc.
This study was a survey of everyone who would participate across 270 courses and 6700 students. We had 90 different instructors that took the survey as well, many of them were returning instructors (there is data on that in the instructor survey), but across the 270 courses there was a good mix (Wiki Ed has data on that, and I could find that if you're really interested, but our statistical analysis showed that there was no variation between new and returning instructors for student responses for the instructors polled - although a different design or question set with that in mind could produce different results).
I like that suggestion and, in (hopefully) future iterations, we can spend more thought on that particular piece when designing surveys (like specifically tracking how many semesters the instructors have been active with Wiki Ed).
In this report it says " the program staff provide Wikipedia training and expertise so the faculty do not need to have any experience editing" which is a big red flag to me. It would be interesting to see the results in an experiment where you first train the faculty and then the faculty carry out the engagement with students. And then see the results in 3 years time when it's a case of "business as usual" rather than "the new thing".
New and returning instructors are encouraged to take (and re-take) a pretty substantial training program (training module completion is listed in the data set for both instructors and students). Faculty are "trained" but there is a limit to the hands-on training that one can offer, of course.
As a general comment, students like the variety of someone new in their classroom. Students do tend to learn more from "real world" assignments than "lab" assignments because the real world is more complex. However, staff and students are often reluctant to have real world assignments significantly influence end-of-term marks/grades because of the uncontrollable variables in the real world assignment that makes it difficult to assess the relative achievement of the students.
Absolutely - Joe Reagle and I just did a presentation for New Media Consortium that recommended that you never grade students on "what remains on Wikipedia" - Wiki Ed has some good suggestions for grading too.
I would expect editing Wikipedia articles to suffer from this problem as each student will be working on different article(s) of different starting size and quality and with different levels of involvement and monitoring by other Wikipedians. It was not clear to me from the report if students were being assessed on this Wikipedia assignment and how important it was to their overall mark/grade.
This was something that across 270 different classes we could not control. Additionally, it was part of the IRB approval that nothing in the study can affect the grade of the student directly (eg: instructors can't give extra credit for taking the survey, they couldn't require students to be present during the focus groups, no grades were reported). This would have been a violation of FERPA in the US, as well as many rules in Canada. So in the end, there was no "assessment" - although each student's work can be traced back to their course and folks can do their own marking and analysis (whether through computational means or otherwise) and compare it to overall data if they like.
In the end, I hope you can make use of the data - as I mentioned above and a few times throughout the report and elsewhere, this research was meant to intersect with a variety of research questions, and I've done my best to open up the data so that folks can investigate their own questions (like looking at quality of work or addressing content gaps) alongside some of the student data.
best,
Zach
-------------------- Zachary J. McDowell, PhD www.zachmcdowell.com
On Tue, Jun 20, 2017 at 8:00 AM, < wiki-research-l-request@lists.wikimedia.org> wrote:
Send Wiki-research-l mailing list submissions to wiki-research-l@lists.wikimedia.org
To subscribe or unsubscribe via the World Wide Web, visit https://lists.wikimedia.org/mailman/listinfo/wiki-research-l or, via email, send a message with subject or body 'help' to wiki-research-l-request@lists.wikimedia.org
You can reach the person managing the list at wiki-research-l-owner@lists.wikimedia.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of Wiki-research-l digest..."
Today's Topics:
- Student Learning Outcomes using Wikipedia-based assignments (Zach McDowell)
- Re: Student Learning Outcomes using Wikipedia-based assignments (Kerry Raymond)
- Research about WikiProject Recommendation (Bowen Yu)
- Re: Research about WikiProject Recommendation (Kerry Raymond)
- Re: Research about WikiProject Recommendation (Jonathan Cardy)
Message: 1 Date: Mon, 19 Jun 2017 18:30:06 -0400 From: Zach McDowell zmcdowell@gmail.com To: wiki-research-l@lists.wikimedia.org Subject: [Wiki-research-l] Student Learning Outcomes using Wikipedia-based assignments Message-ID: <CAMGxLgspVCfbiKfGu=rXXxj6ihTWsaJ=TTDkSJrbkWpr1e+ f6w@mail.gmail.com> Content-Type: text/plain; charset="UTF-8"
Hi Everyone,
For the last year I've been working on a fairly large (social science) research project studying student learning outcomes using Wikipedia based assignments with the Wiki Education Foundation. This was a mixed-methods study designed to address a variety of research questions and provide open data for researchers to dig through, analyze, and utilize in whatever way they deem fit.
Today I am happy to announce that the research report, the data, the codebooks, and many other supporting documents have been released under an open license.
The research report mostly summarizes the preliminary analysis (there were a LOT of questions) of some of the qualitative and quantitative data, but it is also meant to help understand the larger scope of the research project as well. Although this is just a preliminary report, I am working on a few journal publications with this data, so this should lead to more than the report (on my end at least).
If you are interested in student learning, new users, information literacy, or skills transfer, I hope this report and data set finds you well.
Blog post by LiAnna Davis on WMF Blog: https://blog.wikimedia.org/2017/06/19/wikipedia- information-literacy-study/
Full data set (zip file): https://github.com/WikiEducationFoundation/research
Research report (commons): https://commons.wikimedia.org/wiki/File:Student_Learning_ Outcomes_using_Wikipedia-based_Assignments_Fall_2016_Research_Report.pdf
best,
Zach
Zachary J. McDowell, PhD www.zachmcdowell.com
Message: 2 Date: Tue, 20 Jun 2017 16:20:44 +1000 From: "Kerry Raymond" kerry.raymond@gmail.com To: "'Research into Wikimedia content and communities'" wiki-research-l@lists.wikimedia.org Subject: Re: [Wiki-research-l] Student Learning Outcomes using Wikipedia-based assignments Message-ID: 00e501d2e98d$5b13cad0$113b6070$@gmail.com Content-Type: text/plain; charset="utf-8"
This is very interesting data.
One observation I would make is that like many education experiments, it does not control for (what I call) the "highly motivated researcher effect". What I've learned from a lifetime of "new ways to teach" is that the standard experiment is to parachute in a highly motivated researcher into the classroom to introduce the new method, collect data showing improved learning, and then advocate for the new method to be rolled out more widely. However, rolling out more widely involves taking regular teachers (good, bad, and in-between) to learn and apply a new approach, and techniques often fail in the face of teacher lack of enthusiasm to learn anything new, complaints it makes more demands on teachers to use the new method, etc. In this report it says " the program staff provide Wikipedia training and expertise so the faculty do not need to have any experience editing" which is a big red flag to me. It would be interesting to see the results in an experiment where you first train the faculty and then the faculty carry out the engagement with students. And then see the results in 3 years time when it's a case of "business as usual" rather than "the new thing".
As a general comment, students like the variety of someone new in their classroom. Students do tend to learn more from "real world" assignments than "lab" assignments because the real world is more complex. However, staff and students are often reluctant to have real world assignments significantly influence end-of-term marks/grades because of the uncontrollable variables in the real world assignment that makes it difficult to assess the relative achievement of the students. I would expect editing Wikipedia articles to suffer from this problem as each student will be working on different article(s) of different starting size and quality and with different levels of involvement and monitoring by other Wikipedians. It was not clear to me from the report if students were being assessed on this Wikipedia assignment and how important it was to their overall mark/grade.
-----Original Message----- From: Wiki-research-l [mailto:wiki-research-l-bounces@lists.wikimedia.org] On Behalf Of Zach McDowell Sent: Tuesday, 20 June 2017 8:30 AM To: wiki-research-l@lists.wikimedia.org Subject: [Wiki-research-l] Student Learning Outcomes using Wikipedia-based assignments
Hi Everyone,
For the last year I've been working on a fairly large (social science) research project studying student learning outcomes using Wikipedia based assignments with the Wiki Education Foundation. This was a mixed-methods study designed to address a variety of research questions and provide open data for researchers to dig through, analyze, and utilize in whatever way they deem fit.
Today I am happy to announce that the research report, the data, the codebooks, and many other supporting documents have been released under an open license.
The research report mostly summarizes the preliminary analysis (there were a LOT of questions) of some of the qualitative and quantitative data, but it is also meant to help understand the larger scope of the research project as well. Although this is just a preliminary report, I am working on a few journal publications with this data, so this should lead to more than the report (on my end at least).
If you are interested in student learning, new users, information literacy, or skills transfer, I hope this report and data set finds you well.
Blog post by LiAnna Davis on WMF Blog: https://blog.wikimedia.org/2017/06/19/wikipedia- information-literacy-study/
Full data set (zip file): https://github.com/WikiEducationFoundation/research
Research report (commons): https://commons.wikimedia.org/wiki/File:Student_Learning_ Outcomes_using_Wikipedia-based_Assignments_Fall_2016_Research_Report.pdf
best,
Zach
Zachary J. McDowell, PhD www.zachmcdowell.com _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Message: 3 Date: Mon, 19 Jun 2017 23:35:22 -0700 From: Bowen Yu yuxxx856@umn.edu To: Research into Wikimedia content and communities wiki-research-l@lists.wikimedia.org Subject: [Wiki-research-l] Research about WikiProject Recommendation Message-ID: <CAGFZukk-dV+6g4Nphh7AbSCB-smP6Wx=z6kwtvAZL61+J0muwg@ mail.gmail.com> Content-Type: text/plain; charset="UTF-8"
Hi all,
We are preparing to conduct a study about WikiProject recommendations. The goals of our study are (1) to understand the effectiveness of different recommendation algorithms on recruiting new members to WikiProjects, and (2) to evaluate the effectiveness of this intervention on engaging and retaining Wikipedia newcomers.
In this study, we will recommend related editors to the organizers of WikiProjects, and request them to approach and recruit the editors. We will measure the actions and reactions of the organizers and editors for evaluation. More details about our study can be found here on this meta-page https://meta.wikimedia.org/wiki/Research:WikiProject_Recommendation.
While planning the experimental design, we thought to gather more thoughts and suggestions from the community since this study would involve the efforts of some Wikipedians, so we wanted to open it up. Also, if you know of existing work or study in this area, please let us know. Thanks!
Sincerely, Bowen
Message: 4 Date: Tue, 20 Jun 2017 17:25:51 +1000 From: "Kerry Raymond" kerry.raymond@gmail.com To: "'Research into Wikimedia content and communities'" wiki-research-l@lists.wikimedia.org Subject: Re: [Wiki-research-l] Research about WikiProject Recommendation Message-ID: 010e01d2e996$73d6d130$5b847390$@gmail.com Content-Type: text/plain; charset="utf-8"
Looking at the list of WikiProjects you pointed at, they seem to be a mixture of what I would call "process" projects (e.g. Articles for Creation, Deletion Sorting) vs "content" projects (e.g. Military History, Television) vs a third group like "Women in Red" (which is part process, part content).
Generally the "content" projects will tag Talk pages with their WikiProject Banner. But "process" projects don't seem to always do this. For example, I don't think Women in Red has a project banner generally, although I think they do tag articles that arise from specific Edit-a-thons. Some of the process projects seem to use hidden categories for their work.
I would suggest only working with content projects initially. Content projects are more similar to one another in how they operate compared to process projects, and I think it is easier to judge if a user is showing an interest in a content project than in the process project because of standard use of content project banners on articles. So I think you can probably get a better understanding if the referral mechanism is working or not with content projects, whereas I think process projects have a lot of variability in them that may make it difficult to work out if you are seeing success or not.
And at the end of the day, as an encyclopedia, we live or die on our content. Processes are (or at least should be) supportive of good content development but are a second-order effect.
I can certainly see some issues arising from pointing newcomers at process projects as they are unlikely to be aware of the processes at that stage. And indeed some process project do not accept new editors (think of Articles for Creation and new page patrolling). I'd see this as a second project if the content project referral mechanism seems to be working.
Anyhow, that my 10cc!
Kerry
----Original Message----- From: Wiki-research-l [mailto:wiki-research-l-bounces@lists.wikimedia.org] On Behalf Of Bowen Yu Sent: Tuesday, 20 June 2017 4:35 PM To: Research into Wikimedia content and communities < wiki-research-l@lists.wikimedia.org> Subject: [Wiki-research-l] Research about WikiProject Recommendation
Hi all,
We are preparing to conduct a study about WikiProject recommendations. The goals of our study are (1) to understand the effectiveness of different recommendation algorithms on recruiting new members to WikiProjects, and (2) to evaluate the effectiveness of this intervention on engaging and retaining Wikipedia newcomers.
In this study, we will recommend related editors to the organizers of WikiProjects, and request them to approach and recruit the editors. We will measure the actions and reactions of the organizers and editors for evaluation. More details about our study can be found here on this meta-page https://meta.wikimedia.org/wiki/Research:WikiProject_ Recommendation.
While planning the experimental design, we thought to gather more thoughts and suggestions from the community since this study would involve the efforts of some Wikipedians, so we wanted to open it up. Also, if you know of existing work or study in this area, please let us know. Thanks!
Sincerely, Bowen _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Message: 5 Date: Tue, 20 Jun 2017 11:02:21 +0100 From: Jonathan Cardy werespielchequers@gmail.com To: Research into Wikimedia content and communities wiki-research-l@lists.wikimedia.org Subject: Re: [Wiki-research-l] Research about WikiProject Recommendation Message-ID: 33C5D254-B697-4F62-AF74-3948A75750B1@gmail.com Content-Type: text/plain; charset=us-ascii
Hi Bowen,
If you are going to promote wikiprojects by recommendation then you need to test different styles of recommendation. Taking what may still be the two biggest wikiprojects, MILHIST and professional wrestling, what worked as an invitation for either might be quite different than what would work for Opera or chemistry. Tone of voice is important when you are seeking to entice volunteers.
You also need to allow for the effect of different existing recruitment programs. These tend to be subtle, but they will vary, and that variation could mask your project. The most obvious recruitment is via wikiproject tagging of articles, and that isn't necessarily done by people who are active in the project concerned.
Regards
Jonathan
On 20 Jun 2017, at 07:35, Bowen Yu yuxxx856@umn.edu wrote:
Hi all,
We are preparing to conduct a study about WikiProject recommendations.
The
goals of our study are (1) to understand the effectiveness of different recommendation algorithms on recruiting new members to WikiProjects, and (2) to evaluate the effectiveness of this intervention on engaging and retaining Wikipedia newcomers.
In this study, we will recommend related editors to the organizers of WikiProjects, and request them to approach and recruit the editors. We
will
measure the actions and reactions of the organizers and editors for evaluation. More details about our study can be found here on this
meta-page
https://meta.wikimedia.org/wiki/Research:WikiProject_Recommendation.
While planning the experimental design, we thought to gather more
thoughts
and suggestions from the community since this study would involve the efforts of some Wikipedians, so we wanted to open it up. Also, if you
know
of existing work or study in this area, please let us know. Thanks!
Sincerely, Bowen _______________________________________________ Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
Subject: Digest Footer
Wiki-research-l mailing list Wiki-research-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wiki-research-l
End of Wiki-research-l Digest, Vol 142, Issue 13
Thanks, Zach, for the explanation of these things. I realise the goals of WikiEd was not to be a research project but achieve other goals in relation to Wikipedia, in which it seems to be very successful from your data.
I suspect part of my confusion relates to the terms “instructor”, “faculty”, and “program staff” (noting that terminology in North American universities is very different to Australian universities). Can you unpick these for me? Are the “instructors” academic staff of a university or college, or, as I interpreted it, people who would probably identify as Wikipedians who are volunteering to work on the WikiEd program in their local area? I assume “faculty” here means “academic staff member” (in my world, “faculty” is an organisational unit composed of academic staff and non-academic staff). Who are the “program staff”?
Thanks
Kerry
Hi Kerry,
This research project was really conducted to examine a couple different things at once. This project was designed to provide data for researchers trying to understand learning with Wikipedia. Wiki Ed's dashboard was perfect for this because students participation was already were being tracked, so layering research on top of that data (and access to emailing them, cohorts of classes, etcetera) made it an ideal setting for educational research.
As mentioned in the research report, the questions were designed by multiple researchers from a variety of backgrounds, so there isn't really "one" research question, but many approaches to trying to understand what students are learning, including motivations, perceived value, and skills transfer, among other things.
I think the confusion here might be coming from the fact Wiki Education works a little differently than WMF's Education Program - there is a lot of information on how they operate at their website here - https://wikiedu.org/
Wiki Education as an organization isn't a research organization, it is an educational organization with the dual mission of education and improving Wikipedia. They funded the research project but I conducted it through UMass Amherst.
As for terminology, "instructor" is anyone who is an instructor for the class (sometimes TT faculty, sometimes adjunct, sometimes grad student, sometimes librarian). There are usually no additional Wikipedians in the classroom, unless the instructor has a specific contact that they invited in, unbenownst to me.
"Program staff" are folks that work for Wiki Education and support the classes.
In general, Wiki Ed has a "dashboard" and some support staff, but the students are not "taught" by a Wikipedian.
I hope this helps!
best Zach
On Tue, Jun 20, 2017, 5:48 PM Kerry Raymond kerry.raymond@gmail.com wrote:
Thanks, Zach, for the explanation of these things. I realise the goals of WikiEd was not to be a research project but achieve other goals in relation to Wikipedia, in which it seems to be very successful from your data.
I suspect part of my confusion relates to the terms “instructor”, “faculty”, and “program staff” (noting that terminology in North American universities is very different to Australian universities). Can you unpick these for me? Are the “instructors” academic staff of a university or college, or, as I interpreted it, people who would probably identify as Wikipedians who are volunteering to work on the WikiEd program in their local area? I assume “faculty” here means “academic staff member” (in my world, “faculty” is an organisational unit composed of academic staff and non-academic staff). Who are the “program staff”?
Thanks
Kerry
wiki-research-l@lists.wikimedia.org