Okay so followed up with Samir. While the database of questions was selected by he and I, neither one of us did any specific selection beyond randomly selecting 25.
With respect to students going and changing Wikipedia / Uptodate, I very much doubt they would have. There is other students that have found that even when medical students find errors in WP they do not bother fixing them. Both WP and Uptodate change slowly over time.
James
On Thu, Nov 2, 2017 at 4:52 PM, Jean-Philippe Béland jpbeland@wikimedia.ca wrote:
This could have been alleviated by using a dump of Wikipedia at a specific time throughout the study. I don't know if it was done or not, I doubt it since the article do not mention it, I assume they had direct online access to the current Wikipedia at the time of the iterations during the study. Also that would lift one of the concerns in the discussion section about the replicability of the study because Wikipedia evolves, a new study could be completed with the same dump at the time of that study in order to replicate the same results (however I wouldn't see the interest, but just for the sake of having scientifically replicable findings).
JP
On Thu, Nov 2, 2017 at 11:45 AM, Jean-Philippe Béland <jpbeland@wikimedia.ca
wrote:
Hi James,
I finally found time to read the whole article carefully. It is a very well done article and study, in my opinion.
I have one remark/question. Since the study was conducted over a length of time (April 2014 to December 2016), was the data analyzed to see if the increase in the results of good answers in the posttest was higher later during the study (or not) since Wikipedia (and maybe UpToDate, I am not familiar with that resource) evolves with time? Maybe even students who participated in the first iteration of this study went after to improve the related Wikipedia articles, thus obviously having an impact on the results since the information about the specific questions that you retained for the MCQ were "directly" answered on Wikipedia. Is this something that was considered? I do not see that consideration in the discussion section of the article.
Thank you,
JP
On Thu, Nov 2, 2017 at 4:37 AM, James Heilman jmh649@gmail.com wrote:
That bit of the paper could have been a bit clearer. I simple downloaded 100 questions at random from a website that hosts lists of exam question. Am checking with Samir regarding if he did any further selection beyond that.
James
On Thu, Nov 2, 2017 at 2:30 AM, pajz pajzmail@gmail.com wrote:
On 31 October 2017 at 17:09, James Heilman jmh649@gmail.com wrote:
Full study available under an open license at https://mededu.jmir.org/2017/2/e20/
If one gets to chose the questions and assemble the questionnaire then shown to all study participants, I would submit that more or less
arbitrary
study results can be generated by, consciously or subconsciously,
picking
the "right" questions. Curiously, the two people that "reviewed" the questions here were "a Wikipedia editor and administrator," and a "long-term volunteer editor and administrator of Wikipedia" and
"founder of
[...] the Wiki Project Med Foundation."
Not being negative or anything, but if you're trying to scientifically evaluate whether a given exam prep book improves students' grades, would you let the editors of the book prepare the test exam?
Best, Patrik _______________________________________________ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wik
i/Mailing_lists/Guidelines and https://meta.wikimedia.org/wik i/Wikimedia-l
New messages to: Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l,
mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe
-- James Heilman MD, CCFP-EM, Wikipedian
Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wik i/Mailing_lists/Guidelines and https://meta.wikimedia.org/wik i/Wikimedia-l New messages to: Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe
--
Jean-Philippe Béland
[image: Wikimedia Canada] Vice-président — Wikimédia Canada https://ca.wikimedia.org/wiki/Main_Page?uselang=fr, chapitre national soutenant Wikipédia Vice president — Wikimedia Canada https://ca.wikimedia.org/wiki/Main_Page?uselang=en, national chapter supporting Wikipedia 535 avenue Viger Est, Montréal (Québec) H2L 2P3,jpbeland@wikimedia.ca
--
Jean-Philippe Béland
[image: Wikimedia Canada] Vice-président — Wikimédia Canada https://ca.wikimedia.org/wiki/Main_Page?uselang=fr, chapitre national soutenant Wikipédia Vice president — Wikimedia Canada https://ca.wikimedia.org/wiki/Main_Page?uselang=en, national chapter supporting Wikipedia 535 avenue Viger Est, Montréal (Québec) H2L 2P3,jpbeland@wikimedia.ca _______________________________________________ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines and https://meta.wikimedia.org/wiki/Wikimedia-l New messages to: Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe