(Pardon me if I'm doing something wrong regarding the mailing list, I'm a newbie to posting despite lurking for some time)
David Lindsey wrote:
<snip>
I would like to think that we can all agree that Professor Wertheim's critical response is a helpful one, and that adapting the article appropriately would be of benefit to Wikipedia. The entire process of finding an expert to review the article took no more than a half hour of my time and produced a benefit for Wikipedia that substantially exceeds that cost.
</snip>
Yes, the critical response is helpful, and reacting to it would be a good idea.
I do raise my eyebrows at the time required, though. There are plenty of difficulties involved in finding an appropriate expert, contacting them, and—importantly—convincing them to use their time to review the article. It's hard to generalize that most Wikipedians should be able to find a (willing) expert on a subject in so little time, especially for more obscure or less academic topics.
<snip>
Finally, though this idea failed to gain any real traction on wiki, I would like to state my support for the idea of adding a fifth criterion to WP:WIAFA: "5. The article, if possible, has been reviewed by an external subject-matter expert." Even if no such criterion is added, though, I would like to emphasize that it will always, or nearly always, be productive to attempt to find an expert reviewer.
</snip>
Yes, expert reviewers are generally a good thing. I think that there might be some trouble with controversial articles (if the expert is slanted one way or another, a review correcting "mistakes" might not be helpful to NPOV). The dubiousness of Citizendium's homeopathy article makes a good example.
I don't currently believe that expert review should be added as a criterion for Featured articles, however, because of the classic problems with experts. First, how would the interactions/credentials be verified? I can just imagine it now: "I'm submitting this article 'Unobtainium synthesis' to FAC because I think it meets the criteria. Also, the article got an excellent rating on review by Dr. L. Olfake, which she emailed to me the other day." (In case you didn't catch it: "lol fake") Second, how do we avoid Citizendium-like problems? (This is certainly an issue, see also the earlier "Citizendium dead?" thread in this list) Third, would it add systemic bias, or extra hurdles to the process, as Charles Matthews mentions? I'll stop touting the issues, because that's not quite helpful, but there are significant problems with integrating expert review that we ought to address (if not solve) before making it remotely mandatory.
If we were to integrate expert review, I'd start it as another *layer* of Wikipedia. I strongly believe that showing very prominently the level of review a given article—or even a given *revision* thereof—has received, and the perceived level of quality involved, is a good thing. The Wikipedia 1.0 assessment system (Stub, Start, C, B, A, GA, FA…) seems to serve as a decent start for that sort of thing. (In the long run, I'd love to see some system integrating this with some addition to the planned "patrolled revisions" feature that adds passive flags to revisions.) If we were to bolt on an expert review process to the Featured article system, why not make it a new level? I can imagine it now: "FA+". Take a community FA, and give it a (reasonably verified) expert review, and if all outstanding issues are addressed, call it an FA+ instead. Granted, this involves all sorts of instruction creep, but it's just a starting idea. A practical implementation is left as an exercise for the reader. ;)
I think that expert review is a good idea (we like expert participation as long as they're not jerks about it), but I'm cautious about its feasibility in Wikipedia as an "official" process.
</ramble>
Cheers, Nihiltres
Nihiltres wrote:
<snip> I strongly believe that showing very prominently the level of review a given article—or even a given *revision* thereof—has received, and the perceived level of quality involved, is a good thing. The Wikipedia 1.0 assessment system (Stub, Start, C, B, A, GA, FA…) seems to serve as a decent start for that sort of thing.
If we are honest with ourselves, we would admit that we really need levels 1 to 10 for articles. It seems already to be hard to get an A, fairly much impossible to get GA for an "average" topic, and as we know only 1 in 1000 is FA (in round terms). And "expert review" = FA+ is another quite defensible level. I think cutting to the chase, setting substub = 1 and reviewed FA = 10 might be a great timesaver, and help a process in which less "mystique" attached to the whole business. Rebooting with FA = 9 sounds quite fun.
Charles
On 27 April 2010 20:50, Charles Matthews charles.r.matthews@ntlworld.com wrote:
Nihiltres wrote:
<snip> I strongly believe that showing very prominently the level of review a given article—or even a given *revision* thereof—has received, and the perceived level of quality involved, is a good thing. The Wikipedia 1.0 assessment system (Stub, Start, C, B, A, GA, FA…) seems to serve as a decent start for that sort of thing.
If we are honest with ourselves, we would admit that we really need levels 1 to 10 for articles. It seems already to be hard to get an A, fairly much impossible to get GA for an "average" topic, and as we know only 1 in 1000 is FA (in round terms). And "expert review" = FA+ is another quite defensible level. I think cutting to the chase, setting substub = 1 and reviewed FA = 10 might be a great timesaver, and help a process in which less "mystique" attached to the whole business. Rebooting with FA = 9 sounds quite fun.
I realised a few months ago that it had been ages since I'd actually done anything significant in the main namespace, so I decided to have a go at writing an article. With a little help from someone that turned up and started improving the article (in true wiki-fashion), I got it to GA fairly easily. It was at best an "average" topic - it was my local (about 700 year old) church. FAC is very difficult to get through, but GA is entirely doable.
I think adding more levels would make the distinctions more arbitrary, which seems like a bad thing to me. I think we should remove a level, in fact. The current system at the top with A, GA and FA is very confusing. I think GA and A should be merged somehow (perhaps just get rid of A).
Thomas Dalton wrote:
On 27 April 2010 20:50, Charles Matthews charles.r.matthews@ntlworld.com wrote:
Nihiltres wrote:
<snip> I strongly believe that showing very prominently the level of review a given article—or even a given *revision* thereof—has received, and the perceived level of quality involved, is a good thing. The Wikipedia 1.0 assessment system (Stub, Start, C, B, A, GA, FA…) seems to serve as a decent start for that sort of thing.
If we are honest with ourselves, we would admit that we really need levels 1 to 10 for articles. It seems already to be hard to get an A, fairly much impossible to get GA for an "average" topic, and as we know only 1 in 1000 is FA (in round terms). And "expert review" = FA+ is another quite defensible level. I think cutting to the chase, setting substub = 1 and reviewed FA = 10 might be a great timesaver, and help a process in which less "mystique" attached to the whole business. Rebooting with FA = 9 sounds quite fun.
I realised a few months ago that it had been ages since I'd actually done anything significant in the main namespace, so I decided to have a go at writing an article. With a little help from someone that turned up and started improving the article (in true wiki-fashion), I got it to GA fairly easily. It was at best an "average" topic - it was my local (about 700 year old) church. FAC is very difficult to get through, but GA is entirely doable.
I think adding more levels would make the distinctions more arbitrary, which seems like a bad thing to me. I think we should remove a level, in fact. The current system at the top with A, GA and FA is very confusing. I think GA and A should be merged somehow (perhaps just get rid of A).
Well, the research I remember says the transition from B to A makes the most difference to the reader. So I would make that central to any system: from 5 to 6, say. I have seen perfectly decent articles labelled "Start" - I mean articles with say five paras of solid, verifiable factual information. I doubt standards are even across the wiki, but if those are "Start" there have to be a couple of rungs on the ladder below that.; or Start = 3. I see that mathematics uses B+ anyway, so that the lower side has five grades already. There does seem to be some problem with A right now, but abolishing it in such a fashion to reduce incentives to push articles up would really be a bad idea (whatever your anecdotal example says).
Charles
On 27 April 2010 21:33, Charles Matthews charles.r.matthews@ntlworld.com wrote:
Well, the research I remember says the transition from B to A makes the most difference to the reader. So I would make that central to any system: from 5 to 6, say. I have seen perfectly decent articles labelled "Start" - I mean articles with say five paras of solid, verifiable factual information. I doubt standards are even across the wiki, but if those are "Start" there have to be a couple of rungs on the ladder below that.; or Start = 3. I see that mathematics uses B+ anyway, so that the lower side has five grades already. There does seem to be some problem with A right now, but abolishing it in such a fashion to reduce incentives to push articles up would really be a bad idea (whatever your anecdotal example says).
But what is the difference between A and GA? Really, it's minimal (I think A-class requires the content to be essentially complete, GA just requires it to cover all the main points, which isn't much different). You talk about the transition from B to A - is most of that difference to readers between B and GA or between GA and A (I know the ordering isn't perfect, but any A-class article should be able to pass GA with only minimal changes)? I suspect it is between B and GA, so getting rid of A wouldn't have any significant impact.
One alternative is to scrap the entire system and replace it with a points system. We have a few categories like "completeness", "style", "images", "references", etc. and an article gets a certain number of points in each category depending on how good it is. Once an article has the maximum points in each category, it is ready for FAC, which basically is just to confirm the assessment was accurate (the categories should be set up with the FA criteria in mind). This would mean people working on the article know what areas need more work, it gives an incentive to even fairly small improvements and it removes the arbitrary distinctions between different classes of article.
Thomas Dalton wrote:
On 27 April 2010 21:33, Charles Matthews charles.r.matthews@ntlworld.com wrote:
Well, the research I remember says the transition from B to A makes the most difference to the reader. So I would make that central to any system: from 5 to 6, say. I have seen perfectly decent articles labelled "Start" - I mean articles with say five paras of solid, verifiable factual information. I doubt standards are even across the wiki, but if those are "Start" there have to be a couple of rungs on the ladder below that.; or Start = 3. I see that mathematics uses B+ anyway, so that the lower side has five grades already. There does seem to be some problem with A right now, but abolishing it in such a fashion to reduce incentives to push articles up would really be a bad idea (whatever your anecdotal example says).
But what is the difference between A and GA? Really, it's minimal (I think A-class requires the content to be essentially complete, GA just requires it to cover all the main points, which isn't much different). You talk about the transition from B to A - is most of that difference to readers between B and GA or between GA and A (I know the ordering isn't perfect, but any A-class article should be able to pass GA with only minimal changes)? I suspect it is between B and GA, so getting rid of A wouldn't have any significant impact.
[[Talk:Go (game)/GA2]] is the only GA review I have ever looked at: it has many comments (measurements in both metric and imperial, for example) that ar far from your summary.
Charles
On 27 April 2010 23:14, Charles Matthews charles.r.matthews@ntlworld.com wrote:
Thomas Dalton wrote:
On 27 April 2010 21:33, Charles Matthews charles.r.matthews@ntlworld.com wrote:
Well, the research I remember says the transition from B to A makes the most difference to the reader. So I would make that central to any system: from 5 to 6, say. I have seen perfectly decent articles labelled "Start" - I mean articles with say five paras of solid, verifiable factual information. I doubt standards are even across the wiki, but if those are "Start" there have to be a couple of rungs on the ladder below that.; or Start = 3. I see that mathematics uses B+ anyway, so that the lower side has five grades already. There does seem to be some problem with A right now, but abolishing it in such a fashion to reduce incentives to push articles up would really be a bad idea (whatever your anecdotal example says).
But what is the difference between A and GA? Really, it's minimal (I think A-class requires the content to be essentially complete, GA just requires it to cover all the main points, which isn't much different). You talk about the transition from B to A - is most of that difference to readers between B and GA or between GA and A (I know the ordering isn't perfect, but any A-class article should be able to pass GA with only minimal changes)? I suspect it is between B and GA, so getting rid of A wouldn't have any significant impact.
[[Talk:Go (game)/GA2]] is the only GA review I have ever looked at: it has many comments (measurements in both metric and imperial, for example) that ar far from your summary.
Sorry, that bit in brackets wasn't meant to be a summary of the criteria for each class, it was a description of the difference between the classes. Each has lots of other criteria, but they are essentially the same for both.
Thomas Dalton wrote:
Sorry, that bit in brackets wasn't meant to be a summary of the criteria for each class, it was a description of the difference between the classes. Each has lots of other criteria, but they are essentially the same for both.
Getting back to one of the main points: I think we could have a clearer system, certainly, and I think clarity should be asked for on behalf of the readers, who outnumber the writers.
It seems that there are basically two things that go on: material is found for an article on topic T; and then the way the article on T is written gets reviewed in a box-ticking kind of way, mostly for conformity to the Manual and referencing. Which is fair enough. The points at issue seem to be:
- At what level of advancement of the article T should it actually be "commended" to the reader (implicitly) by the rating? - Beyond that level, should the number of rungs of the ladder be made small (fewer but more taxing reviews), or larger (more hurdles, each of which deals with a limited number of matters)?
My vague suggestion for the first part is that "rate on a scale of 1 to 10" is intuitive for just about anyone as reader, but our traditional labels seem more designed for writers. Thinking B+ = 5 and A = 6 at least puts a more normal complexion on what we are talking about. As for the second part, it is not particularly something that bothers me, given the way I have always worked. But making reviewing more "modular" (and predictable, removing the "instruction creep" that moves goalposts) would seem sensible, so I'm for more layers.
After all, professional book production would tend to distinguish editorial input, subediting, and copy editing as phases. The thread is about outside review, which is yet another idea, but with a book would be tried for at an earlier stage, I think.
Charles
Nihiltres wrote:
(Pardon me if I'm doing something wrong regarding the mailing list, I'm a newbie to posting despite lurking for some time)
David Lindsey wrote:
Finally, though this idea failed to gain any real traction on wiki, I would like to state my support for the idea of adding a fifth criterion to WP:WIAFA: "5. The article, if possible, has been reviewed by an external subject-matter expert." Even if no such criterion is added, though, I would like to emphasize that it will always, or nearly always, be productive to attempt to find an expert reviewer.
</snip>
Yes, expert reviewers are generally a good thing. I think that there might be some trouble with controversial articles (if the expert is slanted one way or another, a review correcting "mistakes" might not be helpful to NPOV). The dubiousness of Citizendium's homeopathy article makes a good example.
+1
I don't currently believe that expert review should be added as a criterion for Featured articles, however, because of the classic problems with experts. First, how would the interactions/credentials be verified? I can just imagine it now: "I'm submitting this article 'Unobtainium synthesis' to FAC because I think it meets the criteria. Also, the article got an excellent rating on review by Dr. L. Olfake, which she emailed to me the other day." (In case you didn't catch it: "lol fake") Second, how do we avoid Citizendium-like problems? (This is certainly an issue, see also the earlier "Citizendium dead?" thread in this list) Third, would it add systemic bias, or extra hurdles to the process, as Charles Matthews mentions? I'll stop touting the issues, because that's not quite helpful, but there are significant problems with integrating expert review that we ought to address (if not solve) before making it remotely mandatory.
If we were to integrate expert review, I'd start it as another *layer* of Wikipedia. I strongly believe that showing very prominently the level of review a given article—or even a given *revision* thereof—has received, and the perceived level of quality involved, is a good thing. The Wikipedia 1.0 assessment system (Stub, Start, C, B, A, GA, FA…) seems to serve as a decent start for that sort of thing. (In the long run, I'd love to see some system integrating this with some addition to the planned "patrolled revisions" feature that adds passive flags to revisions.) If we were to bolt on an expert review process to the Featured article system, why not make it a new level? I can imagine it now: "FA+". Take a community FA, and give it a (reasonably verified) expert review, and if all outstanding issues are addressed, call it an FA+ instead. Granted, this involves all sorts of instruction creep, but it's just a starting idea. A practical implementation is left as an exercise for the reader. ;)
I think that expert review is a good idea (we like expert participation as long as they're not jerks about it), but I'm cautious about its feasibility in Wikipedia as an "official" process.
</ramble>
First let me just say generally pretty much every word in your "ramble" is genuinely insightful. Let me share some thoughts it inspired in myself.
To me it seems people who could be tapped for outside input to wikipedia, even when they are people who generally don't want to edit wikipedia actively themselves, shouldn't be limited to academics by any means. Think journalists and professional people in their field doing work as an ordinary day job just as two examples.
I wouldn't limit such input into a single mold either. There could be one system developed for just quickly checking a single fact and asking if they could point to an authoritative source which could be cited in the article. Somebody who knows their shit might have things right handy, and not mind telling what it is, so long as they don't have to insert it into wikipedia themselves, and watch over it to make sure it stays put.
Another system might be recruit such folks to give an impartial summary overview of what problems if any kind a specific article in their field might have, be it bias, unbalance of coverage, facts missing, surfeit of inessential information stuffed in, or simple errors of fact. I do agree that in no way should this kind of system be married to the FA process, for the reason that I don't think there is grounds for limiting it to articles on that level, and of course it would add a new hurdle to the FA process if it was absollutely mandatory, and new hurdles the FA process doesn't really need. And necessarily the resulting summary view of the article could *never* be thought to be genuinely authoritative; that would just be outright impossible. I think somebody said of economics as a science that if you ask for an opinion on a question on their field from 3 experts, you get 7 opinions. The only way you could remotely make that work is if you had for each field a full _panel_ of experts doing the reviewing and I am sure everybody understands how heavy such a system would be.
One interesting idea though, would be in some far future, to actually *hire* fact-checkers for that elusive printed on paper (or other fixed media), which would be genuinely of a higher consistency than the raw product we daily see and will continue to forevermore see on our "unending construction site". I wouldn't hold my breath to seeing it quite next year though. The money just isn't going to be there soon to hire people to even attempt that monumental task, in its entirety.
Yours (sorry about the length of the post),
Jussi-Ville Heiskanen