Today ORES in production was sending out unreasonable amount of timeout
errors. Causing icinga to scream and 14% failure rate on average for ORES
review tool jobs. It turned out that ores workers are logging too much
causing the nodes to run out of disk space. [1] I suspect we had similar
issue in our labs nodes.
I made changes for prod and labs and deployed it today. You can find more
details in the phab card
[1]: https://phabricator.wikimedia.org/T146581
Cheers
Best
Hey,
This is the 22nd weekly update from revision scoring team that we have sent
to this mailing list.
UI work:
- We configured the default threshold for the ORES review tool on
Wikidata to be more strict (higher recall, lower precision)[1]
- We fixed a display issue on Special:Contributions where the filters
would not wrap[2]
Increasing model fitness:
- We finished demonstrating model fitness gains using hash-vector
features[3]. Next, we'll be working to get the hash-vector features
implemented in revscoring/ORES[4].
- We implemented a new strategy for training and testing on all data
using cross-validation[5]. This will both increase the fitness of the
models and make the statistics reported more robust.
Maintenance and robustness
- We fixed an indexing issues in ores_model that prevented the
deployment of updated models[6].
- We did a minor investigation to a short period of degraded service
quality on WMF Labs[7]
1. https://phabricator.wikimedia.org/T144784 -- Change default threshold
for Wikidata to high
2. https://phabricator.wikimedia.org/T143518 -- Filter on user contribs has
nowrap, causing issues
3. https://phabricator.wikimedia.org/T128087 -- [Spike] Investigate
HashingVectorizer
4. https://phabricator.wikimedia.org/T145812 -- Implement ~100 most
important hash vector features in editquality models
5. https://phabricator.wikimedia.org/T142953 -- Train on all data, Report
test statistics on cross-validation
6. https://phabricator.wikimedia.org/T144432 -- oresm_model index should
not be unique
7. https://phabricator.wikimedia.org/T145353 -- Investigate short period of
ores-web-03 insanity
Sincerely,
Aaron from the Revision Scoring team
One of ORES [1] applications is determining article quality. For example,
What would be the best assessment of an article in the given revision.
Users in wikiprojects use ORES data to check if articles need
re-assessment. e.g. if an article is in "Start" level and now good it's
enough to be a "B" article.
As part of Q4 goals, we made a dataset of article quality scores of all
articles in English Wikipedia [2] (Here's the link to download the dataset
[3]) and we are publishing it in figshare as something you can cite [4]
also we are working on publishing monthly data for researchers to track
article quality data change over time. [5]
As a pet project of mine, I always wanted to put these data in a database.
So we can query the database and get much more useful data. For example
quality of articles in category 'History_of_Essex' [6] [7]. The weighed sum
is a measure of quality which is a decimal number between 0 (really stub)
to 5 (a definitely featured article). We have also prediction column which
is a number in this map [8] for example if prediction is 5, it means ORES
thinks it should be a featured article.
I leave more use cases to your imagination :)
I'm looking for a more permanent place to put these data, please tell me if
it's useful for you.
[1] ORES is not a anti-vandalism tool, it's an infrastructure to use AI in
Wikipedia.
[2] https://phabricator.wikimedia.org/T135684
[3] (117 MBs)
https://datasets.wikimedia.org/public-datasets/enwiki/article_quality/wp10-…
[4] https://phabricator.wikimedia.org/T145332
[5] https://phabricator.wikimedia.org/T145655
[6] https://quarry.wmflabs.org/query/12647
[7] https://quarry.wmflabs.org/query/12662
[8]
https://github.com/wiki-ai/wikiclass/blob/3ff2f6c44c52905c7202515c5c8b525fb…
Have fun!
Amir
Amir,
Thanks for this. I mean WOW for lack of better words.
I'm especially impressed with the inclusion of the weighted scores
which allows the observation of small changes in quality. I was going
to suggest that you could do the same thing for the end of every year,
say for the last 5 years, so that we can see the improvement in
articles - but that would be too much to ask. But then I noticed you
are planning on doing this monthly. Double WOW.
Minor quibble - their are lots of disambiguation pages included. I'd
delete those if possible.
Thanks again,
Pete
PS - WOW
Hi Joe,
The particular use case that I have in mind is that a reader, or potential
contributor, wants to know the quality of a particular article. (Note that
this is a slightly different use case from sorting many articles by their
ORES scores, or comparing their ORES scores to the importance and/or
manually assigned article quality grade.) ORES scores are likely to be more
frequently updated than manual article quality grades, and I feel that this
information would be useful to readers and/or potential contributors.
Having the ORES score templates on talk pages, placed there by a bot and
later refreshed weekly, would be adequate for the use case that I have in
mind.
Note that https://upload.wikimedia.org/wikipedia/commons/6/6a/
Wikimedia_Foundation_and_Reboot_New_Readers_Research_-_
Nigeria_%26_India_Highlights_-_July_2016.pdf suggests that WMF Product "Add
an indicator of quality icon (akin to the signal strength icon for mobile
networks) on each article, based on research and analysis on article
contributions and edits." I am thinking that ORES scores, and/or manual
article quality grades, could serve this purpose. English Wikipedia already
displays some article grades on the front pages of articles via the GA or
FA icons, and my impression is that the New Readers Research suggests
making some kind of quality indicator more prominent and on all Wikipedia
wikis. This would involve work that's beyond the scope of what I have in
mind, but I think that it's worth noting for potential future development.
Abbey may wish to comment on this.
Pine
On Fri, Sep 16, 2016 at 2:06 PM, Joe Matazzoni <jmatazzoni(a)wikimedia.org>
wrote:
> The project we’re working on with ORES (Edit Review Improvements
> <https://www.mediawiki.org/wiki/Edit_Review_Improvements> ) is concerned
> only with the Damaging and Good Faith scores at this time. So I don’t know
> too much about the ORES article scores. I believe they have a lot of
> potential, but I’m a little unclear on the general goal of this proposal.
>
> My sense of the ORES article scores is that they’re most useful for people
> who are not (yet) looking at the article. E.g., the scores might be used to
> review AfC backlogs and surface the pages that are ready for re-review. Or
> they might be used to help reviewers engaged in finding and promoting
> Featured Articles. They might conceivably have a place in search results
> some day, as additional data for readers.
>
> My guess is that these scores are based at this time on formal
> characteristics of the article — its length, its conformance with
> formatting standards, whether it includes photos or multimedia, whether it
> has references, etc. These formal characteristics would for the most part,
> I’d think, be obvious to someone already on the article page.
>
> But ORES does not know if a subject is notable, for example. Nor do t
> believe it would know if an article were factually correct, among other
> things.
>
> So, again, what is your goal? Is there a Phabricator task? Good luck!
>
> Joe
>
>
>
> _____________________
>
> *Joe Matazzoni*
> Product Manager, Collaboration
> Wikimedia Foundation, San Francisco
> mobile 202.744.7910
> jmatazzoni(a)wikimedia.org
> *"Imagine a world in which every single human being can freely share in
> the sum of all knowledge." *
>
>
>
>
> On Sep 15, 2016, at 8:57 PM, Pine W <wiki.pine(a)gmail.com> wrote:
>
> Joe, any comment about this?
>
> Toby and Amir, would it make sense to integrate ORES scores into
> https://meta.wikimedia.org/wiki/Community_Tech/PageAssessments, or have
> some other kind of relation between the ability to pull ORES scores from
> the MW API and the PageAssessments API? Or does it make sense to have them
> be separate but working in parallel? I'm an API novice, but the
> functionality of PageAssessments reminds me of what we're discussing here,
> which is why I bring it up.
>
> Thanks,
>
> Pine
>
> On Wed, Aug 31, 2016 at 2:48 PM, Toby Negrin <tnegrin(a)wikimedia.org>
> wrote:
>
>> Hi Pine -- thanks for this. I'm looping in Joe M. from the Collaboration
>> team who can speak to his team's thoughts on ORES scores.
>>
>> It seems likely that my team will devote some resources to making ORES (+
>> the page view API) accessible via the MW API next quarter which will help
>> integration efforts. Our goals go up on Friday and I just got out of a
>> meeting where we agreed to do this and it's unlikely this will change in 2
>> days.
>>
>> As far as New Readers though, I suspect that it's a bit too abstract as
>> of yet. I think we should start with Old Readers :)
>>
>> -Toby
>>
>> On Wed, Aug 31, 2016 at 2:37 PM, Pine W <wiki.pine(a)gmail.com> wrote:
>>
>>> Just to note that https://upload.wikimedia.org/w
>>> ikipedia/commons/6/6a/Wikimedia_Foundation_and_Reboot_New_Re
>>> aders_Research_-_Nigeria_%26_India_Highlights_-_July_2016.pdf suggests
>>> that WMF Product "Add an indicator of quality icon (akin to the signal
>>> strength icon for mobile networks) on each article, based on research and
>>> analysis on article contributions and edits." This is similar to what I
>>> have in mind for talk pages with ORES scores. Adding indicators to the
>>> front page of articles would likely require an RfC which may be more
>>> involved than an RfB to add the score to talk pages. Still, I can see how
>>> having an ORES score, or the Wikipedia Quality score (at least on ENWP)
>>> indicated by an icon at the top of article pages would be useful,
>>> especially if readers can intuitively understand what the indicator is
>>> telling them without needing to look at documentation.
>>>
>>> I think it might be best to start with ORES scores on talk pages. That
>>> fills the need that I have in mind. If someone wants to extend this to
>>> having article quality indicators on the front page of articles, I'd be
>>> happy to collaborate with them.
>>>
>>> Pine
>>>
>>> On Tue, Aug 16, 2016 at 7:32 AM, Pine W <wiki.pine(a)gmail.com> wrote:
>>>
>>>> Thanks. I will add this to my agenda for later this year.
>>>>
>>>> Pine
>>>>
>>>> On Aug 16, 2016 01:48, "Amir Ladsgroup" <ladsgroup(a)gmail.com> wrote:
>>>>
>>>>> Oh, Sorry I missed the question. Yes, it's fairly easy :)
>>>>>
>>>>> Best
>>>>>
>>>>> On Tue, Aug 16, 2016 at 12:15 PM Pine W <wiki.pine(a)gmail.com> wrote:
>>>>>
>>>>>> Pinging again to repeat my question from my previous email.
>>>>>>
>>>>>> Thanks,
>>>>>> Pine
>>>>>>
>>>>>> On Aug 9, 2016 16:04, "Pine W" <wiki.pine(a)gmail.com> wrote:
>>>>>>
>>>>>>> I'd like to work on this if I had the time to learn and to do it.
>>>>>>> Maybe I will in early September or sometime later this year. I've put it on
>>>>>>> my to do list. This would be my first bot; do you think that it's a good
>>>>>>> project for a newbie bot builder?
>>>>>>>
>>>>>>> Pine
>>>>>>>
>>>>>>> On Sat, Aug 6, 2016 at 11:16 AM, Amir Ladsgroup <ladsgroup(a)gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Building that is easy, you just need someone to add these
>>>>>>>> functionalities to the templates and a bot to run it for the first time.
>>>>>>>>
>>>>>>>> Any volunteers?
>>>>>>>>
>>>>>>>> Best
>>>>>>>>
>>>>>>>> On Mon, Aug 1, 2016 at 9:37 PM Pine W <wiki.pine(a)gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Amir,
>>>>>>>>>
>>>>>>>>> I think that I'm following better now.
>>>>>>>>>
>>>>>>>>> The extension option doesn't sound workable for what I have in
>>>>>>>>> mind. I'd like the score to be visible without needing to be logged in,
>>>>>>>>> which I believe means that the feature can't be conditional on having
>>>>>>>>> access to beta features.
>>>>>>>>>
>>>>>>>>> The solution to click on a talk page link for an updated score
>>>>>>>>> sounds more workable. It would be best if the score refreshed automatically
>>>>>>>>> every time someone viewed the talk page for the purpose that I have in
>>>>>>>>> mind, but I don't know what kind of performance penalty and computing
>>>>>>>>> resources that would require. If either of those are nontrivial, then
>>>>>>>>> having a template on the talk page that says something like "Score last
>>>>>>>>> refreshed on TIME DATE. There have been X revisions since that time. Click
>>>>>>>>> here to update the score" would be good for the use case that I have in
>>>>>>>>> mind.
>>>>>>>>>
>>>>>>>>> Pine
>>>>>>>>>
>>>>>>>>> On Mon, Aug 1, 2016 at 1:48 AM, Amir Ladsgroup <
>>>>>>>>> ladsgroup(a)gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hey,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sun, Jul 31, 2016 at 8:16 AM Pine W <wiki.pine(a)gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Amir,
>>>>>>>>>>>
>>>>>>>>>>> Thanks for waiting. As I understand it, we have these options
>>>>>>>>>>> for making the ORES scores visible on article talk pages:
>>>>>>>>>>>
>>>>>>>>>>> 1. A gadget
>>>>>>>>>>> 2. An addition to the ORES extension
>>>>>>>>>>> 3. A bot
>>>>>>>>>>>
>>>>>>>>>>> Is that correct?
>>>>>>>>>>>
>>>>>>>>>> Yes
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> WP:WPP refers to WikiProject Philosophy; did you man something
>>>>>>>>>>> else or am I misunderstanding the abbreviation?
>>>>>>>>>>>
>>>>>>>>>>> My bad, I meant something like this:
>>>>>>>>>> https://en.wikipedia.org/wiki/Template:WikiProject_Physics
>>>>>>>>>>
>>>>>>>>>>> I'm thinking that a gadget might be a workable solution, but my
>>>>>>>>>>> preference would be to have the score displayed on all article talk pages
>>>>>>>>>>> automatically for everyone to see including readers who are not logged in.
>>>>>>>>>>> A bot that posted scores to talk pages and updated the scores on a weekly
>>>>>>>>>>> basis would be good for that purpose.
>>>>>>>>>>>
>>>>>>>>>>> Both are possible solutions but as for bots. Updating the score
>>>>>>>>>> for all articles of a wikiproject (e.g. Wikiproject medicine has 35K
>>>>>>>>>> articles) on weekly basis is not efficient. That bot needs to do 35K edits
>>>>>>>>>> every week and it might make some people mad. We can have an initial
>>>>>>>>>> release and update it on demand (like adding a link to the score in the
>>>>>>>>>> template. If someone wants that score gets updated. Just clicks on the link
>>>>>>>>>> and the bot update it)
>>>>>>>>>>
>>>>>>>>>>> I'm unclear on how an addition to the ORES extension would work;
>>>>>>>>>>> can you explain further what that option would involve?
>>>>>>>>>>>
>>>>>>>>>> Adding it to the ores extension will make things much easier. You
>>>>>>>>>> just need to enable it in your beta features and then in the talk page or
>>>>>>>>>> action=info you will see the score. Implementing it is difficult due to
>>>>>>>>>> storage issues for the database and we might need to get operations
>>>>>>>>>> approval before continuing.
>>>>>>>>>>
>>>>>>>>>> Best
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>>
>>>>>>>>>>> Pine
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Jul 24, 2016 at 11:29 PM, Pine W <wiki.pine(a)gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Amir,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for getting back to me. My mind is a little full at the
>>>>>>>>>>>> moment, and I have some deadlines this week, so it may take me a little
>>>>>>>>>>>> while to get back to you. But I'm flagging this conversation for follow up,
>>>>>>>>>>>> hopefully by the end of this week or early next week.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks for your patience,
>>>>>>>>>>>> Pine
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Jul 22, 2016 at 9:59 PM, Amir Ladsgroup <
>>>>>>>>>>>> ladsgroup(a)gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hey,
>>>>>>>>>>>>> ORES has "article quality" models for English, Russian and
>>>>>>>>>>>>> French Wikipedia. There is a very short introduction in ORES
>>>>>>>>>>>>> <https://meta.wikimedia.org/wiki/Objective_Revision_Evaluation_Service#Artic…>
>>>>>>>>>>>>> description page.
>>>>>>>>>>>>> But if I want to explain it my way. It'll give you a
>>>>>>>>>>>>> prediction of class of the article (A, B, C, FA, GA, stub, start) for a
>>>>>>>>>>>>> given revision. For example for this revision
>>>>>>>>>>>>> <https://en.wikipedia.org/w/index.php?title=Alan_Turing&oldid=730400015> ORES
>>>>>>>>>>>>> predicts
>>>>>>>>>>>>> <https://ores.wikimedia.org/v2/scores/enwiki/?models=wp10&revids=730400015>
>>>>>>>>>>>>> that it's GA (which it is).
>>>>>>>>>>>>> The biggest use case for now is that people produce reports
>>>>>>>>>>>>> of articles needing re
>>>>>>>>>>>>> <https://meta.wikimedia.org/wiki/Research:Revision_scoring_as_a_service/Stor…>
>>>>>>>>>>>>> assessment
>>>>>>>>>>>>> <https://meta.wikimedia.org/wiki/Research:Revision_scoring_as_a_service/Stor…> for
>>>>>>>>>>>>> wikiprojects
>>>>>>>>>>>>> <https://meta.wikimedia.org/wiki/Research:Revision_scoring_as_a_service/Stor…>.
>>>>>>>>>>>>> The other use case is list of popular articles that ORES
>>>>>>>>>>>>> thinks their quality is low
>>>>>>>>>>>>> <https://en.wikipedia.org/wiki/User:DataflowBot/output/Popular_low_quality_a…>
>>>>>>>>>>>>> .
>>>>>>>>>>>>>
>>>>>>>>>>>>> Your use case about ORES prediction is valid the second part
>>>>>>>>>>>>> is a little bit complicated though. For the first part, we have either
>>>>>>>>>>>>> option of a gadget or adding it to the ORES extension. If it's very
>>>>>>>>>>>>> important to let everyone see the scores, we can write a bot to update
>>>>>>>>>>>>> reports weekly in English Wikipedia for articles in a certain Wikiproject
>>>>>>>>>>>>> and we use these data in the WPP templates. Which one do you think suits
>>>>>>>>>>>>> English Wikipedia better?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Best
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sat, Jul 23, 2016 at 2:57 AM Pine W <wiki.pine(a)gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi artificial intelligence people,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Are there baseline ORES scores available for articles? If so,
>>>>>>>>>>>>>> I'm thinking that it would be interesting to have the option to add a
>>>>>>>>>>>>>> template to an article's talk page that shows its ORES score as well as its
>>>>>>>>>>>>>> ORES percentile rank on a particular wiki.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Pine
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> AI mailing list
>>>>>>>>>>>>>> AI(a)lists.wikimedia.org
>>>>>>>>>>>>>> https://lists.wikimedia.org/mailman/listinfo/ai
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> AI mailing list
>>>>>>>>>>>>> AI(a)lists.wikimedia.org
>>>>>>>>>>>>> https://lists.wikimedia.org/mailman/listinfo/ai
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> AI mailing list
>>>>>>>>>>> AI(a)lists.wikimedia.org
>>>>>>>>>>> https://lists.wikimedia.org/mailman/listinfo/ai
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> AI mailing list
>>>>>>>>>> AI(a)lists.wikimedia.org
>>>>>>>>>> https://lists.wikimedia.org/mailman/listinfo/ai
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> AI mailing list
>>>>>>>>> AI(a)lists.wikimedia.org
>>>>>>>>> https://lists.wikimedia.org/mailman/listinfo/ai
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> AI mailing list
>>>>>>>> AI(a)lists.wikimedia.org
>>>>>>>> https://lists.wikimedia.org/mailman/listinfo/ai
>>>>>>>>
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>> AI mailing list
>>>>>> AI(a)lists.wikimedia.org
>>>>>> https://lists.wikimedia.org/mailman/listinfo/ai
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> AI mailing list
>>>>> AI(a)lists.wikimedia.org
>>>>> https://lists.wikimedia.org/mailman/listinfo/ai
>>>>>
>>>>>
>>>
>>
>
>
Joe, any comment about this?
Toby and Amir, would it make sense to integrate ORES scores into
https://meta.wikimedia.org/wiki/Community_Tech/PageAssessments, or have
some other kind of relation between the ability to pull ORES scores from
the MW API and the PageAssessments API? Or does it make sense to have them
be separate but working in parallel? I'm an API novice, but the
functionality of PageAssessments reminds me of what we're discussing here,
which is why I bring it up.
Thanks,
Pine
On Wed, Aug 31, 2016 at 2:48 PM, Toby Negrin <tnegrin(a)wikimedia.org> wrote:
> Hi Pine -- thanks for this. I'm looping in Joe M. from the Collaboration
> team who can speak to his team's thoughts on ORES scores.
>
> It seems likely that my team will devote some resources to making ORES (+
> the page view API) accessible via the MW API next quarter which will help
> integration efforts. Our goals go up on Friday and I just got out of a
> meeting where we agreed to do this and it's unlikely this will change in 2
> days.
>
> As far as New Readers though, I suspect that it's a bit too abstract as of
> yet. I think we should start with Old Readers :)
>
> -Toby
>
> On Wed, Aug 31, 2016 at 2:37 PM, Pine W <wiki.pine(a)gmail.com> wrote:
>
>> Just to note that https://upload.wikimedia.org/w
>> ikipedia/commons/6/6a/Wikimedia_Foundation_and_Reboot_New_
>> Readers_Research_-_Nigeria_%26_India_Highlights_-_July_2016.pdf suggests
>> that WMF Product "Add an indicator of quality icon (akin to the signal
>> strength icon for mobile networks) on each article, based on research and
>> analysis on article contributions and edits." This is similar to what I
>> have in mind for talk pages with ORES scores. Adding indicators to the
>> front page of articles would likely require an RfC which may be more
>> involved than an RfB to add the score to talk pages. Still, I can see how
>> having an ORES score, or the Wikipedia Quality score (at least on ENWP)
>> indicated by an icon at the top of article pages would be useful,
>> especially if readers can intuitively understand what the indicator is
>> telling them without needing to look at documentation.
>>
>> I think it might be best to start with ORES scores on talk pages. That
>> fills the need that I have in mind. If someone wants to extend this to
>> having article quality indicators on the front page of articles, I'd be
>> happy to collaborate with them.
>>
>> Pine
>>
>> On Tue, Aug 16, 2016 at 7:32 AM, Pine W <wiki.pine(a)gmail.com> wrote:
>>
>>> Thanks. I will add this to my agenda for later this year.
>>>
>>> Pine
>>>
>>> On Aug 16, 2016 01:48, "Amir Ladsgroup" <ladsgroup(a)gmail.com> wrote:
>>>
>>>> Oh, Sorry I missed the question. Yes, it's fairly easy :)
>>>>
>>>> Best
>>>>
>>>> On Tue, Aug 16, 2016 at 12:15 PM Pine W <wiki.pine(a)gmail.com> wrote:
>>>>
>>>>> Pinging again to repeat my question from my previous email.
>>>>>
>>>>> Thanks,
>>>>> Pine
>>>>>
>>>>> On Aug 9, 2016 16:04, "Pine W" <wiki.pine(a)gmail.com> wrote:
>>>>>
>>>>>> I'd like to work on this if I had the time to learn and to do it.
>>>>>> Maybe I will in early September or sometime later this year. I've put it on
>>>>>> my to do list. This would be my first bot; do you think that it's a good
>>>>>> project for a newbie bot builder?
>>>>>>
>>>>>> Pine
>>>>>>
>>>>>> On Sat, Aug 6, 2016 at 11:16 AM, Amir Ladsgroup <ladsgroup(a)gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Building that is easy, you just need someone to add these
>>>>>>> functionalities to the templates and a bot to run it for the first time.
>>>>>>>
>>>>>>> Any volunteers?
>>>>>>>
>>>>>>> Best
>>>>>>>
>>>>>>> On Mon, Aug 1, 2016 at 9:37 PM Pine W <wiki.pine(a)gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi Amir,
>>>>>>>>
>>>>>>>> I think that I'm following better now.
>>>>>>>>
>>>>>>>> The extension option doesn't sound workable for what I have in
>>>>>>>> mind. I'd like the score to be visible without needing to be logged in,
>>>>>>>> which I believe means that the feature can't be conditional on having
>>>>>>>> access to beta features.
>>>>>>>>
>>>>>>>> The solution to click on a talk page link for an updated score
>>>>>>>> sounds more workable. It would be best if the score refreshed automatically
>>>>>>>> every time someone viewed the talk page for the purpose that I have in
>>>>>>>> mind, but I don't know what kind of performance penalty and computing
>>>>>>>> resources that would require. If either of those are nontrivial, then
>>>>>>>> having a template on the talk page that says something like "Score last
>>>>>>>> refreshed on TIME DATE. There have been X revisions since that time. Click
>>>>>>>> here to update the score" would be good for the use case that I have in
>>>>>>>> mind.
>>>>>>>>
>>>>>>>> Pine
>>>>>>>>
>>>>>>>> On Mon, Aug 1, 2016 at 1:48 AM, Amir Ladsgroup <ladsgroup(a)gmail.com
>>>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> Hey,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sun, Jul 31, 2016 at 8:16 AM Pine W <wiki.pine(a)gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Amir,
>>>>>>>>>>
>>>>>>>>>> Thanks for waiting. As I understand it, we have these options for
>>>>>>>>>> making the ORES scores visible on article talk pages:
>>>>>>>>>>
>>>>>>>>>> 1. A gadget
>>>>>>>>>> 2. An addition to the ORES extension
>>>>>>>>>> 3. A bot
>>>>>>>>>>
>>>>>>>>>> Is that correct?
>>>>>>>>>>
>>>>>>>>> Yes
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> WP:WPP refers to WikiProject Philosophy; did you man something
>>>>>>>>>> else or am I misunderstanding the abbreviation?
>>>>>>>>>>
>>>>>>>>>> My bad, I meant something like this:
>>>>>>>>> https://en.wikipedia.org/wiki/Template:WikiProject_Physics
>>>>>>>>>
>>>>>>>>>> I'm thinking that a gadget might be a workable solution, but my
>>>>>>>>>> preference would be to have the score displayed on all article talk pages
>>>>>>>>>> automatically for everyone to see including readers who are not logged in.
>>>>>>>>>> A bot that posted scores to talk pages and updated the scores on a weekly
>>>>>>>>>> basis would be good for that purpose.
>>>>>>>>>>
>>>>>>>>>> Both are possible solutions but as for bots. Updating the score
>>>>>>>>> for all articles of a wikiproject (e.g. Wikiproject medicine has 35K
>>>>>>>>> articles) on weekly basis is not efficient. That bot needs to do 35K edits
>>>>>>>>> every week and it might make some people mad. We can have an initial
>>>>>>>>> release and update it on demand (like adding a link to the score in the
>>>>>>>>> template. If someone wants that score gets updated. Just clicks on the link
>>>>>>>>> and the bot update it)
>>>>>>>>>
>>>>>>>>>> I'm unclear on how an addition to the ORES extension would work;
>>>>>>>>>> can you explain further what that option would involve?
>>>>>>>>>>
>>>>>>>>> Adding it to the ores extension will make things much easier. You
>>>>>>>>> just need to enable it in your beta features and then in the talk page or
>>>>>>>>> action=info you will see the score. Implementing it is difficult due to
>>>>>>>>> storage issues for the database and we might need to get operations
>>>>>>>>> approval before continuing.
>>>>>>>>>
>>>>>>>>> Best
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Pine
>>>>>>>>>>
>>>>>>>>>> On Sun, Jul 24, 2016 at 11:29 PM, Pine W <wiki.pine(a)gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Amir,
>>>>>>>>>>>
>>>>>>>>>>> Thanks for getting back to me. My mind is a little full at the
>>>>>>>>>>> moment, and I have some deadlines this week, so it may take me a little
>>>>>>>>>>> while to get back to you. But I'm flagging this conversation for follow up,
>>>>>>>>>>> hopefully by the end of this week or early next week.
>>>>>>>>>>>
>>>>>>>>>>> Thanks for your patience,
>>>>>>>>>>> Pine
>>>>>>>>>>>
>>>>>>>>>>> On Fri, Jul 22, 2016 at 9:59 PM, Amir Ladsgroup <
>>>>>>>>>>> ladsgroup(a)gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hey,
>>>>>>>>>>>> ORES has "article quality" models for English, Russian and
>>>>>>>>>>>> French Wikipedia. There is a very short introduction in ORES
>>>>>>>>>>>> <https://meta.wikimedia.org/wiki/Objective_Revision_Evaluation_Service#Artic…>
>>>>>>>>>>>> description page.
>>>>>>>>>>>> But if I want to explain it my way. It'll give you a prediction
>>>>>>>>>>>> of class of the article (A, B, C, FA, GA, stub, start) for a given
>>>>>>>>>>>> revision. For example for this revision
>>>>>>>>>>>> <https://en.wikipedia.org/w/index.php?title=Alan_Turing&oldid=730400015> ORES
>>>>>>>>>>>> predicts
>>>>>>>>>>>> <https://ores.wikimedia.org/v2/scores/enwiki/?models=wp10&revids=730400015>
>>>>>>>>>>>> that it's GA (which it is).
>>>>>>>>>>>> The biggest use case for now is that people produce reports of
>>>>>>>>>>>> articles needing re
>>>>>>>>>>>> <https://meta.wikimedia.org/wiki/Research:Revision_scoring_as_a_service/Stor…>
>>>>>>>>>>>> assessment
>>>>>>>>>>>> <https://meta.wikimedia.org/wiki/Research:Revision_scoring_as_a_service/Stor…> for
>>>>>>>>>>>> wikiprojects
>>>>>>>>>>>> <https://meta.wikimedia.org/wiki/Research:Revision_scoring_as_a_service/Stor…>.
>>>>>>>>>>>> The other use case is list of popular articles that ORES
>>>>>>>>>>>> thinks their quality is low
>>>>>>>>>>>> <https://en.wikipedia.org/wiki/User:DataflowBot/output/Popular_low_quality_a…>
>>>>>>>>>>>> .
>>>>>>>>>>>>
>>>>>>>>>>>> Your use case about ORES prediction is valid the second part is
>>>>>>>>>>>> a little bit complicated though. For the first part, we have either option
>>>>>>>>>>>> of a gadget or adding it to the ORES extension. If it's very important to
>>>>>>>>>>>> let everyone see the scores, we can write a bot to update reports weekly in
>>>>>>>>>>>> English Wikipedia for articles in a certain Wikiproject and we use these
>>>>>>>>>>>> data in the WPP templates. Which one do you think suits English Wikipedia
>>>>>>>>>>>> better?
>>>>>>>>>>>>
>>>>>>>>>>>> Best
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Sat, Jul 23, 2016 at 2:57 AM Pine W <wiki.pine(a)gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi artificial intelligence people,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Are there baseline ORES scores available for articles? If so,
>>>>>>>>>>>>> I'm thinking that it would be interesting to have the option to add a
>>>>>>>>>>>>> template to an article's talk page that shows its ORES score as well as its
>>>>>>>>>>>>> ORES percentile rank on a particular wiki.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Pine
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> AI mailing list
>>>>>>>>>>>>> AI(a)lists.wikimedia.org
>>>>>>>>>>>>> https://lists.wikimedia.org/mailman/listinfo/ai
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> AI mailing list
>>>>>>>>>>>> AI(a)lists.wikimedia.org
>>>>>>>>>>>> https://lists.wikimedia.org/mailman/listinfo/ai
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> AI mailing list
>>>>>>>>>> AI(a)lists.wikimedia.org
>>>>>>>>>> https://lists.wikimedia.org/mailman/listinfo/ai
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> AI mailing list
>>>>>>>>> AI(a)lists.wikimedia.org
>>>>>>>>> https://lists.wikimedia.org/mailman/listinfo/ai
>>>>>>>>>
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> AI mailing list
>>>>>>>> AI(a)lists.wikimedia.org
>>>>>>>> https://lists.wikimedia.org/mailman/listinfo/ai
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> AI mailing list
>>>>>>> AI(a)lists.wikimedia.org
>>>>>>> https://lists.wikimedia.org/mailman/listinfo/ai
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>> AI mailing list
>>>>> AI(a)lists.wikimedia.org
>>>>> https://lists.wikimedia.org/mailman/listinfo/ai
>>>>>
>>>>
>>>> _______________________________________________
>>>> AI mailing list
>>>> AI(a)lists.wikimedia.org
>>>> https://lists.wikimedia.org/mailman/listinfo/ai
>>>>
>>>>
>>
>
Hey,
This is the 20th weekly update from revision scoring team that we have sent
to this mailing list.
New development:
- We implemented the basic functionality for handling bag of words and
other types of abstract feature vectors in `revscoring`. [1] This required
some changes to some dependencies as well. [2]
- We extended the user-group related features to include more of the
dominant groups outside of English Wikipedia [3] and incremented the models
that changed substantially [4]
Documentation:
- We extended the documentation at mw:Extension:ORES to make it easier
for new developers to work with us. [5]
Resourcing:
- We discussed the teams resourcing needs (hardware, engineering, and
community liaison support) with Wes Moran. [6]
Maintenance and robustness:
- We addressed a variety of issues around caching and how the ORES
extension loads new data
- ORES now returns headers that will disable secondary caching. [7]
- Our maintenance scripts will circumvent caches that do not listen to
no-cache headers. [8, 9]
- We fixed an issue where the ORES review tool would duplicate items in
Special:RecentChanges. [10]
- We standardized the extraction pattern for the enwiktionary model so
that it looks similar to other models. [11]
1. https://phabricator.wikimedia.org/T132580 -- Implement abstraction for
Sparse Feature Vectors
2. https://phabricator.wikimedia.org/T144430 -- Update yamlconf so that
import_path can handle deep attributes
3. https://phabricator.wikimedia.org/T143909 -- Extend user group features
4. https://phabricator.wikimedia.org/T144855 -- Increment ruwiki
editquality models
5. https://phabricator.wikimedia.org/T144676 -- Improve technical
documentation in Extension:ORES in mediawiki.ore
6. https://phabricator.wikimedia.org/T144517 -- ORES and Product:
resourcing discussion
7. https://phabricator.wikimedia.org/T144193 -- Set max-age header to 0
seconds for ORES to quiet secondary caches
8. https://phabricator.wikimedia.org/T144196 -- Get model version needs to
invalidate cache
9. https://phabricator.wikimedia.org/T144195 -- Check model version
replaces every time it runs.
10. https://phabricator.wikimedia.org/T144233 -- Redundant results in ORES
review tool
11. https://phabricator.wikimedia.org/T144605 -- Fix makefile entry for
enwiktionary.rev_reverted.20k_2016.tsv
Sincerely,
Aaron from the Revision Scoring team
Hi artificial intelligence people,
Are there baseline ORES scores available for articles? If so, I'm thinking
that it would be interesting to have the option to add a template to an
article's talk page that shows its ORES score as well as its ORES
percentile rank on a particular wiki.
Pine