AI

ai@lists.wikimedia.org

1 participants
265 discussions

Weekly update #2
by Amir Ladsgroup 02 May '16

02 May '16

The second weekly update on the Revision Scoring project. *New developments* - ORES has graphite dashboard now [1,2] - Deploying new campaigns and testing Wikilabels got easier [3,4] - Revscoring feature extraction got about 13% faster [5] - We deployed new versions of ORES and Wikilabels [6,7] - Wikidata ScoredRevision gadget had a serious issue, it got fixed [8] *Progress in supporting new languages* - Wikidata damaging and goodfaith models are built and deployed [9,10,11] - Dutch damaging and goodfaith models are built and deployed [12] - We are working on langauge utilities of Tamil [13] *Active Labeling campaigns* - Edit quality (damaging and good faith) - Wikipedias: Arabic, Azerbaijani, German, French, Hebrew, Hungarian, Indonesian, Italian, Japanese, Norweigian, Persian (v2), Polish, Spanish, Ukranian, Urdu, Vietnamese - Edit type - English Wikipedia 1. 1. https://grafana.wikimedia.org/dashboard/db/ores 1. 2. https://phabricator.wikimedia.org/T127594 1. 3. https://phabricator.wikimedia.org/T133557 1. 4. https://phabricator.wikimedia.org/T102336 1. 5. https://github.com/wiki-ai/revscoring/pull/268 1. 6. https://phabricator.wikimedia.org/T134032 1. 7. https://phabricator.wikimedia.org/T134174 2. 8. https://phabricator.wikimedia.org/T133903 1. 9. https://phabricator.wikimedia.org/T130274 1. 10. https://phabricator.wikimedia.org/T130301 1. 11. https://lists.wikimedia.org/pipermail/wikidata/2016-May/008641.html 1. 12. https://phabricator.wikimedia.org/T133563 13. https://phabricator.wikimedia.org/T134105 Sincerely, The Revision Scoring team.

1 0

"Damaging" and "goodfaith" models for Wikidata are now online
by Amir Ladsgroup 01 May '16

01 May '16

Hello, TLDR: Vandalism detection model for Wikidata just got much more accurate. Longer version: ORES is designed to handle different types of classification. For example one of under development classification types is "wikiclass" which determines type of edits. If they are adding content, or fixing mistake, etc. The most mature classification of ORES is edit quality. Whether an edit is vandalism or not. We usually have three models: "reverted" model. Training data for this model is obtained automatically. We sample around 20K edits (for Wikidata it was different) and we consider an edit as vandalism if they are reverted within a certain time period after the edit (7 days for Wikidata). On the other hand, "damaging" and "goodfaith" models are more accurate because we sample about 20K edits. Prelabel edits that being made by trusted users such as admins and bots as not harmful to Wikidata/Wikipedia and then we ask users to label the rest. (For Wikidata it was around 4K edits) Since most edits in Wikidata are made by bots and trusted users, We altered this method for Wikidata a bit but the whole process is the same. Don't forget that since it's human judgement, this models are more accurate and useful to damage detection. The ORES extension uses "damaging" model and not "reverted" model, thus having "damaging" model online is a requirement for the extension deployment. People label edits that if an edit is damaging to Wikidata and if the edit is made by good intention. So we have three cases: 1- An edit is harmful to Wikidata but made with good intention. An honest/newbie mistake 2- An edit is harmful and made bad intention. A vandalism 3- A edit with good intention and productive. A "good" edit". Biggest reason to distinguish between honest mistakes and vandalisms is that using anti-vandalism bots caused reducing on new user retention in Wikis [1]. So future anti-vandalism bots should not revert good faith mistakes but report them for human review. One of good things about Wikidata damage detection labeling process is that so many people were involved (we had 38 labelers for Wikidata[2]). Another good thing is that its fitness very high in terms of AI [3]. But since number of damaging edits and not damaging edits are not the same, scores it gives to edits are not intuitive. Let me give you an example: In our damaging model if an edit is scored less than 80% it's probably not vandalism. Actually, in a very huge sampling of human edits we had for reverted model we couldn't find a bad edit with score lower than 93% i.e. If an edit is scored 92% in reverted model, you are pretty sure it's not vandalism. Please reach out to us if you have any questions on using these scores. Please reach out to us if have any questions in general ;) In terms of needed changes, ScoredRevision gadget is set automatically to prefer the damaging model. I just changed my bot in #wikidata-vandalism channel in order to use damaging instead of reverted. If you want to use these models. Check out our docs [4] Sincerely, Revision scoring team [5] [1]: Halfaker, A.; Geiger, R. S.; Morgan, J. T.; Riedl, J. (28 December 2012). "The Rise and Decline of an Open Collaboration System: How Wikipedia's Reaction to Popularity Is Causing Its Decline". *American Behavioral Scientist* *57* (5): 664–688. [2]: https://labels.wmflabs.org/campaigns/wikidatawiki/?campaigns=stats [3]: https://ores.wmflabs.org/scores/wikidatawiki/?model_info [4]: https://ores.wmflabs.org/v2/ [5]: https://meta.wikimedia.org/wiki/Research:Revision_scoring_as_a_service#Team

1 0

Re: [AI] [Wikitech-l] Updates to ORES service & BREAKING CHANGE on April 7th
by Aaron Halfaker 30 Apr '16

30 Apr '16

Hi Mortiz, There's two types of stability you should be aware of: API behavior and model scores. You should expect that the version'd API behavior will remain stable. So, if we choose to make a change to the request or response style, that will appear under the path "v3/" and so forth. So, if you write code against the v2/ API (you shouldn't be writing new code against the v1/ API, but you *can* expect it to be stable), you should expect that it will continue to work as expected. You can see the swagger spec's for the APIs at these endpoints: https://ores.wmflabs.org/v1/spec/ or https://ores.wmflabs.org/v2/spec/ You should expect that the API behavior described will not change. But we may still need to update the models in the future and that would likely change the range of scores slightly. We include versions of the models in the basic API response so that you can cache and invalidate scores that you get from the API. We're still working out the right way to report evaluation metrics to you so that you'll be able to dynamically adjust any thresholds you set in your own application. FWIW, I do not forsee us changing our modeling strategy substantially in the short- or mid-term. It took us ~3 months of work to prepare for the breaking change that was announced in this thread. In the end, we're interested in learning about your needs and concerns so that we can adjust our process and make changes accordingly. So if you have concerns with any of the above please let us know. -Aaron On Sat, Apr 30, 2016 at 5:50 PM, Moritz Schubotz <physik(a)physikerwelt.de> wrote: > Hi Aaron, > > can you say a few words about the stability of the API. > We are working on a scoring model for user contributions, rather than > revisions using Apache Flink. > http://imwa.gehaxelt.in:9090/pdfs/expose.pdf > However, it would be nice to have a somehow compatible API in the end. > > Best > Moritz > > On Thu, Apr 7, 2016 at 10:55 AM, Aaron Halfaker <aaron.halfaker(a)gmail.com> > wrote: > > > FYI, the new models (BREAKING CHANGE) are now deployed. > > > > On Sun, Apr 3, 2016 at 5:38 AM, Aaron Halfaker <aaron.halfaker(a)gmail.com > > > > wrote: > > > > > Hey folks, we have a couple of announcements for you today. First is > that > > > ORES has a large set of new functionality that you might like to take > > > advantage of. We'll also want to talk about a *BREAKING CHANGE on April > > > 7th.* > > > > > > Don't know what ORES is? See > > > > > > http://blog.wikimedia.org/2015/11/30/artificial-intelligence-x-ray-specs/ > > > > > > *New functionality* > > > > > > *Scoring UI* > > > Sometimes you just want to score a few revisions in ORES and > remembering > > > the URL structure is hard. So, we've build a simple scoring > > user-interface > > > <https://ores.wmflabs.org/ui/> that will allow you to more easily > score > > a > > > set of edits. > > > > > > *New API version* > > > We've been consistently getting requests to include more information in > > > ORES' responses. In order to make space for this new information, we > > needed > > > to change the structure of responses. But we wanted to do this without > > > breaking the tools that are already using ORES. So, we've developed a > > > versioning scheme that will allow you to take advantage of new > > > functionality when you are ready. The same old API will continue to be > > > available at https://ores.wmflabs.org/scores/, but we've added two > > > additional paths on top of this. > > > > > > - https://ores.wmflabs.org/v1/scores/ is a mirror of the old > scoring > > > API which will henceforth be referred to as "v1" > > > - https://ores.wmflabs.org/v2/scores/ implements a new response > > format > > > that is consistent between all sub-paths and adds some new > > functionality > > > > > > *Swagger documentation* > > > Curious about the new functionality available in "v2" or maybe what the > > > change was from "v1"? We've implemented a structured description of > both > > > versions of the scoring API using swagger -- which is becoming a > defacto > > > stanard for this sort of thing. Visit https://ores.wmflabs.org/v1/ or > > > https://ores.wmflabs.org/v2/ to see the Swagger user-interface. > > > Visithttps://ores.wmflabs.org/v1/spec/ or > > > https://ores.wmflabs.org/v2/spec/ to get the specification in a > > > machine-readable format. > > > > > > *Feature values & injection* > > > Have you wondered what ORES uses to make it's predictions? You can now > > ask > > > ORES to show you the list of "feature" statistics it uses to score > > > revisions. For example, > > > https://ores.wmflabs.org/v2/scores/enwiki/wp10/34567892/?features will > > > return the score with a mapping of feature values used by the "wp10" > > > article quality model in English Wikipedia to score oldid=34567892 > > > <https://en.wikipedia.org/wiki/Special:Diff/34567892>. You can also > > > "inject" features into the scoring process to see how that affects the > > > prediction. E.g., > > > > > > https://ores.wmflabs.org/v2/scores/enwiki/wp10/34567892?features&feature.wi… > > > > > > *Breaking change -- new models* > > > We've been experimenting with new learning algorithms to make ORES work > > > better and we've found that we get better results with gradient > boosting > > > <https://en.wikipedia.org/wiki/Gradient_boosting> and random forest > > > <https://en.wikipedia.org/wiki/Random_forest> strategies than we do > with > > > the current linear svc > > > <https://en.wikipedia.org/wiki/Support_vector_machine> models. We'd > like > > > to get these new, better models deployed as soon as possible, but with > > the > > > new algorithm comes a change in the range of probabilities returned by > > the > > > model. So, when we deploy this change, any tools that uses hard-coded > > > thresholds on ORES' prediction probabilities will suddenly start > behaving > > > strangely. Regretfully, we haven't found a way around this problem, so > > > we're announcing the change now and we plan to deploy this *BREAKING > > > CHANGE on April 7th*. Please subscribe to the AI mailinglist > > > <https://lists.wikimedia.org/mailman/listinfo/ai> or watch our project > > > page [[:m:ORES <https://meta.wikimedia.org/wiki/ORES>]] to catch > > > announcements of future changes and new functionality. > > > > > > In order to make sure we don't end up in the same situation the next > time > > > we want to change an algorithm, we've included a suite of evaluation > > > statistics with each model. The filter_rate_at_recall(0.9), > > > filter_rate_at_recall(0.75), and recall_at_fpr(0.1) thresholds > represent > > > three critical thresholds (should review, needs review, and definitely > > > damaging -- respectively) that can be used to automatically configure > > your > > > wiki tool. You can find out these thresholds for your model of choice > by > > > adding the ?model_info parameter to requests. So, come breaking > change, > > > we strongly recommend basing your thresholds on these statistics in the > > > future. We'll be working to submit patches to tools that use ORES in > the > > > next week to implement this flexibility. Hopefully, all you'll need to > > do > > > is worth with us on those. > > > > > > -halfak & The Revision Scoring team > > > < > https://meta.wikimedia.org/wiki/Research:Revision_scoring_as_a_service> > > > > > _______________________________________________ > > Wikitech-l mailing list > > Wikitech-l(a)lists.wikimedia.org > > https://lists.wikimedia.org/mailman/listinfo/wikitech-l > > > > > -- > Mit freundlichen Grüßen > Moritz Schubotz > > Telefon (Büro): +49 30 314 22784 > Telefon (Privat):+49 30 488 27330 > E-Mail: schubotz(a)itp.physik.tu-berlin.de > Web: http://www.physikerwelt.de > Skype: Schubi87 > ICQ: 200302764 > Msn: Moritz(a)Schubotz.de > _______________________________________________ > Wikitech-l mailing list > Wikitech-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-l

1 0

Weekly update
by Amir Ladsgroup 25 Apr '16

25 Apr '16

Hello, This is our first weekly update being posted in this mailing list New Developments - Now you can abandon tasks you don't want to review in Wikilabels (T105521) - We collect user-agents in ORES requests (T113754) - Precaching in ORES will be a daemon and more selective (T106638) Progress in supporting new languages - Russian reverted, damaging, and goodfaith models are built. They look good and will be deployed this week. - Hungarian reverted model is built, will be deployed this week. Campaign for goodfaith and damaging is loaded in Wikilabels. - Japanese reverted model are built, but there are still some issues to work out. (T133405) Active Labeling campaigns - Edit quality (damaging and good faith) - Wikipedias: Arabic, Azerbaijani, Dutch, German, French, Hebrew, Hungarian, Indonesian, Italian, Japanese, Norwegian, Persian (v2), Polish, Spanish, Ukrainian, Urdu, Vietnamese - Wikidata - Edit type - English Wikipedia Sincerely, The Revision Scoring team. <https://meta.wikimedia.org/wiki/Research:Revision_scoring_as_a_service#Team>

1 0

Updates to ORES service & BREAKING CHANGE on April 7th
by Aaron Halfaker 07 Apr '16

07 Apr '16

Hey folks, we have a couple of announcements for you today. First is that ORES has a large set of new functionality that you might like to take advantage of. We'll also want to talk about a *BREAKING CHANGE on April 7th.* Don't know what ORES is? See http://blog.wikimedia.org/2015/11/30/artificial-intelligence-x-ray-specs/ *New functionality* *Scoring UI* Sometimes you just want to score a few revisions in ORES and remembering the URL structure is hard. So, we've build a simple scoring user-interface <https://ores.wmflabs.org/ui/> that will allow you to more easily score a set of edits. *New API version* We've been consistently getting requests to include more information in ORES' responses. In order to make space for this new information, we needed to change the structure of responses. But we wanted to do this without breaking the tools that are already using ORES. So, we've developed a versioning scheme that will allow you to take advantage of new functionality when you are ready. The same old API will continue to be available at https://ores.wmflabs.org/scores/, but we've added two additional paths on top of this. - https://ores.wmflabs.org/v1/scores/ is a mirror of the old scoring API which will henceforth be referred to as "v1" - https://ores.wmflabs.org/v2/scores/ implements a new response format that is consistent between all sub-paths and adds some new functionality *Swagger documentation* Curious about the new functionality available in "v2" or maybe what the change was from "v1"? We've implemented a structured description of both versions of the scoring API using swagger -- which is becoming a defacto stanard for this sort of thing. Visit https://ores.wmflabs.org/v1/ or https://ores.wmflabs.org/v2/ to see the Swagger user-interface. Visithttps://ores.wmflabs.org/v1/spec/ or https://ores.wmflabs.org/v2/spec/ to get the specification in a machine-readable format. *Feature values & injection* Have you wondered what ORES uses to make it's predictions? You can now ask ORES to show you the list of "feature" statistics it uses to score revisions. For example, https://ores.wmflabs.org/v2/scores/enwiki/wp10/34567892/?features will return the score with a mapping of feature values used by the "wp10" article quality model in English Wikipedia to score oldid=34567892 <https://en.wikipedia.org/wiki/Special:Diff/34567892>. You can also "inject" features into the scoring process to see how that affects the prediction. E.g., https://ores.wmflabs.org/v2/scores/enwiki/wp10/34567892?features&feature.wi… *Breaking change -- new models* We've been experimenting with new learning algorithms to make ORES work better and we've found that we get better results with gradient boosting <https://en.wikipedia.org/wiki/Gradient_boosting> and random forest <https://en.wikipedia.org/wiki/Random_forest> strategies than we do with the current linear svc <https://en.wikipedia.org/wiki/Support_vector_machine> models. We'd like to get these new, better models deployed as soon as possible, but with the new algorithm comes a change in the range of probabilities returned by the model. So, when we deploy this change, any tools that uses hard-coded thresholds on ORES' prediction probabilities will suddenly start behaving strangely. Regretfully, we haven't found a way around this problem, so we're announcing the change now and we plan to deploy this *BREAKING CHANGE on April 7th*. Please subscribe to the AI mailinglist <https://lists.wikimedia.org/mailman/listinfo/ai> or watch our project page [[:m:ORES <https://meta.wikimedia.org/wiki/ORES>]] to catch announcements of future changes and new functionality. In order to make sure we don't end up in the same situation the next time we want to change an algorithm, we've included a suite of evaluation statistics with each model. The filter_rate_at_recall(0.9), filter_rate_at_recall(0.75), and recall_at_fpr(0.1) thresholds represent three critical thresholds (should review, needs review, and definitely damaging -- respectively) that can be used to automatically configure your wiki tool. You can find out these thresholds for your model of choice by adding the ?model_info parameter to requests. So, come breaking change, we strongly recommend basing your thresholds on these statistics in the future. We'll be working to submit patches to tools that use ORES in the next week to implement this flexibility. Hopefully, all you'll need to do is worth with us on those. -halfak & The Revision Scoring team <https://meta.wikimedia.org/wiki/Research:Revision_scoring_as_a_service>

1 1

Re: [AI] [Research-Internal] Revision Scoring's massive backlog
by Aaron Halfaker 30 Mar '16

30 Mar '16

Fun story, the logs of #wikimedia-dev suggest that I performed 218 actions during the grooming of this backlog. :) On Wed, Mar 30, 2016 at 2:01 PM, Grace Gellerman <ggellerman(a)wikimedia.org> wrote: > Your micro-victory looks great- well done! > > On Wed, Mar 30, 2016 at 10:46 AM, Aaron Halfaker <ahalfaker(a)wikimedia.org> > wrote: > >> Hey folks, >> >> I had a micro victory today, so I thought I'd share. The revision >> scoring project has a huge backlog. I just spent a long time resolving old >> tasks and organizing the *live* tasks by the projects they affect and what >> they need me and my team to do. >> >> See >> https://phabricator.wikimedia.org/tag/revision-scoring-as-a-service-backlog/ >> >> Why you're looking (assuming you care to look at another project's >> backlog), feel free to file new tasks in the "Backlog" or the appropriate >> column. >> >> Note that the columns are not sorted in a nice way. Next time I have >> time for this, I'll be doing prioritization. >> >> -Aaron >> >> _______________________________________________ >> Research-Internal mailing list >> Research-Internal(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/research-internal >> >> > > _______________________________________________ > Research-Internal mailing list > Research-Internal(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/research-internal > >

1 0

Revision Scoring's massive backlog
by Aaron Halfaker 30 Mar '16

30 Mar '16

Hey folks, I had a micro victory today, so I thought I'd share. The revision scoring project has a huge backlog. I just spent a long time resolving old tasks and organizing the *live* tasks by the projects they affect and what they need me and my team to do. See https://phabricator.wikimedia.org/tag/revision-scoring-as-a-service-backlog/ Why you're looking (assuming you care to look at another project's backlog), feel free to file new tasks in the "Backlog" or the appropriate column. Note that the columns are not sorted in a nice way. Next time I have time for this, I'll be doing prioritization. -Aaron

2 1

ORES -- Minor downtime today for redis reboot
by Aaron Halfaker 23 Mar '16

23 Mar '16

Hey folks, As part of setting up production servers, we need to do a change that will affect our hosts in labs (via puppet). This will require a brief downtime event that should last less than 15 minutes. We'll do this at 1400 UTC tomorrow, Tuesday March, 23rd. I'll post an update once we're done. -Aaron

1 1

ORES maintenance and possible issues
by Amir Ladsgroup 19 Mar '16

19 Mar '16

Hey all, We are working on deployment of new version of ORES. This new version includes new version of revscoring, API v2 updates, and new models for arwiki and plwiki. The maintenance is taking more than we anticipated so you might experience some issues or delays requesting scores. We apologize for any inconvenience. We let you know once it's done. Thank you for your patience. Best

2 1

Fwd: Re: [Wikimedia-l] [Wmfall] Inspire Campaign on content curation & review launches today!
by Pine W 29 Feb '16

29 Feb '16

Forwarding the campaign announcement, and ideas from Aaron Halfaker. Pine ---------- Forwarded message ---------- From: "Aaron Halfaker" <ahalfaker(a)wikimedia.org> Date: Feb 29, 2016 16:00 Subject: Re: [Wikimedia-l] [Wmfall] Inspire Campaign on content curation & review launches today! To: "Chris Jethro Schilling" <cschilling(a)wikimedia.org> Cc: "Staff All" <wmfall(a)lists.wikimedia.org>, <glam(a)lists.wikimedia.org>, "Wikimedia Mailing List" <wikimedia-l(a)lists.wikimedia.org> I just finished submitting two ideas that I'd like to advise. https://meta.wikimedia.org/wiki/Grants:IdeaLab/Automated_good-faith_newcome… Build and deploy a machine learning model for flagging newcomers who are editing in good-faith. This has the potential to mitigate some of the secondary, demotivational effects when good-faith newcomers' work passes through curation/review processes. https://meta.wikimedia.org/wiki/Grants:IdeaLab/Fast_and_slow_new_article_re… Concerns about the introduction of spam into Wikipedia has lead Wikipedians towards implementing high speed new article review/curation processes. The speed at which editors tag articles for deletion via these processes is great for dealing with spam, but it might also be faster that good-faith new article creators can build their articles. We could build a machine learning classifier that is tuned to detect spammy article drafts. This would allow the new pages queue to be split into a high-speed spammy article review, and a low-speed article review that allows creators time to make a better first draft. I'll submit some more when I can. :) On Sun, Feb 28, 2016 at 4:56 PM, Chris "Jethro" Schilling < cschilling(a)wikimedia.org> wrote: > Hi everyone, > > I am pleased to announce the launch of the second Inspire Campaign for > IdeaLab.[1] The theme of this campaign is focused on improving tasks > related to content curation & review in our projects: > > <https://meta.wikimedia.org/wiki/Grants:IdeaLab/Inspire> > > Reviewing and organizing tasks are fundamental to all WIkimedia projects, > and these efforts maintain and directly improve the quality of our projects > in addition to increasing the visibility of their content. We invite > everyone to participate by sharing your ideas and proposals on how to > enhance these efforts. Constructive feedback and collaboration on ideas is > encouraged - your skills and advice can elevate a project into action. The > campaign runs until 29 March. > > All proposals are welcome - research projects, technical solutions, > community organizing and outreach, or something completely new! Grants are > available from the Wikimedia Foundation for projects developed during this > campaign that need financial support.[2] Google Hangout sessions are > available in March if you'd like to have a conversation about your ideas.[3] > > Join the Inspire Campaign and let’s work together to improve review and > curation tasks so that we can make our content more meaningful and > accessible. > > With thanks, > > Jethro > > [1] You can learn more about the results of the first Inspire Campaign > here: < > https://meta.wikimedia.org/wiki/Research:Spring_2015_Inspire_campaign> > [2] <https://meta.wikimedia.org/wiki/Grants:Start> > [3] <https://meta.wikimedia.org/wiki/Grants:IdeaLab/Events> (Note: If > another time would work better for you, feel free to e-mail me or ping me > on-wiki). > > --- > Chris "Jethro" Schilling > I JethroBT (WMF) <https://meta.wikimedia.org/wiki/User:I_JethroBT_(WMF)> > Community Organizer, Wikimedia Foundation > <https://wikimediafoundation.org/wiki/Home> > > _______________________________________________ > Wmfall mailing list > Wmfall(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wmfall > > _______________________________________________ Wikimedia-l mailing list, guidelines at: https://meta.wikimedia.org/wiki/Mailing_lists/Guidelines New messages to: Wikimedia-l(a)lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe>

1 0

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

AI