Hello,
Unfortunately we are experiencing spam in our Phabricator instance again and have decided to turn on the requirement for new account approval by Phabricator admins as a mitigation step.
I'm sorry for the inconvenience. We are actively working to address this issue.
Greg
On Sat, Jun 30, 2018 at 8:53 PM Greg Grossmeier greg@wikimedia.org wrote:
Hello,
Unfortunately we are experiencing spam in our Phabricator instance again and have decided to turn on the requirement for new account approval by Phabricator admins as a mitigation step.
I'd request that it please be kept on until we have some spam mitigation tools. At the very least easier revert actions.
I'm sorry for the inconvenience. We are actively working to address this issue.
Greg
-- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | Release Team Manager A18D 1138 8E47 FAC8 1C7D | _______________________________________________ Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
On Sun, Jul 1, 2018 at 7:16 AM Niharika Kohli nkohli@wikimedia.org wrote:
On Sat, Jun 30, 2018 at 8:53 PM Greg Grossmeier greg@wikimedia.org wrote:
Hello,
Unfortunately we are experiencing spam in our Phabricator instance again and have decided to turn on the requirement for new account approval by Phabricator admins as a mitigation step.
I'd request that it please be kept on until we have some spam mitigation tools. At the very least easier revert actions.
Indeed. We should *not* remove the approval process until a better anti-vandalism system is available for phabricator.
Repairing the damage that has been done will require a ton of man-hours.
Cheers, Giuseppe
Hey Greg, Thanks for the update. Should we go head and start manually reverting the vandalism where we see it or is there any possibility of rolling everything back to before the attack?
On Sat, Jun 30, 2018 at 10:57 PM Giuseppe Lavagetto < glavagetto@wikimedia.org> wrote:
On Sun, Jul 1, 2018 at 7:16 AM Niharika Kohli nkohli@wikimedia.org wrote:
On Sat, Jun 30, 2018 at 8:53 PM Greg Grossmeier greg@wikimedia.org wrote:
Hello,
Unfortunately we are experiencing spam in our Phabricator instance again and have decided to turn on the requirement for new account approval by Phabricator admins as a mitigation step.
I'd request that it please be kept on until we have some spam mitigation tools. At the very least easier revert actions.
Indeed. We should *not* remove the approval process until a better anti-vandalism system is available for phabricator.
Repairing the damage that has been done will require a ton of man-hours.
Cheers, Giuseppe -- Giuseppe Lavagetto Senior Technical Operations Engineer, Wikimedia Foundation _______________________________________________ Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
Thanks to all the people who are working (on a weekend!) to fix this.
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore
2018-07-01 5:53 GMT+02:00 Greg Grossmeier greg@wikimedia.org:
Hello,
Unfortunately we are experiencing spam in our Phabricator instance again and have decided to turn on the requirement for new account approval by Phabricator admins as a mitigation step.
I'm sorry for the inconvenience. We are actively working to address this issue.
Greg
-- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | Release Team Manager A18D 1138 8E47 FAC8 1C7D |
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
I wrote a rollback script, currently running as CommunityTechBot https://phabricator.wikimedia.org/p/CommunityTechBot/ and previously Community Tech bot https://phabricator.wikimedia.org/p/Community_Tech_bot/. It seems to work, aside from setting the triage level, which hopefully isn't a huge deal. I can try to fix that later. It is also being slowed down by rate limiting. The script isn't quite shareable yet but when it is I'll publish it. Going to sleep now :)
~Leon
On Sun, Jul 1, 2018 at 2:58 AM Amir E. Aharoni amir.aharoni@mail.huji.ac.il wrote:
Thanks to all the people who are working (on a weekend!) to fix this.
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore
2018-07-01 5:53 GMT+02:00 Greg Grossmeier greg@wikimedia.org:
Hello,
Unfortunately we are experiencing spam in our Phabricator instance again and have decided to turn on the requirement for new account approval by Phabricator admins as a mitigation step.
I'm sorry for the inconvenience. We are actively working to address this issue.
Greg
-- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | Release Team Manager A18D 1138 8E47 FAC8 1C7D |
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
We've got ourselves da MVP!
On Sun, Jul 1, 2018 at 12:51 AM, Leon Ziemba musikanimal@wikimedia.org wrote:
I wrote a rollback script, currently running as CommunityTechBot https://phabricator.wikimedia.org/p/CommunityTechBot/ and previously Community Tech bot https://phabricator.wikimedia.org/p/Community_Tech_bot/. It seems to work, aside from setting the triage level, which hopefully isn't a huge deal. I can try to fix that later. It is also being slowed down by rate limiting. The script isn't quite shareable yet but when it is I'll publish it. Going to sleep now :)
Is this script going to (eventually) undo all of the vandalism? Do users need to do anything else?
On Jul 1, 2018, at 2:31 AM, Max Semenik maxsem.wiki@gmail.com wrote:
We've got ourselves da MVP!
On Sun, Jul 1, 2018 at 12:51 AM, Leon Ziemba musikanimal@wikimedia.org wrote:
I wrote a rollback script, currently running as CommunityTechBot https://phabricator.wikimedia.org/p/CommunityTechBot/ and previously Community Tech bot https://phabricator.wikimedia.org/p/Community_Tech_bot/. It seems to work, aside from setting the triage level, which hopefully isn't a huge deal. I can try to fix that later. It is also being slowed down by rate limiting. The script isn't quite shareable yet but when it is I'll publish it. Going to sleep now :)
-- Best regards, Max Semenik ([[User:MaxSem]]) _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
An update... the bot went to sleep as instructed a few hours after I went to sleep. Bot is now back up and running, with some ~4,500 tasks still to fix.
A few problems: * The new "rate limiting" of the API is rather rigorous. Release engineering tried to whitelist the bot but we had no luck. So, it will take some time to go through everything. * If the bot hits the rate limit while editing a task, all other changes it was going to make to that task didn't happen. Hence you may see only some corrections on some tasks. * The priority level is now being set to "Needs triage". This is because the Conduit API gives me numbers for the priority level, and the edit API wants a string (?!?). I don't know what numbers are for what priorities, so "Needs triage" it is. Older versions of the script left the priority level unchanged, so either way you may wish to review the priorities of your tasks. If you know what the priority number to string mapping is, please tell me :)
Cheers,
~Leon
On Sun, Jul 1, 2018 at 5:32 AM Max Semenik maxsem.wiki@gmail.com wrote:
We've got ourselves da MVP!
On Sun, Jul 1, 2018 at 12:51 AM, Leon Ziemba musikanimal@wikimedia.org wrote:
I wrote a rollback script, currently running as CommunityTechBot https://phabricator.wikimedia.org/p/CommunityTechBot/ and previously Community Tech bot https://phabricator.wikimedia.org/p/Community_Tech_bot/. It seems to work, aside from setting the triage level, which hopefully
isn't a
huge deal. I can try to fix that later. It is also being slowed down by rate limiting. The script isn't quite shareable yet but when it is I'll publish it. Going to sleep now :)
-- Best regards, Max Semenik ([[User:MaxSem]]) _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Hi. Leon, A lot of tasks with no "Needs triage" have some text about changing priority. Can't you use it? Thank you, Igal (User:IKhitron)
2018-07-01 18:46 GMT+03:00 Leon Ziemba musikanimal@wikimedia.org:
An update... the bot went to sleep as instructed a few hours after I went to sleep. Bot is now back up and running, with some ~4,500 tasks still to fix.
A few problems:
- The new "rate limiting" of the API is rather rigorous. Release
engineering tried to whitelist the bot but we had no luck. So, it will take some time to go through everything.
- If the bot hits the rate limit while editing a task, all other changes it
was going to make to that task didn't happen. Hence you may see only some corrections on some tasks.
- The priority level is now being set to "Needs triage". This is because
the Conduit API gives me numbers for the priority level, and the edit API wants a string (?!?). I don't know what numbers are for what priorities, so "Needs triage" it is. Older versions of the script left the priority level unchanged, so either way you may wish to review the priorities of your tasks. If you know what the priority number to string mapping is, please tell me :)
Cheers,
~Leon
On Sun, Jul 1, 2018 at 5:32 AM Max Semenik maxsem.wiki@gmail.com wrote:
We've got ourselves da MVP!
On Sun, Jul 1, 2018 at 12:51 AM, Leon Ziemba musikanimal@wikimedia.org wrote:
I wrote a rollback script, currently running as CommunityTechBot https://phabricator.wikimedia.org/p/CommunityTechBot/ and previously Community Tech bot https://phabricator.wikimedia.org/p/Community_Tech_bot/. It seems to work, aside from setting the triage level, which hopefully
isn't a
huge deal. I can try to fix that later. It is also being slowed down by rate limiting. The script isn't quite shareable yet but when it is I'll publish it. Going to sleep now :)
-- Best regards, Max Semenik ([[User:MaxSem]]) _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
The triage level should now be properly set to what it was before. Thanks to Andre for informing me of the number-to-string mapping. Triage levels from before ~16:00 UTC may be incorrect.
Again there are some issues where the bot hits the rate limiting before it's done with a given task, so *occasionally* it wouldn't have cleaned up everything. I don't think you need to review every task the bot edited, but keep an eye on the tasks you follow. I have slowed down the bot to try to prevent this from happening.
~Leon
On Sun, Jul 1, 2018 at 11:57 AM יגאל חיטרון khitron@post.bgu.ac.il wrote:
Hi. Leon, A lot of tasks with no "Needs triage" have some text about changing priority. Can't you use it? Thank you, Igal (User:IKhitron)
2018-07-01 18:46 GMT+03:00 Leon Ziemba musikanimal@wikimedia.org:
An update... the bot went to sleep as instructed a few hours after I went to sleep. Bot is now back up and running, with some ~4,500 tasks still to fix.
A few problems:
- The new "rate limiting" of the API is rather rigorous. Release
engineering tried to whitelist the bot but we had no luck. So, it will
take
some time to go through everything.
- If the bot hits the rate limit while editing a task, all other changes
it
was going to make to that task didn't happen. Hence you may see only some corrections on some tasks.
- The priority level is now being set to "Needs triage". This is because
the Conduit API gives me numbers for the priority level, and the edit API wants a string (?!?). I don't know what numbers are for what priorities,
so
"Needs triage" it is. Older versions of the script left the priority
level
unchanged, so either way you may wish to review the priorities of your tasks. If you know what the priority number to string mapping is, please tell me :)
Cheers,
~Leon
On Sun, Jul 1, 2018 at 5:32 AM Max Semenik maxsem.wiki@gmail.com
wrote:
We've got ourselves da MVP!
On Sun, Jul 1, 2018 at 12:51 AM, Leon Ziemba <
musikanimal@wikimedia.org>
wrote:
I wrote a rollback script, currently running as CommunityTechBot https://phabricator.wikimedia.org/p/CommunityTechBot/ and
previously
Community Tech bot https://phabricator.wikimedia.org/p/Community_Tech_bot/.
It
seems to work, aside from setting the triage level, which hopefully
isn't a
huge deal. I can try to fix that later. It is also being slowed down
by
rate limiting. The script isn't quite shareable yet but when it is
I'll
publish it. Going to sleep now :)
-- Best regards, Max Semenik ([[User:MaxSem]]) _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Sun, 2018-07-01 at 18:57 +0300, יגאל חיטרון wrote:
Hi. Leon, A lot of tasks with no "Needs triage" have some text about changing priority. Can't you use it?
This has already been fixed in more recent edits by that bot.
andre
Hi Leon. I can't thank you enough for your efforts to help clean things up in Phabricator. I can, however, help make the bot more effective. See below for responses inline.
On Sun, Jul 1, 2018 at 10:47 AM Leon Ziemba musikanimal@wikimedia.org wrote:
An update... the bot went to sleep as instructed a few hours after I went to sleep. Bot is now back up and running, with some ~4,500 tasks still to fix.
A few problems:
- The new "rate limiting" of the API is rather rigorous. Release
engineering tried to whitelist the bot but we had no luck. So, it will take some time to go through everything.
I'm still looking into why the bot hits the rate limit. I'm sure I can come up with a way to get it whitelisted.
- If the bot hits the rate limit while editing a task, all other changes
it was going to make to that task didn't happen. Hence you may see only some corrections on some tasks.
- The priority level is now being set to "Needs triage". This is because
the Conduit API gives me numbers for the priority level, and the edit API wants a string (?!?). I don't know what numbers are for what priorities, so "Needs triage" it is. Older versions of the script left the priority level unchanged, so either way you may wish to review the priorities of your tasks. If you know what the priority number to string mapping is, please tell me :)
If you would like to alter the bot to restore the correct priority, this should help; The priority levels are configured as follows:
{ "10": { "color": "sky", "keywords": [ "lowest" ], "name": "Lowest", "short": "Lowest" }, "25": { "color": "yellow", "keywords": [ "low" ], "name": "Low", "short": "Low" }, "50": { "color": "orange", "keywords": [ "normal" ], "name": "Normal", "short": "Normal" }, "80": { "color": "red", "keywords": [ "high" ], "name": "High", "short": "High" }, "90": { "color": "violet", "keywords": [ "triage" ], "name": "Needs Triage", "short": "Triage" }, "100": { "color": "pink", "keywords": [ "unbreak" ], "name": "Unbreak Now!", "short": "Unbreak!" } }
Cheers,
~Leon
On Sun, Jul 1, 2018 at 5:32 AM Max Semenik maxsem.wiki@gmail.com wrote:
We've got ourselves da MVP!
On Sun, Jul 1, 2018 at 12:51 AM, Leon Ziemba musikanimal@wikimedia.org wrote:
I wrote a rollback script, currently running as CommunityTechBot https://phabricator.wikimedia.org/p/CommunityTechBot/ and previously Community Tech bot https://phabricator.wikimedia.org/p/Community_Tech_bot/. It seems to work, aside from setting the triage level, which hopefully
isn't a
huge deal. I can try to fix that later. It is also being slowed down by rate limiting. The script isn't quite shareable yet but when it is I'll publish it. Going to sleep now :)
-- Best regards, Max Semenik ([[User:MaxSem]]) _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
The bot has now completed it's run. If you see any outstanding tasks that need to be repaired, please give me the task IDs.
The bot ran for roughly 36 hours, repairing at least 4,000 tasks (perhaps many more).
There were some issues with the bot that may still affect your tasks: * The triage level was not restored, or was put in "Needs triage". This was fixed around 16:00 UTC on July 1. Hundreds of tasks were affected. * For most of the bot's run, it was subject to a newly imposed rate limiting. If the rate limit was hit in the middle of repairing a task, the bot may not have fixed everything. Many tasks were affected. This issue was fixed around 15:00 UTC on July 1. * For some tasks, the vandal removed tags as well adding some. The bot did not properly restore the removed tags until around 12:00 UTC on July 2. The number of tasks affected by this is estimated to be low. * Some tasks have "custom fields" that were vandalized, which the bot did not restore. An example is the "due date" on https://phabricator.wikimedia.org/T193593. The number of tasks affected by this should be very low.
If you notice any tasks where the bot didn't fix everything, and you don't want to fix it yourself, just give me the task IDs and I can re-run the bot on those.
Thanks to Andre, Mukunda, and everyone else to helped with this effort.
~Leon
On Sun, Jul 1, 2018 at 8:49 PM Mukunda Modell mmodell@wikimedia.org wrote:
Hi Leon. I can't thank you enough for your efforts to help clean things up in Phabricator. I can, however, help make the bot more effective. See below for responses inline.
On Sun, Jul 1, 2018 at 10:47 AM Leon Ziemba musikanimal@wikimedia.org wrote:
An update... the bot went to sleep as instructed a few hours after I went to sleep. Bot is now back up and running, with some ~4,500 tasks still to fix.
A few problems:
- The new "rate limiting" of the API is rather rigorous. Release
engineering tried to whitelist the bot but we had no luck. So, it will take some time to go through everything.
I'm still looking into why the bot hits the rate limit. I'm sure I can come up with a way to get it whitelisted.
- If the bot hits the rate limit while editing a task, all other changes
it was going to make to that task didn't happen. Hence you may see only some corrections on some tasks.
- The priority level is now being set to "Needs triage". This is because
the Conduit API gives me numbers for the priority level, and the edit API wants a string (?!?). I don't know what numbers are for what priorities, so "Needs triage" it is. Older versions of the script left the priority level unchanged, so either way you may wish to review the priorities of your tasks. If you know what the priority number to string mapping is, please tell me :)
If you would like to alter the bot to restore the correct priority, this should help; The priority levels are configured as follows:
{ "10": { "color": "sky", "keywords": [ "lowest" ], "name": "Lowest", "short": "Lowest" }, "25": { "color": "yellow", "keywords": [ "low" ], "name": "Low", "short": "Low" }, "50": { "color": "orange", "keywords": [ "normal" ], "name": "Normal", "short": "Normal" }, "80": { "color": "red", "keywords": [ "high" ], "name": "High", "short": "High" }, "90": { "color": "violet", "keywords": [ "triage" ], "name": "Needs Triage", "short": "Triage" }, "100": { "color": "pink", "keywords": [ "unbreak" ], "name": "Unbreak Now!", "short": "Unbreak!" } }
Cheers,
~Leon
On Sun, Jul 1, 2018 at 5:32 AM Max Semenik maxsem.wiki@gmail.com wrote:
We've got ourselves da MVP!
On Sun, Jul 1, 2018 at 12:51 AM, Leon Ziemba musikanimal@wikimedia.org wrote:
I wrote a rollback script, currently running as CommunityTechBot https://phabricator.wikimedia.org/p/CommunityTechBot/ and previously Community Tech bot https://phabricator.wikimedia.org/p/Community_Tech_bot/. It seems to work, aside from setting the triage level, which hopefully
isn't a
huge deal. I can try to fix that later. It is also being slowed down by rate limiting. The script isn't quite shareable yet but when it is I'll publish it. Going to sleep now :)
-- Best regards, Max Semenik ([[User:MaxSem]]) _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
Thank you Leon for this heroic effort, and thanks to everyone who helped in this cleanup.
On Mon, Jul 2, 2018, 9:58 AM Leon Ziemba musikanimal@wikimedia.org wrote:
The bot has now completed it's run. If you see any outstanding tasks that need to be repaired, please give me the task IDs.
The bot ran for roughly 36 hours, repairing at least 4,000 tasks (perhaps many more).
There were some issues with the bot that may still affect your tasks:
- The triage level was not restored, or was put in "Needs triage". This
was fixed around 16:00 UTC on July 1. Hundreds of tasks were affected.
- For most of the bot's run, it was subject to a newly imposed rate
limiting. If the rate limit was hit in the middle of repairing a task, the bot may not have fixed everything. Many tasks were affected. This issue was fixed around 15:00 UTC on July 1.
- For some tasks, the vandal removed tags as well adding some. The bot did
not properly restore the removed tags until around 12:00 UTC on July 2. The number of tasks affected by this is estimated to be low.
- Some tasks have "custom fields" that were vandalized, which the bot did
not restore. An example is the "due date" on https://phabricator.wikimedia.org/T193593. The number of tasks affected by this should be very low.
If you notice any tasks where the bot didn't fix everything, and you don't want to fix it yourself, just give me the task IDs and I can re-run the bot on those.
Thanks to Andre, Mukunda, and everyone else to helped with this effort.
~Leon
On Sun, Jul 1, 2018 at 8:49 PM Mukunda Modell mmodell@wikimedia.org wrote:
Hi Leon. I can't thank you enough for your efforts to help clean things up in Phabricator. I can, however, help make the bot more effective. See below for responses inline.
On Sun, Jul 1, 2018 at 10:47 AM Leon Ziemba musikanimal@wikimedia.org wrote:
An update... the bot went to sleep as instructed a few hours after I went to sleep. Bot is now back up and running, with some ~4,500 tasks still to fix.
A few problems:
- The new "rate limiting" of the API is rather rigorous. Release
engineering tried to whitelist the bot but we had no luck. So, it will take some time to go through everything.
I'm still looking into why the bot hits the rate limit. I'm sure I can come up with a way to get it whitelisted.
- If the bot hits the rate limit while editing a task, all other changes
it was going to make to that task didn't happen. Hence you may see only some corrections on some tasks.
- The priority level is now being set to "Needs triage". This is because
the Conduit API gives me numbers for the priority level, and the edit API wants a string (?!?). I don't know what numbers are for what priorities, so "Needs triage" it is. Older versions of the script left the priority level unchanged, so either way you may wish to review the priorities of your tasks. If you know what the priority number to string mapping is, please tell me :)
If you would like to alter the bot to restore the correct priority, this should help; The priority levels are configured as follows:
{ "10": { "color": "sky", "keywords": [ "lowest" ], "name": "Lowest", "short": "Lowest" }, "25": { "color": "yellow", "keywords": [ "low" ], "name": "Low", "short": "Low" }, "50": { "color": "orange", "keywords": [ "normal" ], "name": "Normal", "short": "Normal" }, "80": { "color": "red", "keywords": [ "high" ], "name": "High", "short": "High" }, "90": { "color": "violet", "keywords": [ "triage" ], "name": "Needs Triage", "short": "Triage" }, "100": { "color": "pink", "keywords": [ "unbreak" ], "name": "Unbreak Now!", "short": "Unbreak!" } }
Cheers,
~Leon
On Sun, Jul 1, 2018 at 5:32 AM Max Semenik maxsem.wiki@gmail.com wrote:
We've got ourselves da MVP!
On Sun, Jul 1, 2018 at 12:51 AM, Leon Ziemba <musikanimal@wikimedia.org
wrote:
I wrote a rollback script, currently running as CommunityTechBot https://phabricator.wikimedia.org/p/CommunityTechBot/ and
previously
Community Tech bot https://phabricator.wikimedia.org/p/Community_Tech_bot/.
It
seems to work, aside from setting the triage level, which hopefully
isn't a
huge deal. I can try to fix that later. It is also being slowed down
by
rate limiting. The script isn't quite shareable yet but when it is
I'll
publish it. Going to sleep now :)
-- Best regards, Max Semenik ([[User:MaxSem]]) _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
+100
Also a tip for those of you wanting to clear up the resulting email spam: you can temporarily turn off "conversation mode" in Gmail's settings, then search for messages from CommunityTechBot or the spammer, and delete all these emails without having to delete the threads they belong too.
On Mon, 2 Jul 2018 at 18:18, Joel Aufrecht jaufrecht@wikimedia.org wrote:
Thank you Leon for this heroic effort, and thanks to everyone who helped in this cleanup.
On Mon, Jul 2, 2018, 9:58 AM Leon Ziemba musikanimal@wikimedia.org wrote:
The bot has now completed it's run. If you see any outstanding tasks that need to be repaired, please give me the task IDs.
The bot ran for roughly 36 hours, repairing at least 4,000 tasks (perhaps many more).
There were some issues with the bot that may still affect your tasks:
- The triage level was not restored, or was put in "Needs triage". This
was fixed around 16:00 UTC on July 1. Hundreds of tasks were affected.
- For most of the bot's run, it was subject to a newly imposed rate
limiting. If the rate limit was hit in the middle of repairing a task, the bot may not have fixed everything. Many tasks were affected. This issue was fixed around 15:00 UTC on July 1.
- For some tasks, the vandal removed tags as well adding some. The bot
did not properly restore the removed tags until around 12:00 UTC on July 2. The number of tasks affected by this is estimated to be low.
- Some tasks have "custom fields" that were vandalized, which the bot did
not restore. An example is the "due date" on https://phabricator.wikimedia.org/T193593. The number of tasks affected by this should be very low.
If you notice any tasks where the bot didn't fix everything, and you don't want to fix it yourself, just give me the task IDs and I can re-run the bot on those.
Thanks to Andre, Mukunda, and everyone else to helped with this effort.
~Leon
On Sun, Jul 1, 2018 at 8:49 PM Mukunda Modell mmodell@wikimedia.org wrote:
Hi Leon. I can't thank you enough for your efforts to help clean things up in Phabricator. I can, however, help make the bot more effective. See below for responses inline.
On Sun, Jul 1, 2018 at 10:47 AM Leon Ziemba musikanimal@wikimedia.org wrote:
An update... the bot went to sleep as instructed a few hours after I went to sleep. Bot is now back up and running, with some ~4,500 tasks still to fix.
A few problems:
- The new "rate limiting" of the API is rather rigorous. Release
engineering tried to whitelist the bot but we had no luck. So, it will take some time to go through everything.
I'm still looking into why the bot hits the rate limit. I'm sure I can come up with a way to get it whitelisted.
- If the bot hits the rate limit while editing a task, all other
changes it was going to make to that task didn't happen. Hence you may see only some corrections on some tasks.
- The priority level is now being set to "Needs triage". This is
because the Conduit API gives me numbers for the priority level, and the edit API wants a string (?!?). I don't know what numbers are for what priorities, so "Needs triage" it is. Older versions of the script left the priority level unchanged, so either way you may wish to review the priorities of your tasks. If you know what the priority number to string mapping is, please tell me :)
If you would like to alter the bot to restore the correct priority, this should help; The priority levels are configured as follows:
{ "10": { "color": "sky", "keywords": [ "lowest" ], "name": "Lowest", "short": "Lowest" }, "25": { "color": "yellow", "keywords": [ "low" ], "name": "Low", "short": "Low" }, "50": { "color": "orange", "keywords": [ "normal" ], "name": "Normal", "short": "Normal" }, "80": { "color": "red", "keywords": [ "high" ], "name": "High", "short": "High" }, "90": { "color": "violet", "keywords": [ "triage" ], "name": "Needs Triage", "short": "Triage" }, "100": { "color": "pink", "keywords": [ "unbreak" ], "name": "Unbreak Now!", "short": "Unbreak!" } }
Cheers,
~Leon
On Sun, Jul 1, 2018 at 5:32 AM Max Semenik maxsem.wiki@gmail.com wrote:
We've got ourselves da MVP!
On Sun, Jul 1, 2018 at 12:51 AM, Leon Ziemba < musikanimal@wikimedia.org> wrote:
I wrote a rollback script, currently running as CommunityTechBot https://phabricator.wikimedia.org/p/CommunityTechBot/ and
previously
Community Tech bot https://phabricator.wikimedia.org/p/Community_Tech_bot/.
It
seems to work, aside from setting the triage level, which hopefully
isn't a
huge deal. I can try to fix that later. It is also being slowed down
by
rate limiting. The script isn't quite shareable yet but when it is
I'll
publish it. Going to sleep now :)
-- Best regards, Max Semenik ([[User:MaxSem]]) _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
Yes, though careful because CommunityTechBot shows up as a subscriber to lots of tasks now which are getting comments from other people, so it appears on the Cc list at the bottom of each email notification.
On 2 July 2018 at 19:06, Ed Sanders esanders@wikimedia.org wrote:
+100
Also a tip for those of you wanting to clear up the resulting email spam: you can temporarily turn off "conversation mode" in Gmail's settings, then search for messages from CommunityTechBot or the spammer, and delete all these emails without having to delete the threads they belong too.
On Mon, 2 Jul 2018 at 18:18, Joel Aufrecht jaufrecht@wikimedia.org wrote:
Thank you Leon for this heroic effort, and thanks to everyone who helped in this cleanup.
On Mon, Jul 2, 2018, 9:58 AM Leon Ziemba musikanimal@wikimedia.org wrote:
The bot has now completed it's run. If you see any outstanding tasks that need to be repaired, please give me the task IDs.
The bot ran for roughly 36 hours, repairing at least 4,000 tasks (perhaps many more).
There were some issues with the bot that may still affect your tasks:
- The triage level was not restored, or was put in "Needs triage". This
was fixed around 16:00 UTC on July 1. Hundreds of tasks were affected.
- For most of the bot's run, it was subject to a newly imposed rate
limiting. If the rate limit was hit in the middle of repairing a task, the bot may not have fixed everything. Many tasks were affected. This issue was fixed around 15:00 UTC on July 1.
- For some tasks, the vandal removed tags as well adding some. The bot
did not properly restore the removed tags until around 12:00 UTC on July 2. The number of tasks affected by this is estimated to be low.
- Some tasks have "custom fields" that were vandalized, which the bot
did not restore. An example is the "due date" on https://phabricator.wikimedia.org/T193593. The number of tasks affected by this should be very low.
If you notice any tasks where the bot didn't fix everything, and you don't want to fix it yourself, just give me the task IDs and I can re-run the bot on those.
Thanks to Andre, Mukunda, and everyone else to helped with this effort.
~Leon
On Sun, Jul 1, 2018 at 8:49 PM Mukunda Modell mmodell@wikimedia.org wrote:
Hi Leon. I can't thank you enough for your efforts to help clean things up in Phabricator. I can, however, help make the bot more effective. See below for responses inline.
On Sun, Jul 1, 2018 at 10:47 AM Leon Ziemba musikanimal@wikimedia.org wrote:
An update... the bot went to sleep as instructed a few hours after I went to sleep. Bot is now back up and running, with some ~4,500 tasks still to fix.
A few problems:
- The new "rate limiting" of the API is rather rigorous. Release
engineering tried to whitelist the bot but we had no luck. So, it will take some time to go through everything.
I'm still looking into why the bot hits the rate limit. I'm sure I can come up with a way to get it whitelisted.
- If the bot hits the rate limit while editing a task, all other
changes it was going to make to that task didn't happen. Hence you may see only some corrections on some tasks.
- The priority level is now being set to "Needs triage". This is
because the Conduit API gives me numbers for the priority level, and the edit API wants a string (?!?). I don't know what numbers are for what priorities, so "Needs triage" it is. Older versions of the script left the priority level unchanged, so either way you may wish to review the priorities of your tasks. If you know what the priority number to string mapping is, please tell me :)
If you would like to alter the bot to restore the correct priority, this should help; The priority levels are configured as follows:
{ "10": { "color": "sky", "keywords": [ "lowest" ], "name": "Lowest", "short": "Lowest" }, "25": { "color": "yellow", "keywords": [ "low" ], "name": "Low", "short": "Low" }, "50": { "color": "orange", "keywords": [ "normal" ], "name": "Normal", "short": "Normal" }, "80": { "color": "red", "keywords": [ "high" ], "name": "High", "short": "High" }, "90": { "color": "violet", "keywords": [ "triage" ], "name": "Needs Triage", "short": "Triage" }, "100": { "color": "pink", "keywords": [ "unbreak" ], "name": "Unbreak Now!", "short": "Unbreak!" } }
Cheers,
~Leon
On Sun, Jul 1, 2018 at 5:32 AM Max Semenik maxsem.wiki@gmail.com wrote:
We've got ourselves da MVP!
On Sun, Jul 1, 2018 at 12:51 AM, Leon Ziemba < musikanimal@wikimedia.org> wrote:
> I wrote a rollback script, currently running as CommunityTechBot > https://phabricator.wikimedia.org/p/CommunityTechBot/ and previously > Community > Tech bot https://phabricator.wikimedia.org/p/Community_Tech_bot/. It > seems to work, aside from setting the triage level, which hopefully isn't a > huge deal. I can try to fix that later. It is also being slowed down by > rate limiting. The script isn't quite shareable yet but when it is I'll > publish it. Going to sleep now :) >
-- Best regards, Max Semenik ([[User:MaxSem]]) _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
Yes, though careful because CommunityTechBot shows up as a subscriber to
lots of tasks now which are getting comments from other people, so it appears on the Cc list at the bottom of each email notification.
Oh, oops :/ I think I can automate removing the bot as a subscriber? Will look into that later.
~Leon
On Mon, Jul 2, 2018 at 2:08 PM Alex Monk krenair@gmail.com wrote:
Yes, though careful because CommunityTechBot shows up as a subscriber to lots of tasks now which are getting comments from other people, so it appears on the Cc list at the bottom of each email notification.
On 2 July 2018 at 19:06, Ed Sanders esanders@wikimedia.org wrote:
+100
Also a tip for those of you wanting to clear up the resulting email spam: you can temporarily turn off "conversation mode" in Gmail's settings, then search for messages from CommunityTechBot or the spammer, and delete all these emails without having to delete the threads they belong too.
On Mon, 2 Jul 2018 at 18:18, Joel Aufrecht jaufrecht@wikimedia.org wrote:
Thank you Leon for this heroic effort, and thanks to everyone who helped in this cleanup.
On Mon, Jul 2, 2018, 9:58 AM Leon Ziemba musikanimal@wikimedia.org wrote:
The bot has now completed it's run. If you see any outstanding tasks that need to be repaired, please give me the task IDs.
The bot ran for roughly 36 hours, repairing at least 4,000 tasks (perhaps many more).
There were some issues with the bot that may still affect your tasks:
- The triage level was not restored, or was put in "Needs triage". This
was fixed around 16:00 UTC on July 1. Hundreds of tasks were affected.
- For most of the bot's run, it was subject to a newly imposed rate
limiting. If the rate limit was hit in the middle of repairing a task, the bot may not have fixed everything. Many tasks were affected. This issue was fixed around 15:00 UTC on July 1.
- For some tasks, the vandal removed tags as well adding some. The bot
did not properly restore the removed tags until around 12:00 UTC on July 2. The number of tasks affected by this is estimated to be low.
- Some tasks have "custom fields" that were vandalized, which the bot
did not restore. An example is the "due date" on https://phabricator.wikimedia.org/T193593. The number of tasks affected by this should be very low.
If you notice any tasks where the bot didn't fix everything, and you don't want to fix it yourself, just give me the task IDs and I can re-run the bot on those.
Thanks to Andre, Mukunda, and everyone else to helped with this effort.
~Leon
On Sun, Jul 1, 2018 at 8:49 PM Mukunda Modell mmodell@wikimedia.org wrote:
Hi Leon. I can't thank you enough for your efforts to help clean things up in Phabricator. I can, however, help make the bot more effective. See below for responses inline.
On Sun, Jul 1, 2018 at 10:47 AM Leon Ziemba musikanimal@wikimedia.org wrote:
An update... the bot went to sleep as instructed a few hours after I went to sleep. Bot is now back up and running, with some ~4,500 tasks still to fix.
A few problems:
- The new "rate limiting" of the API is rather rigorous. Release
engineering tried to whitelist the bot but we had no luck. So, it will take some time to go through everything.
I'm still looking into why the bot hits the rate limit. I'm sure I can come up with a way to get it whitelisted.
- If the bot hits the rate limit while editing a task, all other
changes it was going to make to that task didn't happen. Hence you may see only some corrections on some tasks.
- The priority level is now being set to "Needs triage". This is
because the Conduit API gives me numbers for the priority level, and the edit API wants a string (?!?). I don't know what numbers are for what priorities, so "Needs triage" it is. Older versions of the script left the priority level unchanged, so either way you may wish to review the priorities of your tasks. If you know what the priority number to string mapping is, please tell me :)
If you would like to alter the bot to restore the correct priority, this should help; The priority levels are configured as follows:
{ "10": { "color": "sky", "keywords": [ "lowest" ], "name": "Lowest", "short": "Lowest" }, "25": { "color": "yellow", "keywords": [ "low" ], "name": "Low", "short": "Low" }, "50": { "color": "orange", "keywords": [ "normal" ], "name": "Normal", "short": "Normal" }, "80": { "color": "red", "keywords": [ "high" ], "name": "High", "short": "High" }, "90": { "color": "violet", "keywords": [ "triage" ], "name": "Needs Triage", "short": "Triage" }, "100": { "color": "pink", "keywords": [ "unbreak" ], "name": "Unbreak Now!", "short": "Unbreak!" } }
Cheers,
~Leon
On Sun, Jul 1, 2018 at 5:32 AM Max Semenik maxsem.wiki@gmail.com wrote:
> We've got ourselves da MVP! > > On Sun, Jul 1, 2018 at 12:51 AM, Leon Ziemba < > musikanimal@wikimedia.org> > wrote: > > > I wrote a rollback script, currently running as CommunityTechBot > > https://phabricator.wikimedia.org/p/CommunityTechBot/ and > previously > > Community > > Tech bot https://phabricator.wikimedia.org/p/Community_Tech_bot/. > It > > seems to work, aside from setting the triage level, which > hopefully isn't a > > huge deal. I can try to fix that later. It is also being slowed > down by > > rate limiting. The script isn't quite shareable yet but when it is > I'll > > publish it. Going to sleep now :) > > > > -- > Best regards, > Max Semenik ([[User:MaxSem]]) > _______________________________________________ > Wikitech-l mailing list > Wikitech-l@lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
I wrote a short, quick fixer script that is terrible, but is saving me some time in fixing some tasks today.
I figured I'd share the script[0] on this list even thought it's very very (very) alpha and was written very quickly.
Thank you to everyone looking at and thinking about this issue.
-- Tyler
[0]. https://gist.github.com/thcipriani/9b09ca451852ac09ff924220b7770c1b
On 18-07-01 03:51:15, Leon Ziemba wrote:
I wrote a rollback script, currently running as CommunityTechBot https://phabricator.wikimedia.org/p/CommunityTechBot/ and previously Community Tech bot https://phabricator.wikimedia.org/p/Community_Tech_bot/. It seems to work, aside from setting the triage level, which hopefully isn't a huge deal. I can try to fix that later. It is also being slowed down by rate limiting. The script isn't quite shareable yet but when it is I'll publish it. Going to sleep now :)
~Leon
On Sun, Jul 1, 2018 at 2:58 AM Amir E. Aharoni amir.aharoni@mail.huji.ac.il wrote:
Thanks to all the people who are working (on a weekend!) to fix this.
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore
2018-07-01 5:53 GMT+02:00 Greg Grossmeier greg@wikimedia.org:
Hello,
Unfortunately we are experiencing spam in our Phabricator instance again and have decided to turn on the requirement for new account approval by Phabricator admins as a mitigation step.
I'm sorry for the inconvenience. We are actively working to address this issue.
Greg
-- | Greg Grossmeier GPG: B2FA 27B1 F7EB D327 6B8E | | Release Team Manager A18D 1138 8E47 FAC8 1C7D |
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Engineering mailing list Engineering@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/engineering
wikitech-l@lists.wikimedia.org