A village pump in Hungarian Wikipedia had not been archived for a long time before we noticed it. Investigation showed that the bot had hit a warning-type abuse filter upon copying the text to archive and saving. In this case the abuse filter displays a warning and lets the user press Save again if he/she wants to do it anyway. Of course, the archivebot
- did not save the page for the second time (could it?) - did not pass the problematic section and archive the remainder - did not let the owner know about the problem.
It just silently failed, with logging being the only action.
So what would the desired behaviour be in a similar case? Please keep in mind that archivebot is tyoically run with cron or other timing, not in interactive mode.
To see the log click on https://tools.wmflabs.org/ato/log/archive.txt and ctrl f for
ERROR: editpage: abusefilter-warning
Hi,
I believe in this case we should somehow implement 'press Save again' in the bot. I don't know if the MediaWiki API allows this, but if not, I think it should be added (to the AbuseFilter extension?).
If the edit was 'blocked' by an abuse filter, things would be more complicated because we would definitely need an admin to look at the case (and thus need for notification).
A village pump in Hungarian Wikipedia had not been archived for a long time before we noticed it. It just silently failed, with logging being the only action.
I think how to notify the interested parties of archiving failures is a general issue, not just for failures related to abuse filters. I created a Phabricator task for that a while ago. [2]
Best, Yusuke
On Sun, Jun 3, 2018 at 4:07 PM, Bináris wikiposta@gmail.com wrote:
A village pump in Hungarian Wikipedia had not been archived for a long time before we noticed it. Investigation showed that the bot had hit a warning-type abuse filter upon copying the text to archive and saving. In this case the abuse filter displays a warning and lets the user press Save again if he/she wants to do it anyway. Of course, the archivebot
did not save the page for the second time (could it?) did not pass the problematic section and archive the remainder did not let the owner know about the problem.
It just silently failed, with logging being the only action.
So what would the desired behaviour be in a similar case? Please keep in mind that archivebot is tyoically run with cron or other timing, not in interactive mode.
To see the log click on https://tools.wmflabs.org/ato/log/archive.txt and ctrl f for
ERROR: editpage: abusefilter-warning
-- Bináris
pywikibot mailing list pywikibot@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot
I believe that the abuse filter should be changed to allow the bot to edit.
The bot doesn't know whether this is a warning, and or disallow action, nor to know the consequences of hitting the filter for a second time, and expecting a bot to navigate a false positive on a wiki is not reasonable.
-- billinghurst
------ Original Message ------ From: "Yusuke Matsubara" whym@whym.org To: "Pywikibot discussion list" pywikibot@lists.wikimedia.org Sent: 3/06/2018 9:45:39 PM Subject: Re: [pywikibot] Archivebot fails on abuse filter
Hi,
I believe in this case we should somehow implement 'press Save again' in the bot. I don't know if the MediaWiki API allows this, but if not, I think it should be added (to the AbuseFilter extension?).
If the edit was 'blocked' by an abuse filter, things would be more complicated because we would definitely need an admin to look at the case (and thus need for notification).
A village pump in Hungarian Wikipedia had not been archived for a long time before we noticed it. It just silently failed, with logging being the only action.
I think how to notify the interested parties of archiving failures is a general issue, not just for failures related to abuse filters. I created a Phabricator task for that a while ago. [2]
Best, Yusuke
On Sun, Jun 3, 2018 at 4:07 PM, Bináris wikiposta@gmail.com wrote:
A village pump in Hungarian Wikipedia had not been archived for a long time before we noticed it. Investigation showed that the bot had hit a warning-type abuse filter upon copying the text to archive and saving. In this case the abuse filter displays a warning and lets the user press Save again if he/she wants to do it anyway. Of course, the archivebot
did not save the page for the second time (could it?) did not pass the problematic section and archive the remainder did not let the owner know about the problem.
It just silently failed, with logging being the only action.
So what would the desired behaviour be in a similar case? Please keep in mind that archivebot is tyoically run with cron or other timing, not in interactive mode.
To see the log click on https://tools.wmflabs.org/ato/log/archive.txt and ctrl f for
ERROR: editpage: abusefilter-warning
-- Bináris
pywikibot mailing list pywikibot@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot
pywikibot mailing list pywikibot@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot
I think that Pywikibot's save method should allow explicit overriding of a warning raised by abuse filter. It shouldn't be the default state, but it should be possible. Anyway, having a bot regulary hitting AF isn't a good idea - talk with an admin of your wiki.
On Czech Wikipedia, filters that would be hit by archivebot has exception, generally !"bot" in user_groups. This allows cs.wiki to warn users that may or may not know what is happening and allow bots to do their maintenance work without being warned because their operators are expected to know what is ok and what is not.
Cheers, Martin
ne 3. 6. 2018 v 13:49 odesílatel billinghurst billinghurstwiki@gmail.com napsal:
I believe that the abuse filter should be changed to allow the bot to edit.
The bot doesn't know whether this is a warning, and or disallow action, nor to know the consequences of hitting the filter for a second time, and expecting a bot to navigate a false positive on a wiki is not reasonable.
-- billinghurst
------ Original Message ------ From: "Yusuke Matsubara" whym@whym.org To: "Pywikibot discussion list" pywikibot@lists.wikimedia.org Sent: 3/06/2018 9:45:39 PM Subject: Re: [pywikibot] Archivebot fails on abuse filter
Hi,
I believe in this case we should somehow implement 'press Save again' in the bot. I don't know if the MediaWiki API allows this, but if not, I think it should be added (to the AbuseFilter extension?).
If the edit was 'blocked' by an abuse filter, things would be more complicated because we would definitely need an admin to look at the case (and thus need for notification).
A village pump in Hungarian Wikipedia had not been archived for a long time before we noticed it. It just silently failed, with logging being the only action.
I think how to notify the interested parties of archiving failures is a general issue, not just for failures related to abuse filters. I created a Phabricator task for that a while ago. [2]
Best, Yusuke
On Sun, Jun 3, 2018 at 4:07 PM, Bináris wikiposta@gmail.com wrote:
A village pump in Hungarian Wikipedia had not been archived for a long time before we noticed it. Investigation showed that the bot had hit a warning-type abuse filter upon copying the text to archive and saving. In this case the abuse filter displays a warning and lets the user press Save again if he/she wants to do it anyway. Of course, the archivebot
did not save the page for the second time (could it?) did not pass the problematic section and archive the remainder did not let the owner know about the problem.
It just silently failed, with logging being the only action.
So what would the desired behaviour be in a similar case? Please keep in mind that archivebot is tyoically run with cron or other timing, not in interactive mode.
To see the log click on https://tools.wmflabs.org/ato/log/archive.txt and ctrl f for
ERROR: editpage: abusefilter-warning
-- Bináris
pywikibot mailing list pywikibot@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot
pywikibot mailing list pywikibot@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot
pywikibot mailing list pywikibot@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot
IIRC, you can catch the error and try to save the page again. The page will be saved this time.
On Sun, Jun 3, 2018 at 9:12 PM Martin Urbanec martin.urbanec@wikimedia.cz wrote:
I think that Pywikibot's save method should allow explicit overriding of a warning raised by abuse filter. It shouldn't be the default state, but it should be possible.
Hi,
I believe in this case we should somehow implement 'press Save again' in the bot. I don't know if the MediaWiki API allows this, but if not, I think it should be added (to the AbuseFilter extension?).
2018-06-03 19:21 GMT+02:00 Dalba dalba.wiki@gmail.com:
IIRC, you can catch the error and try to save the page again. The page will be saved this time.
Generally this wouldn't be a good idea for all the bots (in framework), but actually for archivebot I don't see any risk in doing so, as it only copies existing texts from one page to another.
2018-06-03 18:41 GMT+02:00 Martin Urbanec martin.urbanec@wikimedia.cz:
I think that Pywikibot's save method should allow explicit overriding of a warning raised by abuse filter. It shouldn't be the default state, but it should be possible. Anyway, having a bot regulary hitting AF isn't a good idea - talk with an admin of your wiki.
No, this is not regular. But just allowing does not solve the original problem that the bot silently failed without notifying the owner. (The solution in this case was a manual archiving.)
On Czech Wikipedia, filters that would be hit by archivebot has exception, generally !"bot" in user_groups. This allows cs.wiki to warn users that may or may not know what is happening and allow bots to do their maintenance work without being warned because their operators are expected to know what is ok and what is not.
In the given case the filter itself caused the problem, and I wrote the filter. Many years ago I didn't think of limiting the filter to main namespace. I opened this thread because the problem is general. !"bot" in user_groups allows ALL bots to avoid the filter, and this is not right.
ne 3. 6. 2018 v 19:46 odesílatel Bináris wikiposta@gmail.com napsal:
2018-06-03 18:41 GMT+02:00 Martin Urbanec martin.urbanec@wikimedia.cz:
I think that Pywikibot's save method should allow explicit overriding of a warning raised by abuse filter. It shouldn't be the default state, but it should be possible. Anyway, having a bot regulary hitting AF isn't a good idea - talk with an admin of your wiki.
No, this is not regular. But just allowing does not solve the original problem that the bot silently failed without notifying the owner. (The solution in this case was a manual archiving.)
Well, failed sliently? Again, I run the same bot in cs.wiki and this is what I can see in logs.
Looking for: {{Šablona:Archivace}} in [[cs:Diskuse:Moravská vlajka]] Processing 1 threads Only 1 (< 2) threads are old enough. Skipping Processing [[cs:Diskuse:Česko]] 13 Threads found on [[cs:Diskuse:Česko]] Looking for: {{Šablona:Archivace}} in [[cs:Diskuse:Česko]] Processing 13 threads 13 Threads found on [[cs:Diskuse:Česko/archiv-02]] Only 3 (< 10) threads are old enough. Skipping Processing [[cs:Wikipedista:ArchivačníBot/test1]] 1 Threads found on [[cs:Wikipedista:ArchivačníBot/test1]] Looking for: {{Šablona:Archivace}} in [[cs:Wikipedista:ArchivačníBot/test1]] Processing 1 threads Only 0 (< 1) threads are old enough. Skipping Processing [[cs:Wikipedista:ArchivačníBot/test3]] 1 Threads found on [[cs:Wikipedista:ArchivačníBot/test3]] Looking for: {{Šablona:Archivace}} in [[cs:Wikipedista:ArchivačníBot/test3]] Processing 1 threads Processing [[cs:Diskuse s wikipedistou:Chmee2]] 4 Threads found on [[cs:Diskuse s wikipedistou:Chmee2]] Looking for: {{Šablona:Archivace}} in [[cs:Diskuse s wikipedistou:Chmee2]] Processing 4 threads Only 0 (< 1) threads are old enough. Skipping Processing [[cs:Diskuse s wikipedistou:HypoBOT]] 1 Threads found on [[cs:Diskuse s wikipedistou:HypoBOT]] Looking for: {{Šablona:Archivace}} in [[cs:Diskuse s wikipedistou:HypoBOT]] Processing 1 threads 2 Threads found on [[cs:Diskuse s wikipedistou:HypoBOT/archiv 2017-1]] Only 1 (< 2) threads are old enough. Skipping Processing [[cs:Diskuse s wikipedistou:JAn Dudík]] 21 Threads found on [[cs:Diskuse s wikipedistou:JAn Dudík]] Looking for: {{Šablona:Archivace}} in [[cs:Diskuse s wikipedistou:JAn Dudík]] Processing 21 threads 138 Threads found on [[cs:Diskuse s wikipedistou:JAn Dudík/archiv 2017-1]] Archiving 8 thread(s). ERROR: Error occurred while processing page [[cs:Diskuse s wikipedistou:JAn Dudík]] ERROR: SpamfilterError: Edit to page [[Diskuse s wikipedistou:JAn Dudík/archiv 2017-1]] rejected by spam filter due to content: twitter.com/search ERROR: SpamfilterError: Edit to page [[Diskuse s wikipedistou:JAn Dudík/archiv 2017-1]] rejected by spam filter due to content: twitter.com/search Traceback (most recent call last): File "/shared/pywikipedia/core/scripts/archivebot.py", line 819, in main archiver.run() File "/shared/pywikipedia/core/scripts/archivebot.py", line 713, in run self.archives[a].update(comment) File "/shared/pywikipedia/core/scripts/archivebot.py", line 531, in update self.save(summary) File "/shared/pywikipedia/core/pywikibot/tools/__init__.py", line 1520, in wrapper return obj(*__args, **__kw) File "/shared/pywikipedia/core/pywikibot/tools/__init__.py", line 1520, in wrapper return obj(*__args, **__kw) File "/shared/pywikipedia/core/pywikibot/page.py", line 1264, in save cc=apply_cosmetic_changes, quiet=quiet, **kwargs) File "/shared/pywikipedia/core/pywikibot/page.py", line 136, in wrapper handle(func, self, *args, **kwargs) File "/shared/pywikipedia/core/pywikibot/page.py", line 127, in handle raise err SpamfilterError: Edit to page [[Diskuse s wikipedistou:JAn Dudík/archiv 2017-1]] rejected by spam filter due to content: twitter.com/search
Framework report problems, bot operators should monitor logs. I know that this is spamfilter and not abusefilter, but Pywikibot behaves the same way even if an abusefilter has rejected given change. See the following manual example:
tools.urbanecmbot@tools-bastion-02 ~ $ python Python 2.7.6 (default, Nov 23 2017, 15:49:48) [GCC 4.8.4] on linux2 Type "help", "copyright", "credits" or "license" for more information.
import pywikibot site = pywikibot.Site('cs', 'wikipedia') page = pywikibot.Page(site, u"Wikipedista:Martin Urbanec/Pískoviště/1") page.text = "This is a test" page.save()
Sleeping for 9.5 seconds, 2018-06-03 22:19:23 ERROR: editpage: abusefilter-disallowed Hit AbuseFilter: Test filter, WARNING: Page [[Wikipedista:Martin Urbanec/Pískoviště/1]] not saved Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/shared/pywikipedia/core/pywikibot/tools/__init__.py", line 1446, in wrapper return obj(*__args, **__kw) File "/shared/pywikipedia/core/pywikibot/tools/__init__.py", line 1446, in wrapper return obj(*__args, **__kw) File "/shared/pywikipedia/core/pywikibot/page.py", line 1270, in save cc=apply_cosmetic_changes, quiet=quiet, **kwargs) File "/shared/pywikipedia/core/pywikibot/page.py", line 127, in wrapper handle(func, self, *args, **kwargs) File "/shared/pywikipedia/core/pywikibot/page.py", line 118, in handle raise err pywikibot.exceptions.PageSaveRelatedError: Page [[cs:Wikipedista:Martin Urbanec/Pískoviště/1]] was not saved.
Of course Pywikibot didn't report it by email, because it will be unexpectable (and, to be precise, Pywikibot do not always have a way to email you - for example, my Pywikibot bot password don't allow emailing (well, maybe it does, I'm not sure, but it isn't required to allow it).
On Czech Wikipedia, filters that would be hit by archivebot has exception, generally !"bot" in user_groups. This allows cs.wiki to warn users that may or may not know what is happening and allow bots to do their maintenance work without being warned because their operators are expected to know what is ok and what is not.
In the given case the filter itself caused the problem, and I wrote the filter. Many years ago I didn't think of limiting the filter to main namespace. I opened this thread because the problem is general. !"bot" in user_groups allows ALL bots to avoid the filter, and this is not right.
pywikibot mailing list pywikibot@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot
2018-06-03 22:23 GMT+02:00 Martin Urbanec martin.urbanec@wikimedia.cz:
Well, failed sliently? Again, I run the same bot in cs.wiki and this is what I can see in logs.
Yes, that's what I wrote in the initial letter with a link to the log. For me, this is quite silent. But your example shows something important: Dalba's solution is not general enough, either. It will work for abusefilter-warning, but not for SpamFilterError. Or for abuse filters that deny edit.
Of course Pywikibot didn't report it by email, because it will be unexpectable (and, to be precise, Pywikibot do not always have a way to email you - for example, my Pywikibot bot password don't allow emailing (well, maybe it does, I'm not sure, but it isn't required to allow it).
Pywikibot should have the option to send a mail to the owner, if he/she wants, and setting the e-mail option for the bot account is worth for this purpose. Of course, it wouldn't be obligatory. I don't think we could expect bot owners to monitor their logs all the time. Why should a human do the work of a computer?
ne 3. 6. 2018 v 22:35 odesílatel Bináris wikiposta@gmail.com napsal:
2018-06-03 22:23 GMT+02:00 Martin Urbanec martin.urbanec@wikimedia.cz:
Well, failed sliently? Again, I run the same bot in cs.wiki and this is what I can see in logs.
Yes, that's what I wrote in the initial letter with a link to the log. For me, this is quite silent.
If somebody writes "fail sliently", I expect "without any notice" automatically. There is a notice, but the notice isn't processed by the operator or their bots.
But your example shows something important: Dalba's solution is not general
enough, either. It will work for abusefilter-warning, but not for SpamFilterError. Or for abuse filters that deny edit.
You cannot save an edit that is being denied. This is what such filters are used to, to prevent some edits. Still, not a framework problem IMHO.
Of course Pywikibot didn't report it by email, because it will be unexpectable (and, to be precise, Pywikibot do not always have a way to email you - for example, my Pywikibot bot password don't allow emailing (well, maybe it does, I'm not sure, but it isn't required to allow it).
Pywikibot should have the option to send a mail to the owner, if he/she wants, and setting the e-mail option for the bot account is worth for this purpose. Of course, it wouldn't be obligatory.
Really? Imagine we have this feature and imagine a bot operator that says "Hey, you can notify the bot operator via email, let's implement SMS as well". Then, other bot operators can require a lot of different notification types. This will make Pywikibot unsustainable. Instead, let's follow a good Unix rule. One program should focus on only one thing. If I'm writing a framework that is interacting with MW API (purpose of Pywikibot), it should interact with MW API and not monitor logs. If I want to monitor logs, I should use another program (generally called log watcher) that focuses on watching/monitoring logs and reporting some patterns to the admin.
I store all logs from my Pywikibot scripts in ~/logs. If I want, I can create a maintenance job to monitor the logs. It will generally execute something like this. It would be relatively easy to modify this to monitor other bots (multiple maintenance jobs, or one running in multiple threads) and/or include the traceback directly in the mail.
tail -Fn0 /data/project/urbanecmbot/logs/archivePages.log | \ while read line ; do echo "$line" | grep -q "ERROR" if [ $? = 0 ] then echo "Fatal error in ArchiveBot, see /data/project/urbanecmbot/logs/archivePages.log for details" | /usr/bin/mail -s "Error notification" martin.urbanec@wikimedia.cz fi done
I don't think we could expect bot owners to monitor their logs all the time. Why should a human do the work of a computer?
I didn't say a human should do it. I said that the bot operator is expected to monitor the logs, either themselfes or using a bot. It is the same for system administrator. The services running in the system log various events into log service, but this is the only thing it does. The log service store the events in logfiles (default setting) or to dedicated log server. If the sysadmin want to know about some events by email/SMS/whatever, they should employ a service that is supposed to monitor for given patterns in given log files.
_______________________________________________
pywikibot mailing list pywikibot@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot
Pywikibot has the concept of a recoverable error. Say your internet connection drops or something else happens, the bot waits a bit and tries again. The same logic could be applied when a abusefilter warning is encountered and just throw an exception if it's not recoverable. We would have to build a bit of abusefilter support in the framework and figure out how to recover. Upload also has some logic with warnings, might be something useful in there to base the logic on.
Maarten
On 03-06-18 22:34, Bináris wrote:
2018-06-03 22:23 GMT+02:00 Martin Urbanec <martin.urbanec@wikimedia.cz mailto:martin.urbanec@wikimedia.cz>:
Well, failed sliently? Again, I run the same bot in cs.wiki and this is what I can see in logs.Yes, that's what I wrote in the initial letter with a link to the log. For me, this is quite silent. But your example shows something important: Dalba's solution is not general enough, either. It will work for abusefilter-warning, but not for SpamFilterError. Or for abuse filters that deny edit.
Of course Pywikibot didn't report it by email, because it will be unexpectable (and, to be precise, Pywikibot do not always have a way to email you - for example, my Pywikibot bot password don't allow emailing (well, maybe it does, I'm not sure, but it isn't required to allow it).Pywikibot should have the option to send a mail to the owner, if he/she wants, and setting the e-mail option for the bot account is worth for this purpose. Of course, it wouldn't be obligatory. I don't think we could expect bot owners to monitor their logs all the time. Why should a human do the work of a computer?
pywikibot mailing list pywikibot@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot
Is abusefilter recoverable? It can even block you without any API-side notice. Do we want to risk it or let bot operators decide on their own?
Best, Martin
po 4. 6. 2018 v 18:25 odesílatel Maarten Dammers maarten@mdammers.nl napsal:
Pywikibot has the concept of a recoverable error. Say your internet connection drops or something else happens, the bot waits a bit and tries again. The same logic could be applied when a abusefilter warning is encountered and just throw an exception if it's not recoverable. We would have to build a bit of abusefilter support in the framework and figure out how to recover. Upload also has some logic with warnings, might be something useful in there to base the logic on.
Maarten
On 03-06-18 22:34, Bináris wrote:
2018-06-03 22:23 GMT+02:00 Martin Urbanec martin.urbanec@wikimedia.cz:
Well, failed sliently? Again, I run the same bot in cs.wiki and this is what I can see in logs.
Yes, that's what I wrote in the initial letter with a link to the log. For me, this is quite silent. But your example shows something important: Dalba's solution is not general enough, either. It will work for abusefilter-warning, but not for SpamFilterError. Or for abuse filters that deny edit.
Of course Pywikibot didn't report it by email, because it will be unexpectable (and, to be precise, Pywikibot do not always have a way to email you - for example, my Pywikibot bot password don't allow emailing (well, maybe it does, I'm not sure, but it isn't required to allow it).
Pywikibot should have the option to send a mail to the owner, if he/she wants, and setting the e-mail option for the bot account is worth for this purpose. Of course, it wouldn't be obligatory. I don't think we could expect bot owners to monitor their logs all the time. Why should a human do the work of a computer?
pywikibot mailing listpywikibot@lists.wikimedia.orghttps://lists.wikimedia.org/mailman/listinfo/pywikibot
pywikibot mailing list pywikibot@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot
Honestly I would leave it as is. The filters that bots should exempt from should be updated on wiki to ignore bots
On Tue, Jun 5, 2018 at 1:09 AM Martin Urbanec martin.urbanec@wikimedia.cz wrote:
Is abusefilter recoverable? It can even block you without any API-side notice. Do we want to risk it or let bot operators decide on their own?
Best, Martin
po 4. 6. 2018 v 18:25 odesílatel Maarten Dammers maarten@mdammers.nl napsal:
Pywikibot has the concept of a recoverable error. Say your internet connection drops or something else happens, the bot waits a bit and tries again. The same logic could be applied when a abusefilter warning is encountered and just throw an exception if it's not recoverable. We would have to build a bit of abusefilter support in the framework and figure out how to recover. Upload also has some logic with warnings, might be something useful in there to base the logic on.
Maarten
On 03-06-18 22:34, Bináris wrote:
2018-06-03 22:23 GMT+02:00 Martin Urbanec martin.urbanec@wikimedia.cz:
Well, failed sliently? Again, I run the same bot in cs.wiki and this is what I can see in logs.
Yes, that's what I wrote in the initial letter with a link to the log. For me, this is quite silent. But your example shows something important: Dalba's solution is not general enough, either. It will work for abusefilter-warning, but not for SpamFilterError. Or for abuse filters that deny edit.
Of course Pywikibot didn't report it by email, because it will be unexpectable (and, to be precise, Pywikibot do not always have a way to email you - for example, my Pywikibot bot password don't allow emailing (well, maybe it does, I'm not sure, but it isn't required to allow it).
Pywikibot should have the option to send a mail to the owner, if he/she wants, and setting the e-mail option for the bot account is worth for this purpose. Of course, it wouldn't be obligatory. I don't think we could expect bot owners to monitor their logs all the time. Why should a human do the work of a computer?
pywikibot mailing listpywikibot@lists.wikimedia.orghttps://lists.wikimedia.org/mailman/listinfo/pywikibot
pywikibot mailing list pywikibot@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot
pywikibot mailing list pywikibot@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot
There are no global filters that will block you, and that is purposeful decision and steward-imposed condition, though there are local filters all over the place that will do so, including meta.
Now the filters have been targetted at spambots, and are well-tuned, though not perfect, so you will need to be on good terms with admins if you are going to force an edit that may be spammy. I would still recommend that you talk to admins about writing abuse filters to allow bots to be excluded fro the impact (pretty easy tweak)
Note that bot edits won't get around title and spam blacklists, which I saw mentioned for twitter.com/search
-- billinghurst
------ Original Message ------ From: "Martin Urbanec" martin.urbanec@wikimedia.cz To: "Pywikibot discussion list" pywikibot@lists.wikimedia.org Sent: 5/06/2018 3:08:48 PM Subject: Re: [pywikibot] Archivebot fails on abuse filter
Is abusefilter recoverable? It can even block you without any API-side notice. Do we want to risk it or let bot operators decide on their own?
Best, Martin
po 4. 6. 2018 v 18:25 odesílatel Maarten Dammers maarten@mdammers.nl napsal:
Pywikibot has the concept of a recoverable error. Say your internet connection drops or something else happens, the bot waits a bit and tries again. The same logic could be applied when a abusefilter warning is encountered and just throw an exception if it's not recoverable. We would have to build a bit of abusefilter support in the framework and figure out how to recover. Upload also has some logic with warnings, might be something useful in there to base the logic on.
Maarten
On 03-06-18 22:34, Bináris wrote:
2018-06-03 22:23 GMT+02:00 Martin Urbanec martin.urbanec@wikimedia.cz:
Well, failed sliently? Again, I run the same bot in cs.wiki and this is what I can see in logs.
Yes, that's what I wrote in the initial letter with a link to the log. For me, this is quite silent. But your example shows something important: Dalba's solution is not general enough, either. It will work for abusefilter-warning, but not for SpamFilterError. Or for abuse filters that deny edit.
Of course Pywikibot didn't report it by email, because it will be unexpectable (and, to be precise, Pywikibot do not always have a way to email you - for example, my Pywikibot bot password don't allow emailing (well, maybe it does, I'm not sure, but it isn't required to allow it).
Pywikibot should have the option to send a mail to the owner, if he/she wants, and setting the e-mail option for the bot account is worth for this purpose. Of course, it wouldn't be obligatory. I don't think we could expect bot owners to monitor their logs all the time. Why should a human do the work of a computer?
pywikibot mailing list pywikibot@lists.wikimedia.orghttps://lists.wikimedia.org/mailman/listinfo/pywikibot
pywikibot mailing list pywikibot@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot
I think how to notify the interested parties of archiving failures is a general issue, not just for failures related to abuse filters. I created a Phabricator task for that a while ago. [2]
Sorry, I forgot the link: [2] https://phabricator.wikimedia.org/T124407 (and there was no [1].)
Best, Yusuke
On Sun, Jun 3, 2018 at 8:45 PM, Yusuke Matsubara whym@whym.org wrote:
Hi,
I believe in this case we should somehow implement 'press Save again' in the bot. I don't know if the MediaWiki API allows this, but if not, I think it should be added (to the AbuseFilter extension?).
If the edit was 'blocked' by an abuse filter, things would be more complicated because we would definitely need an admin to look at the case (and thus need for notification).
A village pump in Hungarian Wikipedia had not been archived for a long time before we noticed it. It just silently failed, with logging being the only action.
I think how to notify the interested parties of archiving failures is a general issue, not just for failures related to abuse filters. I created a Phabricator task for that a while ago. [2]
Best, Yusuke
On Sun, Jun 3, 2018 at 4:07 PM, Bináris wikiposta@gmail.com wrote:
A village pump in Hungarian Wikipedia had not been archived for a long time before we noticed it. Investigation showed that the bot had hit a warning-type abuse filter upon copying the text to archive and saving. In this case the abuse filter displays a warning and lets the user press Save again if he/she wants to do it anyway. Of course, the archivebot
did not save the page for the second time (could it?) did not pass the problematic section and archive the remainder did not let the owner know about the problem.
It just silently failed, with logging being the only action.
So what would the desired behaviour be in a similar case? Please keep in mind that archivebot is tyoically run with cron or other timing, not in interactive mode.
To see the log click on https://tools.wmflabs.org/ato/log/archive.txt and ctrl f for
ERROR: editpage: abusefilter-warning
-- Bináris
pywikibot mailing list pywikibot@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot
Thanks for your proposal but the easiest way for this case is to modify the edit filter to enable such edits. Check bot group, username and edit summary to allow the change.
Best xqt
Von meinem iPhone gesendet
Am 04.06.2018 um 02:06 schrieb Yusuke Matsubara whym@whym.org:
I think how to notify the interested parties of archiving failures is a general issue, not just for failures related to abuse filters. I created a Phabricator task for that a while ago. [2]
Sorry, I forgot the link: [2] https://phabricator.wikimedia.org/T124407 (and there was no [1].)
Best, Yusuke
On Sun, Jun 3, 2018 at 8:45 PM, Yusuke Matsubara whym@whym.org wrote: Hi,
I believe in this case we should somehow implement 'press Save again' in the bot. I don't know if the MediaWiki API allows this, but if not, I think it should be added (to the AbuseFilter extension?).
If the edit was 'blocked' by an abuse filter, things would be more complicated because we would definitely need an admin to look at the case (and thus need for notification).
A village pump in Hungarian Wikipedia had not been archived for a long time before we noticed it. It just silently failed, with logging being the only action.
I think how to notify the interested parties of archiving failures is a general issue, not just for failures related to abuse filters. I created a Phabricator task for that a while ago. [2]
Best, Yusuke
On Sun, Jun 3, 2018 at 4:07 PM, Bináris wikiposta@gmail.com wrote: A village pump in Hungarian Wikipedia had not been archived for a long time before we noticed it. Investigation showed that the bot had hit a warning-type abuse filter upon copying the text to archive and saving. In this case the abuse filter displays a warning and lets the user press Save again if he/she wants to do it anyway. Of course, the archivebot
did not save the page for the second time (could it?) did not pass the problematic section and archive the remainder did not let the owner know about the problem.
It just silently failed, with logging being the only action.
So what would the desired behaviour be in a similar case? Please keep in mind that archivebot is tyoically run with cron or other timing, not in interactive mode.
To see the log click on https://tools.wmflabs.org/ato/log/archive.txt and ctrl f for
ERROR: editpage: abusefilter-warning
-- Bináris
pywikibot mailing list pywikibot@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot
pywikibot mailing list pywikibot@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot