jayvdb created this task. jayvdb added subscribers: Aklapper, pywikipedia-bugs, jayvdb. jayvdb added a project: pywikibot-core.
TASK DESCRIPTION LiveRCPageGenerator stall the travis builds. The only common part of the log is
``` No handlers could be found for logger "socketIO_client" ```
That message doesnt appear on successful runs of this test. My guess is that "socketIO_client" wants to report a problem (possible socket timeout?), but cant due to a bug in their library or in our use of their library.
ar.wp - 2.6 (only failed build for that commit) https://travis-ci.org/wikimedia/pywikibot-core/jobs/45687549
ar.wp - 2.7 & 2.6 https://travis-ci.org/wikimedia/pywikibot-core/jobs/45743553 https://travis-ci.org/wikimedia/pywikibot-core/jobs/45743561
It could be triggered by a low amount of data in the rc stream.
TASK DETAIL https://phabricator.wikimedia.org/T85717
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jayvdb Cc: Aklapper, jayvdb, pywikipedia-bugs
jayvdb added a project: Pywikibot-tests.
TASK DETAIL https://phabricator.wikimedia.org/T85717
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jayvdb Cc: Aklapper, jayvdb, pywikipedia-bugs
jayvdb added a comment.
two more, always ar.wp, and never py3 https://travis-ci.org/wikimedia/pywikibot-core/builds/45747943 https://travis-ci.org/wikimedia/pywikibot-core/builds/45763062
TASK DETAIL https://phabricator.wikimedia.org/T85717
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jayvdb Cc: Aklapper, jayvdb, pywikipedia-bugs
jayvdb added a comment.
Again ar.wp
https://travis-ci.org/wikimedia/pywikibot-core/jobs/46326040
TASK DETAIL https://phabricator.wikimedia.org/T85717
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jayvdb Cc: Aklapper, jayvdb, pywikipedia-bugs
jayvdb added a comment.
And another, also ar.wp
https://travis-ci.org/wikimedia/pywikibot-core/jobs/46238478
TASK DETAIL https://phabricator.wikimedia.org/T85717
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jayvdb Cc: Aklapper, jayvdb, pywikipedia-bugs
jayvdb added a comment.
And another https://travis-ci.org/jayvdb/pywikibot-core/jobs/46413002
TASK DETAIL https://phabricator.wikimedia.org/T85717
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jayvdb Cc: Aklapper, jayvdb, pywikipedia-bugs
jayvdb added a comment.
And another: https://travis-ci.org/wikimedia/pywikibot-core/jobs/46431515
TASK DETAIL https://phabricator.wikimedia.org/T85717
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jayvdb Cc: Aklapper, jayvdb, pywikipedia-bugs
jayvdb added a subscriber: valhallasw.
TASK DETAIL https://phabricator.wikimedia.org/T85717
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jayvdb Cc: Aklapper, jayvdb, valhallasw, pywikipedia-bugs
jayvdb changed the title from "LiveRCPageGenerator stall the builds regularly" to "LiveRCPageGenerator using socketIO_client stall the builds regularly". jayvdb set Security to none.
TASK DETAIL https://phabricator.wikimedia.org/T85717
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jayvdb Cc: Aklapper, jayvdb, valhallasw, pywikipedia-bugs
jayvdb added a comment.
more of the same https://travis-ci.org/wikimedia/pywikibot-core/jobs/46549115 https://travis-ci.org/wikimedia/pywikibot-core/jobs/46581107 https://travis-ci.org/wikimedia/pywikibot-core/jobs/46652170 https://travis-ci.org/wikimedia/pywikibot-core/jobs/46673227
TASK DETAIL https://phabricator.wikimedia.org/T85717
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jayvdb Cc: Aklapper, jayvdb, valhallasw, pywikipedia-bugs
jayvdb added a comment.
After installing a logger "socketIO_client" on , we see the error is:
WARNING:socketIO_client:[connection error] connection closed ()
TASK DETAIL https://phabricator.wikimedia.org/T85717
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jayvdb Cc: Aklapper, jayvdb, valhallasw, pywikipedia-bugs
gerritbot added a project: Patch-For-Review. gerritbot added a comment.
Change 184285 had a related patch set uploaded (by John Vandenberg): Prevent hang in LiveRCPageGenerator
https://gerrit.wikimedia.org/r/184285
https://phabricator.wikimedia.org/tag/patch-for-review/
TASK DETAIL https://phabricator.wikimedia.org/T85717
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: gerritbot Cc: Aklapper, jayvdb, valhallasw, pywikipedia-bugs
gerritbot added a comment.
Change 184285 merged by jenkins-bot: Prevent hang in LiveRCPageGenerator
https://gerrit.wikimedia.org/r/184285
TASK DETAIL https://phabricator.wikimedia.org/T85717
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: gerritbot Cc: Aklapper, jayvdb, valhallasw, pywikipedia-bugs
valhallasw added a comment.
There seem to be two issues here:
- SocketIO_Client's handling of heartbeats (as discussed a few days ago on IRC). https://github.com/invisibleroads/socketIO-client/blob/master/socketIO_clien... handles the case where heartbeat_interval >> .wait() interval, but not the other way around. - One possible way to solve this is by adding an on_hearbeat handler, replying to the servers' heartbeat instead of sending our own. - on reconnect, on_connect is not called. This could be related to T86771: New RC feed is randomly failing: socket.io not re-subscribed? https://phabricator.wikimedia.org/T86771, but in our case, the pypi version is used, which means https://github.com/invisibleroads/socketIO-client/commit/ee425459bd7d98f2df4... is not actually part of the code base yet :/
TASK DETAIL https://phabricator.wikimedia.org/T85717
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: valhallasw Cc: jayvdb, valhallasw, Aklapper, pywikipedia-bugs
gerritbot added a subscriber: gerritbot. gerritbot added a comment.
Change 185200 had a related patch set uploaded (by Merlijn van Deen): RCStream: return heartbeats and handle on_reconnect
https://gerrit.wikimedia.org/r/185200
https://phabricator.wikimedia.org/tag/patch-for-review/
TASK DETAIL https://phabricator.wikimedia.org/T85717
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: gerritbot Cc: gerritbot, jayvdb, valhallasw, Aklapper, pywikipedia-bugs
valhallasw added a comment.
the reason on_connect is not called is the following. Compare
DEBUG:socketIO_client:[transports available] websocket xhr-multipart htmlfile jsonp-polling flashsocket xhr-polling DEBUG:socketIO_client.transports:[transport selected] websocket DEBUG:socketIO_client.transports:[packet sent] 1::/rc: DEBUG:socketIO_client.transports:[packet received] 1:: DEBUG:socketIO_client: [connect] DEBUG:socketIO_client.transports:[packet received] 1::/rc
to
DEBUG:socketIO_client:[transports available] websocket xhr-multipart htmlfile jsonp-polling flashsocket xhr-polling DEBUG:socketIO_client.transports:[packet received] 1:: DEBUG:socketIO_client: [connect] (... nothing)
This means socketio_client doesn't actually register in the /rc namespace, and thus our code is not called.
TASK DETAIL https://phabricator.wikimedia.org/T85717
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: valhallasw Cc: gerritbot, jayvdb, valhallasw, Aklapper, pywikipedia-bugs
gerritbot added a comment.
Change 185200 merged by jenkins-bot: RCStream: return heartbeats and handle on_reconnect
https://gerrit.wikimedia.org/r/185200
TASK DETAIL https://phabricator.wikimedia.org/T85717
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: gerritbot Cc: gerritbot, jayvdb, valhallasw, Aklapper, pywikipedia-bugs
jayvdb closed this task as "Resolved". jayvdb claimed this task. jayvdb added a comment.
This appears to be fixed.
TASK DETAIL https://phabricator.wikimedia.org/T85717
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jayvdb Cc: gerritbot, jayvdb, valhallasw, Aklapper, pywikipedia-bugs
XZise added a subscriber: XZise. XZise added a comment.
Okay since this https://travis-ci.org/wikimedia/pywikibot-core/jobs/52048570 test (which was 7 days ago) all Python 3.3 tests which used setup.py stalled (the nosetests ran fine). It seems that all tests errored the same way. Maybe this is unrelated to this bug, but I'm not familiar with socketIO so I can't really determine that. A new one should be opened then.
Interestingly since then Travis is installing requests-2.5.3 and socketIO-client is using that library so they could be connected.
TASK DETAIL https://phabricator.wikimedia.org/T85717
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jayvdb, XZise Cc: XZise, gerritbot, jayvdb, valhallasw, Aklapper, pywikipedia-bugs
Jdforrester-WMF removed a project: Patch-For-Review.
TASK DETAIL https://phabricator.wikimedia.org/T85717
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jayvdb, Jdforrester-WMF Cc: XZise, gerritbot, jayvdb, valhallasw, Aklapper, pywikipedia-bugs
XZise added a comment.
Regarding my comment 9 days ago: That is fixed and was caused because of https://phabricator.wikimedia.org/T91393#1104004.
TASK DETAIL https://phabricator.wikimedia.org/T85717
REPLY HANDLER ACTIONS Reply to comment or attach files, or !close, !claim, !unsubscribe or !assign <username>.
EMAIL PREFERENCES https://phabricator.wikimedia.org/settings/panel/emailpreferences/
To: jayvdb, XZise Cc: XZise, gerritbot, jayvdb, valhallasw, Aklapper, pywikipedia-bugs
pywikipedia-bugs@lists.wikimedia.org