The test suite is falling into disrepair.
It would be good if +2'ers can suggest changes in the last few months which may have caused these breakages.
We need to investigate these quickly, as other bugs may be lurking behind them, and they prevent the test suite being useful for determining if new proposed changes will create additional regressions.
Running the test suite on my linux workstation shows the following errors:
https://phabricator.wikimedia.org/T128987 testQueriesDirectFromClaim fails with TypeError: Cannot construct a query from a claim of type external-id
https://phabricator.wikimedia.org/T128988 TestWiktionaryGraph.test_octagon failing
https://phabricator.wikimedia.org/T128989 TestLogeventsFactoryGenerator multi event tests failing: 500 not less than or equal to 10
https://phabricator.wikimedia.org/T128990 WikiStats sorting algorithm causing unexpected largest wikisource
https://phabricator.wikimedia.org/T128991 test_pagegen_i18n_input failing: setlocale: LC_ALL: cannot change locale (arz): No such file or directory
https://phabricator.wikimedia.org/T128955 script test for featured.py fails
https://phabricator.wikimedia.org/T128994 Some proofread page tests are writing to the wiki without being declared as write tests
There are other errors occurring on travis builds, which also need investigating, such as https://phabricator.wikimedia.org/T128318 No handlers could be found for logger "pywiki"
Also Xqt has encountered a bug in the test suite for MS Windows https://phabricator.wikimedia.org/T128453 script_tests fails for editarticle in windows environment
Most, but not all, have been fixed. I made a note at https://phabricator.wikimedia.org/T128988#2104901 about the builds which still failed on Travis.
------ André Costa GLAM Developer Wikimedia Sverige On 6 Mar 2016 02:39, "John Mark Vandenberg" jayvdb@gmail.com wrote:
The test suite is falling into disrepair.
It would be good if +2'ers can suggest changes in the last few months which may have caused these breakages.
We need to investigate these quickly, as other bugs may be lurking behind them, and they prevent the test suite being useful for determining if new proposed changes will create additional regressions.
Running the test suite on my linux workstation shows the following errors:
https://phabricator.wikimedia.org/T128987 testQueriesDirectFromClaim fails with TypeError: Cannot construct a query from a claim of type external-id
https://phabricator.wikimedia.org/T128988 TestWiktionaryGraph.test_octagon failing
https://phabricator.wikimedia.org/T128989 TestLogeventsFactoryGenerator multi event tests failing: 500 not less than or equal to 10
https://phabricator.wikimedia.org/T128990 WikiStats sorting algorithm causing unexpected largest wikisource
https://phabricator.wikimedia.org/T128991 test_pagegen_i18n_input failing: setlocale: LC_ALL: cannot change locale (arz): No such file or directory
https://phabricator.wikimedia.org/T128955 script test for featured.py fails
https://phabricator.wikimedia.org/T128994 Some proofread page tests are writing to the wiki without being declared as write tests
There are other errors occurring on travis builds, which also need investigating, such as https://phabricator.wikimedia.org/T128318 No handlers could be found for logger "pywiki"
Also Xqt has encountered a bug in the test suite for MS Windows https://phabricator.wikimedia.org/T128453 script_tests fails for editarticle in windows environment
-- John Vandenberg
pywikibot mailing list pywikibot@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot
With the risk of this being a question with an obvious answer:
Why didn't the two nose tests which are run after a +2 pick up on the errors mentioned in the first mail of this thread? They shouldn't all have been write tests.
And if it is because they are all write tests would there be some way of including these (or a subset of them) into the pre-merge tests?
/André ------ André Costa GLAM Developer Wikimedia Sverige On 10 Mar 2016 21:31, "André Costa" andre.costa@wikimedia.se wrote:
Most, but not all, have been fixed. I made a note at https://phabricator.wikimedia.org/T128988#2104901 about the builds which still failed on Travis.
André Costa GLAM Developer Wikimedia Sverige On 6 Mar 2016 02:39, "John Mark Vandenberg" jayvdb@gmail.com wrote:
The test suite is falling into disrepair.
It would be good if +2'ers can suggest changes in the last few months which may have caused these breakages.
We need to investigate these quickly, as other bugs may be lurking behind them, and they prevent the test suite being useful for determining if new proposed changes will create additional regressions.
Running the test suite on my linux workstation shows the following errors:
https://phabricator.wikimedia.org/T128987 testQueriesDirectFromClaim fails with TypeError: Cannot construct a query from a claim of type external-id
https://phabricator.wikimedia.org/T128988 TestWiktionaryGraph.test_octagon failing
https://phabricator.wikimedia.org/T128989 TestLogeventsFactoryGenerator multi event tests failing: 500 not less than or equal to 10
https://phabricator.wikimedia.org/T128990 WikiStats sorting algorithm causing unexpected largest wikisource
https://phabricator.wikimedia.org/T128991 test_pagegen_i18n_input failing: setlocale: LC_ALL: cannot change locale (arz): No such file or directory
https://phabricator.wikimedia.org/T128955 script test for featured.py fails
https://phabricator.wikimedia.org/T128994 Some proofread page tests are writing to the wiki without being declared as write tests
There are other errors occurring on travis builds, which also need investigating, such as https://phabricator.wikimedia.org/T128318 No handlers could be found for logger "pywiki"
Also Xqt has encountered a bug in the test suite for MS Windows https://phabricator.wikimedia.org/T128453 script_tests fails for editarticle in windows environment
-- John Vandenberg
pywikibot mailing list pywikibot@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot
On Fri, Mar 11, 2016 at 4:04 PM, André Costa andre.costa@wikimedia.se wrote:
With the risk of this being a question with an obvious answer:
Why didn't the two nose tests which are run after a +2 pick up on the errors mentioned in the first mail of this thread?
Wikimedia Foundation CI team has required that our Jenkins test suite does not *access* production wikis at all. So only dry tests are run by Jenkins. The full test suite is only run by the github hook invoking a travis build.
Would it then be possible to identify the tests which work on the test wikis (or could be made to work on them) and have them run on +2?
Even though some tests would still have to be run via the github hook more of them would happen pre-merge.
------ André Costa GLAM Developer Wikimedia Sverige On 11 Mar 2016 12:07, "John Mark Vandenberg" jayvdb@gmail.com wrote:
On Fri, Mar 11, 2016 at 4:04 PM, André Costa andre.costa@wikimedia.se wrote:
With the risk of this being a question with an obvious answer:
Why didn't the two nose tests which are run after a +2 pick up on the
errors
mentioned in the first mail of this thread?
Wikimedia Foundation CI team has required that our Jenkins test suite does not *access* production wikis at all. So only dry tests are run by Jenkins. The full test suite is only run by the github hook invoking a travis build.
-- John Vandenberg
pywikibot mailing list pywikibot@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot
On Sat, Mar 12, 2016 at 2:15 AM, André Costa lokal.profil@gmail.com wrote:
Would it then be possible to identify the tests which work on the test wikis (or could be made to work on them) and have them run on +2?
Yes, that is relatively simple to do with our test runner, and it would be a very good improvement on the current situation.
If you (or someone) could describe what would need to be done I'm happy to help.
Maybe time for this to be a phabricator task though.
/André
------ André Costa GLAM Developer Wikimedia Sverige On 11 Mar 2016 20:37, "John Mark Vandenberg" jayvdb@gmail.com wrote:
On Sat, Mar 12, 2016 at 2:15 AM, André Costa lokal.profil@gmail.com wrote:
Would it then be possible to identify the tests which work on the test
wikis
(or could be made to work on them) and have them run on +2?
Yes, that is relatively simple to do with our test runner, and it would be a very good improvement on the current situation.
-- John Vandenberg
pywikibot mailing list pywikibot@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/pywikibot
Thanks to everyone helping fix the first set of tasks.
There are a few more which are now more visible.
https://phabricator.wikimedia.org/T129763 "beta wikipedia test builds fail"
https://phabricator.wikimedia.org/T129281 "Test test_content_format failing due to new member 'application/vnd.php.serialized' on Wikibase repos" (guess: a Wikibase php change has unintentionally exposed this content format)
https://phabricator.wikimedia.org/T89769 "testQueryApiGetter timeout failure" (Wiki Data Query has become slower?)
https://phabricator.wikimedia.org/T128986 https://phabricator.wikimedia.org/T128994 ProofreadPage is doing a purge without write permission