tl;dr below, frustration ahead.
I've been trying to get Jenkins to run the parsertests of
LabeledSectionTransclusion, but this has been an incredibly frustrating
experience. It shouldn't be too hard, right? Well, read on...
I thought I'd start with the existing 'testextensions' jenkins job [1],
which is failing. The reason it is failing is because of this:
*12:02:56* + php tests/phpunit/phpunit.php --log-junit
junit-phpunit-ext.xml --
extensions/LabeledSectionTransclusion*12:02:56* Unexpected
non-MediaWiki exception encountered, of type
"PHPUnit_Framework_Exception"*12:02:56* exception
'PHPUnit_Framework_Exception' with message 'Neither
"extensions/LabeledSectionTransclusion.php" nor
"extensions/LabeledSectionTransclusion.php" could be opened.' in
/usr/share/php/PHPUnit/Util/Skeleton/Test.php:100
After installing PHPUnit (on itself quite a frustrating experience), I
get this response instead:
valhallasw@lisilwen:~/src/core$ php tests/phpunit/phpunit.php
extensions/LabeledSectionTransclusion
PHPUnit 3.7.10 by Sebastian Bergmann.
Configuration read from /home/valhallasw/src/core/tests/phpunit/suite.xml
Time: 0 seconds, Memory: 18.50Mb
No tests executed!
Which suggests it might be a problem with the Jenkins configuration.
Of course, the tests are also not running, but this is a second issue.
So, where is the Jenkins configuration? Not in Jenkins itself. After
logging in in Jenkins, there /is/ suddenly a new option 'Workspace'
(why is that disabled for logged-out users?), which shows the
extension is checked out in the workspace - so that doesn't explain
why phpunit is unable to locate the tests.
So let's check the configuration. [2] suggests it should be at [3].
Ugh, gitweb, and no github mirror. Under 'jobs', there are some
extensions, but not LST. There is MediaWiki-Tests-Extensions, which
has some XML files that, at least for me, are incomprehensible. And
are they even relevant to this? In any case, they don't tell me why
PHPUnit is unable to find the LST directory, even though it's right
there [4]. I give up.
By the way - I just realized why the job is not in the Jenkins repository:
valhallasw@lisilwen:~/src/jenkins$ grep -R * -e 'mwext'
jobs/.gitignore:/mwext-*
*Really?*
Okay, so then how do I get PHPUnit to run the parsertests? There is
the NewParserTest class, so I was guessing adding this to
LabeledSectionTransclusion.php would work:
class LSTParserTest extends NewParserTest {
function __construct()
{
parent::__construct();
$this->setParserTestFile(__DIR__ . "/lstParserTests.txt");
}
};
Well, that does, er, absolutely nothing. [5] suggests I should move it
to a seperate file and add the following:
$wgHooks['UnitTestsList'][] = 'efFruitsRegisterUnitTests';function
efFruitsRegisterUnitTests( &$files ) {
$testDir = dirname( __FILE__ ) . '/';
$files[] = $testDir . 'AppleTest.php';
return true;}
what it *doesn't* tell you is that this is actually irrelevant when
you run php tests/phpunit/phpunit.php
extensions/LabeledSectionTransclusion - and it's not really clear to
me for when it is relevant. After some struggling, apparently it
detects tests because the filename *ends in Test.php* (which is
documented, well, no-where. After moving the test case there, it still
didn't quite work, but after some more fiddling, something is working!
Okay, so now I have a test case that actually runs when I call the
above command. So, let's push it for review [6] - but that doesn't
actually make Jenkins run the test cases.... so I still don't know
whether this actually fixes it.
[1]
https://integration.mediawiki.org/ci/job/mwext-LabeledSectionTransclusion-t…
[2] http://www.mediawiki.org/wiki/Continuous_integration/Jenkins
[3] https://gerrit.wikimedia.org/r/gitweb?p=integration/jenkins.git;a=tree
[4]
https://integration.mediawiki.org/ci/job/mwext-LabeledSectionTransclusion-t…
[5]
http://www.mediawiki.org/wiki/Manual:PHP_unit_testing/Writing_unit_tests_fo…
So, tl,dr: debugging jenkins failures and getting phpunit to work is a
frustrating experience, because of a lack of documentation and a lack of
'obviousness' (there is not one 'obvious' way to do it). I've updated [5]
with my experiences, but I still have no idea how the magic that is Jenkins
is configured and how it works. It would be great if there would be better
documentation on that.
Some specific things that I think should be fixed:
- Jenkins should not show new (read-only) options to users - make them
public instead
- The Jenkins configuration should also be mirrored on github
- The mwext jobs should be in git (preferrably, because then it's easy to
check them on-line), *or* there should be clear documentation on how to
generate those jobs...
- There should be documentation on how the configuration is actually used
- and preferrably on how to debug failures.
Thanks.
Merlijn
Jenkins has been failing all commits for certain extensions today. I
assume it is somehow related to the accidental addition of submodules
to core master branch. I've been told that some of the tests that
Jenkins runs have been disabled.
I'm assuming someone else will post a followup to explain all the
"somes" above and make sure that all tests are re-enabled and passing
for all affected extensions.
-Niklas
--
Niklas Laxström
I’m glad to announce that Kim Schoonover, Mariya Miteva, Priyanka Nag,
Sucheta Ghoshal, Teresa Cho and Valerie Juarez will join the MediaWiki
community as full-time interns between January and March 2013. They have
been selected as part of the FLOSS Outreach Program for Women.
Check the details at
http://blog.wikimedia.org/2012/12/11/welcome-to-floss-outreach-program-for-…
We wish a happy landing to our new interns and the best luck in their
projects! You’ll be hearing more from them over the next few months.
--
Quim Gil
Technical Contributor Coordinator @ Wikimedia Foundation
http://www.mediawiki.org/wiki/User:Qgil
Wikimedia wikis hosted on the "s3" cluster (pretty much all but the
very large wikis, click on the "s3" box in
https://noc.wikimedia.org/dbtree/ to get a full list) are currently in
read-only mode due to severe replication lag. This problem was
apparently caused by a logic issue in the job queue, which should now
be fixed, but it will still take at least 2-3 hours for replication to
catch up. We apologize for the inconvenience, and are continuing to
monitor the situation.
Erik
--
Erik Möller
VP of Engineering and Product Development, Wikimedia Foundation
Support Free Knowledge: https://wikimediafoundation.org/wiki/Donate
If anyone is interested, or knows of a worthy entrant, the application
deadline for the 2013 Antonia Pizzigati prize, honoring open source
software development in the public interest, has been extended to Friday,
14 December 2012.
http://www.tides.org/impact/awards-prizes/pizzigati-prize/
The Antonio Pizzigati Prize for Software in the Public Interest annually
awards a $10,000 cash grant to one individual who has created or led an
effort to create an open source software product of significant value to
the nonprofit sector and movements for social change.
The Pizzigati Prize honors the brief life of Tony Pizzigati (
http://www.tides.org/impact/awards-prizes/pizzigati-prize/tony/ - he does
not have a Wikipedia entry), an early advocate of open source computing.
-Asher
Hi,
I'm running a robot that makes relatively small changes in ~250KB
pages. Starting from November, the change is made, but I recieve a 504
error. This does not happen when making the change manually and it did
not happen earlier in the year (say, September). The code is a custom,
pywikipediabot-based robot on ro.wp. One example of such change is
http://ro.wikipedia.org/w/index.php?title=Lista_monumentelor_istorice_din_j…
Has something changed recently in the server configuration that would
cause such errors to occur? Is there somethnig I can do to avoid them,
except splitting the pages?
Thanks,
Strainu
Summary: architecture, design, hosting, and APIs for the WMF automated
browser test project are in place, useful testing is happening now, and we
are beginning to build out the test suite for both desktop and mobile
browsers.
About a year ago the WMF created a mandate to implement browser-level
automated testing. I did the preliminary work, and in October 2012 Željko
Filipin joined WMF to be the technical lead for this project. This is an
excellent time to describe the current status of the project and to discuss
where it goes next.
We are using the Ruby programming language for this project because the
tools available for this sort of testing are more sophisticated and better
supported in Ruby than in other languages. While the entire testing
"stack" is remarkable, three aspects are particularly worth noting:
A project called webdriver is in the process of being approved by the W3C
as an internet standard for browser automation. Selenium is the
client-side implementation of the webdriver standard. The Ruby
implementation of selenium-webdriver is built and tested automatically
according to the webdriver standard. As the webdriver standard changes, so
selenium-webdriver in Ruby changes also, automatically. [1]
But selenium is a low-level API for driving browsers, essentially a limited
set of tools to build with. Ruby is unique in also having a standard
high-level API. Today the watir-webdriver project provides a powerful,
well-designed, well-supported high-level API for selenium-webdriver based
on the HTML5 standard.[2]
Cucumber is a high-level API for doing Acceptance Test Driven Development,
or ATDD. Cucumber allows us to create executable specifications in the
form of Given/When/Then statements. For example, here is a working
Cucumber specification for a login test:
Scenario: Log in without entering credentials
Given I am at test2 Log in page
When I log in without entering credentials
Then Log in page should open
And feedback should be You have not specified a valid username.
Cucumber is valuable because it creates a communication channel that does
not usually exist for software development projects. For one thing, casual
users and the greater community can read the Cucumber specifications[3] and
understand what is being tested (and what is failing!). In a similar way,
Cucumber allows non-programming members of the community to contribute to
the test automation project by submitting test scenarios to be automated.
We have created a test backlog page on mediawiki.org[4] for such
scenarios.
The tests themselves are controlled by an instance of the Jenkins
continuous integration server provided by a hosting service called
Cloudbees. The tests are run on hosts managed by Sauce Labs, and are
currently running on Firefox, Chrome, IE6/7/8/9/10, android, iphone, and
ipad, using test2wiki, beta labs, and the mobile web site as targets. As
of today, the status of all the tests are available publicly[5].
Using Cloudbees to host browser tests provides a number of advantages.
Cloudbees provides free and seamless integration with Sauce Labs and all
of their tools. With Cloudbees we have fine control over versions of the
software we use, and we can easily update versions of Ruby gems and Ruby
itself at will. It is possible to publish information between the
Cloudbees Jenkins instance and the WMF Jenkins also, although we have not
yet done this. Cloudbees is economical. Finally, although managed by a
commercial company, every aspect of the integrated service is open source
software. There is little risk in using Cloudbees. The source code for
the tests themselves resides in WMF's gerrit system[6]. If Cloudbees were
to disappear tomorrow, we could re-create the services it provides on WMF
hosts or other hosts, although at some expense both financially and in
terms of maintenance.
Željko and I spent significant time working on the design and
implementation of our first few tests. After that tweaking was essentially
done, the tests we have came fully online November 26th. Since that time
even our few existing tests have identified the following issues:
UploadWizard broken for IE9 only:
https://bugzilla.wikimedia.org/show_bug.cgi?id=42512
UploadWizard mistakenly reverted to broken version on test2, in deployment
pipeline. (no BZ ticket, fixed on the fly by Ryan Kaldari)
AFTv5 user experience issue with IE9:
https://bugzilla.wikimedia.org/show_bug.cgi?id=42551
AFTv5 user experience issue with all IEs, and later breakage in the same
code: https://bugzilla.wikimedia.org/show_bug.cgi?id=42517
The tests have also exposed important issues with our test environments
themselves: https://bugzilla.wikimedia.org/show_bug.cgi?id=42736
And seen just today, UW broken for IE6/IE7:
https://bugzilla.wikimedia.org/show_bug.cgi?id=4292
This week I will be publishing detailed technical design and architecture
documentation for those who might like to contribute beyond simply adding
to the backlog of tests to be implemented. It is worth noting that browser
test automation projects are expensive and often fail, and even WMF has at
least one example of such a failure in its history. In my experience, such
failures are almost always caused by poor design. Željko and I have been
specialists in browser test automation with open source tools since such
tools came into existence, and we intend for this project to continue to be
a world class reference implementation of such a project as it grows.
We will be presenting a walkthrough of the project this Thursday[7].
[1]
https://speakerdeck.com/jarib/automating-130-browser-platform-and-language-…
[2] http://rdoc.info/gems/watir-webdriver/#API_docs
[3] a sophisticated Cucumber test specification in use right now:
https://github.com/wikimedia/qa-browsertests/blob/master/features/uploadwiz…
[4] http://www.mediawiki.org/wiki/Qa/test_backlog
[5] https://wmf.ci.cloudbees.com/ NOTE: current Firefox failures are due to
a bug in selenium to be fixed in next release; IE6/IE7 failure due to
https://bugzilla.wikimedia.org/show_bug.cgi?id=42922 (new behavior today),
IE6 failure because IE6 is not supported for AFTv5 those test should be
turned off.
[6] https://gerrit.wikimedia.org/r/#/admin/projects/qa/browsertests also
https://github.com/wikimedia/qa-browsertests
[7] https://www.mediawiki.org/wiki/Meetings/2012-12-13
Hi, after just a few weeks in my role I have seen more often than not
the pattern of someone willing to help on certain bug report but then
having difficulties finding the right way to do it, and defaulting to
sending an email with questions to e.g. Sumana.
I have tried to write down a meta-answer proposed at
https://www.mediawiki.org/wiki/User:Qgil/Contributing_to_a_bug_report
You can help polishing the text and directing to it anybody needing help
with these first steps.
Andre, feel free moving it to the right location as soon as you are
happy with it.
--
Quim Gil
Technical Contributor Coordinator @ Wikimedia Foundation
http://www.mediawiki.org/wiki/User:Qgil
Hello,
Earlier today I have slightly changed our review / test workflow on
mediawiki/core.git . That will let us do more tests and scale things in
the long term.
The new workflow is described on the wiki (including a flowchart):
https://www.mediawiki.org/wiki/Continuous_integration/Workflow
How it works?
Whenever a patch is submitted, Jenkins will attempt to merge the
patchset against the latest master and abort whenever it fails asking
for the submitter to rebase the patchset.
Jenkins will then verify the PHP files are valid and run JSHint for the
Javascript files.
If everything is fine, jenkins-bot will vote Code-Review +1 to let
everyone know that the change look fine. Else, it will vote Verified -1
preventing the patchset from being submitted.
That is all. The slow unit tests are no more run on patchset submission.
To get the test to run, the patch will have to be reviewed first.
Whenever someone vote CodeReview +2 Jenkins will run the Unit tests,
then if:
- tests are successful, jenkins-bot will vote Verified+1 which let us
merge the patch.
- tests are failing, jenkins-bot vote Verified-1 AND CodeReview -2 to
prevent the patch from being merged.
If all works fine tonight, I will configure jenkins-bot so it merges the
Change automatically whenever the test are successful.
That workflow has been applied to a few extensions this week. I will add
it to more and more extensions and eventually will get all extensions to
receive PHP Linting and a JSHint run.
--
Antoine "hashar" Musso