I believe this is the result of the ongoing intermittent 503 errors on big articles on betalabs (https://bugzilla.wikimedia.org/show_bug.cgi?id=57249).
During the last scrum of scrums, ops declared that they're too overburdened and likely won't have bandwidth for investigating betalabs issues. Antoine, do you have any bandwidth for this? QA folks, perhaps we should consider using a different article for these tests so we don't keep getting these false positive failures.
On Fri, Dec 6, 2013 at 10:09 AM, jenkins-no-reply@cloudbees.com wrote:
See < https://wmf.ci.cloudbees.com/job/MobileFrontend-en.m.wikipedia.beta.wmflabs....
Started by user Chris Mcmahon Building remotely on s-61f7a5c0 in workspace < https://wmf.ci.cloudbees.com/job/MobileFrontend-en.m.wikipedia.beta.wmflabs....
Cloning the remote Git repository Cloning repository https://gerrit.wikimedia.org/r/mediawiki/extensions/MobileFrontend git --version git version 1.7.11.7 Checking out Revision e8593847455adf83472e2f7ef4d7fd81fb9a9aca (origin/master) [MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome] $ /bin/sh -xe /tmp/hudson5691964007505168694.sh
- export MEDIAWIKI_USER=Selenium_user
- MEDIAWIKI_USER=Selenium_user
- export
MEDIAWIKI_PASSWORD_VARIABLE=MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG
- MEDIAWIKI_PASSWORD_VARIABLE=MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG
- export BROWSER_LABEL=chrome
- BROWSER_LABEL=chrome
- export MEDIAWIKI_URL=http://en.m.wikipedia.beta.wmflabs.org/wiki/
- MEDIAWIKI_URL=http://en.m.wikipedia.beta.wmflabs.org/wiki/
- curl -s -o use-ruby
https://repository-cloudbees.forge.cloudbees.com/distributions/ci-addons/rub...
- RUBY_VERSION=2.0.0-p247
- source ./use-ruby
++ [[ -z 2.0.0-p247 ]] ++ ruby187='1.8.7-*' ++ [[ 2.0.0-p247 == 1.8.7-* ]] ++ BASE=/scratch/jenkins/addons ++ mkdir -p /scratch/jenkins/addons ++ '[' '!' -f /scratch/jenkins/addons/use.sh ']' ++ . /scratch/jenkins/addons/use.sh ++ mkdir -p /scratch/jenkins/addons/ruby +++ addon_architecture +++ uname -m ++ ruby_name=ruby-2.0.0-p247-x86_64 ++ '[' '!' -f /scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64.tar.bz2 ']' ++ '[' '!' -f /scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64/bin/ruby ']' ++ export RUBY_HOME=/scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64 ++ RUBY_HOME=/scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64 ++ export PATH=/scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64/bin:/opt/jdk/latest/bin:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/opt/maven/latest/bin:/opt/ant/latest/bin:/opt/android/android-sdk-linux/tools:/opt/gradle/latest/bin:/home/jenkins/bin:/opt/jdk/jdk.latest/bin ++ PATH=/scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64/bin:/opt/jdk/latest/bin:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/opt/maven/latest/bin:/opt/ant/latest/bin:/opt/android/android-sdk-linux/tools:/opt/gradle/latest/bin:/home/jenkins/bin:/opt/jdk/jdk.latest/bin ++ unset GEM_PATH ++ unset GEM_HOME ++ unset BUNDLE_PATH
- gem install bundler --no-ri --no-rdoc
Successfully installed bundler-1.3.5 1 gem installed
- '[' -d tests/browser/ ']'
- cd tests/browser/
- bundle install
Using builder (3.2.2) Using ffi (1.9.3) Using childprocess (0.3.9) Using diff-lcs (1.2.4) Using multi_json (1.8.2) Using gherkin (2.12.2) Using multi_test (0.0.2) Using cucumber (1.3.8) Using i18n (0.6.5) Using faker (1.2.0) Using yml_reader (0.2) Using data_magic (0.16.1) Using json (1.8.1) Using net-http-persistent (2.9) Using page_navigation (0.9) Using rubyzip (1.0.0) Using websocket (1.0.7) Using selenium-webdriver (2.37.0) Using watir-webdriver (0.6.4) Using page-object (0.9.3) Using mime-types (2.0) Using rest-client (1.6.7) Using rspec-expectations (2.14.3) Using syntax (1.0.0) Using mediawiki-selenium (0.1.12) Using bundler (1.3.5) Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed.
- bundle exec cucumber --verbose --profile ci --tags @
en.m.wikipedia.beta.wmflabs.org Code:
- features/support/env.rb
- features/support/hooks.rb
- features/support/modules/url_module.rb
- features/support/pages/article_page.rb
- features/support/pages/beta_page.rb
- features/support/pages/create_article_page.rb
- features/support/pages/create_user_page.rb
- features/support/pages/edit_page.rb
- features/support/pages/home_page.rb
- features/support/pages/language_page.rb
- features/support/pages/login_page.rb
- features/support/pages/notification_page.rb
- features/support/pages/random_page.rb
- features/support/pages/search_page.rb
- features/support/pages/uploads_page.rb
- features/step_definitions/common_steps.rb
- features/step_definitions/create_article_steps.rb
- features/step_definitions/external_links_steps.rb
- features/step_definitions/footer_steps.rb
- features/step_definitions/header_logged_in_steps.rb
- features/step_definitions/header_steps.rb
- features/step_definitions/image_resolves_steps.rb
- features/step_definitions/language_steps.rb
- features/step_definitions/login_required_uploads_steps.rb
- features/step_definitions/login_required_watchlist_steps.rb
- features/step_definitions/menu_opens_page_steps.rb
- features/step_definitions/notification_steps.rb
- features/step_definitions/random_ua_steps.rb
- features/step_definitions/search_steps.rb
- features/step_definitions/uploads_steps.rb
- features/step_definitions/watchlist_steps.rb
Features:
- features/create_article.feature
- features/external_links.feature
- features/footer.feature
- features/footer_logged_in.feature
- features/header.feature
- features/header_logged_in.feature
- features/image_resolves.feature
- features/language.feature
- features/language_logged_in.feature
- features/login_required_uploads.feature
- features/login_required_watchlist.feature
- features/menu_opens_page.feature
- features/notification.feature
- features/random_ua.feature
- features/search.feature
- features/uploads.feature
- features/watchlist.feature
Parsing feature files took 0m0.129s
Build step 'Execute shell' marked build as failure Recording test results
I could look at this (no idea where to though:)
On Fri, Dec 6, 2013 at 9:40 PM, Arthur Richards arichards@wikimedia.orgwrote:
I believe this is the result of the ongoing intermittent 503 errors on big articles on betalabs (https://bugzilla.wikimedia.org/show_bug.cgi?id=57249 ).
During the last scrum of scrums, ops declared that they're too overburdened and likely won't have bandwidth for investigating betalabs issues. Antoine, do you have any bandwidth for this? QA folks, perhaps we should consider using a different article for these tests so we don't keep getting these false positive failures.
On Fri, Dec 6, 2013 at 10:09 AM, jenkins-no-reply@cloudbees.com wrote:
See < https://wmf.ci.cloudbees.com/job/MobileFrontend-en.m.wikipedia.beta.wmflabs....
Started by user Chris Mcmahon Building remotely on s-61f7a5c0 in workspace < https://wmf.ci.cloudbees.com/job/MobileFrontend-en.m.wikipedia.beta.wmflabs....
Cloning the remote Git repository Cloning repository https://gerrit.wikimedia.org/r/mediawiki/extensions/MobileFrontend git --version git version 1.7.11.7 Checking out Revision e8593847455adf83472e2f7ef4d7fd81fb9a9aca (origin/master) [MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome] $ /bin/sh -xe /tmp/hudson5691964007505168694.sh
- export MEDIAWIKI_USER=Selenium_user
- MEDIAWIKI_USER=Selenium_user
- export
MEDIAWIKI_PASSWORD_VARIABLE=MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG
- MEDIAWIKI_PASSWORD_VARIABLE=MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG
- export BROWSER_LABEL=chrome
- BROWSER_LABEL=chrome
- export MEDIAWIKI_URL=http://en.m.wikipedia.beta.wmflabs.org/wiki/
- MEDIAWIKI_URL=http://en.m.wikipedia.beta.wmflabs.org/wiki/
- curl -s -o use-ruby
https://repository-cloudbees.forge.cloudbees.com/distributions/ci-addons/rub...
- RUBY_VERSION=2.0.0-p247
- source ./use-ruby
++ [[ -z 2.0.0-p247 ]] ++ ruby187='1.8.7-*' ++ [[ 2.0.0-p247 == 1.8.7-* ]] ++ BASE=/scratch/jenkins/addons ++ mkdir -p /scratch/jenkins/addons ++ '[' '!' -f /scratch/jenkins/addons/use.sh ']' ++ . /scratch/jenkins/addons/use.sh ++ mkdir -p /scratch/jenkins/addons/ruby +++ addon_architecture +++ uname -m ++ ruby_name=ruby-2.0.0-p247-x86_64 ++ '[' '!' -f /scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64.tar.bz2 ']' ++ '[' '!' -f /scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64/bin/ruby ']' ++ export RUBY_HOME=/scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64 ++ RUBY_HOME=/scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64 ++ export PATH=/scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64/bin:/opt/jdk/latest/bin:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/opt/maven/latest/bin:/opt/ant/latest/bin:/opt/android/android-sdk-linux/tools:/opt/gradle/latest/bin:/home/jenkins/bin:/opt/jdk/jdk.latest/bin ++ PATH=/scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64/bin:/opt/jdk/latest/bin:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/opt/maven/latest/bin:/opt/ant/latest/bin:/opt/android/android-sdk-linux/tools:/opt/gradle/latest/bin:/home/jenkins/bin:/opt/jdk/jdk.latest/bin ++ unset GEM_PATH ++ unset GEM_HOME ++ unset BUNDLE_PATH
- gem install bundler --no-ri --no-rdoc
Successfully installed bundler-1.3.5 1 gem installed
- '[' -d tests/browser/ ']'
- cd tests/browser/
- bundle install
Using builder (3.2.2) Using ffi (1.9.3) Using childprocess (0.3.9) Using diff-lcs (1.2.4) Using multi_json (1.8.2) Using gherkin (2.12.2) Using multi_test (0.0.2) Using cucumber (1.3.8) Using i18n (0.6.5) Using faker (1.2.0) Using yml_reader (0.2) Using data_magic (0.16.1) Using json (1.8.1) Using net-http-persistent (2.9) Using page_navigation (0.9) Using rubyzip (1.0.0) Using websocket (1.0.7) Using selenium-webdriver (2.37.0) Using watir-webdriver (0.6.4) Using page-object (0.9.3) Using mime-types (2.0) Using rest-client (1.6.7) Using rspec-expectations (2.14.3) Using syntax (1.0.0) Using mediawiki-selenium (0.1.12) Using bundler (1.3.5) Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed.
- bundle exec cucumber --verbose --profile ci --tags @
en.m.wikipedia.beta.wmflabs.org Code:
- features/support/env.rb
- features/support/hooks.rb
- features/support/modules/url_module.rb
- features/support/pages/article_page.rb
- features/support/pages/beta_page.rb
- features/support/pages/create_article_page.rb
- features/support/pages/create_user_page.rb
- features/support/pages/edit_page.rb
- features/support/pages/home_page.rb
- features/support/pages/language_page.rb
- features/support/pages/login_page.rb
- features/support/pages/notification_page.rb
- features/support/pages/random_page.rb
- features/support/pages/search_page.rb
- features/support/pages/uploads_page.rb
- features/step_definitions/common_steps.rb
- features/step_definitions/create_article_steps.rb
- features/step_definitions/external_links_steps.rb
- features/step_definitions/footer_steps.rb
- features/step_definitions/header_logged_in_steps.rb
- features/step_definitions/header_steps.rb
- features/step_definitions/image_resolves_steps.rb
- features/step_definitions/language_steps.rb
- features/step_definitions/login_required_uploads_steps.rb
- features/step_definitions/login_required_watchlist_steps.rb
- features/step_definitions/menu_opens_page_steps.rb
- features/step_definitions/notification_steps.rb
- features/step_definitions/random_ua_steps.rb
- features/step_definitions/search_steps.rb
- features/step_definitions/uploads_steps.rb
- features/step_definitions/watchlist_steps.rb
Features:
- features/create_article.feature
- features/external_links.feature
- features/footer.feature
- features/footer_logged_in.feature
- features/header.feature
- features/header_logged_in.feature
- features/image_resolves.feature
- features/language.feature
- features/language_logged_in.feature
- features/login_required_uploads.feature
- features/login_required_watchlist.feature
- features/menu_opens_page.feature
- features/notification.feature
- features/random_ua.feature
- features/search.feature
- features/uploads.feature
- features/watchlist.feature
Parsing feature files took 0m0.129s
Build step 'Execute shell' marked build as failure Recording test results
-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687
That would be awesome if you have some extra cycles, Max!
On Fri, Dec 6, 2013 at 10:50 AM, Max Semenik msemenik@wikimedia.org wrote:
I could look at this (no idea where to though:)
On Fri, Dec 6, 2013 at 9:40 PM, Arthur Richards arichards@wikimedia.orgwrote:
I believe this is the result of the ongoing intermittent 503 errors on big articles on betalabs ( https://bugzilla.wikimedia.org/show_bug.cgi?id=57249).
During the last scrum of scrums, ops declared that they're too overburdened and likely won't have bandwidth for investigating betalabs issues. Antoine, do you have any bandwidth for this? QA folks, perhaps we should consider using a different article for these tests so we don't keep getting these false positive failures.
On Fri, Dec 6, 2013 at 10:09 AM, jenkins-no-reply@cloudbees.com wrote:
See < https://wmf.ci.cloudbees.com/job/MobileFrontend-en.m.wikipedia.beta.wmflabs....
Started by user Chris Mcmahon Building remotely on s-61f7a5c0 in workspace < https://wmf.ci.cloudbees.com/job/MobileFrontend-en.m.wikipedia.beta.wmflabs....
Cloning the remote Git repository Cloning repository https://gerrit.wikimedia.org/r/mediawiki/extensions/MobileFrontend git --version git version 1.7.11.7 Checking out Revision e8593847455adf83472e2f7ef4d7fd81fb9a9aca (origin/master) [MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome] $ /bin/sh -xe /tmp/hudson5691964007505168694.sh
- export MEDIAWIKI_USER=Selenium_user
- MEDIAWIKI_USER=Selenium_user
- export
MEDIAWIKI_PASSWORD_VARIABLE=MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG
MEDIAWIKI_PASSWORD_VARIABLE=MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG
- export BROWSER_LABEL=chrome
- BROWSER_LABEL=chrome
- export MEDIAWIKI_URL=http://en.m.wikipedia.beta.wmflabs.org/wiki/
- MEDIAWIKI_URL=http://en.m.wikipedia.beta.wmflabs.org/wiki/
- curl -s -o use-ruby
https://repository-cloudbees.forge.cloudbees.com/distributions/ci-addons/rub...
- RUBY_VERSION=2.0.0-p247
- source ./use-ruby
++ [[ -z 2.0.0-p247 ]] ++ ruby187='1.8.7-*' ++ [[ 2.0.0-p247 == 1.8.7-* ]] ++ BASE=/scratch/jenkins/addons ++ mkdir -p /scratch/jenkins/addons ++ '[' '!' -f /scratch/jenkins/addons/use.sh ']' ++ . /scratch/jenkins/addons/use.sh ++ mkdir -p /scratch/jenkins/addons/ruby +++ addon_architecture +++ uname -m ++ ruby_name=ruby-2.0.0-p247-x86_64 ++ '[' '!' -f /scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64.tar.bz2 ']' ++ '[' '!' -f /scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64/bin/ruby ']' ++ export RUBY_HOME=/scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64 ++ RUBY_HOME=/scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64 ++ export PATH=/scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64/bin:/opt/jdk/latest/bin:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/opt/maven/latest/bin:/opt/ant/latest/bin:/opt/android/android-sdk-linux/tools:/opt/gradle/latest/bin:/home/jenkins/bin:/opt/jdk/jdk.latest/bin ++ PATH=/scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64/bin:/opt/jdk/latest/bin:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/opt/maven/latest/bin:/opt/ant/latest/bin:/opt/android/android-sdk-linux/tools:/opt/gradle/latest/bin:/home/jenkins/bin:/opt/jdk/jdk.latest/bin ++ unset GEM_PATH ++ unset GEM_HOME ++ unset BUNDLE_PATH
- gem install bundler --no-ri --no-rdoc
Successfully installed bundler-1.3.5 1 gem installed
- '[' -d tests/browser/ ']'
- cd tests/browser/
- bundle install
Using builder (3.2.2) Using ffi (1.9.3) Using childprocess (0.3.9) Using diff-lcs (1.2.4) Using multi_json (1.8.2) Using gherkin (2.12.2) Using multi_test (0.0.2) Using cucumber (1.3.8) Using i18n (0.6.5) Using faker (1.2.0) Using yml_reader (0.2) Using data_magic (0.16.1) Using json (1.8.1) Using net-http-persistent (2.9) Using page_navigation (0.9) Using rubyzip (1.0.0) Using websocket (1.0.7) Using selenium-webdriver (2.37.0) Using watir-webdriver (0.6.4) Using page-object (0.9.3) Using mime-types (2.0) Using rest-client (1.6.7) Using rspec-expectations (2.14.3) Using syntax (1.0.0) Using mediawiki-selenium (0.1.12) Using bundler (1.3.5) Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed.
- bundle exec cucumber --verbose --profile ci --tags @
en.m.wikipedia.beta.wmflabs.org Code:
- features/support/env.rb
- features/support/hooks.rb
- features/support/modules/url_module.rb
- features/support/pages/article_page.rb
- features/support/pages/beta_page.rb
- features/support/pages/create_article_page.rb
- features/support/pages/create_user_page.rb
- features/support/pages/edit_page.rb
- features/support/pages/home_page.rb
- features/support/pages/language_page.rb
- features/support/pages/login_page.rb
- features/support/pages/notification_page.rb
- features/support/pages/random_page.rb
- features/support/pages/search_page.rb
- features/support/pages/uploads_page.rb
- features/step_definitions/common_steps.rb
- features/step_definitions/create_article_steps.rb
- features/step_definitions/external_links_steps.rb
- features/step_definitions/footer_steps.rb
- features/step_definitions/header_logged_in_steps.rb
- features/step_definitions/header_steps.rb
- features/step_definitions/image_resolves_steps.rb
- features/step_definitions/language_steps.rb
- features/step_definitions/login_required_uploads_steps.rb
- features/step_definitions/login_required_watchlist_steps.rb
- features/step_definitions/menu_opens_page_steps.rb
- features/step_definitions/notification_steps.rb
- features/step_definitions/random_ua_steps.rb
- features/step_definitions/search_steps.rb
- features/step_definitions/uploads_steps.rb
- features/step_definitions/watchlist_steps.rb
Features:
- features/create_article.feature
- features/external_links.feature
- features/footer.feature
- features/footer_logged_in.feature
- features/header.feature
- features/header_logged_in.feature
- features/image_resolves.feature
- features/language.feature
- features/language_logged_in.feature
- features/login_required_uploads.feature
- features/login_required_watchlist.feature
- features/menu_opens_page.feature
- features/notification.feature
- features/random_ua.feature
- features/search.feature
- features/uploads.feature
- features/watchlist.feature
Parsing feature files took 0m0.129s
Build step 'Execute shell' marked build as failure Recording test results
-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687
maxsem@deployment-bastion:/data/project/logs$ cat apache-error.log | grep SIGTERM | wc -l 54
The log is full of it:
[Fri Dec 06 14:34:41 2013] [notice] caught SIGTERM, shutting down [Fri Dec 06 14:34:42 2013] [notice] Apache/2.2.22 (Ubuntu) PHP/5.3.10-1ubuntu3.8+wmf2 with Suhosin-Patch configured -- resuming normal operations [Fri Dec 06 14:52:30 2013] [notice] caught SIGTERM, shutting down [Fri Dec 06 14:52:31 2013] [notice] Apache/2.2.22 (Ubuntu) PHP/5.3.10-1ubuntu3.8+wmf2 with Suhosin-Patch configured -- resuming normal operations [Fri Dec 06 15:05:23 2013] [notice] caught SIGTERM, shutting down [Fri Dec 06 15:05:25 2013] [notice] Apache/2.2.22 (Ubuntu) PHP/5.3.10-1ubuntu3.8+wmf2 with Suhosin-Patch configured -- resuming normal operations [Fri Dec 06 15:22:13 2013] [notice] caught SIGTERM, shutting down [Fri Dec 06 15:22:14 2013] [notice] Apache/2.2.22 (Ubuntu) PHP/5.3.10-1ubuntu3.8+wmf2 with Suhosin-Patch configured -- resuming normal operations [Fri Dec 06 15:34:19 2013] [notice] caught SIGTERM, shutting down [Fri Dec 06 15:34:20 2013] [notice] Apache/2.2.22 (Ubuntu) PHP/5.3.10-1ubuntu3.8+wmf2 with Suhosin-Patch configured -- resuming normal operations
Not sure what restarts it, but that must be it.
On Fri, Dec 6, 2013 at 10:15 PM, Arthur Richards arichards@wikimedia.orgwrote:
That would be awesome if you have some extra cycles, Max!
On Fri, Dec 6, 2013 at 10:50 AM, Max Semenik msemenik@wikimedia.orgwrote:
I could look at this (no idea where to though:)
On Fri, Dec 6, 2013 at 9:40 PM, Arthur Richards arichards@wikimedia.orgwrote:
I believe this is the result of the ongoing intermittent 503 errors on big articles on betalabs ( https://bugzilla.wikimedia.org/show_bug.cgi?id=57249).
During the last scrum of scrums, ops declared that they're too overburdened and likely won't have bandwidth for investigating betalabs issues. Antoine, do you have any bandwidth for this? QA folks, perhaps we should consider using a different article for these tests so we don't keep getting these false positive failures.
On Fri, Dec 6, 2013 at 10:09 AM, jenkins-no-reply@cloudbees.com wrote:
See < https://wmf.ci.cloudbees.com/job/MobileFrontend-en.m.wikipedia.beta.wmflabs....
Started by user Chris Mcmahon Building remotely on s-61f7a5c0 in workspace < https://wmf.ci.cloudbees.com/job/MobileFrontend-en.m.wikipedia.beta.wmflabs....
Cloning the remote Git repository Cloning repository https://gerrit.wikimedia.org/r/mediawiki/extensions/MobileFrontend git --version git version 1.7.11.7 Checking out Revision e8593847455adf83472e2f7ef4d7fd81fb9a9aca (origin/master) [MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome] $ /bin/sh -xe /tmp/hudson5691964007505168694.sh
- export MEDIAWIKI_USER=Selenium_user
- MEDIAWIKI_USER=Selenium_user
- export
MEDIAWIKI_PASSWORD_VARIABLE=MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG
MEDIAWIKI_PASSWORD_VARIABLE=MEDIAWIKI_PASSWORD_SELENIUM_USER_WMFLABS_ORG
- export BROWSER_LABEL=chrome
- BROWSER_LABEL=chrome
- export MEDIAWIKI_URL=http://en.m.wikipedia.beta.wmflabs.org/wiki/
- MEDIAWIKI_URL=http://en.m.wikipedia.beta.wmflabs.org/wiki/
- curl -s -o use-ruby
https://repository-cloudbees.forge.cloudbees.com/distributions/ci-addons/rub...
- RUBY_VERSION=2.0.0-p247
- source ./use-ruby
++ [[ -z 2.0.0-p247 ]] ++ ruby187='1.8.7-*' ++ [[ 2.0.0-p247 == 1.8.7-* ]] ++ BASE=/scratch/jenkins/addons ++ mkdir -p /scratch/jenkins/addons ++ '[' '!' -f /scratch/jenkins/addons/use.sh ']' ++ . /scratch/jenkins/addons/use.sh ++ mkdir -p /scratch/jenkins/addons/ruby +++ addon_architecture +++ uname -m ++ ruby_name=ruby-2.0.0-p247-x86_64 ++ '[' '!' -f /scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64.tar.bz2 ']' ++ '[' '!' -f /scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64/bin/ruby ']' ++ export RUBY_HOME=/scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64 ++ RUBY_HOME=/scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64 ++ export PATH=/scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64/bin:/opt/jdk/latest/bin:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/opt/maven/latest/bin:/opt/ant/latest/bin:/opt/android/android-sdk-linux/tools:/opt/gradle/latest/bin:/home/jenkins/bin:/opt/jdk/jdk.latest/bin ++ PATH=/scratch/jenkins/addons/ruby/ruby-2.0.0-p247-x86_64/bin:/opt/jdk/latest/bin:/usr/lib64/qt-3.3/bin:/usr/lib64/ccache:/usr/local/bin:/usr/bin:/opt/maven/latest/bin:/opt/ant/latest/bin:/opt/android/android-sdk-linux/tools:/opt/gradle/latest/bin:/home/jenkins/bin:/opt/jdk/jdk.latest/bin ++ unset GEM_PATH ++ unset GEM_HOME ++ unset BUNDLE_PATH
- gem install bundler --no-ri --no-rdoc
Successfully installed bundler-1.3.5 1 gem installed
- '[' -d tests/browser/ ']'
- cd tests/browser/
- bundle install
Using builder (3.2.2) Using ffi (1.9.3) Using childprocess (0.3.9) Using diff-lcs (1.2.4) Using multi_json (1.8.2) Using gherkin (2.12.2) Using multi_test (0.0.2) Using cucumber (1.3.8) Using i18n (0.6.5) Using faker (1.2.0) Using yml_reader (0.2) Using data_magic (0.16.1) Using json (1.8.1) Using net-http-persistent (2.9) Using page_navigation (0.9) Using rubyzip (1.0.0) Using websocket (1.0.7) Using selenium-webdriver (2.37.0) Using watir-webdriver (0.6.4) Using page-object (0.9.3) Using mime-types (2.0) Using rest-client (1.6.7) Using rspec-expectations (2.14.3) Using syntax (1.0.0) Using mediawiki-selenium (0.1.12) Using bundler (1.3.5) Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed.
- bundle exec cucumber --verbose --profile ci --tags @
en.m.wikipedia.beta.wmflabs.org Code:
- features/support/env.rb
- features/support/hooks.rb
- features/support/modules/url_module.rb
- features/support/pages/article_page.rb
- features/support/pages/beta_page.rb
- features/support/pages/create_article_page.rb
- features/support/pages/create_user_page.rb
- features/support/pages/edit_page.rb
- features/support/pages/home_page.rb
- features/support/pages/language_page.rb
- features/support/pages/login_page.rb
- features/support/pages/notification_page.rb
- features/support/pages/random_page.rb
- features/support/pages/search_page.rb
- features/support/pages/uploads_page.rb
- features/step_definitions/common_steps.rb
- features/step_definitions/create_article_steps.rb
- features/step_definitions/external_links_steps.rb
- features/step_definitions/footer_steps.rb
- features/step_definitions/header_logged_in_steps.rb
- features/step_definitions/header_steps.rb
- features/step_definitions/image_resolves_steps.rb
- features/step_definitions/language_steps.rb
- features/step_definitions/login_required_uploads_steps.rb
- features/step_definitions/login_required_watchlist_steps.rb
- features/step_definitions/menu_opens_page_steps.rb
- features/step_definitions/notification_steps.rb
- features/step_definitions/random_ua_steps.rb
- features/step_definitions/search_steps.rb
- features/step_definitions/uploads_steps.rb
- features/step_definitions/watchlist_steps.rb
Features:
- features/create_article.feature
- features/external_links.feature
- features/footer.feature
- features/footer_logged_in.feature
- features/header.feature
- features/header_logged_in.feature
- features/image_resolves.feature
- features/language.feature
- features/language_logged_in.feature
- features/login_required_uploads.feature
- features/login_required_watchlist.feature
- features/menu_opens_page.feature
- features/notification.feature
- features/random_ua.feature
- features/search.feature
- features/uploads.feature
- features/watchlist.feature
Parsing feature files took 0m0.129s
Build step 'Execute shell' marked build as failure Recording test results
-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687
-- Arthur Richards Software Engineer, Mobile [[User:Awjrichards]] IRC: awjr +1-415-839-6885 x6687
Le 06/12/13 18:40, Arthur Richards a écrit :
I believe this is the result of the ongoing intermittent 503 errors on big articles on betalabs (https://bugzilla.wikimedia.org/show_bug.cgi?id=57249).
During the last scrum of scrums, ops declared that they're too overburdened and likely won't have bandwidth for investigating betalabs issues. Antoine, do you have any bandwidth for this? QA folks, perhaps we should consider using a different article for these tests so we don't keep getting these false positive failures.
<snip>
Hello,
Is there any reason to use the Barack Obama article for testing? The reason that article has grown popular is that I used it to stress test the Cite extension something like 3 or 4 years ago. At that time that was an article with probably the most {{cite}} and it highlighted we needed a better template system (short story: we have LUA now).
I am not sure what the feature tests are trying to achieve, but they should probably hit another article. But I am probably missing a point.
As noticed on the bug report, the article takes up to 45 seconds to parse on beta. There is surely a bunch of things we can optimize to make parsing faster there. Last time I looked at it, it is because cache are usually cold since we have not much traffic on beta.
The varnish text cache are supposed to have a 180s first byte timeout. I did confirm a few hours ago that the text cache in beta does have the 180s timeout.
During the last scrum of scrums, ops declared that they're too overburdened
<rant> If we hired 10-15 more junior ops people to handle the basic tasks instead of attempting to hire one rock star for half the expected wage in SF... We might have an ops team less overwhelmed :-D </rant>
To conclude, I will attempt to investigate that issue on Monday. A possibility would be to have a sleep.php that would take a sleeping time value in parameter. Can then attempt to find out whether the timeout is respected, aka query:
sleep?60 sleep?180 ..
cheers,
On Fri, Dec 6, 2013 at 3:01 PM, Antoine Musso amusso@wikimedia.org wrote:
Is there any reason to use the Barack Obama article for testing? The reason that article has grown popular is that I used it to stress test the Cite extension something like 3 or 4 years ago. At that time that was an article with probably the most {{cite}} and it highlighted we needed a better template system (short story: we have LUA now).
One of the big reasons we've used the Obama article to test against is because it's long and complex. Historically, we've had issues that would only surface in particular cases which could often be replicated on highly complex articles, but not necessarily on really simple articles. Is there another article that we could use that meets the criteria of complex and long, but will not cause the 500 errors like Obama article? If so, then let's try that instead. I suspect we do not actually know the answer to this question and worry that we can try finding other workarounds for this problem that will ultimately pile up and cause even more headaches.
The one thing I do know is that we need to do something, and do it soon. If our tests keep failing because of an architectural issue rather an issue in our code, the tests lose their utility - we stop paying attention to failures when they are false positives (or false positive-ish).
Given what I've heard form Antoine and folks in ops, this issue is not likely to be resolved in the near term. Who is the 'owner' of betalabs anyway - ops? platform? QA?
*Chris*, what do you think we should do to resolve this asap?
To conclude, I will attempt to investigate that issue on Monday.
*Antoine*, did you have any luck?
A possibility would be to have a sleep.php that would take a sleeping time value in parameter. Can then attempt to find out whether the timeout is respected, aka query:
sleep?60 sleep?180 ..
I think we should either use another article or really fix the issue. Yet another hack feel like it will just introduce further fragility and possibly cause issues with test validity.
On Mon, Dec 9, 2013 at 3:14 PM, Arthur Richards arichards@wikimedia.orgwrote:
Given what I've heard form Antoine and folks in ops, this issue is not likely to be resolved in the near term. Who is the 'owner' of betalabs anyway - ops? platform? QA?
*Chris*, what do you think we should do to resolve this asap?
To conclude, I will attempt to investigate that issue on Monday.
*Antoine*, did you have any luck?
A little forensics makes me think that the 503 from Varnish on a number of hosts began to appear on or about 20 Nov, first reported on meta, but noted in a possible Varnish configuration problem on 26 Nov on the ops mail list.
As for who 'owns' beta, it is more a matter of 'who owns what' on beta. We seem to be surfacing some tricky Varnish config problem. So: who is it that owns the Varnish configuration on beta, especially since we seem to be seeing the 503 error on other wikis as well, but under-reported?