Hello all,
I want to apologize for the current slowness in our CI stack. Everything is working fine in the sense that nothing is broken.
We simply did not adequately prepare for the switch to php5.5 being the new minimum version for MediaWiki. This resulted in a large influx of jobs on a set of our Jenkins executers that was not a large enough pool. Timo and Antoine have created more executers to help mitigate [0].
More explicitly: due to package version availability in Ubuntu, our php5.3 jobs run on our pool of Ubuntu Precis hosts, while the php5.5/hhvm jobs run on our pool of Ubuntu Trusty hosts.
Both sets of jobs are in our pipeline to migrate to the new Nodepool system that allows burstable availability of executers.
Again, apologies for the delays but know that the jobs will complete just not as fast as you are accustomed to.
Greg
[0] https://phabricator.wikimedia.org/T126423
Le 10/02/2016 22:06, Greg Grossmeier a écrit :
Hello all,
I want to apologize for the current slowness in our CI stack. Everything is working fine in the sense that nothing is broken.
We simply did not adequately prepare for the switch to php5.5 being the new minimum version for MediaWiki. This resulted in a large influx of jobs on a set of our Jenkins executers that was not a large enough pool. Timo and Antoine have created more executers to help mitigate [0].
More explicitly: due to package version availability in Ubuntu, our php5.3 jobs run on our pool of Ubuntu Precis hosts, while the php5.5/hhvm jobs run on our pool of Ubuntu Trusty hosts.
Both sets of jobs are in our pipeline to migrate to the new Nodepool system that allows burstable availability of executers.
Again, apologies for the delays but know that the jobs will complete just not as fast as you are accustomed to.
Greg
Hello,
As a follow, yesterday we created new instance turns out 2GB memory is not enough to run some of the MediaWiki jobs. Since yesterday you might have encountered errors such as:
* Access denied for user 'jenkins_mw0'@'localhost' * Disk I/O Error * File that went away mysteriously * Long delays in jobs
etc
A few minutes ago I have deleted all those 2GB slaves and spawned bigger ones. Provisioning is ongoing as I am writing. Will get them pooled in a couple/three hours.
I do not have proof: but PHP 5.5 on Trusty might have its share of issue as well when stress tests with our test suites. After all we never massively ran tests against 5.5 ;-}
cheers and be careful!
<quote name="Greg Grossmeier" date="2016-02-10" time="13:06:03 -0800">
Hello all,
I want to apologize for the current slowness in our CI stack. Everything is working fine in the sense that nothing is broken.
Quick update on this issue.
CI response times should be back to normal (or much closer to normal) after we addressed most of the issues with this yesterday (by adding more Ubuntu Trusty Jenkins slaves, the migration to Nodepool is a longer term goal).
Thank you for your understanding,
Greg
wikitech-l@lists.wikimedia.org