A series of upgrades and changes have left instances with
'role::puppetmaster::standalone' applied in a broken state. This is
unfortunate because Puppet is unable to fix itself. There is a small
manual update required.
On the project specific puppet master (instance with
role::puppetmaster:;standalone applied):
0. Make sure your copy of the operation puppet repo is current at
/var/lib/git/operations/puppet
1. edit /etc/puppet/puppet.conf to replace 'default_manifest =
$confdir/manifests/site.pp' with 'default_manifest = $confdir/manifests/'
2. restart apache (service apache2 restart)
3. puppet agent --test
Puppet should now able to fix itself and clients should run fine. Places
where this role is applied can be found at
https://tools.wmflabs.org/openstack-browser/puppetclass/role::puppetmaster:…
--
Chase Pettet
chasemp on phabricator <https://phabricator.wikimedia.org/p/chasemp/> and
IRC
There is a bit of overload [0] on labsdb1003 right now; that is normal
considering there is only 1 server left of the original 3. However, it is
now a good opportunity to start using the new servers [1] so your queries
can be 10x faster!
Another tip is, if you have regularly executed queries (for example, in a
cron), to setup some locking mechanism so one makes sure the same query is
not running multiple times on at the same time (e.g. if for some reasons
they become slower than usual).
Please ask questions if you have them about how to do any of these.
Regards,
[0] <url:
https://grafana.wikimedia.org/dashboard/file/server-board.json?panelId=12&f…
>
[1] <url:
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#Connecting_to_t…
>
--
Jaime Crespo
<http://wikimedia.org>
Sorry for cross-posting!
Reminder: Technical Advice IRC meeting again **tomorrow, Wednesday 4-5 pm
UTC** on #wikimedia-tech.
The Technical Advice IRC meeting is open for all volunteer developers,
topics and questions. This can be anything from "how to get started" over
"who would be the best contact for X" to specific questions on your project.
If you know already what you would like to discuss or ask, please add your
topic to the next meeting: https://www.mediawiki.org/wiki
/Technical_Advice_IRC_Meeting
This meeting is an offer by WMDE’s tech team. Hosts of tomorrows meeting
are: @Thiemo_WMDE & @Amir1.
Hope to see you there!
Michi (for WMDE’s tech team)
--
Michael F. Schönitzer
Wikimedia Deutschland e.V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Tel. (030) 219 158 26-0
http://wikimedia.de
Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen
Wissens frei teilhaben kann. Helfen Sie uns dabei!
http://spenden.wikimedia.de/
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/681/51985.
Sorry for cross-posting!
Reminder: Technical Advice IRC meeting again **tomorrow, Wednesday 4-5 pm
UTC** on #wikimedia-tech.
The Technical Advice IRC meeting is open for all volunteer developers,
topics and questions. This can be anything from "how to get started" over
"who would be the best contact for X" to specific questions on your project.
If you know already what you would like to discuss or ask, please add your
topic to the next meeting: https://www.mediawiki.org/
wiki/Technical_Advice_IRC_Meeting
This meeting is an offer by WMDE’s tech team. Hosts of tomorrows meeting
are: @Lucas_WMDE & @Thiemo_WMDE.
Hope to see you there!
Michi (for WMDE’s tech team)
--
Michael F. Schönitzer
Wikimedia Deutschland e.V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Tel. (030) 219 158 26-0
http://wikimedia.de
Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen
Wissens frei teilhaben kann. Helfen Sie uns dabei!
http://spenden.wikimedia.de/
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/681/51985.
As discussed previously in this list [1] and on phabricator [2], I've
just removed the Ubuntu Trusty image as a default option when creating
new VMs. This is part of a longterm foundation-wide process to
standardize on Debian as the distribution of choice.
Existing Trusty VMs are unaffected by this change, as are present
ToolForge workflows. WMCS operators still have the ability to create
Trusty VMs in a pinch, so if you need one please create a phabricator
task with an explanation of what you need and why and we'll create it as
soon as we're able.
-Andrew
[1] https://lists.wikimedia.org/pipermail/cloud/2017-October/000056.html
[2] https://phabricator.wikimedia.org/T161899
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
Sorry for cross-posting!
Reminder: Technical Advice IRC meeting again **tomorrow, Wednesday 4-5 pm
UTC** on #wikimedia-tech.
The Technical Advice IRC meeting is open for all volunteer developers,
topics and questions. This can be anything from "how to get started" over
"who would be the best contact for X" to specific questions on your project.
If you know already what you would like to discuss or ask, please add your
topic to the next meeting: https://www.mediawiki.org/wiki/Technical_
Advice_IRC_Meeting
This meeting is an offer by WMDE’s tech team. Hosts of tomorrows meeting
are: @addshore & @CFisch_WMDE.
Hope to see you there!
Michi (for WMDE’s tech team)
--
Michael F. Schönitzer
Wikimedia Deutschland e.V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Tel. (030) 219 158 26-0
http://wikimedia.de
Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen
Wissens frei teilhaben kann. Helfen Sie uns dabei!
http://spenden.wikimedia.de/
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/681/51985.
TL;DR:
* c1.labsdb (labsdb1001.eqiad.wmnet) is down due to hardware issues
* *.labsdb are pointing to c3.labsdb (labsdb1003.eqiad.wmnet)
The physical server behind c1.labsdb (labsdb1001.eqiad.wmnet)
experienced a hard drive failure around 2017-11-01T03:30 UTC. This
failure is preventing the MySQL service on that host from starting.
The *.labsdb service names that were pointed at that server have been
updated to point to c3.labsdb (labsdb1003.eqiad.wmnet) instead.
See <https://phabricator.wikimedia.org/T179464> for more information
and additional updates.
Expect slower than normal performance as all traffic is handled by a
single server. Now would be a great time to update the configuration
for your tools to use the new database cluster [0][1].
[0]: https://phabricator.wikimedia.org/phame/post/view/70/new_wiki_replica_serve…
[1]: https://wikitech.wikimedia.org/wiki/Wiki_Replica_c1_and_c3_shutdown
Bryan
--
Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org>
[[m:User:BDavis_(WMF)]] Manager, Cloud Services Boise, ID USA
irc: bd808 v:415.839.6885 x6855
_______________________________________________
Wikimedia Cloud Services announce mailing list
Cloud-announce(a)lists.wikimedia.org (formerly labs-announce(a)lists.wikimedia.org)
https://lists.wikimedia.org/mailman/listinfo/cloud-announce
Sorry for cross-posting!
Reminder: Technical Advice IRC meeting again **tomorrow 4-5 pm UTC** on
#wikimedia-tech.
The Technical Advice IRC meeting is open for all volunteer developers,
topics and questions. This can be anything from "how to get started" over
"who would be the best contact for X" to specific questions on your project.
If you know already what you would like to discuss or ask, please add your
topic to the next meeting:
https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting
This meeting is an offer by WMDE’s tech team. Hosts of tomorrows meeting
are: @addshore & @CFisch_WMDE.
Hope to see you there!
Michi (for WMDE’s tech team)
--
Michael F. Schönitzer
Wikimedia Deutschland e.V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Tel. (030) 219 158 26-0
http://wikimedia.de
Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen
Wissens frei teilhaben kann. Helfen Sie uns dabei!
http://spenden.wikimedia.de/
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/681/51985.
Sorry for cross-posting!
Reminder: Technical Advice IRC meeting again **today 3-4 pm UTC** on
#wikimedia-tech.
The Technical Advice IRC meeting is open for all volunteer developers,
topics and questions. This can be anything from "how to get started" over
"who would be the best contact for X" to specific questions on your project.
If you know already what you would like to discuss or ask, please add your
topic to the next meeting: https://www.mediawiki.org/wiki/Technical_
Advice_IRC_Meeting
This meeting is an offer by WMDE’s tech team. Hosts of todays meeting are:
@addshore & @CFisch_WMDE.
Hope to see you there!
Michi (for WMDE’s tech team)
--
Michael F. Schönitzer
Wikimedia Deutschland e.V. | Tempelhofer Ufer 23-24 | 10963 Berlin
Tel. (030) 219 158 26-0
http://wikimedia.de
Stellen Sie sich eine Welt vor, in der jeder Mensch an der Menge allen
Wissens frei teilhaben kann. Helfen Sie uns dabei!
http://spenden.wikimedia.de/
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/681/51985.
The Cloud Services team has two positions open that might be
interesting to some members of our technical community. We are
especially hoping to find someone from our existing community of
technical contributors for the Technical Support position.
== Cloud Services Technical Support (Contract) ==
Type: Contract
Hours: 20 hours per week
Duration: 3 months (with possibility for extension)
The Wikimedia Cloud Services team is looking for a part-time Technical
Support specialist to help the Wikimedia movement's volunteer
developers to rapidly clarify and resolve their technical questions
and issues.
Responsibilities:
* Actively monitor various IRC channels, wikipages, and mailing lists,
for support requests
* Help volunteer developers to diagnose and resolve technical issues
with our IaaS and PaaS infrastructure and their code
* Communicate clearly and patiently with individuals from our
international communities of volunteer developers
* Write, verify, and improve documentation used by volunteer
developers and technical staff
* Ensure positive and constructive discussions with the community and
the Wikimedia Foundation
* Work and communicate effectively within a small team distributed
across multiple time zones
Full job posting: <http://grnh.se/h5gjvl1>
== Operations Engineer (Cloud Services) ==
Responsibilities:
* Perform day-to-day operational tasks on Wikimedia’s Cloud Services
infrastructure (deployment, maintenance, configuration,
troubleshooting)
* Support volunteer and staff developers using Infrastructure as a
Service (IaaS) and Platform as a Service (PaaS) products
* Assist in the architectural design of new services and making them
operate at scale
* Assist in or lead incident response, diagnosis and followup on
system outages or alerts across our stack
* On-call support and off-hours coverage in a 24x7 environment
Full job posting: <http://grnh.se/1xb8om1>
If you are interested in either job, please read the full job postings
and apply online. Feel free to share the links with people you may
know as well via your favorite social media platforms, email, or just
old fashioned voice communications. :) You can find these and other
jobs with the Wikimedia Foundation at
<https://wikimediafoundation.org/wiki/Work_with_us>.
Bryan
--
Bryan Davis Wikimedia Foundation <bd808(a)wikimedia.org>
[[m:User:BDavis_(WMF)]] Manager, Cloud Services Boise, ID USA
irc: bd808 v:415.839.6885 x6855