Hello,
I always believed that our servers where monitored 24/7? But nobody seems to be arround to fix a core part in our systems?
Huib
---------- Forwarded message ---------- From: rupert THURNER rupert.thurner@gmail.com Date: Sat, Aug 10, 2013 at 5:24 PM Subject: [Wikimania-l] git.wikimedia.org dead due to wikimania ;) To: "Wikimania general list (open subscription)" < wikimania-l@lists.wikimedia.org>
it seems git.wikimedia.org is down due: * no volunteer has access * no paid person works on weekends * everybody else is at wikimania
rupert.
_______________________________________________ Wikimania-l mailing list Wikimania-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l
On Sat, Aug 10, 2013 at 5:31 PM, Huib Laurens sterkebak@gmail.com wrote:
Hello,
I always believed that our servers where monitored 24/7? But nobody seems to be arround to fix a core part in our systems?
Huib
---------- Forwarded message ---------- From: rupert THURNER rupert.thurner@gmail.com Date: Sat, Aug 10, 2013 at 5:24 PM Subject: [Wikimania-l] git.wikimedia.org dead due to wikimania ;) To: "Wikimania general list (open subscription)" < wikimania-l@lists.wikimedia.org>
it seems git.wikimedia.org is down due:
- no volunteer has access
- no paid person works on weekends
- everybody else is at wikimania
rupert.
Doesn't this imply that nobody is at Wikimania?
Wikimania-l mailing list Wikimania-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikimania-l
-- Met vriendelijke groet,
Huib Laurens _______________________________________________ Wikimedia-l mailing list Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l, mailto:wikimedia-l-request@lists.wikimedia.org?subject=unsubscribe
Huib Laurens, 10/08/2013 17:31:
I always believed that our servers where monitored 24/7?
Not magically, despite the amount of unicorns bought by the WMF. You're looking for https://bugzilla.wikimedia.org/show_bug.cgi?id=51983
But nobody seems to be arround to fix a core part in our systems?
No development is really prevented by gitblit being down; business as usual continues on gerrit.wikimedia.org and this additional reliability is actually what they wanted to achieve by moving gitblit (formerly gitweb) to a separate server.
Nemo
Huib Laurens wrote:
I always believed that our servers were monitored 24/7? But nobody seems to be around to fix a core part in our systems?
Hi Huib.
You've been around quite a long time, so it shouldn't be new information to you that the appropriate mailing list for an issue like this is wikitech-l (where there's already an ongoing thread), not wikimedia-l. And you should also know that for issues like this, the best place to search is Bugzilla, as both Nemo and myself have now pointed out (specifically https://bugzilla.wikimedia.org/51769).
You seem to be a bit confused about core v. non-core services. In the Wikimedia world, core functionality generally means that the wikis are accessible for, at the very minimum, read access. Peripheral services and sites have varying levels of criticality, though there's no rating system in which git.wikimedia.org (a simple Git repository viewer) would be considered core (though it being completely down can be considered critical, as it makes development work more tedious and annoying).
Obviously the operations team has a number of monitoring systems in place. The public monitoring dashboard is here: http://status.wikimedia.org.
MZMcBride
I maintained 24/7 support with a team of 6. WMF has 150 staff and does not have weekend support. The tail is wagging the dog.
We have weekend support - for core services, which constitute our MediaWiki instances. Git, however, is not a core service - as Max accurately notes, while it makes development finicky and frustrating, it doesn't impact on our core mission, which is to provide the sum of human knowledge, for everybody. It being down for a short period of time is not critical - Wikipedia being down? Well, I think we'd agree that's a pretty serious problem.
I'd like to think Engineering do a pretty good job at uptime for core services - when was the last time you saw Wikipedia down for any extended period of time? - regardless of what day of the week it is. I recall a few months ago that we had weird, inconsistent cacheing problems that were causing trouble for users - Leslie Carr, one of our Ops engineers, worked 11 hours on Friday and (iirc) 14 hours over the weekend to get it fixed. This isn't the only story in that vein - where there are critical problems to critical services, they tend to be fixed as promptly as possible, and staffers pull out all of the stops to make it happen. But it's not realistic for ops and dev/ops to maintain 24/7 uptime not just for our core services, but for every single one of the miscellaneous instances we host. If we had 150 engineers in Platform and Operations, then I might be singing a different tune, but unfortunately we don't ;).
On the other hand, if you would like to donate enough money to pay for a team of 6 to maintain our git instance 24/7, I'm sure we could set something up :P
On 12 August 2013 20:19, Richard Farmbrough richard@farmbrough.co.ukwrote:
I maintained 24/7 support with a team of 6. WMF has 150 staff and does not have weekend support. The tail is wagging the dog.
______________________________**_________________ Wikimedia-l mailing list Wikimedia-l@lists.wikimedia.**org Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/**mailman/listinfo/wikimedia-lhttps://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:wikimedia-l-request@**lists.wikimedia.orgwikimedia-l-request@lists.wikimedia.org ?subject=**unsubscribe>
A quick update I just sent to wikitech-l (where this conversation should be happening):
http://lists.wikimedia.org/pipermail/wikitech-l/2013-August/071171.html
Full text below:
----------------
As you all are aware, git.wikimedia.org (powered by GitBlit) has been unstable. The root cause is basically that there is insufficient caching mechanisms in place exacerbated by a lack of robust rules for web indexers (Google, Bing, etc).
Currently, the solution is to disallow all indexers from indexing git.wikimedia.org via the robots.txt file (https://git.wikimedia.org/robots.txt). We hope to open up indexing in a more controlled fashion soon.
For now, git.wikimedia.org should be stable for everyday use.
You can follow along with the caching developments on bug 49371: https://bugzilla.wikimedia.org/show_bug.cgi?id=49371
Thanks to Chad and Ariel, especially, who have been working to make sure this is resolved quickly and permanently.
Greg
On Tue, 13 Aug 2013, Oliver Keyes wrote:
I'd like to think Engineering do a pretty good job at uptime for core services - when was the last time you saw Wikipedia down for any extended period of time? - regardless of what day of the week it is.
Wikipedia uptime is certianly much better these days than it was around 2005/6 when there were at least two websites dedicated to reporting its status. Back then it wasn't uncommon to experience major slowdowns, periods of non-responsiveness, extended read only periods and edits failing (sometimes silently) due to (iirc) overloaded servers. I can't remember the last time I saw anything like that, but it certainly wasn't 2013.
The folks responsible for this certainly deserve kudos for their efforts.
---- Chris McKenna
cmckenna@sucs.org www.sucs.org/~cmckenna
The essential things in life are seen not with the eyes, but with the heart
Antoine de Saint Exupery
Chris McKenna, 14/08/2013 00:00:
On Tue, 13 Aug 2013, Oliver Keyes wrote:
I'd like to think Engineering do a pretty good job at uptime for core services - when was the last time you saw Wikipedia down for any extended period of time? - regardless of what day of the week it is.
Wikipedia uptime is certianly much better these days than it was around 2005/6 when there were at least two websites dedicated to reporting its status. Back then it wasn't uncommon to experience major slowdowns, periods of non-responsiveness, extended read only periods and edits failing (sometimes silently) due to (iirc) overloaded servers. I can't remember the last time I saw anything like that, but it certainly wasn't 2013.
Btw it would be nice to have stats on uptime, or at least some assessment of the progress for the "stabilizing infrastructure" 5 years strategic goal. I couldn't find any recently, I asked both on mailing lists and on Meta. https://meta.wikimedia.org/wiki/Talk:Wikimedia_budget#Stabilizing_the_infrastructure
Nemo
http://status.wikimedia.org is from external monitors
Sent from my mobile. Please excuse the brevity and typos. On Aug 14, 2013 4:44 AM, "Federico Leva (Nemo)" nemowiki@gmail.com wrote:
Chris McKenna, 14/08/2013 00:00:
On Tue, 13 Aug 2013, Oliver Keyes wrote:
I'd like to think Engineering do a pretty good job at uptime for core services - when was the last time you saw Wikipedia down for any extended period of time? - regardless of what day of the week it is.
Wikipedia uptime is certianly much better these days than it was around 2005/6 when there were at least two websites dedicated to reporting its status. Back then it wasn't uncommon to experience major slowdowns, periods of non-responsiveness, extended read only periods and edits failing (sometimes silently) due to (iirc) overloaded servers. I can't remember the last time I saw anything like that, but it certainly wasn't 2013.
Btw it would be nice to have stats on uptime, or at least some assessment of the progress for the "stabilizing infrastructure" 5 years strategic goal. I couldn't find any recently, I asked both on mailing lists and on Meta. <https://meta.wikimedia.org/**wiki/Talk:Wikimedia_budget#** Stabilizing_the_infrastructurehttps://meta.wikimedia.org/wiki/Talk:Wikimedia_budget#Stabilizing_the_infrastructure **>
Nemo
______________________________**_________________ Wikimedia-l mailing list Wikimedia-l@lists.wikimedia.**org Wikimedia-l@lists.wikimedia.org Unsubscribe: https://lists.wikimedia.org/**mailman/listinfo/wikimedia-lhttps://lists.wikimedia.org/mailman/listinfo/wikimedia-l, <mailto:wikimedia-l-request@**lists.wikimedia.orgwikimedia-l-request@lists.wikimedia.org ?subject=**unsubscribe>
Leslie Carr, 14/08/2013 22:30:
http://status.wikimedia.org is from external monitors
I don't see any annual or multi-year stats there and I don't think they monitor readon-only vs. rw state.
Nemo
On 14/08/13 07:40, Oliver Keyes wrote:
We have weekend support - for core services, which constitute our MediaWiki instances. Git, however, is not a core service - as Max accurately notes, while it makes development finicky and frustrating,
I didn't know anyone used git.wikimedia.org. I think I've only visited it once.
We're not talking about Git being down, just about one of the two web viewers being down (the other being GitHub). Web viewers of source code are a luxury -- they're handy when you're not a developer and so couldn't be bothered to clone the repository, but they're hardly a human right.
Despite that, note that Leslie Carr did work on fixing it on the weekend.
-- Tim Starling
2013/8/14 Tim Starling tstarling@wikimedia.org:
On 14/08/13 07:40, Oliver Keyes wrote:
We have weekend support - for core services, which constitute our MediaWiki instances. Git, however, is not a core service - as Max accurately notes, while it makes development finicky and frustrating,
I didn't know anyone used git.wikimedia.org. I think I've only visited it once.
I sometimes use it to point people to code, especially new volunteers who are starting to learn the code.
Other people very frequently send me git.wikimedia.org links when they are asking me about the code.
So yes, it's not as important as keeping *.wikipedia.org online, but it is definitely far from negligible.
-- Amir Elisha Aharoni · אָמִיר אֱלִישָׁע אַהֲרוֹנִי http://aharoni.wordpress.com “We're living in pieces, I want to live in peace.” – T. Moore
wikimedia-l@lists.wikimedia.org