As noted in the server admin log [1], Phabricator is currently down due to a network outage impacting one of our racks in the Ashburn data-center. We're investigating and will aim to restore service ASAP.
Erik
[1] https://wikitech.wikimedia.org/wiki/Server_Admin_Log
"ASAP"? when it's already hitting approx. five hours of down time?
On 30 November 2014 at 18:14, Erik Moeller erik@wikimedia.org wrote:
As noted in the server admin log [1], Phabricator is currently down due to a network outage impacting one of our racks in the Ashburn data-center. We're investigating and will aim to restore service ASAP.
Erik
[1] https://wikitech.wikimedia.org/wiki/Server_Admin_Log
-- Erik Möller VP of Product & Strategy, Wikimedia Foundation _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Hoi, Right ? so it is thanksgiving et al.. Be thankful that it is seen, It is not Wikipedia or any of the projects...so relax.. eat some left over turkey.. Thanks, GerardM
On 30 November 2014 at 09:59, K. Peachey p858snake@gmail.com wrote:
"ASAP"? when it's already hitting approx. five hours of down time?
On 30 November 2014 at 18:14, Erik Moeller erik@wikimedia.org wrote:
As noted in the server admin log [1], Phabricator is currently down due
to
a network outage impacting one of our racks in the Ashburn data-center. We're investigating and will aim to restore service ASAP.
Erik
[1] https://wikitech.wikimedia.org/wiki/Server_Admin_Log
-- Erik Möller VP of Product & Strategy, Wikimedia Foundation _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
This last line in the conversation strikes me as dry, useless and submissive. We should not ever "neglect" a sister project (including wikimedia projects, phabricator, wikitech, tools, etc), as small as their userbase might seem. For many, weekends are the volunteering or coding time and if the website they use for it was off, such people would be frustrated.
Would be interesting to see how to set up multi-server instance of FAB. http://blog.iweb.com/en/2012/02/how-to-distribute-website-load-across-multip... What I don't understand is how to decentralise the database.
Happy Thanksgiving to all, of course...
-- svetlana
On Sun, 30 Nov 2014, at 20:13, Gerard Meijssen wrote:
Hoi, Right ? so it is thanksgiving et al.. Be thankful that it is seen, It is not Wikipedia or any of the projects...so relax.. eat some left over turkey.. Thanks, GerardM
On 30 November 2014 at 09:59, K. Peachey p858snake@gmail.com wrote:
"ASAP"? when it's already hitting approx. five hours of down time?
On 30 November 2014 at 18:14, Erik Moeller erik@wikimedia.org wrote:
As noted in the server admin log [1], Phabricator is currently down due
to
a network outage impacting one of our racks in the Ashburn data-center. We're investigating and will aim to restore service ASAP.
Erik
[1] https://wikitech.wikimedia.org/wiki/Server_Admin_Log
-- Erik Möller VP of Product & Strategy, Wikimedia Foundation _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Thanksgiving is only celebrated at this time in the US. Many of us dont celebrate it.
That said downtime happens, and its a non-essential service during non working hours. Well it may be frustrating, its not the end of the world. If anyone is despretely looking for a bug to fix, they can ask on irc, im sure the regulars can think of a hundred bugs off the top of their head.
I dont think Peachy was grumbling so much as looking for an accurate time frame for the solution.
Re selveta's comment about distributing the db: there is standard ways of doing that (e.g. simplest would be to just use db replication, and switch master on failure), im not sure if phab is important enough to warrant that. I would probably lean to no it isnt personally. Obviously that would be an operations call.
--bawolff On Nov 30, 2014 5:13 AM, "Gerard Meijssen" gerard.meijssen@gmail.com wrote:
Hoi, Right ? so it is thanksgiving et al.. Be thankful that it is seen, It is not Wikipedia or any of the projects...so relax.. eat some left over turkey.. Thanks, GerardM
On 30 November 2014 at 09:59, K. Peachey p858snake@gmail.com wrote:
"ASAP"? when it's already hitting approx. five hours of down time?
On 30 November 2014 at 18:14, Erik Moeller erik@wikimedia.org wrote:
As noted in the server admin log [1], Phabricator is currently down
due
to
a network outage impacting one of our racks in the Ashburn
data-center.
We're investigating and will aim to restore service ASAP.
Erik
[1] https://wikitech.wikimedia.org/wiki/Server_Admin_Log
-- Erik Möller VP of Product & Strategy, Wikimedia Foundation _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Hoi, The argument about "non-working" hours is problematic. When the only thing that counts are the working hours of staff in the USA you may be right. As it is, WIkimedia Germany has staff working at other times and they are affected. Affected are the non-professionals as well..
My advise, do not go there. It is broken and it needs fixing. Thanks, GerardM
On 30 November 2014 at 19:02, Brian Wolff bawolff@gmail.com wrote:
Thanksgiving is only celebrated at this time in the US. Many of us dont celebrate it.
That said downtime happens, and its a non-essential service during non working hours. Well it may be frustrating, its not the end of the world. If anyone is despretely looking for a bug to fix, they can ask on irc, im sure the regulars can think of a hundred bugs off the top of their head.
I dont think Peachy was grumbling so much as looking for an accurate time frame for the solution.
Re selveta's comment about distributing the db: there is standard ways of doing that (e.g. simplest would be to just use db replication, and switch master on failure), im not sure if phab is important enough to warrant that. I would probably lean to no it isnt personally. Obviously that would be an operations call.
--bawolff On Nov 30, 2014 5:13 AM, "Gerard Meijssen" gerard.meijssen@gmail.com wrote:
Hoi, Right ? so it is thanksgiving et al.. Be thankful that it is seen, It is not Wikipedia or any of the projects...so relax.. eat some left over turkey.. Thanks, GerardM
On 30 November 2014 at 09:59, K. Peachey p858snake@gmail.com wrote:
"ASAP"? when it's already hitting approx. five hours of down time?
On 30 November 2014 at 18:14, Erik Moeller erik@wikimedia.org wrote:
As noted in the server admin log [1], Phabricator is currently down
due
to
a network outage impacting one of our racks in the Ashburn
data-center.
We're investigating and will aim to restore service ASAP.
Erik
[1] https://wikitech.wikimedia.org/wiki/Server_Admin_Log
-- Erik Möller VP of Product & Strategy, Wikimedia Foundation _______________________________________________ Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
Wikitech-l mailing list Wikitech-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l
On Nov 30, 2014 2:07 PM, "Gerard Meijssen" gerard.meijssen@gmail.com wrote:
Hoi, The argument about "non-working" hours is problematic. When the only thing that counts are the working hours of staff in the USA you may be right. As it is, WIkimedia Germany has staff working at other times and they are affected. Affected are the non-professionals as well..
My advise, do not go there. It is broken and it needs fixing. Thanks, GerardM
I just simply meant that it didnt happen in the middle of the normal day of work for the people responsible for fixing it (not neccesarily the people affected), so there is probably going to be less relavent people around (given its a weekend and a US holiday, although i imagine there are still people "on-call") hence we should be gracious with our expectations. I in no way meant to suggest it shouldnt be fixed or that it shouldnt be fixed quickly (in fact i was trying to argue against the "go eat turkey" setiment)
I didnt think WM-DE had people working on satutdays... but volunteers certainly do work on saturdays and were affected.
--bawolff
Hi, let me recycle this reply posted initially at "Determine phabricator.wikimedia.org service level" - https://phabricator.wikimedia.org/T76381
Currently Phabricator is getting the same service level that Bugzilla had. Looking at the whole Wikimedia picture, I think this is the most sensible option. I don't see any strong reason to change it.
Bugzilla was down unexpectedly several times in the past years, and if Ops was able to react quicker it's just because we were luckier with the cause, timing and location of the breaks. If we would have Bugzilla instead of Phabricator in the rack that went down this weekend, the service provided by Ops would have been exactly the same.
We can reopen this discussion when planning the migration of code review and (eventually) continuous integration. For now, I think we are good. This is the opinion of the Engineering Community team. If this works also for Operations and Platform Engineering, then we can resolve this task.
PS: About the downtime itself, 5 hours on a weekend is clearly unfortunate, but imho nothing that should make us revise the current service level. Was anybody unable to work, arms crossed? Was any project delayed? I'm counting volunteers as much as employees. Personally I learned about the downtime only in wikitech-l, having used Phabricator on Saturday-Sunday night at 1am CET, and then on Sunday at 1pm.
wikitech-l@lists.wikimedia.org