Outage info

List overview All Threads
Download

newer

older

Re: [Wikitech-l] [MediaWiki-CVS]...

<noinclude>{{:Page}}</noinclude>...

Brion Vibber

18 Mar 2008 18 Mar '08

1:29 a.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 We had some sort of network outage starting a bit after midnight UTC; heavy packet loss (33%-39% range) was visible on a number of networks, and the Tampa servers became difficult to reach for a time. Additionally, there was some DST packet overload on our primary load balancer, making the wikis inaccessible through it. Packet loss was largely reduced by about 01:05 UTC; some poking at the load balancer apparently reset things to where it was happy with the network again by around 01:18 UTC. Sites are accessible and seem reasonably happy at this time. - -- brion vibber (brion @ wikimedia.org)

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkffGwsACgkQwRnhpk1wk47FGQCg0zfrzTU4A8UTq3dKMhu+NDN2 0c8An01aTNr+oWq+M4/m66PVRTC+YJmc =DaL1 -----END PGP SIGNATURE-----

Show replies by date

Jay R. Ashworth

18 Mar 18 Mar

1:32 a.m.

On Mon, Mar 17, 2008 at 06:29:47PM -0700, Brion Vibber wrote:

...

Sites are accessible and seem reasonably happy at this time.

Don't anthropomorphize computers... they *hate* that. :-) And thanks to whomever did those rack-diagram tables on wikitech -- I just stole them for my new job. Leuksman's SSL cert pukes on my Firefox, BTW; is it self-signed? Cheers, -- jra -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com '87 e24 St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274 Those who cast the vote decide nothing. Those who count the vote decide everything. -- (Joseph Stalin)

Brion Vibber

1:35 a.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jay R. Ashworth wrote:

...

On Mon, Mar 17, 2008 at 06:29:47PM -0700, Brion Vibber wrote:

Sites are accessible and seem reasonably happy at this time.

It's self-signed and probably installed wrong. ;) Note that Firefox 3 is *very* pissy about certs it doesn't like, but you can poke in an exception if you follow the directions. :) - -- brion

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkffHGsACgkQwRnhpk1wk44dQQCgkfHgev4K9ZGBP073D4xZx7E4 tRAAn3DSqffF7GCaKc79PHd+FXXMUIWa =ce5X -----END PGP SIGNATURE-----

Jay R. Ashworth

1:40 a.m.

On Mon, Mar 17, 2008 at 06:35:39PM -0700, Brion Vibber wrote:

...

And thanks to whomever did those rack-diagram tables on wikitech -- I just stole them for my new job. Leuksman's SSL cert pukes on my Firefox, BTW; is it self-signed?

It's self-signed and probably installed wrong. ;)

I *think* the machine I was on might have had 1.0.7 (one of the two did), but it accepted my override just fine.

...

Note that Firefox 3 is *very* pissy about certs it doesn't like, but you can poke in an exception if you follow the directions. :)

I was *just* about to install b4 on my SuSE box; we'll see how that goes. 1.11 installed like a champ, BTW: I have two notes on the installer script; who should they go to? BZ? A person? Cheers, -- jra -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com '87 e24 St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274 Those who cast the vote decide nothing. Those who count the vote decide everything. -- (Joseph Stalin)

Brion Vibber

1:48 a.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jay R. Ashworth wrote:

...

On Mon, Mar 17, 2008 at 06:35:39PM -0700, Brion Vibber wrote:

And thanks to whomever did those rack-diagram tables on wikitech -- I just stole them for my new job. Leuksman's SSL cert pukes on my Firefox, BTW; is it self-signed?

It's self-signed and probably installed wrong. ;)

I *think* the machine I was on might have had 1.0.7 (one of the two did), but it accepted my override just fine.

Note that Firefox 3 is *very* pissy about certs it doesn't like, but you can poke in an exception if you follow the directions. :)

I was *just* about to install b4 on my SuSE box; we'll see how that goes. 1.11 installed like a champ, BTW: I have two notes on the installer script; who should they go to? BZ? A person?

Bugzilla would be ideal; people work better when they've got a tracker to refer to. :D - -- brion

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkffH4YACgkQwRnhpk1wk44ucQCeNd0yal8yqy1zqxz1OXzS42hC 8MgAoIaiixyVK0AcMMYiZWmVPP/1Vy34 =mMky -----END PGP SIGNATURE-----

Jay R. Ashworth

2:35 a.m.

On Mon, Mar 17, 2008 at 06:48:54PM -0700, Brion Vibber wrote:

...

1.11 installed like a champ, BTW: I have two notes on the installer script; who should they go to? BZ? A person?

Bugzilla would be ideal; people work better when they've got a tracker to refer to. :D

https://bugzilla.wikimedia.org/show_bug.cgi?id=13409 https://bugzilla.wikimedia.org/show_bug.cgi?id=13410 Cheers, -- jra -- Jay R. Ashworth Baylink jra(a)baylink.com Designer The Things I Think RFC 2100 Ashworth & Associates http://baylink.pitas.com '87 e24 St Petersburg FL USA http://photo.imageinc.us +1 727 647 1274 Those who cast the vote decide nothing. Those who count the vote decide everything. -- (Joseph Stalin)

David Gerard

10:25 a.m.

On 18/03/2008, Jay R. Ashworth <jra(a)baylink.com> wrote:

...

On Mon, Mar 17, 2008 at 06:35:39PM -0700, Brion Vibber wrote:

...

> Note that Firefox 3 is *very* pissy about certs it doesn't like, but you > can poke in an exception if you follow the directions. :)

...

I was *just* about to install b4 on my SuSE box; we'll see how that goes.

I'm running Minefield (FF3 nightlies) on Windows and Linux at present; I heartily recommend it to all. Faster, smaller, better in every way. It really does get very pissy about non-Verisigned certs, though. - d.

James R.

1:33 a.m.

Thanks to all the dev's, especially brion, for your fast responses. Teamwork is the key :-) -j. E Administrator - English Wikipedia Online Team Member - Wikimania 2008 Email: e.wikipedia(a)gmail.com Your continued donations keep Wikipedia running! Support the Wikimedia Foundation today. Visit http://donate.wikimedia.org/ -------------------------------------------------- From: "Brion Vibber" <brion(a)wikimedia.org> Sent: Tuesday, March 18, 2008 11:29 AM To: "Wikimedia developers" <wikitech-l(a)lists.wikimedia.org> Subject: [Wikitech-l] Outage info

...

We had some sort of network outage starting a bit after midnight UTC; heavy packet loss (33%-39% range) was visible on a number of networks, and the Tampa servers became difficult to reach for a time. Additionally, there was some DST packet overload on our primary load balancer, making the wikis inaccessible through it. Packet loss was largely reduced by about 01:05 UTC; some poking at the load balancer apparently reset things to where it was happy with the network again by around 01:18 UTC. Sites are accessible and seem reasonably happy at this time. - -- brion vibber (brion @ wikimedia.org) _______________________________________________ Wikitech-l mailing list Wikitech-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikitech-l

David Gerard

4:30 p.m.

On 18/03/2008, Brion Vibber <brion(a)wikimedia.org> wrote:

...

Someone suggests we pissed off the operators of the Storm botnet: http://nailo19.wordpress.com/2008/03/17/wikipedia-hacked/ [[:en:Storm botnet]] was the Featured Article on Saturday ... - d.

Simetrical

9:22 p.m.

On Tue, Mar 18, 2008 at 12:30 PM, David Gerard <dgerard(a)gmail.com> wrote:

...

Someone suggests we pissed off the operators of the Storm botnet: http://nailo19.wordpress.com/2008/03/17/wikipedia-hacked/ [[:en:Storm botnet]] was the Featured Article on Saturday ...

It affected a lot more than Wikipedia (although, any effective DDoS against Wikipedia would do that). But really, Wikipedia gets 50,000 req/s peak. The Wikipedia article gives estimates of the Storm botnet size that range from 160,000 to 50 million. Even 10,000 computers sending a continuous stream of requests could manage hundreds of times normal request load. If Storm wanted Wikipedia down, they could have done a damn sight better than they did. And what would they hope to accomplish? I imagine it's just a coincidence.

5906

days inactive

5906

days old

wikitech-l@lists.wikimedia.org

Manage subscription

9 comments

5 participants

tags (0)

participants (5)

Brion Vibber
David Gerard
James R.
Jay R. Ashworth
Simetrical