status

List overview All Threads
Download

newer

older

an idea

Populated the spatial_ref_sys...

River Tarnell

24 Aug 2009 24 Aug '09

3:31 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 hi, the Toolserver is currently offline due to an issue with the NFS server. we are working on the issue and have a ticket open with Sun, but at this point there is no ETA. if it looks like the issue will take some time to resolve, we will restore the last /home backup onto another server and use that. however, that means losing about 12 hours worth of changes, so we will try to avoid it if possible. - river.

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (HP-UX) iEYEARECAAYFAkqSsmMACgkQIXd7fCuc5vKobACfcRyaQMlECY0tIXfTvXTH7IeH NPQAn2UcW83zd3hTnk8wRU08cvWgD9Ni =8yRk -----END PGP SIGNATURE-----

Show replies by date

Pietrodn

24 Aug 24 Aug

5:40 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Il giorno 24/ago/09, alle ore 17:31, River Tarnell ha scritto:

...

the Toolserver is currently offline due to an issue with the NFS server. we are working on the issue and have a ticket open with Sun, but at this point there is no ETA.

Sounds very bad. :-( Is the filesystem blocked both read and write? Can you please add a notice to the "404" pages, so that our tools' users know what's happening? The webserver can't serve pages so the "DaB's notice" can't be displayed...

...

if it looks like the issue will take some time to resolve, we will restore the last /home backup onto another server and use that. however, that means losing about 12 hours worth of changes, so we will try to avoid it if possible.

Thank you for your work, as always. Regards, Pietrodn powerpdn(a)gmail.com

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iEYEARECAAYFAkqS0KsACgkQ36hWg9C8ELFx6gCcDWoBJ79G9rDPNKw0aRxgLR50 w4oAn2cTHjgb8y/eGurmT3jKQX11N2sj =8Trq -----END PGP SIGNATURE-----

River Tarnell

5:50 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 River Tarnell:

...

the Toolserver is currently offline due to an issue with the NFS server. we are working on the issue and have a ticket open with Sun, but at this point there is no ETA.

update: hyacinth is now online without /home. this means stable is up, but the rest of the Toolserver is still offline. at this point we understand the issue but not the cause. i have two separate plans to restore service; preferably by recovering /home, but if that is not possible within the next few hours, i will rebuild the server and restore from the last backup, dated 2009-08-24 at 5:00AM UTC. once service is restored i will provide a post-mortem report on the issue. - river.

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (HP-UX) iEYEARECAAYFAkqS0uwACgkQIXd7fCuc5vJqGQCfa0HGhDuB0P6W6UhCmHP39y3l tyYAn2v5WGMcTwVYZxvBbV4Drb9s02t+ =TtTc -----END PGP SIGNATURE-----

Amir sarabadani

7:02 p.m.

When toolserver going on? On 8/24/09, River Tarnell <river(a)loreley.flyingparchment.org.uk> wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 River Tarnell:

the Toolserver is currently offline due to an issue with the NFS server. we are working on the issue and have a ticket open with Sun, but at this point there is no ETA.

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (HP-UX) iEYEARECAAYFAkqS0uwACgkQIXd7fCuc5vJqGQCfa0HGhDuB0P6W6UhCmHP39y3l tyYAn2v5WGMcTwVYZxvBbV4Drb9s02t+ =TtTc -----END PGP SIGNATURE-----

_______________________________________________ Toolserver-l mailing list (Toolserver-l(a)lists.wikimedia.org) https://lists.wikimedia.org/mailman/listinfo/toolserver-l Posting guidelines for this list: https://wiki.toolserver.org/view/Mailing_list_etiquette

-- امیر سرآبادانی

River Tarnell

10:05 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 after further investigation we have determined that the best course of action is to restore from a backup. i have imported the last /home backup (2009-08-24 05:00 UTC) to hemlock, and temporarily mounted this as /home, so the Toolserver is now up. i will rebuild the NFS server now (which will take a couple of hours at least) and then start migrating data back to it. there will be some further downtime while this is done. i will be making some changes to reduce issues like this in the future, which i will document later. - river.

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (HP-UX) iEYEARECAAYFAkqTDqUACgkQIXd7fCuc5vLPnACgvbJeY1eN1+YM7JjmZTf5ttb6 UiAAoL3ouML+WtkR3jmmLw74m8MBA8hW =/fjE -----END PGP SIGNATURE-----

River Tarnell

25 Aug 25 Aug

8:33 a.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 hi, i've finished repairing hyacinth (the NFS server) and i'm about to start copying /home back. this will take an hour or two during which the Toolserver will be offline again. assuming there are no problems, this should be the last downtime. - river.

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (HP-UX) iEYEARECAAYFAkqTocYACgkQIXd7fCuc5vJIbQCdExO0K9vKWQk2jccA5eLV70vG B4MAnRt4udwzuwrdg+5EsoYvkAGZI0lp =X7Hr -----END PGP SIGNATURE-----

River Tarnell

5 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 hi, the maintenance is now mostly finished and the Toolserver is back up. i'm now working on bringing stable back. - river.

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (HP-UX) iEYEARECAAYFAkqUGLsACgkQIXd7fCuc5vIzVgCfS6sXP87MByFZ7k9tAFgh0+LW D1IAoIVdhc9ZVFPFJJL6vA6v4b0bv4wb =Ih9j -----END PGP SIGNATURE-----

River Tarnell

6:31 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 hi, maintenance on stable is now finished and it should be up. the system was reinstalled, so if you notice anything odd (like missing configuration or software) please let me know. - river.

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (HP-UX) iEYEARECAAYFAkqULeQACgkQIXd7fCuc5vIVgACeJicqczYen79m1VTrF31uNDh1 pqcAoL1Bwqib9O2Os2bhDLY4u4FDNvst =1LjX -----END PGP SIGNATURE-----

emijrp

8:52 p.m.

Thanks River. I think that this isn't the correct place for this request, but since some time ago this tool[1] doesn't find commonswiki edits. Perhaps it is due to the recent commonswiki database movement. Regards [1] http://toolserver.org/~vvv/sulutil.php?user=Emijrp 2009/8/25 River Tarnell <river(a)loreley.flyingparchment.org.uk>

...

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (HP-UX) iEYEARECAAYFAkqULeQACgkQIXd7fCuc5vIVgACeJicqczYen79m1VTrF31uNDh1 pqcAoL1Bwqib9O2Os2bhDLY4u4FDNvst =1LjX -----END PGP SIGNATURE-----

DaB.

10:08 p.m.

Hello, Am Dienstag 25 August 2009 22:52:47 schrieb emijrp:

...

I think that this isn't the correct place for this request,

please contact the author of the tool (it's vvv). See [1]. Sincerly, DaB. [1] https://wiki.toolserver.org/view/If_you_found_a_bug -- wp-blog.de

Peter Körner

9:57 p.m.

...

the maintenance is now mostly finished and the Toolserver is back up. i'm now working on bringing stable back.

could you please take a look at cassini? I'm unable to login and also [1] seems to be not available. Both may be a problem with the /home server, i think. Thank you! Peter

River Tarnell

10 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Peter Körner:

...

could you please take a look at cassini? I'm unable to login and also [1] seems to be not available.

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (HP-UX) iEYEARECAAYFAkqUXvkACgkQIXd7fCuc5vIxRwCgqE9UDT/X9O+DAzfpRAQugR2x z74An0CM8zBKwGAihFmlyadAxMbbsHPj =mc+Z -----END PGP SIGNATURE-----

Peter Körner

10:02 p.m.

...

Peter Körner:

could you please take a look at cassini? I'm unable to login and also [1] seems to be not available.

Oh., yay, that's like forgetting an attachment on a mail :) It should have been referring to http://cassini.toolserver.org/~mazder/ which is up again now, too. Thank you! Peter

Peter Körner

24 Aug 24 Aug

7:39 p.m.

hi river, i'm sorry to hear that you got so much trouble with the servers, the replication and all those other things in the past few weeks. I can imagine how annoying this is. If i'd live near you, i'll send you some beer/wine/cola/coffee or what ever you'd prefer, to keep you up. But unfortunately it's a long way from germany so I'm just able to pat you on the back. Thank you! Peter

River Tarnell

25 Aug 25 Aug

10:46 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 for those who don't read journal.ts.o, a write-up of the outage is available at <https://confluence.toolserver.org/display/tech/Platform+outage+2009-08-24> - river.

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (HP-UX) iEYEARECAAYFAkqUadkACgkQIXd7fCuc5vLtMQCguYECbndpyibJU5Lu6KaYLoX0 c4YAnRubVoGLKzEP3E2wZ6dfYGovI0ua =GVZd -----END PGP SIGNATURE-----

Shaurabh Bharti

26 Aug 26 Aug

1:07 a.m.

Hullo River, Thanks for bringing the server back. I could not access https://phpmyadmin.toolserver.org/index.php though. Would be great if you could fix it. -- Regards, Shaurabh Bharti http://infosysblogs.com/web2 http://sbharti.blogspot.com On Wed, Aug 26, 2009 at 4:16 AM, River Tarnell < river(a)loreley.flyingparchment.org.uk> wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 for those who don't read journal.ts.o, a write-up of the outage is available at < https://confluence.toolserver.org/display/tech/Platform+outage+2009-08-24&g… - river.

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (HP-UX) iEYEARECAAYFAkqUadkACgkQIXd7fCuc5vLtMQCguYECbndpyibJU5Lu6KaYLoX0 c4YAnRubVoGLKzEP3E2wZ6dfYGovI0ua =GVZd -----END PGP SIGNATURE-----

River Tarnell

4:54 a.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Shaurabh Bharti:

...

I could not access https://phpmyadmin.toolserver.org/index.php though. Would be great if you could fix it.

this is fixed now, but in future please use JIRA (https://jira.toolserver.org) to report issues. - river.

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (HP-UX) iEYEARECAAYFAkqUv/MACgkQIXd7fCuc5vId1wCfRPwgDv270+68BMeb2aBn5+Eg iuIAnjtgO67ikChiruZtI4LnGDMEh3FG =AOyy -----END PGP SIGNATURE-----

Platonides

10:46 p.m.

River Tarnell wrote:

...

for those who don't read journal.ts.o, a write-up of the outage is available at <https://confluence.toolserver.org/display/tech/Platform+outage+2009-08-24> - river.

...

This issue also highlighted the problems of having the NFS server as a single point of failure. We already have a plan to make NFS redundant, but this will need to

wait until next year until

...

funds are available to implement it.

How are you planning to make NFS redundant?

River Tarnell

11:01 p.m.

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Platonides:

...

How are you planning to make NFS redundant?

we plan to use a disk array for storage and two head nodes to provide NFS access. the array has two controllers which provide redundant service to the host (active/passive). each host is connected to both controllers (4 cables in total), redundantly to each other, and redundantly to the network. each head will run Solaris with Sun Cluster HA-NFS software to provide active/passive NFS service to clients with transparent failover. this allows us to perform maintenance on the array and either NFS head without affecting service, and in case or hardware or software failure of the active head, the system will failover to the passive head. - river.

...PGP SIGNATURE...

-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (HP-UX) iEYEARECAAYFAkqVvuUACgkQIXd7fCuc5vJRzQCeJVjXyFaCZR7z6ZYS6WAycpRE +4AAnRPSQLO5goTVfQM8VYHPz9c4CNhi =Ag2Q -----END PGP SIGNATURE-----

5370

days inactive

5372

days old

toolserver-l@lists.wikimedia.org

Manage subscription

18 comments

8 participants

tags (0)

participants (8)

Amir sarabadani
DaB.
emijrp
Peter Körner
Pietrodn
Platonides
River Tarnell
Shaurabh Bharti