[Foundation-l] Wikimedia mirrors

[George Herbert]

On Thu, Sep 16, 2010 at 12:58 AM, John Vandenberg <jayvdb at gmail.com> wrote:
> On Thu, Sep 16, 2010 at 5:40 PM, Federico Leva (Nemo)
> <nemowiki at gmail.com> wrote:
>> John Vandenberg, 16/09/2010 03:00:
>>> English, French, German, Italian, Polish, Portugeuse, Swedish and
>>> Chinese Wikipedia all appear to have some mirrors, but are any of them
>>> reliable enough to be used for disaster recovery?
>>
>> Obviously not, at least Italian ones.
>>
>>> The smaller projects are easier to backup, as they are smaller.  I am
>>> sure that with a little effort and coordination, chapters,
>>> universities and similar organisations would be willing to routinely
>>> backup a subset of projects, and combined we would have multiple
>>> current backups of all projects.
>>
>> I agree. Now we have only this:
>> http://www.balkaninsight.com/en/main/news/21606/
>
> Kudos to Milos & Wikimedia Serbia!!
>
>> How many TB are needed? I don't know what's the average, but e.g. right
>> now my university should have about 50 TB of free disk space (which is
>> not so much, after all).
>
> The key would be to allow the mirrors to delete their mirror when they
> need to use their excess storage capability.  If they let us know in
> advance that they are reclaiming the space, another organisation with
> excess storage capability can take over.


I appreciate all the enthusiasm in thread, but (speaking for myself as
an individual, and IT consultant who does things like business
continuity and disaster recovery planning consulting among other
infrastructure work) this is a core operational competency role that
the Foundation needs to ensure is handled in house as part of the
routine IT operations.  And, as I understand it now, it is, though I
have only had high level discussions with some of the Foundation staff
about this and not seen the server configs myself so I can't
personally attest to the status.

Database and file backups need to be in (at least) 2 locations, and my
understanding is that there are complete redundant copies at the
Amsterdam datacenter now, and that the new main datacenter in Virginia
will continue this.

If a third location is needed, the current HQ in San Francisco is
plenty far enough away from the other 2 locations to provide excellent
DR capability.  If there's need for a datacenter / fast net access
redundant copy in SF or the Bay Area, a rack or few U of a shared rack
would be enough for a fileserver, and that's available at multiple
excellently connected locations in the Bay Area.

Disaster Recovery is not something the Foundation should attempt to
crowdsource.  I recommend it be left to professionals whose job it is
and who have prior experience in the field.  If you haven't watched
major services drop, datacenters burn down, software environments melt
down, and spent years working to ensure that those don't happen again,
you really don't have a good feel for the type and magnitude of the
risks and the sorts of tools to employ to try and mitigate them.

If there's interest in an offline discussion on IT disasters and
disaster recovery and reliability engineering, I can do that, but it
should be offline from Foundation-L...


-- 
-george william herbert
george.herbert at gmail.com