Thanks Tomasz for the great work. I think most 2 old backups with the latest
are more than enough. Most people use the latest as you have mentioned.
For huge dump a month window is reasonable as well but if this is pushed to
two weeks, that would be perfect but I wonder whether people can really
benefit from the dump every two weeks because the download, decompress, and
processing of some huge dumps needs a lot of time.
bilal
On Thu, May 14, 2009 at 9:02 PM, Tomasz Finc <tfinc(a)wikimedia.org> wrote:
Now that we are generating all but the biggest of
wiki's reliably I'd
like to start the discussion of retention for older data base dumps.
If we can reliably stick to a two week window for each wiki's dump
iteration, how many dumps would back would it make sense to keep?
Most clients that I've talked to only need the latest and simply look at
the older ones in case the newest dump failed a step.
If there are other retention cases then I'd love to hear them and figure
what's feasible to do.
Operations wise I'd be thinking of keeping somewhere between 1-5 of the
previous dumps and then archiving copies of each dump at 6month windows
for permanent storage. Doing that for all of the current dumps is way
more space then we have currently available but that's also why were
working on funding for those storage servers.
Is that overkill or simply not enough? let know.
--tomasz