[Labs-l] Building our ZIM farm @wmflabs

[Gabriel Wicke]

Federico,

Francium is meant as a processing and storage server. The HTML dump part is
relatively light on CPU and naturally heavy on storage. ZIM based on this
will be more CPU-consuming, but should still be doable with a single
hardware node.

The current status is that the hardware is up, but we don't have shell
access to the server yet. I hope that this can be resolved soon.

Gabriel

On Tue, Mar 31, 2015 at 12:05 AM, Federico Leva (Nemo) <nemowiki at gmail.com>
wrote:

> Any news on this?
>
> > == Next steps ==
> >
> > We want to complete our effort and mirror the biggest Wikipedia
> > projects. Unfortunately, we have reached the limits of a traditional
> > usage of wmflabs. We need more quota and to experiment with the NFS
> > storage because an x-large instance in not able to mirror more than
> > 1.5 millions of articles at a time. How might that be made possible?
>
> Per https://phabricator.wikimedia.org/T91853#1113701 and
> https://phabricator.wikimedia.org/T91853#1129560 , francium seems a
> storage server rather than a processing server: it can't be used for
> mwoffliner tasks. https://phabricator.wikimedia.org/T57503 is a separate
> task.
>         Emmanuel, I think the Labs instances are only meant to store the
> data they're currently processing, while the ZIM files as such keep ending
> up on http://download.kiwix.org/ + mirrors: right?
>         Andrew, if so the best option IMHO is to increase the local disk
> quota of (at least) one of the instances from 160 to 500 GB. Then Emmanuel
> can start testing an en.wiki run and see how it goes. virt1011, currently
> used by mwoffliner2, seems to afford it. https://ganglia.wikimedia.org/
> latest/graph_all_periods.php?c=Virtualization%20cluster%
> 20eqiad&h=virt1011.eqiad.wmnet&r=hour&z=default&jr=&js=
> &st=1427785147&v=1188.316&m=disk_free&vl=GB&ti=Disk%
> 20Space%20Available&z=large
>
> Nemo
>
>
> _______________________________________________
> Labs-l mailing list
> Labs-l at lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/labs-l
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.wikimedia.org/pipermail/labs-l/attachments/20150331/0eac8a4a/attachment.html>