There are three things going on here:
1) Automatic updating of the maxmind IP4 database -- Ottomata created many many moons ago a wonderful puppet module that does this.

2) Automatic updating of the maxmind IP6 database -- Ottomata's puppet module does not do this because Maxmind does not sell IP6 databases, it was never in scope of the puppet module to automatically update this. Christian's bug report https://bugzilla.wikimedia.org/show_bug.cgi?id=54191 refers to this but it should not be a bug; at best it's an infrastructure task. 

3) Copying of all the maxmind databases to HDFS. The problem that we face here is that even though the Maxmind IP4 database was updated on the local fs, it was not copied to HDFS. We never spent time to figure out a way to make jobs easily reproducible and how to version the different maxmind databases; ottomata has some thoughts about this with symlinking. Bug https://bugzilla.wikimedia.org/show_bug.cgi?id=53762 refers to this problem.


So: bug 54191 and 53762 do not depend on each other. Bug 54191 is not really a bug, once we buy a license we can add this to the puppet module.

An idea: let the PM manage the bugs :)

D




On Mon, Sep 16, 2013 at 4:43 PM, Toby Negrin <tnegrin@wikimedia.org> wrote:
I'm confused -- did you just fix this?


On Mon, Sep 16, 2013 at 4:40 PM, Andrew Otto <otto@wikimedia.org> wrote:
We do update the v4 database.  Not sure how often Maxmind update's their DBs, but puppet will run the geoipupdate script weekly.  The DBs get distributed to puppet client agents whenever the agent runs (roughly every 30 minutes).




root@stafford:~# tail -n 10 /var/log/geoipupdate.log
Sun Jul 14 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP
Sun Jul 21 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP
Sun Jul 28 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP
Sun Aug  4 03:30:02 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP
Sun Aug 11 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP
Sun Aug 18 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP
Sun Aug 25 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP
Sun Sep  1 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP
Sun Sep  8 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP
Sun Sep 15 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP
root@stafford:~# ls -l /var/lib/puppet/volatile/GeoIP
total 47300
-rw-r--r-- 1 root root 40128176 Sep 15 03:30 GeoIPCity.dat
-rw-r--r-- 1 root root   946656 Sep 15 03:30 GeoIP.dat
-rw-r--r-- 1 root root  7355209 Sep 15 03:30 GeoIPRegion.dat




On Sep 16, 2013, at 4:37 PM, Toby Negrin <tnegrin@wikimedia.org> wrote:

AFAIK we don't update the V4 database either. When we fix this, we should fix both. I'll link, but not dup the bugs.


On Mon, Sep 16, 2013 at 4:02 PM, Diederik van Liere <dvanliere@wikimedia.org> wrote:
Those two issues are separate: automatic updating ip6 database and getting the databases into HDFS. But I also would like to remind us that we should not conflate bugs and infrastructure tasks: the fact that ip6 is not automatic updated is not so much of a bug: we don't have a subscription (because it does not exist) it was never written in the first place. Let's really use bugzilla for bugs.  D


On Mon, Sep 16, 2013 at 3:59 PM, Christian Aistleitner <christian@quelltextlich.at> wrote:
Hi Toby,

On Mon, Sep 16, 2013 at 03:28:30PM -0700, Toby Negrin wrote:
> thanks Christian -- should I combine this with the bug for the IPV4
> database?
>
> https://bugzilla.wikimedia.org/show_bug.cgi?id=53762

Sorry. Seems I got confused what 53762 was about. To me, it read like
it's about getting up-to-date GeoIP databases into HDFS, and about
generally making older database versions available.

Bug 54191 was meant to cover getting up-to-date IPv6 databases onto
the plain file system of stats1 and stats1002. If that's implicitly
included in 53762, then obviously: Yes, let's join them.

Best regards,
Christian


--
---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ----
                           Companies' registry: 360296y in Linz
Christian Aistleitner
Gruendbergstrasze 65a        Email:  christian@quelltextlich.at
4040 Linz, Austria           Phone:          +43 732 / 26 95 63
                             Fax:            +43 732 / 26 95 63
                             Homepage: http://quelltextlich.at/
---------------------------------------------------------------

_______________________________________________
Analytics mailing list
Analytics@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics



_______________________________________________
Analytics mailing list
Analytics@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics


_______________________________________________
Analytics mailing list
Analytics@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics



_______________________________________________
Analytics mailing list
Analytics@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics