There are three things going on here:
1) Automatic updating of the maxmind IP4 database -- Ottomata created many
many moons ago a wonderful puppet module that does this.
2) Automatic updating of the maxmind IP6 database -- Ottomata's puppet
module does not do this because Maxmind does not sell IP6 databases, it was
never in scope of the puppet module to automatically update this.
Christian's bug report
https://bugzilla.wikimedia.org/show_bug.cgi?id=54191refers to this but
it should not be a bug; at best it's an infrastructure
task.
3) Copying of all the maxmind databases to HDFS. The problem that we face
here is that even though the Maxmind IP4 database was updated on the local
fs, it was not copied to HDFS. We never spent time to figure out a way to
make jobs easily reproducible and how to version the different maxmind
databases; ottomata has some thoughts about this with symlinking. Bug
https://bugzilla.wikimedia.org/show_bug.cgi?id=53762 refers to this problem.
So: bug 54191 and 53762 do not depend on each other. Bug 54191 is not
really a bug, once we buy a license we can add this to the puppet module.
An idea: let the PM manage the bugs :)
D
On Mon, Sep 16, 2013 at 4:43 PM, Toby Negrin <tnegrin(a)wikimedia.org> wrote:
I'm confused -- did you just fix this?
On Mon, Sep 16, 2013 at 4:40 PM, Andrew Otto <otto(a)wikimedia.org> wrote:
We do update the v4 database. Not sure how often
Maxmind update's their
DBs, but puppet will run the geoipupdate script weekly. The DBs get
distributed to puppet client agents whenever the agent runs (roughly every
30 minutes).
root@stafford:~# tail -n 10 /var/log/geoipupdate.log
Sun Jul 14 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files
into /var/lib/puppet/volatile/GeoIP
Sun Jul 21 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files
into /var/lib/puppet/volatile/GeoIP
Sun Jul 28 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files
into /var/lib/puppet/volatile/GeoIP
Sun Aug 4 03:30:02 UTC 2013: geoipupdate downloading MaxMind .dat files
into /var/lib/puppet/volatile/GeoIP
Sun Aug 11 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files
into /var/lib/puppet/volatile/GeoIP
Sun Aug 18 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files
into /var/lib/puppet/volatile/GeoIP
Sun Aug 25 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files
into /var/lib/puppet/volatile/GeoIP
Sun Sep 1 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files
into /var/lib/puppet/volatile/GeoIP
Sun Sep 8 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files
into /var/lib/puppet/volatile/GeoIP
Sun Sep 15 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files
into /var/lib/puppet/volatile/GeoIP
root@stafford:~# ls -l /var/lib/puppet/volatile/GeoIP
total 47300
-rw-r--r-- 1 root root 40128176 Sep 15 03:30 GeoIPCity.dat
-rw-r--r-- 1 root root 946656 Sep 15 03:30 GeoIP.dat
-rw-r--r-- 1 root root 7355209 Sep 15 03:30 GeoIPRegion.dat
On Sep 16, 2013, at 4:37 PM, Toby Negrin <tnegrin(a)wikimedia.org> wrote:
AFAIK we don't update the V4 database either. When we fix this, we should
fix both. I'll link, but not dup the bugs.
On Mon, Sep 16, 2013 at 4:02 PM, Diederik van Liere <
dvanliere(a)wikimedia.org> wrote:
Those two issues are separate: automatic updating
ip6 database and
getting the databases into HDFS. But I also would like to remind us that we
should not conflate bugs and infrastructure tasks: the fact that ip6 is not
automatic updated is not so much of a bug: we don't have a subscription
(because it does not exist) it was never written in the first place. Let's
really use bugzilla for bugs. D
On Mon, Sep 16, 2013 at 3:59 PM, Christian Aistleitner <
christian(a)quelltextlich.at> wrote:
Hi Toby,
On Mon, Sep 16, 2013 at 03:28:30PM -0700, Toby Negrin wrote:
> thanks Christian -- should I combine this with the bug for the IPV4
> database?
>
>
https://bugzilla.wikimedia.org/show_bug.cgi?id=53762
Sorry. Seems I got confused what 53762 was about. To me, it read like
it's about getting up-to-date GeoIP databases into HDFS, and about
generally making older database versions available.
Bug 54191 was meant to cover getting up-to-date IPv6 databases onto
the plain file system of stats1 and stats1002. If that's implicitly
included in 53762, then obviously: Yes, let's join them.
Best regards,
Christian
--
---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ----
Companies' registry: 360296y in Linz
Christian Aistleitner
Gruendbergstrasze 65a Email: christian(a)quelltextlich.at
4040 Linz, Austria Phone: +43 732 / 26 95 63
Fax: +43 732 / 26 95 63
Homepage:
http://quelltextlich.at/
---------------------------------------------------------------
_______________________________________________
Analytics mailing list
Analytics(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics
_______________________________________________
Analytics mailing list
Analytics(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics
_______________________________________________
Analytics mailing list
Analytics(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics
_______________________________________________
Analytics mailing list
Analytics(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/analytics