How often do we get updates of the GeoIP IPv6 databases?

List overview All Threads
Download

newer

older

HTTPS enabled by default for...

WikiMetrics use case: user-level...

Christian Aistleitner

12 Sep 2013 12 Sep '13

noon

Hi, while we obviously do not see too much IPv6 traffic, I am getting more and more concerned about the fact that we do not see updates of the GeoIPv6 databases on at least stat1 and stat1002. Those databases still agree with the file Diederik gave me back in July. How often do we expect to see updates of the GeoIPv6 database? Best regards, Christian -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Gruendbergstrasze 65a Email: christian(a)quelltextlich.at 4040 Linz, Austria Phone: +43 732 / 26 95 63 Fax: +43 732 / 26 95 63 Homepage: http://quelltextlich.at/ ---------------------------------------------------------------

Attachments:

signature.asc (application/pgp-signature — 490 bytes)

Show replies by date

Christian Aistleitner

16 Sep 16 Sep

10:27 p.m.

Hi, On Thu, Sep 12, 2013 at 02:00:41PM +0200, quelltextlich e.U. - Christian Aistleitner wrote:

...

How often do we expect to see updates of the GeoIPv6 database?

Talking this over with Maxmind, their IPv6 data is only available at GeoLite-level. Our file seems to be such a free variant as well. We could just update the databases using newer GeoLite databases. They are available from: http://dev.maxmind.com/geoip/legacy/geolite/ Since discussion showed that we would actually expect to have automatic updates for the IPv6 GeoIP databases, I filed bug 54191 [1] to track this issue. Best regards, Christian [1] https://bugzilla.wikimedia.org/show_bug.cgi?id=54191 -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Gruendbergstrasze 65a Email: christian(a)quelltextlich.at 4040 Linz, Austria Phone: +43 732 / 26 95 63 Fax: +43 732 / 26 95 63 Homepage: http://quelltextlich.at/ ---------------------------------------------------------------

Toby Negrin

10:28 p.m.

New subject: How often do we get updates of the GeoIP IPv6 databases?

thanks Christian -- should I combine this with the bug for the IPV4 database? https://bugzilla.wikimedia.org/show_bug.cgi?id=53762 -Toby On Mon, Sep 16, 2013 at 3:27 PM, Christian Aistleitner < christian(a)quelltextlich.at> wrote:

...

Hi, On Thu, Sep 12, 2013 at 02:00:41PM +0200, quelltextlich e.U. - Christian Aistleitner wrote:

How often do we expect to see updates of the GeoIPv6 database?

Christian Aistleitner

10:59 p.m.

Hi Toby, On Mon, Sep 16, 2013 at 03:28:30PM -0700, Toby Negrin wrote:

...

thanks Christian -- should I combine this with the bug for the IPV4 database? https://bugzilla.wikimedia.org/show_bug.cgi?id=53762

Sorry. Seems I got confused what 53762 was about. To me, it read like it's about getting up-to-date GeoIP databases into HDFS, and about generally making older database versions available. Bug 54191 was meant to cover getting up-to-date IPv6 databases onto the plain file system of stats1 and stats1002. If that's implicitly included in 53762, then obviously: Yes, let's join them. Best regards, Christian -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Gruendbergstrasze 65a Email: christian(a)quelltextlich.at 4040 Linz, Austria Phone: +43 732 / 26 95 63 Fax: +43 732 / 26 95 63 Homepage: http://quelltextlich.at/ ---------------------------------------------------------------

Diederik van Liere

11:02 p.m.

New subject: How often do we get updates of the GeoIP IPv6 databases?

Those two issues are separate: automatic updating ip6 database and getting the databases into HDFS. But I also would like to remind us that we should not conflate bugs and infrastructure tasks: the fact that ip6 is not automatic updated is not so much of a bug: we don't have a subscription (because it does not exist) it was never written in the first place. Let's really use bugzilla for bugs. D On Mon, Sep 16, 2013 at 3:59 PM, Christian Aistleitner < christian(a)quelltextlich.at> wrote:

...

Hi Toby, On Mon, Sep 16, 2013 at 03:28:30PM -0700, Toby Negrin wrote:

thanks Christian -- should I combine this with the bug for the IPV4 database? https://bugzilla.wikimedia.org/show_bug.cgi?id=53762

Toby Negrin

11:37 p.m.

New subject: How often do we get updates of the GeoIP IPv6 databases?

AFAIK we don't update the V4 database either. When we fix this, we should fix both. I'll link, but not dup the bugs. On Mon, Sep 16, 2013 at 4:02 PM, Diederik van Liere <dvanliere(a)wikimedia.org

...

wrote:

> Those two issues are separate: automatic updating ip6 database and getting > the databases into HDFS. But I also would like to remind us that we should > not conflate bugs and infrastructure tasks: the fact that ip6 is not > automatic updated is not so much of a bug: we don't have a subscription > (because it does not exist) it was never written in the first place. Let's > really use bugzilla for bugs. > D > > > On Mon, Sep 16, 2013 at 3:59 PM, Christian Aistleitner < > christian(a)quelltextlich.at

...

wrote:

> >> Hi Toby, >> >> On Mon, Sep 16, 2013 at 03:28:30PM -0700, Toby Negrin wrote: >> > thanks Christian -- should I combine this with the bug for the IPV4 >> > database? >> > >> > https://bugzilla.wikimedia.org/show_bug.cgi?id=53762 >> >> Sorry. Seems I got confused what 53762 was about. To me, it read like >> it's about getting up-to-date GeoIP databases into HDFS, and about >> generally making older database versions available. >> >> Bug 54191 was meant to cover getting up-to-date IPv6 databases onto >> the plain file system of stats1 and stats1002. If that's implicitly >> included in 53762, then obviously: Yes, let's join them. >> >> Best regards, >> Christian >> >> >> -- >> ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- >> Companies' registry: 360296y in Linz >> Christian Aistleitner >> Gruendbergstrasze 65a Email: christian(a)quelltextlich.at >> 4040 Linz, Austria Phone: +43 732 / 26 95 63 >> Fax: +43 732 / 26 95 63 >> Homepage: http://quelltextlich.at/ >> --------------------------------------------------------------- >> >> _______________________________________________ >> Analytics mailing list >> Analytics(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/analytics >> >> >

Andrew Otto

11:40 p.m.

New subject: How often do we get updates of the GeoIP IPv6 databases?

We do update the v4 database. Not sure how often Maxmind update's their DBs, but puppet will run the geoipupdate script weekly. The DBs get distributed to puppet client agents whenever the agent runs (roughly every 30 minutes). root@stafford:~# tail -n 10 /var/log/geoipupdate.log Sun Jul 14 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Jul 21 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Jul 28 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Aug 4 03:30:02 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Aug 11 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Aug 18 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Aug 25 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Sep 1 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Sep 8 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Sep 15 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP root@stafford:~# ls -l /var/lib/puppet/volatile/GeoIP total 47300 -rw-r--r-- 1 root root 40128176 Sep 15 03:30 GeoIPCity.dat -rw-r--r-- 1 root root 946656 Sep 15 03:30 GeoIP.dat -rw-r--r-- 1 root root 7355209 Sep 15 03:30 GeoIPRegion.dat On Sep 16, 2013, at 4:37 PM, Toby Negrin <tnegrin(a)wikimedia.org> wrote:

...

AFAIK we don't update the V4 database either. When we fix this, we should fix both. I'll link, but not dup the bugs. On Mon, Sep 16, 2013 at 4:02 PM, Diederik van Liere <dvanliere(a)wikimedia.org> wrote: Those two issues are separate: automatic updating ip6 database and getting the databases into HDFS. But I also would like to remind us that we should not conflate bugs and infrastructure tasks: the fact that ip6 is not automatic updated is not so much of a bug: we don't have a subscription (because it does not exist) it was never written in the first place. Let's really use bugzilla for bugs. D On Mon, Sep 16, 2013 at 3:59 PM, Christian Aistleitner <christian(a)quelltextlich.at> wrote: Hi Toby, On Mon, Sep 16, 2013 at 03:28:30PM -0700, Toby Negrin wrote:

thanks Christian -- should I combine this with the bug for the IPV4 database? https://bugzilla.wikimedia.org/show_bug.cgi?id=53762

Toby Negrin

11:43 p.m.

New subject: How often do we get updates of the GeoIP IPv6 databases?

I'm confused -- did you just fix this? On Mon, Sep 16, 2013 at 4:40 PM, Andrew Otto <otto(a)wikimedia.org> wrote:

...

Hi Toby, On Mon, Sep 16, 2013 at 03:28:30PM -0700, Toby Negrin wrote:

thanks Christian -- should I combine this with the bug for the IPV4 database? https://bugzilla.wikimedia.org/show_bug.cgi?id=53762

_______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

Diederik van Liere

11:50 p.m.

New subject: How often do we get updates of the GeoIP IPv6 databases?

There are three things going on here: 1) Automatic updating of the maxmind IP4 database -- Ottomata created many many moons ago a wonderful puppet module that does this. 2) Automatic updating of the maxmind IP6 database -- Ottomata's puppet module does not do this because Maxmind does not sell IP6 databases, it was never in scope of the puppet module to automatically update this. Christian's bug report https://bugzilla.wikimedia.org/show_bug.cgi?id=54191refers to this but it should not be a bug; at best it's an infrastructure task. 3) Copying of all the maxmind databases to HDFS. The problem that we face here is that even though the Maxmind IP4 database was updated on the local fs, it was not copied to HDFS. We never spent time to figure out a way to make jobs easily reproducible and how to version the different maxmind databases; ottomata has some thoughts about this with symlinking. Bug https://bugzilla.wikimedia.org/show_bug.cgi?id=53762 refers to this problem. So: bug 54191 and 53762 do not depend on each other. Bug 54191 is not really a bug, once we buy a license we can add this to the puppet module. An idea: let the PM manage the bugs :) D On Mon, Sep 16, 2013 at 4:43 PM, Toby Negrin <tnegrin(a)wikimedia.org> wrote:

...

I'm confused -- did you just fix this? On Mon, Sep 16, 2013 at 4:40 PM, Andrew Otto <otto(a)wikimedia.org> wrote:

Hi Toby, On Mon, Sep 16, 2013 at 03:28:30PM -0700, Toby Negrin wrote: > thanks Christian -- should I combine this with the bug for the IPV4 > database? > > https://bugzilla.wikimedia.org/show_bug.cgi?id=53762 Sorry. Seems I got confused what 53762 was about. To me, it read like it's about getting up-to-date GeoIP databases into HDFS, and about generally making older database versions available. Bug 54191 was meant to cover getting up-to-date IPv6 databases onto the plain file system of stats1 and stats1002. If that's implicitly included in 53762, then obviously: Yes, let's join them. Best regards, Christian -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Gruendbergstrasze 65a Email: christian(a)quelltextlich.at 4040 Linz, Austria Phone: +43 732 / 26 95 63 Fax: +43 732 / 26 95 63 Homepage: http://quelltextlich.at/ --------------------------------------------------------------- _______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

_______________________________________________ Analytics mailing list Analytics(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics

Christian Aistleitner

17 Sep 17 Sep

12:39 a.m.

Hi, thanks Diederik for this nice breakdown of the different issues. On Mon, Sep 16, 2013 at 04:50:57PM -0700, Diederik van Liere wrote:

...

[ Bug 54191 ] [...], once we buy a license we can add this to the puppet module.

There is no need to buy a license. Let's continue that discussion on https://bugzilla.wikimedia.org/show_bug.cgi?id=54191 Have fun, Christian -- ---- quelltextlich e.U. ---- \\ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Gruendbergstrasze 65a Email: christian(a)quelltextlich.at 4040 Linz, Austria Phone: +43 732 / 26 95 63 Fax: +43 732 / 26 95 63 Homepage: http://quelltextlich.at/ ---------------------------------------------------------------

3875

days inactive

3880

days old

analytics@lists.wikimedia.org

Manage subscription

9 comments

4 participants

tags (0)

participants (4)

Andrew Otto
Christian Aistleitner
Diederik van Liere
Toby Negrin