Hi,
while we obviously do not see too much IPv6 traffic, I am getting more and more concerned about the fact that we do not see updates of the GeoIPv6 databases on at least stat1 and stat1002.
Those databases still agree with the file Diederik gave me back in July.
How often do we expect to see updates of the GeoIPv6 database?
Best regards, Christian
Hi,
On Thu, Sep 12, 2013 at 02:00:41PM +0200, quelltextlich e.U. - Christian Aistleitner wrote:
How often do we expect to see updates of the GeoIPv6 database?
Talking this over with Maxmind, their IPv6 data is only available at GeoLite-level. Our file seems to be such a free variant as well. We could just update the databases using newer GeoLite databases. They are available from: http://dev.maxmind.com/geoip/legacy/geolite/
Since discussion showed that we would actually expect to have automatic updates for the IPv6 GeoIP databases, I filed bug 54191 [1] to track this issue.
Best regards, Christian
[1] https://bugzilla.wikimedia.org/show_bug.cgi?id=54191
thanks Christian -- should I combine this with the bug for the IPV4 database?
https://bugzilla.wikimedia.org/show_bug.cgi?id=53762
-Toby
On Mon, Sep 16, 2013 at 3:27 PM, Christian Aistleitner < christian@quelltextlich.at> wrote:
Hi,
On Thu, Sep 12, 2013 at 02:00:41PM +0200, quelltextlich e.U. - Christian Aistleitner wrote:
How often do we expect to see updates of the GeoIPv6 database?
Talking this over with Maxmind, their IPv6 data is only available at GeoLite-level. Our file seems to be such a free variant as well. We could just update the databases using newer GeoLite databases. They are available from: http://dev.maxmind.com/geoip/legacy/geolite/
Since discussion showed that we would actually expect to have automatic updates for the IPv6 GeoIP databases, I filed bug 54191 [1] to track this issue.
Best regards, Christian
[1] https://bugzilla.wikimedia.org/show_bug.cgi?id=54191
-- ---- quelltextlich e.U. ---- \ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Gruendbergstrasze 65a Email: christian@quelltextlich.at 4040 Linz, Austria Phone: +43 732 / 26 95 63 Fax: +43 732 / 26 95 63 Homepage: http://quelltextlich.at/
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Hi Toby,
On Mon, Sep 16, 2013 at 03:28:30PM -0700, Toby Negrin wrote:
thanks Christian -- should I combine this with the bug for the IPV4 database?
Sorry. Seems I got confused what 53762 was about. To me, it read like it's about getting up-to-date GeoIP databases into HDFS, and about generally making older database versions available.
Bug 54191 was meant to cover getting up-to-date IPv6 databases onto the plain file system of stats1 and stats1002. If that's implicitly included in 53762, then obviously: Yes, let's join them.
Best regards, Christian
Those two issues are separate: automatic updating ip6 database and getting the databases into HDFS. But I also would like to remind us that we should not conflate bugs and infrastructure tasks: the fact that ip6 is not automatic updated is not so much of a bug: we don't have a subscription (because it does not exist) it was never written in the first place. Let's really use bugzilla for bugs. D
On Mon, Sep 16, 2013 at 3:59 PM, Christian Aistleitner < christian@quelltextlich.at> wrote:
Hi Toby,
On Mon, Sep 16, 2013 at 03:28:30PM -0700, Toby Negrin wrote:
thanks Christian -- should I combine this with the bug for the IPV4 database?
Sorry. Seems I got confused what 53762 was about. To me, it read like it's about getting up-to-date GeoIP databases into HDFS, and about generally making older database versions available.
Bug 54191 was meant to cover getting up-to-date IPv6 databases onto the plain file system of stats1 and stats1002. If that's implicitly included in 53762, then obviously: Yes, let's join them.
Best regards, Christian
-- ---- quelltextlich e.U. ---- \ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Gruendbergstrasze 65a Email: christian@quelltextlich.at 4040 Linz, Austria Phone: +43 732 / 26 95 63 Fax: +43 732 / 26 95 63 Homepage: http://quelltextlich.at/
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
AFAIK we don't update the V4 database either. When we fix this, we should fix both. I'll link, but not dup the bugs.
On Mon, Sep 16, 2013 at 4:02 PM, Diederik van Liere <dvanliere@wikimedia.org
wrote:
Those two issues are separate: automatic updating ip6 database and getting the databases into HDFS. But I also would like to remind us that we should not conflate bugs and infrastructure tasks: the fact that ip6 is not automatic updated is not so much of a bug: we don't have a subscription (because it does not exist) it was never written in the first place. Let's really use bugzilla for bugs. D
On Mon, Sep 16, 2013 at 3:59 PM, Christian Aistleitner < christian@quelltextlich.at> wrote:
Hi Toby,
On Mon, Sep 16, 2013 at 03:28:30PM -0700, Toby Negrin wrote:
thanks Christian -- should I combine this with the bug for the IPV4 database?
Sorry. Seems I got confused what 53762 was about. To me, it read like it's about getting up-to-date GeoIP databases into HDFS, and about generally making older database versions available.
Bug 54191 was meant to cover getting up-to-date IPv6 databases onto the plain file system of stats1 and stats1002. If that's implicitly included in 53762, then obviously: Yes, let's join them.
Best regards, Christian
-- ---- quelltextlich e.U. ---- \ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Gruendbergstrasze 65a Email: christian@quelltextlich.at 4040 Linz, Austria Phone: +43 732 / 26 95 63 Fax: +43 732 / 26 95 63 Homepage: http://quelltextlich.at/
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
We do update the v4 database. Not sure how often Maxmind update's their DBs, but puppet will run the geoipupdate script weekly. The DBs get distributed to puppet client agents whenever the agent runs (roughly every 30 minutes).
root@stafford:~# tail -n 10 /var/log/geoipupdate.log Sun Jul 14 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Jul 21 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Jul 28 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Aug 4 03:30:02 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Aug 11 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Aug 18 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Aug 25 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Sep 1 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Sep 8 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Sep 15 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP root@stafford:~# ls -l /var/lib/puppet/volatile/GeoIP total 47300 -rw-r--r-- 1 root root 40128176 Sep 15 03:30 GeoIPCity.dat -rw-r--r-- 1 root root 946656 Sep 15 03:30 GeoIP.dat -rw-r--r-- 1 root root 7355209 Sep 15 03:30 GeoIPRegion.dat
On Sep 16, 2013, at 4:37 PM, Toby Negrin tnegrin@wikimedia.org wrote:
AFAIK we don't update the V4 database either. When we fix this, we should fix both. I'll link, but not dup the bugs.
On Mon, Sep 16, 2013 at 4:02 PM, Diederik van Liere dvanliere@wikimedia.org wrote: Those two issues are separate: automatic updating ip6 database and getting the databases into HDFS. But I also would like to remind us that we should not conflate bugs and infrastructure tasks: the fact that ip6 is not automatic updated is not so much of a bug: we don't have a subscription (because it does not exist) it was never written in the first place. Let's really use bugzilla for bugs. D
On Mon, Sep 16, 2013 at 3:59 PM, Christian Aistleitner christian@quelltextlich.at wrote: Hi Toby,
On Mon, Sep 16, 2013 at 03:28:30PM -0700, Toby Negrin wrote:
thanks Christian -- should I combine this with the bug for the IPV4 database?
Sorry. Seems I got confused what 53762 was about. To me, it read like it's about getting up-to-date GeoIP databases into HDFS, and about generally making older database versions available.
Bug 54191 was meant to cover getting up-to-date IPv6 databases onto the plain file system of stats1 and stats1002. If that's implicitly included in 53762, then obviously: Yes, let's join them.
Best regards, Christian
-- ---- quelltextlich e.U. ---- \ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Gruendbergstrasze 65a Email: christian@quelltextlich.at 4040 Linz, Austria Phone: +43 732 / 26 95 63 Fax: +43 732 / 26 95 63 Homepage: http://quelltextlich.at/
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
I'm confused -- did you just fix this?
On Mon, Sep 16, 2013 at 4:40 PM, Andrew Otto otto@wikimedia.org wrote:
We do update the v4 database. Not sure how often Maxmind update's their DBs, but puppet will run the geoipupdate script weekly. The DBs get distributed to puppet client agents whenever the agent runs (roughly every 30 minutes).
root@stafford:~# tail -n 10 /var/log/geoipupdate.log Sun Jul 14 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Jul 21 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Jul 28 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Aug 4 03:30:02 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Aug 11 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Aug 18 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Aug 25 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Sep 1 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Sep 8 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Sep 15 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP root@stafford:~# ls -l /var/lib/puppet/volatile/GeoIP total 47300 -rw-r--r-- 1 root root 40128176 Sep 15 03:30 GeoIPCity.dat -rw-r--r-- 1 root root 946656 Sep 15 03:30 GeoIP.dat -rw-r--r-- 1 root root 7355209 Sep 15 03:30 GeoIPRegion.dat
On Sep 16, 2013, at 4:37 PM, Toby Negrin tnegrin@wikimedia.org wrote:
AFAIK we don't update the V4 database either. When we fix this, we should fix both. I'll link, but not dup the bugs.
On Mon, Sep 16, 2013 at 4:02 PM, Diederik van Liere < dvanliere@wikimedia.org> wrote:
Those two issues are separate: automatic updating ip6 database and getting the databases into HDFS. But I also would like to remind us that we should not conflate bugs and infrastructure tasks: the fact that ip6 is not automatic updated is not so much of a bug: we don't have a subscription (because it does not exist) it was never written in the first place. Let's really use bugzilla for bugs. D
On Mon, Sep 16, 2013 at 3:59 PM, Christian Aistleitner < christian@quelltextlich.at> wrote:
Hi Toby,
On Mon, Sep 16, 2013 at 03:28:30PM -0700, Toby Negrin wrote:
thanks Christian -- should I combine this with the bug for the IPV4 database?
Sorry. Seems I got confused what 53762 was about. To me, it read like it's about getting up-to-date GeoIP databases into HDFS, and about generally making older database versions available.
Bug 54191 was meant to cover getting up-to-date IPv6 databases onto the plain file system of stats1 and stats1002. If that's implicitly included in 53762, then obviously: Yes, let's join them.
Best regards, Christian
-- ---- quelltextlich e.U. ---- \ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Gruendbergstrasze 65a Email: christian@quelltextlich.at 4040 Linz, Austria Phone: +43 732 / 26 95 63 Fax: +43 732 / 26 95 63 Homepage: http://quelltextlich.at/
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
There are three things going on here: 1) Automatic updating of the maxmind IP4 database -- Ottomata created many many moons ago a wonderful puppet module that does this.
2) Automatic updating of the maxmind IP6 database -- Ottomata's puppet module does not do this because Maxmind does not sell IP6 databases, it was never in scope of the puppet module to automatically update this. Christian's bug report https://bugzilla.wikimedia.org/show_bug.cgi?id=54191refers to this but it should not be a bug; at best it's an infrastructure task.
3) Copying of all the maxmind databases to HDFS. The problem that we face here is that even though the Maxmind IP4 database was updated on the local fs, it was not copied to HDFS. We never spent time to figure out a way to make jobs easily reproducible and how to version the different maxmind databases; ottomata has some thoughts about this with symlinking. Bug https://bugzilla.wikimedia.org/show_bug.cgi?id=53762 refers to this problem.
So: bug 54191 and 53762 do not depend on each other. Bug 54191 is not really a bug, once we buy a license we can add this to the puppet module.
An idea: let the PM manage the bugs :)
D
On Mon, Sep 16, 2013 at 4:43 PM, Toby Negrin tnegrin@wikimedia.org wrote:
I'm confused -- did you just fix this?
On Mon, Sep 16, 2013 at 4:40 PM, Andrew Otto otto@wikimedia.org wrote:
We do update the v4 database. Not sure how often Maxmind update's their DBs, but puppet will run the geoipupdate script weekly. The DBs get distributed to puppet client agents whenever the agent runs (roughly every 30 minutes).
root@stafford:~# tail -n 10 /var/log/geoipupdate.log Sun Jul 14 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Jul 21 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Jul 28 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Aug 4 03:30:02 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Aug 11 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Aug 18 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Aug 25 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Sep 1 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Sep 8 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP Sun Sep 15 03:30:01 UTC 2013: geoipupdate downloading MaxMind .dat files into /var/lib/puppet/volatile/GeoIP root@stafford:~# ls -l /var/lib/puppet/volatile/GeoIP total 47300 -rw-r--r-- 1 root root 40128176 Sep 15 03:30 GeoIPCity.dat -rw-r--r-- 1 root root 946656 Sep 15 03:30 GeoIP.dat -rw-r--r-- 1 root root 7355209 Sep 15 03:30 GeoIPRegion.dat
On Sep 16, 2013, at 4:37 PM, Toby Negrin tnegrin@wikimedia.org wrote:
AFAIK we don't update the V4 database either. When we fix this, we should fix both. I'll link, but not dup the bugs.
On Mon, Sep 16, 2013 at 4:02 PM, Diederik van Liere < dvanliere@wikimedia.org> wrote:
Those two issues are separate: automatic updating ip6 database and getting the databases into HDFS. But I also would like to remind us that we should not conflate bugs and infrastructure tasks: the fact that ip6 is not automatic updated is not so much of a bug: we don't have a subscription (because it does not exist) it was never written in the first place. Let's really use bugzilla for bugs. D
On Mon, Sep 16, 2013 at 3:59 PM, Christian Aistleitner < christian@quelltextlich.at> wrote:
Hi Toby,
On Mon, Sep 16, 2013 at 03:28:30PM -0700, Toby Negrin wrote:
thanks Christian -- should I combine this with the bug for the IPV4 database?
Sorry. Seems I got confused what 53762 was about. To me, it read like it's about getting up-to-date GeoIP databases into HDFS, and about generally making older database versions available.
Bug 54191 was meant to cover getting up-to-date IPv6 databases onto the plain file system of stats1 and stats1002. If that's implicitly included in 53762, then obviously: Yes, let's join them.
Best regards, Christian
-- ---- quelltextlich e.U. ---- \ ---- Christian Aistleitner ---- Companies' registry: 360296y in Linz Christian Aistleitner Gruendbergstrasze 65a Email: christian@quelltextlich.at 4040 Linz, Austria Phone: +43 732 / 26 95 63 Fax: +43 732 / 26 95 63 Homepage: http://quelltextlich.at/
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Analytics mailing list Analytics@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/analytics
Hi,
thanks Diederik for this nice breakdown of the different issues.
On Mon, Sep 16, 2013 at 04:50:57PM -0700, Diederik van Liere wrote:
[ Bug 54191 ] [...], once we buy a license we can add this to the puppet module.
There is no need to buy a license. Let's continue that discussion on https://bugzilla.wikimedia.org/show_bug.cgi?id=54191
Have fun, Christian