Maps-l August 2011

maps-l@lists.wikimedia.org

7 participants
7 discussions

tirex "dirty" queue is turned off?
by Kay Drangmeister 30 Aug '11

30 Aug '11

Hi, after returning from my canada holidays, I see that the tirex "dirty" rendering queue is inactive. Has it been deactivated by purpose or is this an error? Tirex Master Status (updated=2011-08-21 11:06:12) Master server: started=2011-08-10 21:36:48 pid=17815 Queue: Prio Size Maxsize Age 1 0 233 2 151144 151144 8792:53-14495:54 10 37 42 30:42-14204:24 all 151181 151181 30:42-14495:54 Buckets: (load=1.61) Name Priority Rend. MaxRd Maxld Active Can Queued Age missing 1- 1 3 6 20 yes yes 0 *dirty 2- 9 0 4 8 no no 151144 8792:53-14495:54 bulk 10- 19 0 3 6 yes no 37 30:42-14204:24 background 20- 0 3 4 yes no 0 ^see line with the * Active=no Kind regards, Kay

4 12

Re: [Maps-l] Maps-l Digest, Vol 28, Issue 4
by maksumial8biz＠yandex.ru 29 Aug '11

29 Aug '11

---------- Отправлено с моего телефона Nokia ------Исходное сообщение------ От: <maps-l-request(a)lists.wikimedia.org> Кому: <maps-l(a)lists.wikimedia.org>,<maps-l-request(a)lists.wikimedia.org> Дата: 29 Август 2011 г. 19:36:09 GMT+00:00 Тема: Maps-l Digest, Vol 28, Issue 4 Send Maps-l mailing list submissions to maps-l(a)lists.wikimedia.org To subscribe or unsubscribe via the World Wide Web, visit https://lists.wikimedia.org/mailman/listinfo/maps-l or, via email, send a message with subject or body 'help' to maps-l-request(a)lists.wikimedia.org You can reach the person managing the list at maps-l-owner(a)lists.wikimedia.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Maps-l digest..." Today's Topics: 1. tirex "dirty" queue is turned off? (Kay Drangmeister) 2. Re: tirex "dirty" queue is turned off? (Peter K?rner) 3. Re: tirex "dirty" queue is turned off? (Tim Alder) 4. Re: tirex "dirty" queue is turned off? (Kay Drangmeister) 5. Re: tirex "dirty" queue is turned off? (Tim Alder) 6. Starting of re-rendering all tiles (Tim Alder) 7. Re: Starting of re-rendering all tiles (Kay Drangmeister) 8. Re: Starting of re-rendering all tiles (Tim Alder) 9. Re: Starting of re-rendering all tiles (Peter K?rner) 10. Re: Starting of re-rendering all tiles (Kai Krueger) 11. Re: Starting of re-rendering all tiles (Tim Alder) ---------------------------------------------------------------------- Message: 1 Date: Sun, 21 Aug 2011 13:22:59 +0200 From: Kay Drangmeister <kay(a)drangmeister.net> Subject: [Maps-l] tirex "dirty" queue is turned off? To: Map integration <maps-l(a)lists.wikimedia.org> Message-ID: <4E50EA93.6010902(a)drangmeister.net> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Hi, after returning from my canada holidays, I see that the tirex "dirty" rendering queue is inactive. Has it been deactivated by purpose or is this an error? Tirex Master Status (updated=2011-08-21 11:06:12) Master server: started=2011-08-10 21:36:48 pid=17815 Queue: Prio Size Maxsize Age 1 0 233 2 151144 151144 8792:53-14495:54 10 37 42 30:42-14204:24 all 151181 151181 30:42-14495:54 Buckets: (load=1.61) Name Priority Rend. MaxRd Maxld Active Can Queued Age missing 1- 1 3 6 20 yes yes 0 *dirty 2- 9 0 4 8 no no 151144 8792:53-14495:54 bulk 10- 19 0 3 6 yes no 37 30:42-14204:24 background 20- 0 3 4 yes no 0 ^see line with the * Active=no Kind regards, Kay ------------------------------ Message: 2 Date: Sun, 21 Aug 2011 13:24:34 +0200 From: Peter K?rner <osm-lists(a)mazdermind.de> Subject: Re: [Maps-l] tirex "dirty" queue is turned off? To: maps-l(a)lists.wikimedia.org Message-ID: <4E50EAF2.4000505(a)mazdermind.de> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Am 21.08.2011 13:22, schrieb Kay Drangmeister: > Hi, > > after returning from my canada holidays, I see that the tirex "dirty" > rendering queue is inactive. Has it been deactivated by purpose or is > this an error? > someome suggested to turn down rendering until the database has catched up. I did this by disabling the dirty queue, so that the missing tiles are still rendered. Peter ------------------------------ Message: 3 Date: Sun, 21 Aug 2011 14:11:08 +0200 From: Tim Alder <tim.alder(a)s2002.tu-chemnitz.de> Subject: Re: [Maps-l] tirex "dirty" queue is turned off? To: Map integration <maps-l(a)lists.wikimedia.org> Message-ID: <4E50F5DC.8000106(a)s2002.tu-chemnitz.de> Content-Type: text/plain; charset=ISO-8859-1; format=flowed The plan is that if we catched up we will reactivated the expire-script and begin than to render all tiles z=9-15 systematically, after this we will rendering systematically all tiles that are older than a year (expired) and repeat this if we are done. I'm not sure what we do with z=16-18, I would say we do this like in the past. Greetings Tim Am 21.08.2011 13:24, schrieb Peter K?rner: >> > > someome suggested to turn down rendering until the database has catched > up. I did this by disabling the dirty queue, so that the missing tiles > are still rendered. > > Peter ------------------------------ Message: 4 Date: Mon, 22 Aug 2011 22:19:03 +0200 From: Kay Drangmeister <kay(a)drangmeister.net> Subject: Re: [Maps-l] tirex "dirty" queue is turned off? To: Map integration <maps-l(a)lists.wikimedia.org> Message-ID: <4E52B9B7.6020107(a)drangmeister.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Am 21.08.2011 13:24, schrieb Peter K?rner: > Am 21.08.2011 13:22, schrieb Kay Drangmeister: >> tirex "dirty" rendering queue is inactive. > someome suggested to turn down rendering until the database has catched > up. I did this by disabling the dirty queue, Thanks for the explanation (is there a wiki page about the current status showing this?). When is the projected date/time the DB should have catched up? The age of the queue suggest it's been more than 11 days already... Is there a monitoring possibility? Thanks, Kay ------------------------------ Message: 5 Date: Mon, 22 Aug 2011 22:36:38 +0200 From: Tim Alder <tim.alder(a)s2002.tu-chemnitz.de> Subject: Re: [Maps-l] tirex "dirty" queue is turned off? To: Map integration <maps-l(a)lists.wikimedia.org> Message-ID: <4E52BDD6.4010306(a)s2002.tu-chemnitz.de> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Hello Kay, in this mailing list you would find the answer. Peter wrote a little tool to show the replag: http://toolserver.org/~mazder/tirex-replag/?refresh=0&human=1 So we have now a replag of 4 days and I would say we need 3 days to be up-to-date. Unfortunately is munin-graph again out of order: http://munin.toolserver.org/OSM/ptolemy/replication_delay2.html and jira-request TS-1125 after nearly a month still untouched. :-( I believe it would be difficult for us to keep the wiki up-to-data, so let's use this list. But I know that's not the optimal medium if somebody is coming for a long vacation or so, so it's not the problem that you ask. Greetings Kolossos Am 22.08.2011 22:19, schrieb Kay Drangmeister: > Am 21.08.2011 13:24, schrieb Peter K?rner: >> Am 21.08.2011 13:22, schrieb Kay Drangmeister: >>> tirex "dirty" rendering queue is inactive. >> someome suggested to turn down rendering until the database has catched >> up. I did this by disabling the dirty queue, > > Thanks for the explanation (is there a wiki page about the current > status showing this?). When is the projected date/time the DB should > have catched up? The age of the queue suggest it's been more than > 11 days already... Is there a monitoring possibility? > > Thanks, > Kay > > > _______________________________________________ > Maps-l mailing list > Maps-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/maps-l > ------------------------------ Message: 6 Date: Sat, 27 Aug 2011 23:11:04 +0200 From: Tim Alder <tim.alder(a)s2002.tu-chemnitz.de> Subject: [Maps-l] Starting of re-rendering all tiles To: Map integration <maps-l(a)lists.wikimedia.org> Message-ID: <4E595D68.7040402(a)s2002.tu-chemnitz.de> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Hello, we have now an up-to-date database, and expiring is working again. I start to rendering all tiles z=9-18 systematically, that are watched in the last 64 days and older than 3 days (because expire didn't work a long time). I put additional z=16-18 in the batch because it is else difficult to handle in different processes. We rendering 150-200 tiles/min what is really nice. There are 3 Mio. tiles in the batch so it will need around 11 days. After this we need to re-render only tiles that are older than a year because expiring is again working, so there will be than much fewer tiles to handle. Greetings Kolossos Am 21.08.2011 14:11, schrieb Tim Alder: > The plan is that if we catched up we will reactivated the expire-script > and begin than to render all tiles z=9-15 systematically, after this we > will rendering systematically all tiles that are older than a year > (expired) and repeat this if we are done. I'm not sure what we do with > z=16-18, I would say we do this like in the past. > > Greetings Tim > > Am 21.08.2011 13:24, schrieb Peter K?rner: >>> > >> someome suggested to turn down rendering until the database has catched >> up. I did this by disabling the dirty queue, so that the missing tiles >> are still rendered. >> >> Peter > ------------------------------ Message: 7 Date: Sat, 27 Aug 2011 23:31:15 +0200 From: "Kay Drangmeister" <kay(a)drangmeister.net> Subject: Re: [Maps-l] Starting of re-rendering all tiles To: "Map integration" <maps-l(a)lists.wikimedia.org> Message-ID: <op.v0v52dh9hiwlwc(a)afa-30100.empolis.local> Content-Type: text/plain; charset=iso-8859-15; format=flowed; delsp=yes Hi, Am 27.08.2011, 23:11 Uhr, schrieb Tim Alder <tim.alder(a)s2002.tu-chemnitz.de>: > Hello, we have now an up-to-date database, and expiring is working again. Looks great. http://toolserver.org/~mazder/tirex-status/?short=1&extended=0&refresh=0 shows a new queue "systema", I guess that's exactly it. One observation: currently putting re-rendered tiles into the queue is a bit quicker than the queue rendering tiles, so the queue grows slowly over time. Nothing to worry I guess, tho'. But: should the AGE of the tiles not gradually decrease, as the queue is supposed to render oldest tiles first? And is "dirty" now separate from the "systema" queue (as is apparent)? If I try to manually dirty a metatile, I don't see it appearing anywhere on the page linked above. Kind regards, Kay ------------------------------ Message: 8 Date: Sun, 28 Aug 2011 00:09:31 +0200 From: Tim Alder <tim.alder(a)s2002.tu-chemnitz.de> Subject: Re: [Maps-l] Starting of re-rendering all tiles To: Map integration <maps-l(a)lists.wikimedia.org> Message-ID: <4E596B1B.70800(a)s2002.tu-chemnitz.de> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Am 27.08.2011 23:31, schrieb Kay Drangmeister: > ... But: should the AGE of the > tiles not gradually decrease, as the queue is supposed to render oldest > tiles first? No, as far I know the queue is only order by numbers of requests. > > And is "dirty" now separate from the "systema" queue (as is apparent)? > If I try to manually dirty a metatile, I don't see it appearing anywhere > on the page linked above. Yes, that's really still a problem. I didn't find a trick how to set dirty-tiles to prio=3 or to deactivated the dirty-bucket and push the list under "bulk" (prio=10) into tirex. Greetings Kolossos ------------------------------ Message: 9 Date: Mon, 29 Aug 2011 10:22:14 +0200 From: Peter K?rner <osm-lists(a)mazdermind.de> Subject: Re: [Maps-l] Starting of re-rendering all tiles To: Map integration <maps-l(a)lists.wikimedia.org> Message-ID: <4E5B4C36.5080303(a)mazdermind.de> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Am 27.08.2011 23:31, schrieb Kay Drangmeister: > Hi, > > Am 27.08.2011, 23:11 Uhr, schrieb Tim Alder<tim.alder(a)s2002.tu-chemnitz.de>: >> Hello, we have now an up-to-date database, and expiring is working again. > > Looks great. > http://toolserver.org/~mazder/tirex-status/?short=1&extended=0&refresh=0 > shows a new queue "systema", I guess that's exactly it. > > One observation: currently putting re-rendered tiles into the queue is > a bit quicker than the queue rendering tiles, so the queue grows slowly > over time. Nothing to worry I guess, tho'. But: should the AGE of the > tiles not gradually decrease, as the queue is supposed to render oldest > tiles first? The queue is sorted by the number of requests received for this tile, not by age. > And is "dirty" now separate from the "systema" queue (as is apparent)? > If I try to manually dirty a metatile, I don't see it appearing anywhere > on the page linked above. when tirex receives a dirty message from mod_tile (metatile oder then planet-timestamp), it enqueues it with prio 2, so it's put into the systema bucket. This can be changed in source [1] (i guess the second int in the array is the priority for cmdRender). Also, if you enqueue a tile that is already in one of the queues, it's not added a second time but just moved up a little in the queue. Peter [1] <http://trac.openstreetmap.org/browser/applications/utils/tirex/lib/Tirex/So…> ------------------------------ Message: 10 Date: Mon, 29 Aug 2011 08:40:49 -0600 From: Kai Krueger <kakrueger(a)gmail.com> Subject: Re: [Maps-l] Starting of re-rendering all tiles To: Map integration <maps-l(a)lists.wikimedia.org> Message-ID: <4E5BA4F1.1070101(a)gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed On 08/29/2011 02:22 AM, Peter K?rner wrote: > Am 27.08.2011 23:31, schrieb Kay Drangmeister: >> Hi, >> >> Am 27.08.2011, 23:11 Uhr, schrieb Tim Alder<tim.alder(a)s2002.tu-chemnitz.de>: >>> Hello, we have now an up-to-date database, and expiring is working again. >> >> Looks great. >> http://toolserver.org/~mazder/tirex-status/?short=1&extended=0&refresh=0 >> shows a new queue "systema", I guess that's exactly it. >> >> One observation: currently putting re-rendered tiles into the queue is >> a bit quicker than the queue rendering tiles, so the queue grows slowly >> over time. Nothing to worry I guess, tho'. But: should the AGE of the >> tiles not gradually decrease, as the queue is supposed to render oldest >> tiles first? > The queue is sorted by the number of requests received for this tile, > not by age. Will this not quickly mess up the systematic ordering of the systema queue? Is it possible to turn off this reordering? Has this got something to do with that throughput has dropped from 160 - 200 down to (still better than before) 60 - 70 tiles / minute? But overall, it does seem the systematicity is working. > >> And is "dirty" now separate from the "systema" queue (as is apparent)? >> If I try to manually dirty a metatile, I don't see it appearing anywhere >> on the page linked above. > > when tirex receives a dirty message from mod_tile (metatile oder then > planet-timestamp), it enqueues it with prio 2, so it's put into the > systema bucket. This can be changed in source [1] (i guess the second > int in the array is the priority for cmdRender). Perhaps the mappings from mod_tile render request types to tirex priority levels could be made configurable. Alternatively, one could tune the mod_tile config to send requests as cmdDirty rather than cmdRender (the latter assumes the request can be rendered on the fly, which isn't the case if about 300k tiles are ahead of it in the queue) mod_tiles cmdDirty maps onto tirex prio level 10 by default. Kai P.S. In the run.log file, at the end of each import, there are messages of the form 26799 import done [2011-08-29 02:11:54] 26799 expiring tiles [2011-08-29 02:11:54] 26799 [error] tile_expiry error [2011-08-29 02:11:55] 26799 resetting state Does the "resetting state" have anything to do with that the replag is not going down? > > Also, if you enqueue a tile that is already in one of the queues, it's > not added a second time but just moved up a little in the queue. > > Peter > > > [1] > <http://trac.openstreetmap.org/browser/applications/utils/tirex/lib/Tirex/So…> > > _______________________________________________ > Maps-l mailing list > Maps-l(a)lists.wikimedia.org > https://lists.wikimedia.org/mailman/listinfo/maps-l ------------------------------ Message: 11 Date: Mon, 29 Aug 2011 21:36:07 +0200 From: Tim Alder <tim.alder(a)s2002.tu-chemnitz.de> Subject: Re: [Maps-l] Starting of re-rendering all tiles To: Map integration <maps-l(a)lists.wikimedia.org> Message-ID: <4E5BEA27.8040006(a)s2002.tu-chemnitz.de> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Am 29.08.2011 16:40, schrieb Kai Krueger: > Will this not quickly mess up the systematic ordering of the systema > queue? Is it possible to turn off this reordering? > > Has this got something to do with that throughput has dropped from >160 - 200 down to (still better than before) 60 - 70 tiles / minute? Hello, my theory is that we started with rendering around Alaska. This area is relative empty and stretched by mercator-projection. Now we are somewhere in Canada/USA incl. NYC with high density of geo-objects. Following the z-curve we should come back to empty areas in the north several times, and render again very fast. If not, my theory is false. In general: Tirex should be able to order the queue by z-curve, especially if you have more than one style (not sure how it can reach the end of it), it would makes it for me much easier to handle it. And Tirex should be able to not order the queue. >> when tirex receives a dirty message from mod_tile (metatile oder then >> planet-timestamp), it enqueues it with prio 2, so it's put into the >> systema bucket. This can be changed in source [1] (i guess the second >> int in the array is the priority for cmdRender). I change the parameter to 4 in ~/tirex/lib/vendor_perl/5.12/Tirex/Source/ModTile.pm It seems that it need a restart of tirex, what I don't want to do in the moment. > > Alternatively, one could tune the mod_tile config to send requests as > cmdDirty rather than cmdRender (the latter assumes the request can be > rendered on the fly, which isn't the case if about 300k tiles are ahead > of it in the queue) mod_tiles cmdDirty maps onto tirex prio level 10 by > default. No idea where to do this. > > > Kai > > P.S. In the run.log file, at the end of each import, there are messages > of the form > > 26799 import done > [2011-08-29 02:11:54] 26799 expiring tiles > [2011-08-29 02:11:54] 26799 [error] tile_expiry error > [2011-08-29 02:11:55] 26799 resetting state > > Does the "resetting state" have anything to do with that the replag is > not going down? > Would be important to know. It seems to work at beginning. ------------------------------ _______________________________________________ Maps-l mailing list Maps-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l End of Maps-l Digest, Vol 28, Issue 4 *************************************

1 0

tirex running but no db-queries
by Tim Alder 16 Aug '11

16 Aug '11

Hello, we have a running rendering-queue but Mapnik rendering nothing and we have no queries on postgresql. The problem seems to come after rendering some tiles of powermap in Zoom-level 6, (can not work in my eyes) . I only monitoring this because I push 17.000 tiles in the queue to render this systematically, but this seems not the reason for the problems we have now. Greetings Tim

4 14

Project documentation / wiki pages
by Julian Picht 15 Aug '11

15 Aug '11

Hi, as I needed to setup a local tirex tile server to try out some scripts which need direct access to the metatiles, I was looking for any information about how it's done on the toolservers. Sadly, I had to find out that nearly all documentation is out of date. Please, somebody who knows how it's all set up, update the pages or create some new. These are the pages I found when I searched for "OpenStreetMap". I marked them as {{outdated}} because they clearly are. - https://wiki.toolserver.org/view/OpenStreetMap_server - 17:31, 8 March 2010 - http://meta.wikimedia.org/wiki/Maps_server_setup_tasks - 16:56, 19 September 2009 - https://wiki.toolserver.org/view/Geographical_tools - 21:13, 1 February 2010 - https://wiki.toolserver.org/view/OpenStreetMap_server/Setup_notes - 04:22, 19 November 2009 Cheers, Julian

2 1

ptolemy- tirex out of order
by Tim Alder 09 Aug '11

09 Aug '11

Tirex doesn't work since yesterday, this means we rendering nothing in the moment. So I open a JIRA-ticket: https://jira.toolserver.org/browse/TS-1133 2 other tickets are also open: https://jira.toolserver.org/browse/TS-1125 https://jira.toolserver.org/browse/TS-1127 Greetings Kolossos

3 3

tool for systematically rendering
by Tim Alder 08 Aug '11

08 Aug '11

I used the last rainy weekends to write a little python-tool[1] to bring the rendering processes for each day in a systematically order. This ordering by geometrical position should reduce our load on harddiscs by reducing random-requests, better usage of disk-cache and preventing out-of-memory (during low-level rendering in different areas of the world at the same time). The tools goes through all styles, from z=9 to z=15. It checks all metatile-files and looks to access-time and modify-time of each tile. So the tile must have for re-rendering: *watched in the last 64 days *and must be older than a year (expiring process makes tiles much older). *in an specified area of the earth This specified area depends on zoomlevel. z=15 will be renderd to 50% so it switch day by day from northern to southern hemissphere. z=14 will be renderd to 25% each day (NW,SW,NE,SE). this going further until z=9 where we render 1.56% and completed the world after 64 days. The system is so that the areas of lower zoomlevels are included in areas of higher zoomlevels. Lower zoomlevels cost us more and changes a harder to see, so I think an update-interval of 2 month for z=9 is acceptable. At the end, I have a list of tiles that need a rerendering, I order this tiles by z-curve[2] and export it in an format that should be usable for tirex-batch (example: map=default z=15 x=5072 y=11208). This batch-process should starting at beginning of each night (lower server load). z=16-18 would be updated like now on request, because we have no problems there. The scanning process of the meta-tiles needs on ptolemy nearly 1 hour. Perhaps somebody can check my code. I know that the tiles are not on ptolemy, so my scan is running over the network to willow (afaik), not optimal, but it seems acceptable. In the moment we render 10.000 tiles in nearly 8hours[3], afterwards we should be much faster. How can we bring the tool to live-system? Some ideas to optimize the tool/rendering process? Greetings Kolossos [1] public-sourcecode: http://toolserver.org/~osm/batchjob-creator/dir-search_source.php place of tool: /home/project/o/s/m/osm/tools/batchjob-creator/ [2]http://en.wikipedia.org/wiki/Z-order_%28curve%29 [3]tail -n -10000 tirex/var/log/jobs.log|less

3 4

Coordinates
by Lars Aronsson 03 Aug '11

03 Aug '11

What different projects and success stories do we have for adding geographic coordinates to existing articles in various languages of Wikipedia? In some languages there are WikiProjects, but I think they were more active some years ago. I tried to count now, and I think the Swedish language Wikipedia has 32,600 geographic coordinates for its 404,000 articles or one coordinate per 12.4 articles. Is that reasonable or is it far below the average? I need a success story to learn from. -- Lars Aronsson (lars(a)aronsson.se) Aronsson Datateknik - http://aronsson.se

2 3

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Maps-l August 2011