Hi all!
I'd like to continue with the Shape extraction for Wikipedia articles, like Kolossos has begun with the
http://wiki.openstreetmap.org/wiki/Query-to-map
Query-to-Map-project. There is pretty much already there, even the complete endresult, like I mentioned in
https://wiki.toolserver.org/view/GeoShape#Frontend_Examples
What I'm thinking about is: I don't think the Osm2pgsql database schema which we have (I think) is suitable for the long-term for the reasons mentioned in the same article further below,
https://wiki.toolserver.org/view/GeoShape#Comparasion_osm2pgsql_and_osmosis
mainly because I cant find a good changemanagement for derivate tables, which can be worked around maybe or which can be omitted creating a new Shape table every now and then.
Personally I like the lonvia approach best, it's a Osmosis snapshot schema which is quite up to date to my opinion and its a ready GPL package. On the other hand, it's in phyton which may not be 'everybodys darling' [PK], and it's rendering functions are not needed for this purpose here. In one thing I tend to agree with Stephan Plepelits:
Unfortunately we have to import the database twice ... once with osm2pgsql to have a database structure which is usable for mapnik. A second time with osmosis, because we want to have all possibly available data. http://gitorious.org/openstreetbrowser/openstreetbrowser/blobs/master/DOCU
What do you think?
Guverneu/Bernhard
Hi,
While it is generally possible to "dirty" some tiles on toolserver, I tried
with this one:
http://toolserver.org/tiles/parking-bw/4/8/5.png
(i.e. appended /dirty to it), but reading the .../status, it always says:
Tile is clean. Last rendered at Sun Jul 24 03:47:23 2011
How can that be? It must always be possible to manually request a rerender.
What is the difference between tiles that can be dirtyfied and this one?
Kind regards,
Kay
Hello,
I updated /osm/libs/openlayers/latest to 2.11.
Now OpenLayers supports mobile multitouch devices!
Some optimisation in our code seems necessary to run fast, like:
http://openlayers.org/dev/examples/mobile.html or
http://www.openstreetmap.org
Perhaps somebody want to play with it.
Thanks to OpenLayers!
Greetings Kolossos
Hi
could someone (Peter) please update my parking styles on ptolemy?
I'd do it myself, but https://jira.toolserver.org/browse/TS-1107
is still unresolved.
The styles are here: /home/kayd/deploy
Samples are here: /home/kayd/map-*.png
Thanks,
Kay
Hi.
I noticed that a lot has been done re-organizing the rendering queues on
the toolserver. YAY!
http://munin.toolserver.org/OSM/ptolemy/tirex_status_queued_requests.html
looks now very colorful - mostly green (background).
Thanks to everyone working on this. Many of the tiles I waited for have
been rendered now.
However - there seems to be some buggish behaviour: looking at
http://toolserver.org/~mazder/tirex-status/?short=1&extended=0&refresh=0
it currently shows:
Tirex Master Status (updated=2011-09-16 15:33:12)
Master server:
started=2011-09-11 06:28:53 pid=4623
Queue:
Prio Size Maxsize Age
1 0 427
2 65 181 0:01-1:20 <- *** Size increasing ***
10 3025 3614 7742:24-7647:13 <- *** Size decreasing ***
99 50149 50149 1:26-7744:02
all 53239 53323 0:01-7744:02
Buckets: (load=3.75)
Name Priority Rendering MaxRend Maxload Active Can Queued Age
missing 1- 1 1 6 20 yes yes 0
systema 2- 2 4 5 8 yes no 65 0:01-1:20
dirty 3- 9 0 4 8 yes no 0
bulk 10- 19 0 6 6 yes yes 3025 7742:24-7647:13
background 20- 0 3 4 yes no 50149 1:26-7744:02
Currently rendering: (num=5)
Map X Y Z Prio Age
default 164568 85216 18 1 1
hikebike 33904 22840 16 2 6
hikebike 33912 22840 16 2 4
default 4224 2856 13 2 3
default 8448 5712 14 2 2
From what I see, queue systema (prio 2) is filled (65) and rendering. But
when I refresh the page, I can see that the "bulk" queue Size is decreasing.
So tirex must (realy?) be rendering Prio 10 tiles as well, but it's never
showing in the "Currently rendering" table, there I can see only Prio 1 and
2 tiles. How is this possible?
Kind regards,
Kay
----------
Отправлено с моего телефона Nokia
------Исходное сообщение------
От: <maps-l-request(a)lists.wikimedia.org>
Кому: <maps-l(a)lists.wikimedia.org>,<maps-l-request(a)lists.wikimedia.org>
Дата: 16 Сентябрь 2011 г. 15:40:34 GMT+00:00
Тема: Maps-l Digest, Vol 29, Issue 4
Send Maps-l mailing list submissions to
maps-l(a)lists.wikimedia.org
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.wikimedia.org/mailman/listinfo/maps-l
or, via email, send a message with subject or body 'help' to
maps-l-request(a)lists.wikimedia.org
You can reach the person managing the list at
maps-l-owner(a)lists.wikimedia.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Maps-l digest..."
Today's Topics:
1. Re: Starting of re-rendering all tiles (Tim Alder)
2. Re: Starting of re-rendering all tiles (Stephan Knauss)
3. Re: Starting of re-rendering all tiles (Kai Krueger)
4. Re: Starting of re-rendering all tiles +Performance (Tim Alder)
5. Re: Starting of re-rendering all tiles (Stephan Knauss)
6. Re: Starting of re-rendering all tiles (Julian Picht)
7. Re: Starting of re-rendering all tiles (Tim Alder)
8. Other OpenLayer maps (OpenSeaMap) (church.of.emacs.ml)
9. Re: Other OpenLayer maps (OpenSeaMap) (Tim Alder)
10. Possible tirex render queue bug? (Kay Drangmeister)
----------------------------------------------------------------------
Message: 1
Date: Sun, 11 Sep 2011 11:49:55 +0200
From: Tim Alder <tim.alder(a)s2002.tu-chemnitz.de>
Subject: Re: [Maps-l] Starting of re-rendering all tiles
To: Map integration <maps-l(a)lists.wikimedia.org>
Message-ID: <4E6C8443.6070905(a)s2002.tu-chemnitz.de>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Am 10.09.2011 01:44, schrieb Kay Drangmeister:
> Hi Kolossos,
>
> http://munin.toolserver.org/OSM/ptolemy/tirex_status_queued_requests.html
>
> What is filling the "dirty" queue? The re-render should only
> fill the "systema" queue, shouldn't it? Currently the "dirty"
> queue is 11k entries.
mod_tile is still running and filling the queue. I restarted tirex an
hour ago and set prio of this dirty files to 99. I had the hope that it
so ignored by tirex, but doesn't work.
We are rendering now high density areas around amsterdam and running
with many styles in timeouts at zoom-level 9 (x=264 y=160).
I will think about to remove zoom-level 9 from my list. Other ideas?
What's not good is that we have in the moment 70 seq. scans and 12 index
scans.
"vacuum full" was broken after 2.5 day.
I hope somebody else could install the postgres_upgrade_fix.
>
> Next question, what is filling the bulk queue (4k entries)? :-)
I don't know.
> how is the age being calculated in the queues?
I don't know. Is this important?
Greetings Kolossos
>
> Kind regards,
> Kay
>
>
------------------------------
Message: 2
Date: Sun, 11 Sep 2011 19:33:39 +0200
From: Stephan Knauss <toolserver(a)stephans-server.de>
Subject: Re: [Maps-l] Starting of re-rendering all tiles
To: maps-l(a)lists.wikimedia.org
Message-ID: <4E6CF0F3.7050602(a)stephans-server.de>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
On 11.09.2011 11:49, Tim Alder wrote:
> I hope somebody else could install the postgres_upgrade_fix.
was there an upgrade? If so then I understand the wiki we need a
re-import as the db can not be repaired. The script is to prevent damage.
Stephan
------------------------------
Message: 3
Date: Mon, 12 Sep 2011 10:29:15 -0600
From: Kai Krueger <kakrueger(a)gmail.com>
Subject: Re: [Maps-l] Starting of re-rendering all tiles
To: Map integration <maps-l(a)lists.wikimedia.org>
Message-ID: <4E6E335B.5080209(a)gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
On 9/11/11 3:49 AM, Tim Alder wrote:
>
>
> What's not good is that we have in the moment 70 seq. scans and 12 index
> scans.
>
> "vacuum full" was broken after 2.5 day.
> I hope somebody else could install the postgres_upgrade_fix.
That's not so good either.
If I understood it correctly the postgres_upgrade_fix is only relevant
if one updates postgresql. However, as far as I know, postgresql wasn't
upgraded and it is still running 8.3. So I doubt this will fix anything.
So the question is how do we fix the database corruption? Is it
necessary to do a full reimport?
In the mean time, until the corruption is fixed, can we turn off
diff-imports. They are always erroring anyway and are adding load to the
db, that could probably better be spent on rendering.
Kai
------------------------------
Message: 4
Date: Mon, 12 Sep 2011 23:22:19 +0200
From: Tim Alder <tim.alder(a)s2002.tu-chemnitz.de>
Subject: Re: [Maps-l] Starting of re-rendering all tiles +Performance
To: Map integration <maps-l(a)lists.wikimedia.org>
Message-ID: <4E6E780B.3080908(a)s2002.tu-chemnitz.de>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
I stopped load-import.
Tim
Am 12.09.2011 18:29, schrieb Kai Krueger:
> In the mean time, until the corruption is fixed, can we turn off
> diff-imports. They are always erroring anyway and are adding load to the
> db, that could probably better be spent on rendering.
------------------------------
Message: 5
Date: Mon, 12 Sep 2011 23:46:35 +0200
From: Stephan Knauss <toolserver(a)stephans-server.de>
Subject: Re: [Maps-l] Starting of re-rendering all tiles
To: maps-l(a)lists.wikimedia.org
Message-ID: <4E6E7DBB.9030705(a)stephans-server.de>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
On 12.09.2011 18:29, Kai Krueger wrote:
>> I hope somebody else could install the postgres_upgrade_fix.
> That's not so good either.
> If I understood it correctly the postgres_upgrade_fix is only relevant
> if one updates postgresql. However, as far as I know, postgresql wasn't
> upgraded and it is still running 8.3. So I doubt this will fix anything.
I had asked before. So if no upgrade happened this can't be the cause of
the corruption. The script would also only prevent it from getting
corrupted.
If we don't have a backup then I sound like the db needs to be rebuild.
BTW: PostgreSQL 9.1 was just released...
and osm2pgsql main development branch no longer supports intarray. This
migration also needed a re-import.
Is this a hint that we should do a big upgrade?
Could wait for the rendering to complete so we have some tiles still to
deliver while the db rebuilds for a day or two.
Stephan
------------------------------
Message: 6
Date: Mon, 12 Sep 2011 23:51:19 +0200
From: Julian Picht <julian.picht(a)gmail.com>
Subject: Re: [Maps-l] Starting of re-rendering all tiles
To: Map integration <maps-l(a)lists.wikimedia.org>
Message-ID:
<CAMqQAJ_9AnKbV6eqMZxMMaT2FQxBGD-z75HQSPe5ehj1unMfFw(a)mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
Couldn't we install the 9.1 and the new osm2pgsql in parallel, do a fresh
import there and than switch over? Or isn't there enough disk space?
Julian
2011/9/12 Stephan Knauss <toolserver(a)stephans-server.de>
> On 12.09.2011 18:29, Kai Krueger wrote:
> >> I hope somebody else could install the postgres_upgrade_fix.
> > That's not so good either.
> > If I understood it correctly the postgres_upgrade_fix is only relevant
> > if one updates postgresql. However, as far as I know, postgresql wasn't
> > upgraded and it is still running 8.3. So I doubt this will fix anything.
> I had asked before. So if no upgrade happened this can't be the cause of
> the corruption. The script would also only prevent it from getting
> corrupted.
> If we don't have a backup then I sound like the db needs to be rebuild.
>
> BTW: PostgreSQL 9.1 was just released...
> and osm2pgsql main development branch no longer supports intarray. This
> migration also needed a re-import.
>
> Is this a hint that we should do a big upgrade?
> Could wait for the rendering to complete so we have some tiles still to
> deliver while the db rebuilds for a day or two.
>
>
> Stephan
>
> _______________________________________________
> Maps-l mailing list
> Maps-l(a)lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/maps-l
>
Hello,
we haved now reached with the re-rendering the point z-curve=0.25.
So we are now rendering the emptiness of north norway. But this has no
visible effect on the rendering speed. The rendering performance
correlate very good with the queue length. I found now the parameter
-num in tirex-batch to limit the queue length. I will restart the
process for tiles with z>0.25 and limit the queue to a value of around
5.000.
"vacuum full" is now running more than 1 day. If it's not done tomorrow,
I believe I will stop it. Ok?
Greetings Kolossos
Hello,
maintenance on OSM.org is now over but our replag of ptolemy is still
growing.
We are not able to contact the diff-server from OpenStreetMap:
osm@ptolemy:~$ ping planet.openstreetmap.org
no answer from planet.openstreetmap.org
Ping works fine from nightshade or my laptop.
I hope nobody block us.
We change nothing on the config.
Greetings Kolossos
----------
Отправлено с моего телефона Nokia
------Исходное сообщение------
От: <maps-l-request(a)lists.wikimedia.org>
Кому: <maps-l(a)lists.wikimedia.org>,<maps-l-request(a)lists.wikimedia.org>
Дата: 5 Сентябрь 2011 г. 20:43:16 GMT+00:00
Тема: Maps-l Digest, Vol 29, Issue 1
Send Maps-l mailing list submissions to
maps-l(a)lists.wikimedia.org
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.wikimedia.org/mailman/listinfo/maps-l
or, via email, send a message with subject or body 'help' to
maps-l-request(a)lists.wikimedia.org
You can reach the person managing the list at
maps-l-owner(a)lists.wikimedia.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Maps-l digest..."
Today's Topics:
1. Re: Starting of re-rendering all tiles (Peter K?rner)
2. Re: Maps-l Digest, Vol 28, Issue 4 (maksumial8biz(a)yandex.ru)
3. Re: Starting of re-rendering all tiles (Tim Alder)
4. replag is growing /no contact to planet.osm.org (Tim Alder)
5. Re: replag is growing /no contact to planet.osm.org (Grant Slater)
----------------------------------------------------------------------
Message: 1
Date: Tue, 30 Aug 2011 00:05:44 +0200
From: Peter K?rner <osm-lists(a)mazdermind.de>
Subject: Re: [Maps-l] Starting of re-rendering all tiles
To: maps-l(a)lists.wikimedia.org
Message-ID: <4E5C0D38.9010104(a)mazdermind.de>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Hi
Am 29.08.2011 21:36, schrieb Tim Alder:
>> P.S. In the run.log file, at the end of each import, there are messages
>> of the form
>>
>> 26799 import done
>> [2011-08-29 02:11:54] 26799 expiring tiles
>> [2011-08-29 02:11:54] 26799 [error] tile_expiry error
>> [2011-08-29 02:11:55] 26799 resetting state
>>
>> Does the "resetting state" have anything to do with that the replag is
>> not going down?
>>
>
> Would be important to know. It seems to work at beginning.
The load-next script had the expire.rb script enabled which is the wrong
method of expiring. The correct one follows some lines after the
expire.rb statement. I commented out the bad commands.
I also lowered the maxInterval setting from 24h to 6h, so we can check
sooner if this fixed the problem.
Peter
------------------------------
Message: 2
Date: Mon, 29 Aug 2011 23:48:27 +0000
From: "maksumial8biz(a)yandex.ru" <maksumial8biz(a)yandex.ru>
Subject: Re: [Maps-l] Maps-l Digest, Vol 28, Issue 4
To: <maps-l(a)lists.wikimedia.org>,<maps-l-request(a)lists.wikimedia.org>
Message-ID: <20110830034829.mSUiXZKJ(a)smtp14.mail.yandex.net>
Content-Type: TEXT/PLAIN; charset=UTF-8
----------
?????????? ? ????? ???????? Nokia
------???????? ?????????------
??: <maps-l-request(a)lists.wikimedia.org>
????: <maps-l(a)lists.wikimedia.org>,<maps-l-request(a)lists.wikimedia.org>
????: 29 ?????? 2011 ?. 19:36:09 GMT+00:00
????: Maps-l Digest, Vol 28, Issue 4
Send Maps-l mailing list submissions to
maps-l(a)lists.wikimedia.org
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.wikimedia.org/mailman/listinfo/maps-l
or, via email, send a message with subject or body 'help' to
maps-l-request(a)lists.wikimedia.org
You can reach the person managing the list at
maps-l-owner(a)lists.wikimedia.org
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Maps-l digest..."
Today's Topics:
1. tirex "dirty" queue is turned off? (Kay Drangmeister)
2. Re: tirex "dirty" queue is turned off? (Peter K?rner)
3. Re: tirex "dirty" queue is turned off? (Tim Alder)
4. Re: tirex "dirty" queue is turned off? (Kay Drangmeister)
5. Re: tirex "dirty" queue is turned off? (Tim Alder)
6. Starting of re-rendering all tiles (Tim Alder)
7. Re: Starting of re-rendering all tiles (Kay Drangmeister)
8. Re: Starting of re-rendering all tiles (Tim Alder)
9. Re: Starting of re-rendering all tiles (Peter K?rner)
10. Re: Starting of re-rendering all tiles (Kai Krueger)
11. Re: Starting of re-rendering all tiles (Tim Alder)
----------------------------------------------------------------------
Message: 1
Date: Sun, 21 Aug 2011 13:22:59 +0200
From: Kay Drangmeister <kay(a)drangmeister.net>
Subject: [Maps-l] tirex "dirty" queue is turned off?
To: Map integration <maps-l(a)lists.wikimedia.org>
Message-ID: <4E50EA93.6010902(a)drangmeister.net>
Content-Type: text/plain; charset=ISO-8859-15; format=flowed
Hi,
after returning from my canada holidays, I see that the tirex "dirty"
rendering queue is inactive. Has it been deactivated by purpose or is
this an error?
Tirex Master Status (updated=2011-08-21 11:06:12)
Master server:
started=2011-08-10 21:36:48 pid=17815
Queue:
Prio Size Maxsize Age
1 0 233
2 151144 151144 8792:53-14495:54
10 37 42 30:42-14204:24
all 151181 151181 30:42-14495:54
Buckets: (load=1.61)
Name Priority Rend. MaxRd Maxld Active Can Queued Age
missing 1- 1 3 6 20 yes yes 0
*dirty 2- 9 0 4 8 no no 151144 8792:53-14495:54
bulk 10- 19 0 3 6 yes no 37 30:42-14204:24
background 20- 0 3 4 yes no 0
^see line with the * Active=no
Kind regards,
Kay
------------------------------
Message: 2
Date: Sun, 21 Aug 2011 13:24:34 +0200
From: Peter K?rner <osm-lists(a)mazdermind.de>
Subject: Re: [Maps-l] tirex "dirty" queue is turned off?
To: maps-l(a)lists.wikimedia.org
Message-ID: <4E50EAF2.4000505(a)mazdermind.de>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Am 21.08.2011 13:22, schrieb Kay Drangmeister:
> Hi,
>
> after returning from my canada holidays, I see that the tirex "dirty"
> rendering queue is inactive. Has it been deactivated by purpose or is
> this an error?
>
someome suggested to turn down rendering until the database has catched
up. I did this by disabling the dirty queue, so that the missing tiles
are still rendered.
Peter
------------------------------
Message: 3
Date: Sun, 21 Aug 2011 14:11:08 +0200
From: Tim Alder <tim.alder(a)s2002.tu-chemnitz.de>
Subject: Re: [Maps-l] tirex "dirty" queue is turned off?
To: Map integration <maps-l(a)lists.wikimedia.org>
Message-ID: <4E50F5DC.8000106(a)s2002.tu-chemnitz.de>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
The plan is that if we catched up we will reactivated the expire-script
and begin than to render all tiles z=9-15 systematically, after this we
will rendering systematically all tiles that are older than a year
(expired) and repeat this if we are done. I'm not sure what we do with
z=16-18, I would say we do this like in the past.
Greetings Tim
Am 21.08.2011 13:24, schrieb Peter K?rner:
>> >
> someome suggested to turn down rendering until the database has catched
> up. I did this by disabling the dirty queue, so that the missing tiles
> are still rendered.
>
> Peter
------------------------------
Message: 4
Date: Mon, 22 Aug 2011 22:19:03 +0200
From: Kay Drangmeister <kay(a)drangmeister.net>
Subject: Re: [Maps-l] tirex "dirty" queue is turned off?
To: Map integration <maps-l(a)lists.wikimedia.org>
Message-ID: <4E52B9B7.6020107(a)drangmeister.net>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Am 21.08.2011 13:24, schrieb Peter K?rner:
> Am 21.08.2011 13:22, schrieb Kay Drangmeister:
>> tirex "dirty" rendering queue is inactive.
> someome suggested to turn down rendering until the database has catched
> up. I did this by disabling the dirty queue,
Thanks for the explanation (is there a wiki page about the current
status showing this?). When is the projected date/time the DB should
have catched up? The age of the queue suggest it's been more than
11 days already... Is there a monitoring possibility?
Thanks,
Kay
------------------------------
Message: 5
Date: Mon, 22 Aug 2011 22:36:38 +0200
From: Tim Alder <tim.alder(a)s2002.tu-chemnitz.de>
Subject: Re: [Maps-l] tirex "dirty" queue is turned off?
To: Map integration <maps-l(a)lists.wikimedia.org>
Message-ID: <4E52BDD6.4010306(a)s2002.tu-chemnitz.de>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Hello Kay,
in this mailing list you would find the answer.
Peter wrote a little tool to show the replag:
http://toolserver.org/~mazder/tirex-replag/?refresh=0&human=1
So we have now a replag of 4 days and I would say we need 3 days to
be up-to-date.
Unfortunately is munin-graph again out of order:
http://munin.toolserver.org/OSM/ptolemy/replication_delay2.html
and jira-request TS-1125 after nearly a month still untouched. :-(
I believe it would be difficult for us to keep the wiki up-to-data, so
let's use this list. But I know that's not the optimal medium if
somebody is coming for a long vacation or so, so it's not the problem
that you ask.
Greetings Kolossos
Am 22.08.2011 22:19, schrieb Kay Drangmeister:
> Am 21.08.2011 13:24, schrieb Peter K?rner:
>> Am 21.08.2011 13:22, schrieb Kay Drangmeister:
>>> tirex "dirty" rendering queue is inactive.
>> someome suggested to turn down rendering until the database has catched
>> up. I did this by disabling the dirty queue,
>
> Thanks for the explanation (is there a wiki page about the current
> status showing this?). When is the projected date/time the DB should
> have catched up? The age of the queue suggest it's been more than
> 11 days already... Is there a monitoring possibility?
>
> Thanks,
> Kay
>
>
> _______________________________________________
> Maps-l mailing list
> Maps-l(a)lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/maps-l
>
------------------------------
Message: 6
Date: Sat, 27 Aug 2011 23:11:04 +0200
From: Tim Alder <tim.alder(a)s2002.tu-chemnitz.de>
Subject: [Maps-l] Starting of re-rendering all tiles
To: Map integration <maps-l(a)lists.wikimedia.org>
Message-ID: <4E595D68.7040402(a)s2002.tu-chemnitz.de>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Hello, we have now an up-to-date database, and expiring is working again.
I start to rendering all tiles z=9-18 systematically, that are watched
in the last 64 days and older than 3 days (because expire didn't work a
long time). I put additional z=16-18 in the batch because it is else
difficult to handle in different processes.
We rendering 150-200 tiles/min what is really nice.
There are 3 Mio. tiles in the batch so it will need around 11 days.
After this we need to re-render only tiles that are older than a year
because expiring is again working, so there will be than much fewer
tiles to handle.
Greetings Kolossos
Am 21.08.2011 14:11, schrieb Tim Alder:
> The plan is that if we catched up we will reactivated the expire-script
> and begin than to render all tiles z=9-15 systematically, after this we
> will rendering systematically all tiles that are older than a year
> (expired) and repeat this if we are done. I'm not sure what we do with
> z=16-18, I would say we do this like in the past.
>
> Greetings Tim
>
> Am 21.08.2011 13:24, schrieb Peter K?rner:
>>> >
>> someome suggested to turn down rendering until the database has catched
>> up. I did this by disabling the dirty queue, so that the missing tiles
>> are still rendered.
>>
>> Peter
>
------------------------------
Message: 7
Date: Sat, 27 Aug 2011 23:31:15 +0200
From: "Kay Drangmeister" <kay(a)drangmeister.net>
Subject: Re: [Maps-l] Starting of re-rendering all tiles
To: "Map integration" <maps-l(a)lists.wikimedia.org>
Message-ID: <op.v0v52dh9hiwlwc(a)afa-30100.empolis.local>
Content-Type: text/plain; charset=iso-8859-15; format=flowed;
delsp=yes
Hi,
Am 27.08.2011, 23:11 Uhr, schrieb Tim Alder <tim.alder(a)s2002.tu-chemnitz.de>:
> Hello, we have now an up-to-date database, and expiring is working again.
Looks great.
http://toolserver.org/~mazder/tirex-status/?short=1&extended=0&refresh=0
shows a new queue "systema", I guess that's exactly it.
One observation: currently putting re-rendered tiles into the queue is
a bit quicker than the queue rendering tiles, so the queue grows slowly
over time. Nothing to worry I guess, tho'. But: should the AGE of the
tiles not gradually decrease, as the queue is supposed to render oldest
tiles first?
And is "dirty" now separate from the "systema" queue (as is apparent)?
If I try to manually dirty a metatile, I don't see it appearing anywhere
on the page linked above.
Kind regards,
Kay
------------------------------
Message: 8
Date: Sun, 28 Aug 2011 00:09:31 +0200
From: Tim Alder <tim.alder(a)s2002.tu-chemnitz.de>
Subject: Re: [Maps-l] Starting of re-rendering all tiles
To: Map integration <maps-l(a)lists.wikimedia.org>
Message-ID: <4E596B1B.70800(a)s2002.tu-chemnitz.de>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Am 27.08.2011 23:31, schrieb Kay Drangmeister:
> ... But: should the AGE of the
> tiles not gradually decrease, as the queue is supposed to render oldest
> tiles first?
No, as far I know the queue is only order by numbers of requests.
>
> And is "dirty" now separate from the "systema" queue (as is apparent)?
> If I try to manually dirty a metatile, I don't see it appearing anywhere
> on the page linked above.
Yes, that's really still a problem. I didn't find a trick how to set
dirty-tiles to prio=3 or to deactivated the dirty-bucket and push the
list under "bulk" (prio=10) into tirex.
Greetings Kolossos
------------------------------
Message: 9
Date: Mon, 29 Aug 2011 10:22:14 +0200
From: Peter K?rner <osm-lists(a)mazdermind.de>
Subject: Re: [Maps-l] Starting of re-rendering all tiles
To: Map integration <maps-l(a)lists.wikimedia.org>
Message-ID: <4E5B4C36.5080303(a)mazdermind.de>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Am 27.08.2011 23:31, schrieb Kay Drangmeister:
> Hi,
>
> Am 27.08.2011, 23:11 Uhr, schrieb Tim Alder<tim.alder(a)s2002.tu-chemnitz.de>:
>> Hello, we have now an up-to-date database, and expiring is working again.
>
> Looks great.
> http://toolserver.org/~mazder/tirex-status/?short=1&extended=0&refresh=0
> shows a new queue "systema", I guess that's exactly it.
>
> One observation: currently putting re-rendered tiles into the queue is
> a bit quicker than the queue rendering tiles, so the queue grows slowly
> over time. Nothing to worry I guess, tho'. But: should the AGE of the
> tiles not gradually decrease, as the queue is supposed to render oldest
> tiles first?
The queue is sorted by the number of requests received for this tile,
not by age.
> And is "dirty" now separate from the "systema" queue (as is apparent)?
> If I try to manually dirty a metatile, I don't see it appearing anywhere
> on the page linked above.
when tirex receives a dirty message from mod_tile (metatile oder then
planet-timestamp), it enqueues it with prio 2, so it's put into the
systema bucket. This can be changed in source [1] (i guess the second
int in the array is the priority for cmdRender).
Also, if you enqueue a tile that is already in one of the queues, it's
not added a second time but just moved up a little in the queue.
Peter
[1]
<http://trac.openstreetmap.org/browser/applications/utils/tirex/lib/Tirex/So…>
------------------------------
Message: 10
Date: Mon, 29 Aug 2011 08:40:49 -0600
From: Kai Krueger <kakrueger(a)gmail.com>
Subject: Re: [Maps-l] Starting of re-rendering all tiles
To: Map integration <maps-l(a)lists.wikimedia.org>
Message-ID: <4E5BA4F1.1070101(a)gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
On 08/29/2011 02:22 AM, Peter K?rner wrote:
> Am 27.08.2011 23:31, schrieb Kay Drangmeister:
>> Hi,
>>
>> Am 27.08.2011, 23:11 Uhr, schrieb Tim Alder<tim.alder(a)s2002.tu-chemnitz.de>:
>>> Hello, we have now an up-to-date database, and expiring is working again.
>>
>> Looks great.
>> http://toolserver.org/~mazder/tirex-status/?short=1&extended=0&refresh=0
>> shows a new queue "systema", I guess that's exactly it.
>>
>> One observation: currently putting re-rendered tiles into the queue is
>> a bit quicker than the queue rendering tiles, so the queue grows slowly
>> over time. Nothing to worry I guess, tho'. But: should the AGE of the
>> tiles not gradually decrease, as the queue is supposed to render oldest
>> tiles first?
> The queue is sorted by the number of requests received for this tile,
> not by age.
Will this not quickly mess up the systematic ordering of the systema
queue? Is it possible to turn off this reordering?
Has this got something to do with that throughput has dropped from 160 -
200 down to (still better than before) 60 - 70 tiles / minute? But
overall, it does seem the systematicity is working.
>
>> And is "dirty" now separate from the "systema" queue (as is apparent)?
>> If I try to manually dirty a metatile, I don't see it appearing anywhere
>> on the page linked above.
>
> when tirex receives a dirty message from mod_tile (metatile oder then
> planet-timestamp), it enqueues it with prio 2, so it's put into the
> systema bucket. This can be changed in source [1] (i guess the second
> int in the array is the priority for cmdRender).
Perhaps the mappings from mod_tile render request types to tirex
priority levels could be made configurable.
Alternatively, one could tune the mod_tile config to send requests as
cmdDirty rather than cmdRender (the latter assumes the request can be
rendered on the fly, which isn't the case if about 300k tiles are ahead
of it in the queue) mod_tiles cmdDirty maps onto tirex prio level 10 by
default.
Kai
P.S. In the run.log file, at the end of each import, there are messages
of the form
26799 import done
[2011-08-29 02:11:54] 26799 expiring tiles
[2011-08-29 02:11:54] 26799 [error] tile_expiry error
[2011-08-29 02:11:55] 26799 resetting state
Does the "resetting state" have anything to do with that the replag is
not going down?
>
> Also, if you enqueue a tile that is already in one of the queues, it's
> not added a second time but just moved up a little in the queue.
>
> Peter
>
>
> [1]
> <http://trac.openstreetmap.org/browser/applications/utils/tirex/lib/Tirex/So…>
>
> _______________________________________________
> Maps-l mailing list
> Maps-l(a)lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/maps-l
------------------------------
Message: 11
Date: Mon, 29 Aug 2011 21:36:07 +0200
From: Tim Alder <tim.alder(a)s2002.tu-chemnitz.de>
Subject: Re: [Maps-l] Starting of re-rendering all tiles
To: Map integration <maps-l(a)lists.wikimedia.org>
Message-ID: <4E5BEA27.8040006(a)s2002.tu-chemnitz.de>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Am 29.08.2011 16:40, schrieb Kai Krueger:
> Will this not quickly mess up the systematic ordering of the systema
> queue? Is it possible to turn off this reordering?
>
> Has this got something to do with that throughput has dropped from
>160 - 200 down to (still better than before) 60 - 70 tiles / minute?
Hello, my theory is that we started with rendering around Alaska.
This area is relative empty and stretched by mercator-projection.
Now we are somewhere in Canada/USA incl. NYC with high density of
geo-objects.
Following the z-curve we should come back to empty areas in the north
several times, and render again very fast. If not, my theory is false.
In general:
Tirex should be able to order the queue by z-curve, especially if you
have more than one style (not sure how it can reach the end of it), it
would makes it for me much easier to handle it. And Tirex should be able
to not order the queue.
>> when tirex receives a dirty message from mod_tile (metatile oder then
>> planet-timestamp), it enqueues it with prio 2, so it's put into the
>> systema bucket. This can be changed in source [1] (i guess the second
>> int in the array is the priority for cmdRender).
I change the parameter to 4 in
~/tirex/lib/vendor_perl/5.12/Tirex/Source/ModTile.pm
It seems that it need a restart of tirex, what I don't want to do in the
moment.
>
> Alternatively, one could tune the mod_tile config to send requests as
> cmdDirty rather than cmdRender (the latter assumes the request can be
> rendered on the fly, which isn't the case if about 300k tiles are ahead
> of it in the queue) mod_tiles cmdDirty maps onto tirex prio level 10 by
> default.
No idea where to do this.
>
>
> Kai
>
> P.S. In the run.log file, at the end of each import, there are messages
> of the form
>
> 26799 import done
> [2011-08-29 02:11:54] 26799 expiring tiles
> [2011-08-29 02:11:54] 26799 [error] tile_expiry error
> [2011-08-29 02:11:55] 26799 resetting state
>
> Does the "resetting state" have anything to do with that the replag is
> not going down?
>
Would be important to know. It seems to work at beginning.
------------------------------
_______________________________________________
Maps-l mailing list
Maps-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/maps-l
End of Maps-l Digest, Vol 28, Issue 4
*************************************
------------------------------
Message: 3
Date: Tue, 30 Aug 2011 22:57:06 +0200
From: Tim Alder <tim.alder(a)s2002.tu-chemnitz.de>
Subject: Re: [Maps-l] Starting of re-rendering all tiles
To: Map integration <maps-l(a)lists.wikimedia.org>
Message-ID: <4E5D4EA2.6070603(a)s2002.tu-chemnitz.de>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Hello,
we are rendering now somewhere in the middle of atlantic[1].
So I would like to know why we are not so fast as at the begining?
If I look in psql the most queries are <IDLE>.
We are rendering now 60 Metatiles/minute, so we are not bad, but I would
like to understand.
Greetings Tim
[1]
http://openstreetmap.gryph.de/bigmap.cgi?zoom=14&xmin=7656&xmax=7664&ymin=5…
------------------------------
Message: 4
Date: Mon, 05 Sep 2011 19:38:24 +0200
From: Tim Alder <tim.alder(a)s2002.tu-chemnitz.de>
Subject: [Maps-l] replag is growing /no contact to planet.osm.org
To: Map integration <maps-l(a)lists.wikimedia.org>
Message-ID: <4E650910.40104(a)s2002.tu-chemnitz.de>
Content-Type: text/plain; charset=ISO-8859-15; format=flowed
Hello,
maintenance on OSM.org is now over but our replag of ptolemy is still
growing.
We are not able to contact the diff-server from OpenStreetMap:
osm@ptolemy:~$ ping planet.openstreetmap.org
no answer from planet.openstreetmap.org
Ping works fine from nightshade or my laptop.
I hope nobody block us.
We change nothing on the config.
Greetings Kolossos
------------------------------
Message: 5
Date: Mon, 5 Sep 2011 21:42:31 +0100
From: Grant Slater <openstreetmap(a)firefishy.com>
Subject: Re: [Maps-l] replag is growing /no contact to planet.osm.org
To: Map integration <maps-l(a)lists.wikimedia.org>
Message-ID:
<CAGXuowhCVB6e_KJgHS5g3VeOOKvEwbyLiF1L1o1gq_Mj5CiFHg(a)mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1
On 5 September 2011 18:38, Tim Alder <tim.alder(a)s2002.tu-chemnitz.de> wrote:
> Hello,
> maintenance on OSM.org is now over but our replag of ptolemy is still
> growing.
>
> We are not able to contact the diff-server from OpenStreetMap:
> ? ? ? ?osm@ptolemy:~$ ping planet.openstreetmap.org
> ? ? ? ?no answer from planet.openstreetmap.org
>
> Ping works fine from nightshade or my laptop.
>
> I hope nobody block us.
> We change nothing on the config.
>
I am part of the OSM sysadmin team. There are no blocks from our side,
although ping replies are rate limited which may be causing the ping
issue.
Tried (or similar) from the host?: w3m http://planet.osm.org/
/ Grant
------------------------------
_______________________________________________
Maps-l mailing list
Maps-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/maps-l
End of Maps-l Digest, Vol 29, Issue 1
*************************************
