Hi,
I'm sorry about the state of the OSM Toolserver. I said I would fix it, but
I didn't. However, I will finally be having a look at this tomorrow (or
possibly Thursday). Before I do that, there's a couple of things I'd like
to address.
Firstly, PostgreSQL. On cassini, some users had either root access or
access to the postgres user. I don't feel comfortable doing this on ptolemy
(and it would also mean using separate database passwords, etc. just for
that server). However, I can give out full (owner) access to the 'osm'
database.
Rather than give this to a single user, I think it makes sense to create a
multi-maintainer project to handle OSM databases. An MMT[0] is a role
account that multiple users have access to. The 'osm' MMT will have owner
access to the osm database on ptolemy. Since this does not confer any
additional privileges, and the OSM data is not private, we can easily add
people to the MMT (Peter, Tim?), and these people will be responsible for
ensuring the database is available.
This is better than the current situation where only I have access, since
I'm not particularly familiar with either OSM or the PostgreSQL extensions
it uses. However, final responsibility for the OSM database (as well as
software installation/changes) will still lie with me.
If this sounds acceptable to everyone, I will drop the current outdated osm
database on ptolemy, and create a new, empty database and the MMT. People
who want access to the MMT should let me know (by replying to this mail with
your TS username). Those people can then start on the process of importing
the database and setting up replication.
Secondly: renderd/mod_tile. I'm open to suggestions here. Currently, I
plan to run renderd and mod_tile on ptolemy, which means only I will have
access. However, there are quite a few portability issues with the OSM
software which I've had to fix. I would appreciate it if someone with
commit access to OSM's SVN repository would be able to work with me to
integrate these patches, to make things easier in future.
I will wait until PostgreSQL is working well before starting on rendered.
Regards,
River.
[0] https://wiki.toolserver.org/view/Multi-maintainer_projects
forwarding to dev,
we're talking about a standard osm2pgsql setup. Does anyone know what's
going on with the "pending" index?
Peter
-------- Original-Nachricht --------
Betreff: Re: [Maps-l] OSM Toolserver - spatial_ref_sys
Datum: Mon, 22 Feb 2010 16:19:02 +0000
Von: Kai Krueger <kakrueger(a)gmail.com>
An: Map integration <maps-l(a)lists.wikimedia.org>
CC: Peter Körner <osm-lists(a)mazdermind.de>
On 22/02/10 12:39, Peter Körner wrote:
> So we're nearly done, we're sth. around one hour behind the main DB.
>
> > Great, but again it seems rather slow.
> The problem is, that the "Going over pending ways / relations" phase
> takes too long for small window sizes.
>
> Importing 2 Minutes of changes takes around 15 seconds for the
> "Processing: Node(k) Way(k) Relation(k)" phase and 5 minutes for the
> "Going over pending.." phase. That's just too long.
The "Going over pending.." phase amongst others executes a SELECT id FROM
planet_osm_ways WHERE pending; which seems to require a seq_scan on
planet_osm_ways.
However, a single seq scan on ways seems to take on the order of 8
minutes in my
test.
Likewise, seq_scans on the other tables also take very long. A sec_scan
on planet_osm_point takes about a minute, on planet_osm_polygon about 3
minutes and on planet_osm_line it seems to take over ten minutes for a
single seq_scan. Given that for low zoom levels (before the spatial
indices kick in), there are several queries per tile requiring full
table scans, this doesn't look good for rendering performance. I don't
have any comparison numbers from other servers that have the full planet
loaded to see if that is normal, but perhaps we can do some comparisons
with cassini?
Kai
P.S. there does seem to be an index on "pending" but trying to run some
queries from psql, it didn't seem to be used.
>
> I'm constantly adjusting the window size to find the best match between
> runtime and window size but as I'd like to have a 1 minute window, as we
> have it on cassini, we'll need to tweak a little somewhere.
Hello all,
because of an kernel-update, I will reboot nightshade and cassini tonight
(~0:30 o'clock UTC). The downtime should be 5 minutes, but it can expand up to
30min.
The process can be follow on [1].
Sincerly,
DaB.
[1] https://jira.toolserver.org/browse/MNT-289
--
wp-blog.de
Hi
As I announced the hourly and minute diffs on planet.openstreetmap.org
has been dropped in favor of the replication diffs. It seems, noone
changed ptolemy to the replication scheme, so it's far out of up2date.
select max(id) from planet_osm_nodes ;
on ptolemy results in 580146831, while cassini, which uses the
minute-replicate diffs says 622311437.
It seems, ptolemy stopped replication around 04/12/2009, 00:00, which is
the creation time of 580146831 + 1 (see [1]).
As I said before I'd love to set everything up on ptolemy just as I did
on cassini. I'd also volunteer to set up the rendering on ortelius.
I'll talk about the Wikimedia Toolserver and OSM on the FOSSGIS 2010 but
I'm really unsure what I should tell the audience. The Software and Data
seems not to be maintained (I'm talking about the OSM Software Stack,
not the Server as such -- they are of course well maintained) and
volunteers, that already proved to be able to fill this gap, don't
receive any response. This is so sad.
Peter
[1] <http://www.openstreetmap.org/browse/node/580146832>
Hello again.
At Sunday 31 January 2010 14:14:13 DaB. wrote:
> I can give no link to a jira-ticket, because jira is down since yesterday.
after jira is back now (thanks to River), the ticket-url is
https://jira.toolserver.org/browse/MNT-205. I will also reboot cassini for the
same case too, so I added my original mail below for the OSM-folks.
Sincerly,
DaB.
>Hello all,
>
>for an important kernel-update, nightshade will reboot around 23 o'clock UTC.
>The downtime should be short, but all running programms on nightshade will of
>corse stop. So please make sure that they restart after the reboot or move
>them temporary to willow (caution. solaris! ;)).
>
>I can give no link to a jira-ticket, because jira is down since yesterday. I
>will send a email after the reboot to inform you all.
>
>Sincerly,
>DaB.
>
--
wp-blog.de