Maps-l November 2009

maps-l@lists.wikimedia.org

12 participants
14 discussions

Restart of apache and updating of php on cassini
by DaB. 27 Nov '09

27 Nov '09

Hello all, the debian-security-folks updated php because of some security-problems (if you like details, I added the report below). The update requires an update and restart of apache too. For this reason, I will update and restart php and apache on cassini tonight (between 1 and 2 o'clock UTC) . The "downtime" of apache should be only a few minute at maximum. You can see the progress at [1]. Sincerly, DaB. [1] https://jira.toolserver.org/browse/MNT-16 --- News for php5 (php-pear php5 php5-cgi php5-cli php5-common php5-curl php5- gd php5-mysql php5-pgsql) --- php5 (5.2.6.dfsg.1-1+lenny4) stable-security; urgency=high * Maximum number of file uploads per request limited To prevent Denial of Service attacks by exhausting the number of available temporary file names, the max_file_uploads option introduced in PHP 5.3.1 has been backported. Due to the nature of this new option a default limit has been set to 50, hoping it is sensible enough to not to cause disruptions on existing services. The value of this new limit can be changed in the php.ini file. If you installed the php5-suhosin extension there was a limiting mechanism in place already. In this case you may want to make sure the new limit imposed by PHP itself is not smaller than suhosin's. -- Raphael Geissert <geissert(a)debian.org> Sat, 21 Nov 2009 18:13:48 -0600 --- Changes for php5 (php-pear php5 php5-cgi php5-cli php5-common php5-curl php5-gd php5-mysql php5-pgsql) --- php5 (5.2.6.dfsg.1-1+lenny4) stable-security; urgency=high * CVE-2009-2687: DoS via malformed JPEG images with invalid offset fields (Closes: #535888) * CVE-2009-2626: remote memory disclosure via ini_* functions (Closes: #540605) * CVE-2009-3292: multiple missing checks processing exif image data * CVE-2009-3291: improper handling of nul character in CommonName fields of X509 certificates * max_file_uploads: prevent, by limiting, temporary files exhaustion DoS * Add an entry to debian/NEWS about the new per-request file uploads limit -- Raphael Geissert <geissert(a)debian.org> Sat, 21 Nov 2009 18:28:12 -0600 -- wp-blog.de

1 0

Load on cassini
by Peter Körner 27 Nov '09

27 Nov '09

Hi Atm. the load on cassini is around 20 and the diff imports are fighting to stay up. I think it's cmarq's hillshade-generator takes the most resources on cassini. top reports ~57.1% IO waiting so I think there's a lot of disk going on. I tried to do a "du -hs /mnt/user-store/osm_hillshading" but ir ran loooong until I canceled it. What I think is going on is that the generate_tiles script creates a real huge number of single pngs. Unlike mod_tile it is not combining the small tiles into larger chunks (mod_tile packs 8x8 tiles into a metatile). Also unless mod_tile it renders each and every tile in each and every zoomlevel, while mod_tile only renders tiles that are really shown (e.g. it leaves out most of the atlantic at zoomlevel 18, as nobody would view the empty sea at this zoomlevel. Both problems together create a real huge number of files in a single directory which is not good for most filesystems. To make things worse, /mnt/user-store is connected to cassini via NFS, so there's another bottleneck. I don't know it this really is a problem, but I think it's worth talking about it. Peter

4 13

OSM Toolserver, cassini / ptolemy status
by River Tarnell 26 Nov '09

26 Nov '09

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 hi, per previous discussions, we've now reinstalled ptolemy as the OSM Toolserver database, and i'm now working on setting up PostgreSQL and the mapnik database (which is what cassini has). cassini will remain up for now, but will go offline once ptolemy is ready, since it will become Wikimedia's production OSM database. users cannot log into ptolemy. instead, it will be accessed from the normal Toolserver. - river. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (HP-UX) iEYEARECAAYFAksLUf8ACgkQIXd7fCuc5vIhUgCgk1ATOtLNun34E8u8ZLyCfJ+f jR8An1JRpjcUOpwrNqld1lhi+dXkfrkt =Nvwn -----END PGP SIGNATURE-----

4 14

PosgreSQL in Munin
by Peter Körner 15 Nov '09

15 Nov '09

Hi Maybe we could add PostgreSQL to the Munin-Stats? http://muninpgplugins.projects.postgresql.org/ Just think that it could be useful.. Peter

2 1

Errors from /sql/diffs-load-next in cron
by Ævar Arnfjörð Bjarmason 14 Nov '09

14 Nov '09

Why does this happen? ---------- Forwarded message ---------- From: Cron Daemon <root(a)cassini.toolserver.org> Date: Sat, Nov 14, 2009 at 11:35 AM Subject: Cron <root@cassini> /sql/diffs/load-next To: root(a)cassini.toolserver.org ERROR: List of process IDs must follow -p. ********* simple selection ********* ********* selection by list ********* -A all processes -C by command name -N negate selection -G by real group ID (supports names) -a all w/ tty except session leaders -U by real user ID (supports names) -d all except session leaders -g by session OR by effective group name -e all processes -p by process ID T all processes on this terminal -s processes in the sessions given a all w/ tty, including other users -t by tty g OBSOLETE -- DO NOT USE -u by effective user ID (supports names) r only running processes U processes for specified users x processes w/o controlling ttys t by tty *********** output format ********** *********** long options *********** -o,o user-defined -f full --Group --User --pid --cols --ppid -j,j job control s signal --group --user --sid --rows --info -O,O preloaded -o v virtual memory --cumulative --format --deselect -l,l long u user-oriented --sort --tty --forest --version -F extra full X registers --heading --no-heading --context ********* misc options ********* -V,V show version L list format codes f ASCII art forest -m,m,-L,-T,H threads S children in sum -y change -l format -M,Z security data c true command name -c scheduling class -w,w wide output n numeric WCHAN,UID -H process hierarchy

4 6

Re-Enabling the diffs with another state-file
by Peter Körner 11 Nov '09

11 Nov '09

Hi Ævar I'm sorry I have to bother you so much. Can you please execute the following commands on cassini? mv /sql/diffs/state.txt /sql/diffs/state-old.txt mv /sql/diffs/state-new.txt /sql/diffs/state.txt rm /sql/diffs/stop this will re-enable the updates from a state before the error, so we won't loose changes. I patched osmosis with woodpacks patch [1] to cut of relations with more than 32767 members, as those kill as well osmosis as osm2pgsql. The Owner of /sql/diffs/state.txt is root, so I'm not able to do it myself. Maybe we should run the update script under a user that is part of the mapper group. I have to go now but I'll look at it tomorrow morning. Peter [1] Source Patch: http://www.remote.org/frederik/tmp/remove_excessive_members.diff replace Character.MAX_VALUE with 32767 before patching Binary Package: http://www.remote.org/frederik/tmp/osmosis-patched-to-drop-excessive-member…

1 0

renderd crashing on cassini
by Ævar Arnfjörð Bjarmason 11 Nov '09

11 Nov '09

renderd is crashing regularly on cassini. Here's the renderd -f output: """ [...] renderd[23606]: DEBUG: Got command Dirty fd(35) xml(en), z(10), x(528), y(338) renderd[23606]: DEBUG: Connection 18, fd 40 closed, now 20 left renderd[23606]: DEBUG: Got command Dirty fd(39) xml(en), z(10), x(528), y(339) renderd[23606]: DEBUG: Connection 18, fd 35 closed, now 19 left renderd[23606]: DEBUG: Connection 18, fd 39 closed, now 18 left Warning 1: TIFFReadDirectory:/mnt/user-store/osm_hillshading/N45E010-N49E014/N45E010-N49E014_hill.tif: Wrong "StripByteCounts" field, ignoring and calculating from imagelength Warning 1: TIFFReadDirectory:/mnt/user-store/osm_hillshading/N45E010-N49E014/N45E010-N49E014_hill.tif: Wrong "StripByteCounts" field, ignoring and calculating from imagelength terminate called after throwing an instance of 'mapnik::datasource_exception' what(): PSQL error: ERROR: invalid input syntax for type numeric: " " Full sql was: 'SELECT AsBinary("way",'NDR') AS geom,"importance_real","name","natural" from (SELECT *, (to_number(importance, '9.9')) as importance_real FROM planet_osm_point WHERE "natural" IN ('peak')) AS peak WHERE "way" && SetSRID('BOX3D(1210762.528037192 5750510.511950379,1254790.256329453 5794538.240242641)'::box3d, 900913)' Aborted """ And from the postgresql main log: """ 2009-11-11 19:57:12 UTC ERROR: invalid input syntax for type numeric: " " 2009-11-11 19:57:12 UTC STATEMENT: SELECT AsBinary("way",'NDR') AS geom,"importance_real","name","natural" from (SELECT *, (to_number(importance, '9.9')) as importance_real FROM planet_osm_point WHERE "natural" IN ('peak')) AS peak WHERE "way" && SetSRID('BOX3D(1210762.528037192 5750510.511950379,1254790.256329453 5794538.240242641)'::box3d, 900913) 2009-11-11 19:57:13 UTC LOG: unexpected EOF on client connection """ Don't have time to debug this now, meanwhile I'm restarting renderd every 5 minutes in cron..

2 2

Perms of some dirs in /sql/mod_tile/ on cassini
by Peter Körner 11 Nov '09

11 Nov '09

Hi (Avar?) There are some dirs in /sql/mod_tile/ that are owned by root:root and thus can not be written by mod_tile. This is the reason why [1] always returns a 404, while e.g. [2] or [3] don't. Can so. with sudo perms please run find /sql/mod_tile/ -user root | xargs chown www-data:www-data on cassini? Peter [1] http://cassini.toolserver.org/tiles/osm-like/de/4/8/7.png [2] http://cassini.toolserver.org/tiles/osm-like/zh/4/8/7.png [3] http://cassini.toolserver.org/tiles/osm-like/ar/4/8/7.png

2 3

Multilingular tile rendering back up on Cassini
by Ævar Arnfjörð Bjarmason 10 Nov '09

10 Nov '09

The multilingular tiles are back, e.g.: http://cassini.toolserver.org/tile-browse/browse-en.html?zoom=2&lat=40&lon=0 http://cassini.toolserver.org/tile-browse/browse-is.html?zoom=2&lat=40&lon=0 I kicked it a bit, which involved: * Recompiling mapnik/mod_tile/renderd from svn HEAD & installing. Mapnik now has gdal support and more goodies * rm -rf of the old tiles We don't have cmarqu's hikingmap up yet though. mod_tile catches it but renderd never gets the requests, we're trying to find out why. Also the text for the countries (and presumably all text) is a bit too low, compare the English map above with the main OSM one: http://www.openstreetmap.org/?zoom=2&lat=40&lon=0 How were we on updates to the DB anyway?

3 11

Planet-Update on Cassini
by Peter Körner 09 Nov '09

09 Nov '09

Hey ho! Finally, after the last import got interrupted, we got a fresh planet import on cassini with the additional fields. It took around 28 hours because of the new text-fields, i think. As cassini has no java, I can't run osmosis and start the diff imports, but as river & marcin agreed on a plan to make ptolem the new toolserver-db, we should maybe wait for this to happen before putting any more work in cassini. The script I did the import with can be fetched from here: http://cassini.toolserver.org/~mazder/planet-update Maybe it will be helpful un ptolemy, too. Peter

3 4

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

Maps-l November 2009