Hello all,
I have developed a client-side tile renderer for the WikiMiniAtlas
last week. It enables me to support virtually limitless zooming in.
GeoJSON data is generated from the osm-mapnik database and transmitted
to the browser, where map tiles are rendered into canvas elements.
This allows me to change the map style without and server side changes
(re-rendering, cache purging etc.). I'm working on a set of styles to
emphasize certain structures on the map (subways, tunnels, bridges,
raillines, monuments, etc. ).
Kolossos just posted about the visualization of time dependent data.
My client side rendered tiles would support start_date and end_date
fairly effortlessly. I am transmitting all data to the client anyways
and could check for these time tags as well and display buildings
accordingly (no prerendered tiles necessary, minimizing space
requirements on the server).
The client-side tiles are active now in WikiMiniAtlas so check it out
if you are interested.
Daniel
P.S.: just keep it in the back of your head: with WikiMiniAtlas there
is another project bringing OSM data into Wikipedia...
Hello everyone,
After just writing last week that everything is running stable on the
OpenStreetMap tile rendering server, the next day, the postgresql
database seems to have gotten corrupted and is only partially functional
anymore.
Due to the database corruption, replication (diff imports) are suspended
for the moment and most rendering of new map tiles is disabled as well.
This will effect both WIWOSM and the osm tiles showen in the osm_gadget.
Any changes that occurred in the last 3 days or so as well future
updates to the OSM database won't show up in either until the
toolserver-osm database can be fixed again. This is unfortunately likely
going to take a few days, if not weeks should a full new import be
necessary due to the corruption.
More technically:
Initially queries seemed to fail every couple of hours with the error
message
"DETAIL: The postmaster has commanded this server process to roll back
the current transaction and exit, because another server process exited
abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and
repeat your command.
The connection to the server was lost. Attempting reset: Failed."
But otherwise more or less "work".
Now though diff imports quickly fail, and e.g. the query "SELECT * FROM
planet_ways WHERE id = 67780465;" consistently results in the error
"unexpected chunk number 0 (expected 1) for toast value 28214399 in
pg_toast_3406700" which sounds like definite database corruption.
I have no idea what caused the issues, but they seemed to start in the
evening of the 23rd.
It appears that the planet_ways table definitely has problems, but I
don't know if the rendering tables are corrupt as well. Potentially this
means that processes that work on the other tables still work, although
I'd treat them with care, as it may well be that they are corrupt as well.
i will see if reindexing the tables or vacuuming them will help to
recover the corruption.
If not, then a full reimport will presumably be necessary.
This is a bit of an awkward time for this, as due to the license change
in OSM, there are no up-to-date planet files at the moment (last one was
3 weeks ago) and will likely not resume until the license change is
over. When that will be is not yet entirely clear. So that might add
another couple of weeks of delay until a new import can take place.
Hopefully this can be resolved as soon as possible, but it will likely
be an inconvenience to WIWOSM and tile rendering.
Kai
Hello, I fwd this to maps-l because I believe mapstory.org is really
interesting to showing developments on OpenStreetMaps inside Wikipedia.
At Wikimania we talk also about how we can bring history more in the
focus of OSM, so promoting the "start_date"[1] for buildings would be a
good start.
Greetings Kolossos
[1] http://wiki.openstreetmap.org/wiki/Key:start_date
-------- Original-Nachricht --------
Betreff: creating a MapStory Labs project
Datum: Fri, 27 Jul 2012 18:13:30 -0500
Von: Nitin Gadia <nittyjee(a)gmail.com>
Antwort an: Wikimedia Labs <labs-l(a)lists.wikimedia.org>
An: labs-l(a)lists.wikimedia.org
Kopie (CC): Christopher Tucker <christopher.tucker(a)gmail.com>
Newsgruppen: gmane.org.wikimedia.labs
Hello,
We recently attended Wikimania in Washington DC, where we met Erik
Möller <http://en.wikipedia.org/wiki/Erik_M%C3%B6ller>, who showed us
Wikimedia Labs, telling us it would be a good fit. We're interested in
helping create a Wikimedia Labs project around serving /mapstories/. A
mapstory is a type of map that shows change over time, which is, like
wikipedia or openstreetmap (OSM, www.openstreetmap.org
<http://www.openstreetmap.org>), being developed in an online community
built on the same standards of openness, at /www.mapstory.org/
<http://www.mapstory.org>. We are aware that wikimedia and openstreetmap
(OSM) are collaborating to serve OSM data through wikimedia to be seen
on wikipedia articles, and would love to do the same. MapStory is
inspired by both openstreetmap and wikipedia, and is being built on the
same standards of openness, and intends to be yet another companion in
the open knowledge community, and is to be governed in similar ways by
the MapStory Foundation. At the moment, MapStory is invitation-based, as
it is being developed, but it will be fully open to anyone's registration.
Examples of mapstories include:
*Events: Yellowstone Fire: http://mapstory.org/maps/149
*Trends: Population Growth: http://mapstory.org/maps/162
*General historical change: Africa: http://mapstory.org/maps/153,
walmart: http://mapstory.org/maps/60, NYC subway:
http://mapstory.org/maps/117/view
*Large scale mapstories - some mapstories will be enormous, allowing you
to eventually type in a time, and see the way the world was then. So, a
user might type in "1900", and see all the roads, buildings, land use,
and landscape down to very local levels, like you can with OSM now.
The integration of mapstories into wikipedia and other wikimedia
projects would be extremely fruitful. I can imagine that any article
about political entities, events, and statistics can, and probably will,
have a mapstory. I'll reply to this message with some examples in a moment.
*
What do we need to do in order for us to establish a MapStory labs
project, or add ourselves to an existing one? *Unfortunately, I am not a
developer myself, but we can at least get started while the developers
crank out the core mapstory functionality.
I've cc-ed Christopher Tucker, the founder of MapStory.
Please reply-all if you can :)
Thanks,
Nitin Gadia
Ames, Iowa, USA
MapStory Foundation
/www.mapstory.org/ <http://www.mapstory.org>
Hello everyone,
Overall running the Wikipedia-OpenStreetMap server on the toolserver
cluster (in particular on server Ptolemy) has worked reasonably well and
hasn't needed too much maintenance attention recently anymore. It also
seems to have handled serving map tiles to the OSM-gadget in various
Wikipedia's including the German, Spanish and Russian Wikipedia amongst
others pretty well and more recently also WIWOSM.
However, there remain a number of issues that have never been resolved
entirely satisfactorily on Ptolemy
1) There is still a memory leak in tirex-master as well as a creep in
CPU usage over time. This has to some degree been solved by simply
restarting Tirex every 12 hours in a cron job, very much limiting the
scope of the memory leak and to a lesser degree also the CPU usage
creep. This however means that the request queues are dropped every time.
2) The socket between tirex and mod_tile / render_list always gets
closed before the the successful acknowledgment can be sent from tirex.
This means the requester can't tell if the rendering was successful. In
mod_tile this results in returning http 404 errors for tiles that need
rendering, instead of returning the tiles that were rendered on the fly.
In render_list I got around the issue by simply always assuming the
render request was successful and reconnecting to the socket for each
request.
3) The socket between tirex and mod_tile / render_list refuses
connection for a (random) subset of connection requests. This results in
quite a number of rendering requests from mod_tile being dropped as it
can't connect to the tirex socket. In render_list, I could again work
around the problem by sleeping for n seconds and then retrying the
connection until it eventually succeeds.
4) The performance of postgresql is still below what can be expected
from the server of the specs of Ptolemy. While clustering the rendering
tables by the geometry column several months ago brought up the
performance to a level that it can now more or less keep up with the
limited re-rendering load put on the server from Wikipedia, low-zoom
rendering is still exceptionally slow. Also it can barely keep up with
replication from the OSM servers, and not infrequently drops behind
during busy times. Other servers with much slower I/O performance on the
other hand seem to have no problem keeping up with diff-imports.
Non of these issues are directly critical, but would be nice to solve.
Next to the OpenStreetMap server hosted by the toolserver, the wikimedia
foundation is planning on hosting their own OpenStreetMap tile-server,
at least for the mobile wikipedia client, but presumably also for
inclusion in the "standard" wikipedias that are currently served by the
toolserver. If I understand correctly, they have already purchased the
hardware and are awaiting provisioning until somewhat can puppetize the
OSM tileserver rendering stack.
Secondly, if I understand it correctly, the toolserver cluster is slowly
moving back from Solaris to Debian.
Now with the OpenStreetMap switch of license from CC-BY-SA to ODbL for
the raw map data soon to be completed. (The database has now mostly been
purged of data for which the OSMF did not get permission to relicense),
my understanding is that the OSMF will likely recommend everyone to do a
fresh import of a new ODbL licensed planet for legal purposes, rather
than to apply the soon to be ODbL licensed diffs to a base CC-BY-SA
database.
One question is, would this forced full re-import of the OpenStreetMap
database, which last time took approximately 4 days, be a good
opportunity to change things in the setup? For example could this be
used to migrate from Solaris to Debian? Some of the above mentioned
issues might be solved by a change of OS. Furthermore, it should be
easier and better tested to upgrade the OSM rendering stack.
Unfortunately key components in Debian Squeeze appear to be quite old,
including postgres with version 8.4, mapnik with version 0.7 and the
boost library (which would be necessary to compile mapnik 2.0) (They are
sufficiently recent in Debian Sid, or Ubuntu 12.04), so that they would
likely need back ports of self compilation.
However, an issue is that ptolemy is in productive use in several key
wikipedia's serving the map tiles. So a replacement would be necessary
before being able to take it down for upgrades. At least for the tiles
(I am not sure what the requirement is for WIWOSM), the main issue is
serving tiles. (Re-)rendering would be suspended for several days anyway
during an import. If I understand it correctly, the tiles are currently
stored on the toolserver SAN. All that would be needed would be a apache
webserver with mod_tile installed.
So the main question is, would this be a good time to change things? Are
there any other problems / issues that need fixing or improving? Or
should we simply re-import a fresh ODbL planet into the existing setup,
once the license change has completed?
Kai
Hello,
WIWOSM is activated now, but if it not work in your language version of
Wikipedia it can be that the article name will not transfer from
Wikipedia to map aplication. Check this on articles like "Berlin".
For repair this there are two ways:
*Add the article title to the URL of the map inside MediaWiki:Common.js:
+ '&title=' + mw.util.wikiUrlencode( mw.config.get( 'wgTitle' ) )
*Better way seems the installation by using Metawiki:
http://de.wikipedia.org/wiki/Hilfe:OpenStreetMap/en#Installation
This keeps you up-to-date and clean-up your local javaScript.
Greetings Tim alias Kolossos
Hello everyone,
as many of you will already know, OpenStreetMap has been planning to
change its license from CC-BY-SA 2.0 to OdBL 1.0 for a long time now
(several years). It appears that the date of the actual change is now
approaching fast and that the intended plan[3] suggests that the change
will begin on the 27th of March and be complete by the 1st of April.
As, unlike Wikipedia, with its change to CC-BY-SA, OSM so far has not
had a clause in the license to switch to a different license, it had to
independently and separately ask all of its contributors to relicense
their data to the new license.
In a project with more than 200.000 contributors reaching all of them
and getting them to agree to a new license is an (near) impossible task.
Therefore despite OSM(F)'s best effort to reach as many mappers as
possible, obviously not all could be reached and convinced to agree.
As OSMF does not hold the copyright (the individual mappers do), it can
only re-license the content for which it has permission and has to
delete all other data.
Worldwide currently 2% - 3% of data are affected[1], but for some
regions or countries 10% or more of data have to be deleted.
The question for Wikipedia (in the sense of the maps rendered on the
toolserver and embedded in a number of language wikipedia's) now is how
to handle the license transition?
My suggestion is to freeze the toolserver's openstreetmap database on
the 27th and stop all data updates for now. After the dust settles on
the license change and the impact of the deleted data become more
apparent, one can reevaluate the situation and decide at what point the
missing updates are worth more than the deleted data. At that point a
fresh import of the OSM database would be necessary after which the
updates can resume. My guess would be that it would take a few weeks or
perhaps month for the most visible damage to the data to be fixed, but
it is very hard to predict this so far. An overview of the current
situation can be seen on cleanmap [2], which shows either what the map
will look like after the license change, or on badmap which data needs
to be removed or modified before the license change.
After the re-import, it will also require a slight adaptation of the
attribution to reflect the license change to OdBL. The tiles (which are
embedded in Wikipedia) I believe can still be published under CC-BY-SA
(it is only the data that is under OdBL, not the produced work) and I
don't think it will directly impact (or prevent) any of the current
other usage of OpenStreetMap data in Wikipedia. However, I don't know if
the Wikimedia legal council has or wants to say anything about it and /
or give its OK for continued inclusion?
What do others think? How should the OSM license change and the
resulting removal of data from its database be handled in Wikipedia? Do
people agree with the above suggestion? Does it have any other
consequences? Further thoughts?
Kai
[1] http://odbl.poole.ch/
[2] http://cleanmap.poole.ch/
[3] http://www.osmfoundation.org/wiki/License/Rebuild_Plan
[4] http://www.osmfoundation.org/wiki/License/About_The_License_Change
I activate now WIWOSM[1] on all language versions. But it's still
beta-status because it's not running on IE (we work on that.).
I will promote this feature at WIKIMANIA, but perhaps it's possible for
you also to talk with other users to make it more popular.
We have in the moment 168.000 Wikipedia-tags in OSM but nearly 2 Mio.
coordinates in Wikipedia, so there is a lot of work to do.
To connect both projects we create two tools:
*Wikipedia-Plugin in JOSM
*http://wiki.openstreetmap.org/wiki/JOSM/Plugins/RemoteControl/Add-tags
(Add-tags runs now faster by using Overpass API for types: nodes, ways
and relations instead of points, lines and areas.)
Greetings Kolossos
[1] http://wiki.openstreetmap.org/wiki/WIWOSM
[2] http://taginfo.openstreetmap.org/keys/wikipedia