Hello there, long time no see:)
In the last few days I've been working on the project of getting OpenStreetMap onto Wikimedia as outlined here:
http://techblog.wikimedia.org/2009/04/openstreetmap-maps-will-be-added-to-wi...
Unfortunately I wasn't able to hack on it sooner (but of course other people have been working on it too!) and the project has been somewhat held up by the WM-DE servers being delayed.
Anyway, one thing standing between us and world domination is rendering those static maps, I'm going to implement this but first I'd like to get comments on *how* we'd like to do it, so I've written a plan for doing it:
http://www.mediawiki.org/wiki/Extension:SlippyMap/Static_map_generation
Would generating static images like that be fine for Wikimedia purposes or is this totally crazy? I think it would be just fine, but then again I did write the Cite extension so take that with a grain of salt:)
And to spam a bit: if getting pretty OpenStreetMap maps deployed is something you'd like to happen sooner than later head over to our development page:
http://www.mediawiki.org/wiki/Extension:SlippyMap#Development
I'm working off the Bugzilla tasklist which should be an approximate indication of stuff that needs to be done.
Ævar Arnfjörð Bjarmason wrote:
Hello there, long time no see:)
In the last few days I've been working on the project of getting OpenStreetMap onto Wikimedia as outlined here:
Woohoo!
Anyway, one thing standing between us and world domination is rendering those static maps, I'm going to implement this but first I'd like to get comments on *how* we'd like to do it, so I've written a plan for doing it:
http://www.mediawiki.org/wiki/Extension:SlippyMap/Static_map_generation
I added a couple quickie notes on the talk page...
It might be good to use a Google Maps Static-like model here; the MediaWiki end can just make itself an <img> and stick all the relevant info onto the URL, calling out to the static map server.
Having a basic time-based expiration on rendered images is probably fine for our needs (and easy to handle, especially if we let the squids deal with keeping the static images around!)
-- brion
On Wed, Jul 15, 2009 at 12:42 AM, Brion Vibberbrion@wikimedia.org wrote:
Ævar Arnfjörð Bjarmason wrote:
Hello there, long time no see:)
In the last few days I've been working on the project of getting OpenStreetMap onto Wikimedia as outlined here:
Woohoo!
Anyway, one thing standing between us and world domination is rendering those static maps, I'm going to implement this but first I'd like to get comments on *how* we'd like to do it, so I've written a plan for doing it:
http://www.mediawiki.org/wiki/Extension:SlippyMap/Static_map_generation
I added a couple quickie notes on the talk page...
Ah, thanks, to reply to this:
Static maps of arbitrary size presumably have much the same problem here as the map tile images. How is it handled there?"
The same way I want to do static map generation. Just put an arbitrary expiry time on it & serve it to the client.
It might be good to use a Google Maps Static-like model here; the MediaWiki end can just make itself an <img> and stick all the relevant info onto the URL, calling out to the static map server.
That's a very good idea, but I'd been assuming that I wouldn't be able to do that -- that each apache server was supposed to do things like EasyTimeline generation / image rescaling and likewise static map generation on its own & write it to the shared image NFS.
But just being able to call out to an existing server makes things a whole lot easier.
Then we can just run dedicated rendering machines with the apaches being dumb about all this crazy map stuff.
This also means that we can set up the Wikimedia Deutschland servers to do tile rendering and then easily test it on a big wiki (like dewiki) by just enabling the SlippyMap extension which won't do anything more fancy than point static/tile URLs to the external service.
So yay!
Either storing it on local disk there or simply leaving it to be cached by upper-level squids
Throwing it at the squids and making it their problem would be simpler for *me* at least.
Ævar Arnfjörð Bjarmason wrote:
On Wed, Jul 15, 2009 at 12:42 AM, Brion Vibberbrion@wikimedia.org wrote:
Static maps of arbitrary size presumably have much the same problem here as the map tile images. How is it handled there?"
The same way I want to do static map generation. Just put an arbitrary expiry time on it & serve it to the client.
Handy that!
It might be good to use a Google Maps Static-like model here; the MediaWiki end can just make itself an <img> and stick all the relevant info onto the URL, calling out to the static map server.
That's a very good idea, but I'd been assuming that I wouldn't be able to do that -- that each apache server was supposed to do things like EasyTimeline generation / image rescaling and likewise static map generation on its own & write it to the shared image NFS.
But just being able to call out to an existing server makes things a whole lot easier.
Yep... this way is usually easier to manage for us, and also helps to isolate services for performance and security reasons. If the map rendering goes crazy and uses ten bajillion gigs of RAM and disk space, we don't want that taking down the site. :) If it's isolated and nothing depends on synchronous result returns, then you just don't get new maps until it's fixed.
For things like the PDF rendering where we need to be able to return data through the main web servers, isolation is still better -- we just need a timeout on fetches to make sure we don't get a million reqs stuck if the service hangs.
This also means that we can set up the Wikimedia Deutschland servers to do tile rendering and then easily test it on a big wiki (like dewiki) by just enabling the SlippyMap extension which won't do anything more fancy than point static/tile URLs to the external service.
So yay!
:D
Either storing it on local disk there or simply leaving it to be cached by upper-level squids
Throwing it at the squids and making it their problem would be simpler for *me* at least.
Hehehe...
-- brion
I wrote:
For things like the PDF rendering where we need to be able to return data through the main web servers, isolation is still better -- we just need a timeout on fetches to make sure we don't get a million reqs stuck if the service hangs.
Funny story: the first prototype of our Lucene search server went live around Christmas 2004 and immediately hung the whole site thanks to an overloaded server and an incredibly long default timeout for HTTP fetches from PHP. :)
Ahh, those heady early cowboy days...
-- brion
wikitech-l@lists.wikimedia.org