Hi folks! I am the Engineering Community Manager at the Wikimedia Foundation, which means I'm a liaison for people who want to improve the technical capabilities of our projects. I spoke with a few of you offlist (including at Wikimania) about maps and Wikimedia. While I'm still getting up to speed on the proposals and tools people have made, I wanted to give you an update regarding what's happening with Wikimedia's planned OpenStreetMap tileserver.
We know that we will set up a production tileserver. That is, Wikimedia plans to have an OpenStreetMap tileserver, in production (that is, fully supported by our Operations department and with uptime expectations similar to those for our main sites). We need this not only for the tools community but also for use in our official mobile apps. Brandon Black is the contact for this. He is a member of the Operations team at WMF. In order to set up a production-level tileserver, we will also need a production OSM database, so he is working on figuring out how best to grab that data from the official OSM database regularly (I am oversimplifying here).
Brandon has already started on the project of setting up the production tileserver. I know that he will be posting regular updates at https://wikitech.wikimedia.org/wiki/OSM_Tileserver but he is still in the investigation and architecture phase so he does not have a timeline yet. I'm sorry, I know that is frustrating.
Very soon after we have the production tileserver running, we plan to replicate that OSM database to Wikimedia Labs for use on Tool Labs, or do similar SQL import magic. People using Labs will be able to draw data directly from that replica, and will be able to grab tiles from the production tileserver. I'm the contact on this but since Coren (Marc-Andre Pelletier) is the domain expert on this particular bit of database replication, please go ahead and include both of us on any questions.
Thanks, all.
Hello Sumana (and Brandon and Marc-André),
There's been a mention of getting 2 copies of the "OSM database", one for the use of the production tileserver and the other for the use of Wikimedia Labs. Is this correct?
What I'd like to know is if this is the raw OSM database, which contains the raw OSM data that is suitable for editing, or a rendering OSM database suitable for rendering map tiles. The rendering database is usually loaded with a program like osm2pgsql[1] from raw OSM data that is either in XML or PBF file formats.
The problem with a rendering OSM database is that this is optimized for rendering map tiles, which is excellent for the production tileserver. However, this database is (mostly) unsuitable for other purposes such as routing and (probably) spatial analysis. This may hamper possible use cases of developers on Wikimedia Labs wanting to explore using OSM data for various purposes.
[1] http://wiki.openstreetmap.org/wiki/Osm2pgsql
Thanks, Eugene
On Fri, Aug 23, 2013 at 3:47 AM, Sumana Harihareswara <sumanah@wikimedia.org
wrote:
Hi folks! I am the Engineering Community Manager at the Wikimedia Foundation, which means I'm a liaison for people who want to improve the technical capabilities of our projects. I spoke with a few of you offlist (including at Wikimania) about maps and Wikimedia. While I'm still getting up to speed on the proposals and tools people have made, I wanted to give you an update regarding what's happening with Wikimedia's planned OpenStreetMap tileserver.
We know that we will set up a production tileserver. That is, Wikimedia plans to have an OpenStreetMap tileserver, in production (that is, fully supported by our Operations department and with uptime expectations similar to those for our main sites). We need this not only for the tools community but also for use in our official mobile apps. Brandon Black is the contact for this. He is a member of the Operations team at WMF. In order to set up a production-level tileserver, we will also need a production OSM database, so he is working on figuring out how best to grab that data from the official OSM database regularly (I am oversimplifying here).
Brandon has already started on the project of setting up the production tileserver. I know that he will be posting regular updates at https://wikitech.wikimedia.org/wiki/OSM_Tileserver but he is still in the investigation and architecture phase so he does not have a timeline yet. I'm sorry, I know that is frustrating.
Very soon after we have the production tileserver running, we plan to replicate that OSM database to Wikimedia Labs for use on Tool Labs, or do similar SQL import magic. People using Labs will be able to draw data directly from that replica, and will be able to grab tiles from the production tileserver. I'm the contact on this but since Coren (Marc-Andre Pelletier) is the domain expert on this particular bit of database replication, please go ahead and include both of us on any questions.
Thanks, all.
-- Sumana Harihareswara Engineering Community Manager Wikimedia Foundation
Hello, I hope I can answer some parts of your question. Yes, there will be 2 copies of the "OSM database".
The database for rendering contains also tables with some "raw data" like nodes, ways and relations, because we need them for updating. But there will be not all indexes on this raw data so that you can't use it for everything.
I believe the project is ambitious enough also if we only concentrate us on rendering. I mean it's in the moment hard to predict how many requests we will get per second.
The ambitious whishlist is there: https://wikitech.wikimedia.org/wiki/OSM_Tileserver#Future_Work All things in this list are things with direct profit for Wikipedia. This profit I can't see if we would have databases for routing, OverpassAPI, user-statistics or Nominatim. If somebody can explain the reasons, we should be open for it, otherwise I think it's better to concentrate on a flexible(hstore) and high-performance rendering database. If our database on Toolserver would be faster, I know that we could perform with this database-layout a lot of more geospatial analysis, because we can use all features of PostGIS.
Greetings Tim alias Kolossos
Am 22.08.2013 23:03, schrieb Eugene Alvin Villar:
Hello Sumana (and Brandon and Marc-André),
There's been a mention of getting 2 copies of the "OSM database", one for the use of the production tileserver and the other for the use of Wikimedia Labs. Is this correct?
What I'd like to know is if this is the raw OSM database, which contains the raw OSM data that is suitable for editing, or a rendering OSM database suitable for rendering map tiles. The rendering database is usually loaded with a program like osm2pgsql[1] from raw OSM data that is either in XML or PBF file formats.
The problem with a rendering OSM database is that this is optimized for rendering map tiles, which is excellent for the production tileserver. However, this database is (mostly) unsuitable for other purposes such as routing and (probably) spatial analysis. This may hamper possible use cases of developers on Wikimedia Labs wanting to explore using OSM data for various purposes.
[1] http://wiki.openstreetmap.org/wiki/Osm2pgsql
Thanks, Eugene
On Fri, Aug 23, 2013 at 8:11 PM, Tim Alder tim.alder@s2002.tu-chemnitz.dewrote:
The ambitious whishlist is there: https://wikitech.wikimedia.org/wiki/OSM_Tileserver#Future_Work
Hi, Just wondering whether there has been much discussion of using a different style. The "future work" section assumes we'd use the default OSM style. I've been playing around a lot with TileMill over the last few months and IMHO:
- it's really not hard to make an alternative style - the default OSM style sort of makes sense as a general purpose navigational map (show every street name, and pretty much anything with a name=*), but doesn't make much sense for Wikipedia (which would mostly be using maps to show a single item in context, rather than navigation) - there's probably not much benefit in replicating the default OSM style (ie, there's no efficiency gained) - the default OSM style is ugly. :)
Why not a distinctive Wikipedia style, or 2, or 5?
Steve
On 08/26/2013 12:13 AM, Steve Bennett wrote:
On Fri, Aug 23, 2013 at 8:11 PM, Tim Alder <tim.alder@s2002.tu-chemnitz.de mailto:tim.alder@s2002.tu-chemnitz.de> wrote:
The ambitious whishlist is there: https://wikitech.wikimedia.org/wiki/OSM_Tileserver#Future_WorkHi, Just wondering whether there has been much discussion of using a different style. The "future work" section assumes we'd use the default OSM style. I've been playing around a lot with TileMill over the last few months and IMHO:
I think the intention has always been to have the option to use different styles that are more appropriate for Wikipedia than the "default OSM" style. However, I think it is important to have a solid technical infrastructure first to base any such options off of. Although on the old toolserver tileserver setup, there have been a number of styles, we never managed to get the performance of that server high enough to make those styles reliably available. Indeed, if I am not mistaken, the OSM gadget has now moved back to using the osm.org tile infrastructure instead of the one on toolserver, as it wasn't reliable enough for production. Without the use of the same style as on osm.org that wouldn't have been an option.
The production ready infrastructure to be run by WMF is also turning out to be a longer process than originally expected and so imho the focus should be to get something reliably up and running first. The less unknown parts, the quicker this is likely to happen. So concentrating on replicating the standard tile server infrastructure seems preferable.
Once the reliable infrastructure is in place, the tools to embed this in Wikipedia and Wikipedia Mobile are solid and it is much better understood what kind of load Wikipedia generates, I am sure the question of different styles will come up once again. As you say, there are likely better styles for Wikipedia's purpose than the default OSM style. This is also part of the reason why the plans are to have both a production infrastructure and an infrastructure in labs. The latter being a place to experiment with things like new styles or new techniques to present map data (like e.g. vector tiles) that can eventually be moved into the production system.
Kai
- it's really not hard to make an alternative style
- the default OSM style sort of makes sense as a general purpose
navigational map (show every street name, and pretty much anything with a name=*), but doesn't make much sense for Wikipedia (which would mostly be using maps to show a single item in context, rather than navigation)
- there's probably not much benefit in replicating the default OSM
style (ie, there's no efficiency gained)
- the default OSM style is ugly. :)
Why not a distinctive Wikipedia style, or 2, or 5?
Steve
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
On 08/22/2013 01:47 PM, Sumana Harihareswara wrote:
We know that we will set up a production tileserver. That is, Wikimedia plans to have an OpenStreetMap tileserver, in production (that is, fully supported by our Operations department and with uptime expectations similar to those for our main sites). We need this not only for the tools community but also for use in our official mobile apps. Brandon Black is the contact for this. He is a member of the Operations team at WMF. In order to set up a production-level tileserver, we will also need a production OSM database, so he is working on figuring out how best to grab that data from the official OSM database regularly (I am oversimplifying here).
With "grab that data from the official OSM database", do you mean the question of how to scale across multiple servers and whether to couple those servers or operate them entirely independently and do the load balancing on the front end? Or are there other issues still open? If there are problems or missing features with osm2pgsql and mod_tile / renderd there is a chance I would be able to fix and incorporate them.
Kai
On 08/22/2013 03:47 PM, Sumana Harihareswara wrote:
Hi folks! I am the Engineering Community Manager at the Wikimedia Foundation, which means I'm a liaison for people who want to improve the technical capabilities of our projects. I spoke with a few of you offlist (including at Wikimania) about maps and Wikimedia. While I'm still getting up to speed on the proposals and tools people have made, I wanted to give you an update regarding what's happening with Wikimedia's planned OpenStreetMap tileserver.
We know that we will set up a production tileserver. That is, Wikimedia plans to have an OpenStreetMap tileserver, in production (that is, fully supported by our Operations department and with uptime expectations similar to those for our main sites). We need this not only for the tools community but also for use in our official mobile apps. Brandon Black is the contact for this. He is a member of the Operations team at WMF. In order to set up a production-level tileserver, we will also need a production OSM database, so he is working on figuring out how best to grab that data from the official OSM database regularly (I am oversimplifying here).
Brandon has already started on the project of setting up the production tileserver. I know that he will be posting regular updates at https://wikitech.wikimedia.org/wiki/OSM_Tileserver but he is still in the investigation and architecture phase so he does not have a timeline yet. I'm sorry, I know that is frustrating.
Very soon after we have the production tileserver running, we plan to replicate that OSM database to Wikimedia Labs for use on Tool Labs, or do similar SQL import magic. People using Labs will be able to draw data directly from that replica, and will be able to grab tiles from the production tileserver. I'm the contact on this but since Coren (Marc-Andre Pelletier) is the domain expert on this particular bit of database replication, please go ahead and include both of us on any questions.
Thanks, all.
Maps-l folks: just a quick update saying that Brandon is still working on setting up the production tileserver, has been posting updates at https://wikitech.wikimedia.org/wiki/OSM_Tileserver , and sorry, it isn't up yet.
I am about to start a sabbatical and will not be onlist; please feel free to ask Quim Gil qgil@wikimedia.org for any help or about any concerns you have regarding Wikimedia-related tech!
Thanks; looking forward to coming back in January.
Hi Sumana!
I'm the guy working at opengeoserver.org mentioned in [0]. The funding from Wikimedia Germany for the current hardware will run out at the end of the year. So I would like to say hi and ask about the current WMF OSM Tileserver status?
I wouldn't consider opengeoserver to be production ready an a Wikipedia kind of scale, but I would love to keep the server alive within e.g. the WikiLabs infrastructue.
cu andreas
[0] https://wikitech.wikimedia.org/wiki/OSM_Tileserver#Future_Work
2013/8/22 Sumana Harihareswara sumanah@wikimedia.org:
Hi folks! I am the Engineering Community Manager at the Wikimedia Foundation, which means I'm a liaison for people who want to improve the technical capabilities of our projects. I spoke with a few of you offlist (including at Wikimania) about maps and Wikimedia. While I'm still getting up to speed on the proposals and tools people have made, I wanted to give you an update regarding what's happening with Wikimedia's planned OpenStreetMap tileserver.
We know that we will set up a production tileserver. That is, Wikimedia plans to have an OpenStreetMap tileserver, in production (that is, fully supported by our Operations department and with uptime expectations similar to those for our main sites). We need this not only for the tools community but also for use in our official mobile apps. Brandon Black is the contact for this. He is a member of the Operations team at WMF. In order to set up a production-level tileserver, we will also need a production OSM database, so he is working on figuring out how best to grab that data from the official OSM database regularly (I am oversimplifying here).
Brandon has already started on the project of setting up the production tileserver. I know that he will be posting regular updates at https://wikitech.wikimedia.org/wiki/OSM_Tileserver but he is still in the investigation and architecture phase so he does not have a timeline yet. I'm sorry, I know that is frustrating.
Very soon after we have the production tileserver running, we plan to replicate that OSM database to Wikimedia Labs for use on Tool Labs, or do similar SQL import magic. People using Labs will be able to draw data directly from that replica, and will be able to grab tiles from the production tileserver. I'm the contact on this but since Coren (Marc-Andre Pelletier) is the domain expert on this particular bit of database replication, please go ahead and include both of us on any questions.
Thanks, all.
-- Sumana Harihareswara Engineering Community Manager Wikimedia Foundation
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
On Wed, Nov 6, 2013 at 3:03 AM, Andreas Trawoeger atrawog@opengeoserver.org wrote:
So I would like to say hi and ask about the current WMF OSM Tileserver status?
CC'ing my colleague Ken Snider who's currently supporting this project to answer this.
--tomasz
Hello, as info for other people we use opengeoserver-tiles already in Wikipedia's OSM-Gadget as optional Satellite-layer.
I see opengeoserver-tiles of aerial images in the same category like our hillshading tiles from Toolserver, which are also looking for a new home. Both kinds of tiles need only static storage because they are not updated in the way of OSM-tiles. A grouping like metatiles could be useful.
About what a volume of images we are talking in the moment, how much can it be in the future? (I know we could get some terabytes of Landsat from NASA.)
As described above I see this tiles in an other category than OSM-tiles, but the status of new OSM-Servers can be found at: https://bugzilla.wikimedia.org/show_bug.cgi?id=33980
Greetings Tim alias Kolossos
Am 06.11.2013 12:03, schrieb Andreas Trawoeger:
Hi Sumana!
I'm the guy working at opengeoserver.org mentioned in [0]. The funding from Wikimedia Germany for the current hardware will run out at the end of the year. So I would like to say hi and ask about the current WMF OSM Tileserver status?
I wouldn't consider opengeoserver to be production ready an a Wikipedia kind of scale, but I would love to keep the server alive within e.g. the WikiLabs infrastructue.
cu andreas
[0] https://wikitech.wikimedia.org/wiki/OSM_Tileserver#Future_Work
Hi Tim!
OpenGeoServer.org currently stores 523 GB raw spatial data plus 166 GB cached data from other spatial data sources and I'm currently looking into how to best process Landsat data.
Landsat data processing needs a completely different technical infrastructure than OpenGeoServer.org which focus on processing static GeoTiff + data caching from other spatial web services so I would like to resolve the future of OpenGeoServer before I start doing the next spatial web-service.
cu andreas
2013/11/6 Tim Alder tim@alder-digital.de:
Hello, as info for other people we use opengeoserver-tiles already in Wikipedia's OSM-Gadget as optional Satellite-layer.
I see opengeoserver-tiles of aerial images in the same category like our hillshading tiles from Toolserver, which are also looking for a new home. Both kinds of tiles need only static storage because they are not updated in the way of OSM-tiles. A grouping like metatiles could be useful.
About what a volume of images we are talking in the moment, how much can it be in the future? (I know we could get some terabytes of Landsat from NASA.)
As described above I see this tiles in an other category than OSM-tiles, but the status of new OSM-Servers can be found at: https://bugzilla.wikimedia.org/show_bug.cgi?id=33980
Greetings Tim alias Kolossos
Am 06.11.2013 12:03, schrieb Andreas Trawoeger:
Hi Sumana!
I'm the guy working at opengeoserver.org mentioned in [0]. The funding from Wikimedia Germany for the current hardware will run out at the end of the year. So I would like to say hi and ask about the current WMF OSM Tileserver status?
I wouldn't consider opengeoserver to be production ready an a Wikipedia kind of scale, but I would love to keep the server alive within e.g. the WikiLabs infrastructue.
cu andreas
[0] https://wikitech.wikimedia.org/wiki/OSM_Tileserver#Future_Work
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
On 11/06/2013 03:03 AM, Andreas Trawoeger wrote:
Hi Sumana!
I'm the guy working at opengeoserver.org mentioned in [0].
Hi Andreas, I'm the guy tryng to cover Sumana's responsibilities while she is on a Sabbatical.
I don't have anything to add to the comments made here and at the related bug report, but I didn't want to leave the impression that nobody in Sumana's end was listening. :)
Like others have said, I believe the best approach now (and always) is regular and transparent communication and documentation of ongoing / pending tasks. This way contributors like you know what is going on and perhaps are able to help.
Brandon Black, cc'd, is working on OSM related things from WMF Operations' side of things.
Also see: https://bugzilla.wikimedia.org/show_bug.cgi?id=33980
Greg
<quote name="Andreas Trawoeger" date="2013-11-06" time="12:03:35 +0100">
Hi Sumana!
I'm the guy working at opengeoserver.org mentioned in [0]. The funding from Wikimedia Germany for the current hardware will run out at the end of the year. So I would like to say hi and ask about the current WMF OSM Tileserver status?
I wouldn't consider opengeoserver to be production ready an a Wikipedia kind of scale, but I would love to keep the server alive within e.g. the WikiLabs infrastructue.
cu andreas
[0] https://wikitech.wikimedia.org/wiki/OSM_Tileserver#Future_Work
2013/8/22 Sumana Harihareswara sumanah@wikimedia.org:
Hi folks! I am the Engineering Community Manager at the Wikimedia Foundation, which means I'm a liaison for people who want to improve the technical capabilities of our projects. I spoke with a few of you offlist (including at Wikimania) about maps and Wikimedia. While I'm still getting up to speed on the proposals and tools people have made, I wanted to give you an update regarding what's happening with Wikimedia's planned OpenStreetMap tileserver.
We know that we will set up a production tileserver. That is, Wikimedia plans to have an OpenStreetMap tileserver, in production (that is, fully supported by our Operations department and with uptime expectations similar to those for our main sites). We need this not only for the tools community but also for use in our official mobile apps. Brandon Black is the contact for this. He is a member of the Operations team at WMF. In order to set up a production-level tileserver, we will also need a production OSM database, so he is working on figuring out how best to grab that data from the official OSM database regularly (I am oversimplifying here).
Brandon has already started on the project of setting up the production tileserver. I know that he will be posting regular updates at https://wikitech.wikimedia.org/wiki/OSM_Tileserver but he is still in the investigation and architecture phase so he does not have a timeline yet. I'm sorry, I know that is frustrating.
Very soon after we have the production tileserver running, we plan to replicate that OSM database to Wikimedia Labs for use on Tool Labs, or do similar SQL import magic. People using Labs will be able to draw data directly from that replica, and will be able to grab tiles from the production tileserver. I'm the contact on this but since Coren (Marc-Andre Pelletier) is the domain expert on this particular bit of database replication, please go ahead and include both of us on any questions.
Thanks, all.
-- Sumana Harihareswara Engineering Community Manager Wikimedia Foundation
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l