Hi to all. We are currently using, for a project, your maps tiles (we like your color schema) As we would like to put them in our CDN to offload your servers, do you have a safe method to download pre-rendered map tiles or should I download the whole OSM data file and generate all tiles manually ?
If possible, we prefere to not generate all tiles manually, due to lack of resources on our side.
Any idea ? Thank you
Hello Gandalf,
I am not a member of the Wikimedia Maps team, but I assume that like many other organizations that render map tiles, they do not pre-render these tiles because that is a waste of disk space. Please refer to the following page for more information: https://wiki.openstreetmap.org/wiki/Tile_disk_usage
In particular, pre-rendering all tiles would use approximately 54 TB of storage and majority of them are never viewed by the end user. So most map tile servers generally just render tiles as needed and then cache them.
Hope this helps.
On Wed, Oct 3, 2018 at 3:34 AM Gandalf Corvotempesta < gandalf.corvotempesta@gmail.com> wrote:
Hi to all. We are currently using, for a project, your maps tiles (we like your color schema) As we would like to put them in our CDN to offload your servers, do you have a safe method to download pre-rendered map tiles or should I download the whole OSM data file and generate all tiles manually ?
If possible, we prefere to not generate all tiles manually, due to lack of resources on our side.
Any idea ? Thank you
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
Il giorno mer 3 ott 2018 alle ore 08:55 Eugene Alvin Villar seav80@gmail.com ha scritto:
I am not a member of the Wikimedia Maps team, but I assume that like many other organizations that render map tiles, they do not pre-render these tiles because that is a waste of disk space. Please refer to the following page for more information: https://wiki.openstreetmap.org/wiki/Tile_disk_usage
That's true.
In particular, pre-rendering all tiles would use approximately 54 TB of storage and majority of them are never viewed by the end user. So most map tile servers generally just render tiles as needed and then cache them.
Do you have any idea on how to put wikimedia tiles under a CDN like Cloudflare ? Current tiles are hosting at about 220ms from here. I've thought to regenerate them on the fly with a local tileserver but we have to import the whole europe/world map to Postgresql and this takes weeks and tons of ram that we don't have.
Hello !
I'm working on the maps infrastructure for Wikimedia Maps. As guessed by Eugene, we don't pregenerate tiles. You can find some info on our infrastructure on wikitech [1] (as always, documentation should be better than it is).
We generate vector tiles that we store in Cassandra, regenerating those vector tiles when we import changes from OSM. Raster tiles (.png) are generated on the fly from the vector tiles and cached by Varnish. That rasterization process is cheap, so some level of cache miss are not an issue. That being said, please don't try to download all our tiles by iterating over them!
We don't provide dumps of our vector tiles either, as this is not that helpful, since you would still have to build the rendering infrastructure. At that point, you're probably better off building the full stack.
Options that might work for you:
* Have a proxy on your side that would cache our tiles. Depending on your access patterns and the cache duration, this might be enough. * Build your own rendering stack, reusing our style.
Note that Paul Norman was working on a new style for WMF, which has a number of improvements about our current style. I recommend you have a look into it as well! It is available on his github account [2]. See his blog post about it [3].
Good luck!
Guillaume
[1] https://wikitech.wikimedia.org/wiki/Maps [2] https://github.com/pnorman/bolder [3] http://www.paulnorman.ca/blog/2018/08/more-work-on-bolder/
On Wed, Oct 3, 2018 at 12:01 PM Gandalf Corvotempesta gandalf.corvotempesta@gmail.com wrote:
Il giorno mer 3 ott 2018 alle ore 08:55 Eugene Alvin Villar seav80@gmail.com ha scritto:
I am not a member of the Wikimedia Maps team, but I assume that like many other organizations that render map tiles, they do not pre-render these tiles because that is a waste of disk space. Please refer to the following page for more information: https://wiki.openstreetmap.org/wiki/Tile_disk_usage
That's true.
In particular, pre-rendering all tiles would use approximately 54 TB of storage and majority of them are never viewed by the end user. So most map tile servers generally just render tiles as needed and then cache them.
Do you have any idea on how to put wikimedia tiles under a CDN like Cloudflare ? Current tiles are hosting at about 220ms from here. I've thought to regenerate them on the fly with a local tileserver but we have to import the whole europe/world map to Postgresql and this takes weeks and tons of ram that we don't have.
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
Il giorno mer 3 ott 2018 alle ore 23:19 Guillaume Lederrey glederrey@wikimedia.org ha scritto:
Hello !
I'm working on the maps infrastructure for Wikimedia Maps. As guessed by Eugene, we don't pregenerate tiles. You can find some info on our infrastructure on wikitech [1] (as always, documentation should be better than it is).
Hi. Sorry for the late reply and sorry to reach you in private, but I have some questions that I prefere to not make public.
We generate vector tiles that we store in Cassandra, regenerating those vector tiles when we import changes from OSM. Raster tiles (.png) are generated on the fly from the vector tiles and cached by Varnish. That rasterization process is cheap, so some level of cache miss are not an issue. That being said, please don't try to download all our tiles by iterating over them!
Ok.
We don't provide dumps of our vector tiles either, as this is not that helpful, since you would still have to build the rendering infrastructure. At that point, you're probably better off building the full stack.
What do you use to build the vector tiles ? Do you directly start for OSM dumps ? What we are looking for is something pre-generated so that we don't have to extract the whole OSM dump (we don't have enough computing power to parse that huge file). It would be ok, for us, to have an updated mbtiles that will be rendered and cached on-the-fly (we already have tried that with success, but we are unable to find an updated mbtiles dump and, as wrote above, we are unable to generate that dumps on own own)
Options that might work for you:
- Have a proxy on your side that would cache our tiles. Depending on
your access patterns and the cache duration, this might be enough.
We also tried that by using cloudflare (keep the following private):
$ curl -I https://thumbnails.take2me.it/w_256/h_256/url/https://maps.wikimedia.org/osm... HTTP/2 200 date: Sun, 11 Nov 2018 08:32:10 GMT content-type: image/jpeg content-length: 15128 set-cookie: __cfduid=d44f565b88a285b2334ea1699eddca8461541925130; expires=Mon, 11-Nov-19 08:32:10 GMT; path=/; domain=.take2me.it; HttpOnly cache-control: max-age=31536000, public expires: Mon, 11 Nov 2019 08:31:49 GMT last-modified: Sun, 11 Nov 2018 08:31:49 GMT cf-cache-status: HIT expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct" server: cloudflare cf-ray: 477f68222d554334-MXP
as you can see, it's a cache hit (1 year) from cloudflare, so that tile would be fetched from cloudflare and not from your server. Probably, by respecting your actual cache expire (1day) we can have an always updated map.
How much time do you need to update your maps with updated data from OSM ? 1 day ? 1 Month? Just to know...
By caching your tiles has some advantages (almost-zero infrastructure on our side) but also some disadvantage: we can't change the style. Any suggestion on how to generate raster tiles (are much faster to elaborate than vectors, on some smartphones) without having to generate the mbtiles from raw OSM data ? Are wrote above, we can't handle that huge file and the only MBTiles I have found is 1 year old, in example: https://openmaptiles.com/downloads/tileset/osm/europe/albania/?usage=educati...
So, to recap: we are able to cache your tiles on our side, but we would like to generate our own raster tiles. We are able to generate raster tiles on our own, starting from a pre-computed mbtiles, but that file is outdated. How can we generate (or download) an updated mbtiles without using the whole OSM dump?
Another thing to keep private: we are using your maps on our procution site. If I tell you the referral, could you please set that as whitelist (or at least, warn me before any suspensions ? We don't want to see our website down because you dropped our access to your maps)
Thank you in advance Alessandro.
Just as an fyi, you sent this to a public list, so anything you wanted to keep private is now fully public. It's also archived and the archive is public and probably already mirrored by other sites.
Στις Κυρ, 11 Νοε 2018, 10:45 π.μ. ο χρήστης Gandalf Corvotempesta < gandalf.corvotempesta@gmail.com> έγραψε:
Il giorno mer 3 ott 2018 alle ore 23:19 Guillaume Lederrey glederrey@wikimedia.org ha scritto:
Hello !
I'm working on the maps infrastructure for Wikimedia Maps. As guessed by Eugene, we don't pregenerate tiles. You can find some info on our infrastructure on wikitech [1] (as always, documentation should be better than it is).
Hi. Sorry for the late reply and sorry to reach you in private, but I have some questions that I prefere to not make public.
We generate vector tiles that we store in Cassandra, regenerating those vector tiles when we import changes from OSM. Raster tiles (.png) are generated on the fly from the vector tiles and cached by Varnish. That rasterization process is cheap, so some level of cache miss are not an issue. That being said, please don't try to download all our tiles by iterating over them!
Ok.
We don't provide dumps of our vector tiles either, as this is not that helpful, since you would still have to build the rendering infrastructure. At that point, you're probably better off building the full stack.
What do you use to build the vector tiles ? Do you directly start for OSM dumps ? What we are looking for is something pre-generated so that we don't have to extract the whole OSM dump (we don't have enough computing power to parse that huge file). It would be ok, for us, to have an updated mbtiles that will be rendered and cached on-the-fly (we already have tried that with success, but we are unable to find an updated mbtiles dump and, as wrote above, we are unable to generate that dumps on own own)
Options that might work for you:
- Have a proxy on your side that would cache our tiles. Depending on
your access patterns and the cache duration, this might be enough.
We also tried that by using cloudflare (keep the following private):
$ curl -I https://thumbnails.take2me.it/w_256/h_256/url/https://maps.wikimedia.org/osm... HTTP/2 200 date: Sun, 11 Nov 2018 08:32:10 GMT content-type: image/jpeg content-length: 15128 set-cookie: __cfduid=d44f565b88a285b2334ea1699eddca8461541925130; expires=Mon, 11-Nov-19 08:32:10 GMT; path=/; domain=.take2me.it; HttpOnly cache-control: max-age=31536000, public expires: Mon, 11 Nov 2019 08:31:49 GMT last-modified: Sun, 11 Nov 2018 08:31:49 GMT cf-cache-status: HIT expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct" server: cloudflare cf-ray: 477f68222d554334-MXP
as you can see, it's a cache hit (1 year) from cloudflare, so that tile would be fetched from cloudflare and not from your server. Probably, by respecting your actual cache expire (1day) we can have an always updated map.
How much time do you need to update your maps with updated data from OSM ? 1 day ? 1 Month? Just to know...
By caching your tiles has some advantages (almost-zero infrastructure on our side) but also some disadvantage: we can't change the style. Any suggestion on how to generate raster tiles (are much faster to elaborate than vectors, on some smartphones) without having to generate the mbtiles from raw OSM data ? Are wrote above, we can't handle that huge file and the only MBTiles I have found is 1 year old, in example: https://openmaptiles.com/downloads/tileset/osm/europe/albania/?usage=educati...
So, to recap: we are able to cache your tiles on our side, but we would like to generate our own raster tiles. We are able to generate raster tiles on our own, starting from a pre-computed mbtiles, but that file is outdated. How can we generate (or download) an updated mbtiles without using the whole OSM dump?
Another thing to keep private: we are using your maps on our procution site. If I tell you the referral, could you please set that as whitelist (or at least, warn me before any suspensions ? We don't want to see our website down because you dropped our access to your maps)
Thank you in advance Alessandro.
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
I know, i've removed what i Need to keep private before posting but i forgot to remove the original phrases
In the original email there were some ip adresses that must not be disclosed but i've changed the email a bit...
Il giorno dom 11 nov 2018, 12:38 Alexandros Kosiaris < akosiaris@wikimedia.org> ha scritto:
Just as an fyi, you sent this to a public list, so anything you wanted to keep private is now fully public. It's also archived and the archive is public and probably already mirrored by other sites.
Στις Κυρ, 11 Νοε 2018, 10:45 π.μ. ο χρήστης Gandalf Corvotempesta < gandalf.corvotempesta@gmail.com> έγραψε:
Il giorno mer 3 ott 2018 alle ore 23:19 Guillaume Lederrey glederrey@wikimedia.org ha scritto:
Hello !
I'm working on the maps infrastructure for Wikimedia Maps. As guessed by Eugene, we don't pregenerate tiles. You can find some info on our infrastructure on wikitech [1] (as always, documentation should be better than it is).
Hi. Sorry for the late reply and sorry to reach you in private, but I have some questions that I prefere to not make public.
We generate vector tiles that we store in Cassandra, regenerating those vector tiles when we import changes from OSM. Raster tiles (.png) are generated on the fly from the vector tiles and cached by Varnish. That rasterization process is cheap, so some level of cache miss are not an issue. That being said, please don't try to download all our tiles by iterating over them!
Ok.
We don't provide dumps of our vector tiles either, as this is not that helpful, since you would still have to build the rendering infrastructure. At that point, you're probably better off building the full stack.
What do you use to build the vector tiles ? Do you directly start for OSM dumps ? What we are looking for is something pre-generated so that we don't have to extract the whole OSM dump (we don't have enough computing power to parse that huge file). It would be ok, for us, to have an updated mbtiles that will be rendered and cached on-the-fly (we already have tried that with success, but we are unable to find an updated mbtiles dump and, as wrote above, we are unable to generate that dumps on own own)
Options that might work for you:
- Have a proxy on your side that would cache our tiles. Depending on
your access patterns and the cache duration, this might be enough.
We also tried that by using cloudflare (keep the following private):
$ curl -I https://thumbnails.take2me.it/w_256/h_256/url/https://maps.wikimedia.org/osm... HTTP/2 200 date: Sun, 11 Nov 2018 08:32:10 GMT content-type: image/jpeg content-length: 15128 set-cookie: __cfduid=d44f565b88a285b2334ea1699eddca8461541925130; expires=Mon, 11-Nov-19 08:32:10 GMT; path=/; domain=.take2me.it; HttpOnly cache-control: max-age=31536000, public expires: Mon, 11 Nov 2019 08:31:49 GMT last-modified: Sun, 11 Nov 2018 08:31:49 GMT cf-cache-status: HIT expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct" server: cloudflare cf-ray: 477f68222d554334-MXP
as you can see, it's a cache hit (1 year) from cloudflare, so that tile would be fetched from cloudflare and not from your server. Probably, by respecting your actual cache expire (1day) we can have an always updated map.
How much time do you need to update your maps with updated data from OSM ? 1 day ? 1 Month? Just to know...
By caching your tiles has some advantages (almost-zero infrastructure on our side) but also some disadvantage: we can't change the style. Any suggestion on how to generate raster tiles (are much faster to elaborate than vectors, on some smartphones) without having to generate the mbtiles from raw OSM data ? Are wrote above, we can't handle that huge file and the only MBTiles I have found is 1 year old, in example: https://openmaptiles.com/downloads/tileset/osm/europe/albania/?usage=educati...
So, to recap: we are able to cache your tiles on our side, but we would like to generate our own raster tiles. We are able to generate raster tiles on our own, starting from a pre-computed mbtiles, but that file is outdated. How can we generate (or download) an updated mbtiles without using the whole OSM dump?
Another thing to keep private: we are using your maps on our procution site. If I tell you the referral, could you please set that as whitelist (or at least, warn me before any suspensions ? We don't want to see our website down because you dropped our access to your maps)
Thank you in advance Alessandro.
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
On Sun, Nov 11, 2018 at 9:45 AM Gandalf Corvotempesta gandalf.corvotempesta@gmail.com wrote:
Il giorno mer 3 ott 2018 alle ore 23:19 Guillaume Lederrey glederrey@wikimedia.org ha scritto:
Hello !
I'm working on the maps infrastructure for Wikimedia Maps. As guessed by Eugene, we don't pregenerate tiles. You can find some info on our infrastructure on wikitech [1] (as always, documentation should be better than it is).
Hi. Sorry for the late reply and sorry to reach you in private, but I have some questions that I prefere to not make public.
It looks like this was a fail :) But I think you removed private infos. Please double check.
We generate vector tiles that we store in Cassandra, regenerating those vector tiles when we import changes from OSM. Raster tiles (.png) are generated on the fly from the vector tiles and cached by Varnish. That rasterization process is cheap, so some level of cache miss are not an issue. That being said, please don't try to download all our tiles by iterating over them!
Ok.
We don't provide dumps of our vector tiles either, as this is not that helpful, since you would still have to build the rendering infrastructure. At that point, you're probably better off building the full stack.
What do you use to build the vector tiles ?
We are using our own stack (Kartotherian [1]), which is based on mapnik [2]. There are a number of other alternatives possible.
Do you directly start for OSM dumps ?
Yes, we start from OSM dumps, that we load in a postgres database. We update that dump daily.
What we are looking for is something pre-generated so that we don't have to extract the whole OSM dump (we don't have enough computing power to parse that huge file). It would be ok, for us, to have an updated mbtiles that will be rendered and cached on-the-fly (we already have tried that with success, but we are unable to find an updated mbtiles dump and, as wrote above, we are unable to generate that dumps on own own)
Options that might work for you:
- Have a proxy on your side that would cache our tiles. Depending on
your access patterns and the cache duration, this might be enough.
We also tried that by using cloudflare (keep the following private):
$ curl -I https://thumbnails.take2me.it/w_256/h_256/url/https://maps.wikimedia.org/osm... HTTP/2 200 date: Sun, 11 Nov 2018 08:32:10 GMT content-type: image/jpeg content-length: 15128 set-cookie: __cfduid=d44f565b88a285b2334ea1699eddca8461541925130; expires=Mon, 11-Nov-19 08:32:10 GMT; path=/; domain=.take2me.it; HttpOnly cache-control: max-age=31536000, public expires: Mon, 11 Nov 2019 08:31:49 GMT last-modified: Sun, 11 Nov 2018 08:31:49 GMT cf-cache-status: HIT expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct" server: cloudflare cf-ray: 477f68222d554334-MXP
as you can see, it's a cache hit (1 year) from cloudflare, so that tile would be fetched from cloudflare and not from your server. Probably, by respecting your actual cache expire (1day) we can have an always updated map.
How much time do you need to update your maps with updated data from OSM ? 1 day ? 1 Month? Just to know...
At the moment, we are updating every 2 days. We'd like to increase this frequency, but we need some work on our side to make that happen.
By caching your tiles has some advantages (almost-zero infrastructure on our side) but also some disadvantage: we can't change the style. Any suggestion on how to generate raster tiles (are much faster to elaborate than vectors, on some smartphones) without having to generate the mbtiles from raw OSM data ? Are wrote above, we can't handle that huge file and the only MBTiles I have found is 1 year old, in example: https://openmaptiles.com/downloads/tileset/osm/europe/albania/?usage=educati...
So, to recap: we are able to cache your tiles on our side, but we would like to generate our own raster tiles. We are able to generate raster tiles on our own, starting from a pre-computed mbtiles, but that file is outdated. How can we generate (or download) an updated mbtiles without using the whole OSM dump?
One of the problem is that we already apply some style based decisions when generating our vector tiles. For example, we already filter out features that will not be rendered at some zoom levels. So basing a style on our vector tiles will limit you in what you can render. Overall, those vector tiles are an implementation detail, coupled to our style and not meant to be reused. The format is likely to change without warning, which would break whatever integration you might have.
If you need your own style, are limited on resources and don't want to rely on our raster tiles, I'm not sure what the solution could be. There are a few maps provider that will take care of the infrastructure for you and let you provide your own style. But that's not really our mission. We provide maps with the goal of having them integrate into the various Wikipedia projects. You are very welcomed to reuse those maps if they work for you. But we're not a general purpose maps provider.
Another thing to keep private: we are using your maps on our procution site. If I tell you the referral, could you please set that as whitelist (or at least, warn me before any suspensions ? We don't want to see our website down because you dropped our access to your maps)
As long as you follow the Maps Terms of Use [3], we should not have any reason to block you. We always reserve the right to block any traffic that we see as abusive or that might be a risk to the stability of our infrastructure. But we will make reasonable efforts to contact you in case of trouble.
Thank you in advance Alessandro.
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
[1] https://github.com/kartotherian/kartotherian [2] https://mapnik.org/ [3] https://foundation.wikimedia.org/wiki/Maps_Terms_of_Use
Il giorno lun 12 nov 2018 alle ore 11:28 Guillaume Lederrey glederrey@wikimedia.org ha scritto:
It looks like this was a fail :) But I think you removed private infos. Please double check.
Yes, i've removed private infos before sending. :)
One of the problem is that we already apply some style based decisions when generating our vector tiles. For example, we already filter out features that will not be rendered at some zoom levels. So basing a style on our vector tiles will limit you in what you can render. Overall, those vector tiles are an implementation detail, coupled to our style and not meant to be reused. The format is likely to change without warning, which would break whatever integration you might have.
If you need your own style, are limited on resources and don't want to rely on our raster tiles, I'm not sure what the solution could be. There are a few maps provider that will take care of the infrastructure for you and let you provide your own style. But that's not really our mission. We provide maps with the goal of having them integrate into the various Wikipedia projects. You are very welcomed to reuse those maps if they work for you. But we're not a general purpose maps provider.
Currently, I'm able to put the whole osm dump to a postgis database, by using a desktop computer (we don't have that computing power on our cloud).
But, as we don't have the ability to use the postgis database on our frontend server (they are cloud located and tons of GB ram needed for the import are too expensive), we have to convert from postgis to mbtiles. We don't know how to do that. mbtiles are usefull, it's a single small-size file comparted to tons of pre computed png tiles.
What I would like to do is to "dump" the whole pgsql database to an mbtiles file, from a command line (this has to be scripted and automated) so that our tile server will fetch and serve tiles directly from that.
Any hint or suggestions ?
On Mon, Nov 12, 2018 at 4:48 PM Gandalf Corvotempesta gandalf.corvotempesta@gmail.com wrote:
Il giorno lun 12 nov 2018 alle ore 11:28 Guillaume Lederrey glederrey@wikimedia.org ha scritto:
It looks like this was a fail :) But I think you removed private infos. Please double check.
Yes, i've removed private infos before sending. :)
One of the problem is that we already apply some style based decisions when generating our vector tiles. For example, we already filter out features that will not be rendered at some zoom levels. So basing a style on our vector tiles will limit you in what you can render. Overall, those vector tiles are an implementation detail, coupled to our style and not meant to be reused. The format is likely to change without warning, which would break whatever integration you might have.
If you need your own style, are limited on resources and don't want to rely on our raster tiles, I'm not sure what the solution could be. There are a few maps provider that will take care of the infrastructure for you and let you provide your own style. But that's not really our mission. We provide maps with the goal of having them integrate into the various Wikipedia projects. You are very welcomed to reuse those maps if they work for you. But we're not a general purpose maps provider.
Currently, I'm able to put the whole osm dump to a postgis database, by using a desktop computer (we don't have that computing power on our cloud).
But, as we don't have the ability to use the postgis database on our frontend server (they are cloud located and tons of GB ram needed for the import are too expensive), we have to convert from postgis to mbtiles. We don't know how to do that. mbtiles are usefull, it's a single small-size file comparted to tons of pre computed png tiles.
What I would like to do is to "dump" the whole pgsql database to an mbtiles file, from a command line (this has to be scripted and automated) so that our tile server will fetch and serve tiles directly from that.
Any hint or suggestions ?
That sounds like a question that should be asked on one of the OSM mailing lists. They have much more expertise in this area! (long way to say: "I don't know").
Good luck!
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l