On Sun, Nov 11, 2018 at 9:45 AM Gandalf Corvotempesta
<gandalf.corvotempesta(a)gmail.com> wrote:
Il giorno mer 3 ott 2018 alle ore 23:19 Guillaume Lederrey
<glederrey(a)wikimedia.org> ha scritto:
Hello !
I'm working on the maps infrastructure for Wikimedia Maps. As guessed
by Eugene, we don't pregenerate tiles. You can find some info on our
infrastructure on wikitech [1] (as always, documentation should be
better than it is).
Hi.
Sorry for the late reply and sorry to reach you in private, but I have some
questions that I prefere to not make public.
It looks like this was a fail :) But I think you removed private
infos. Please double check.
We generate
vector tiles that we store in Cassandra, regenerating
those vector tiles when we import changes from OSM. Raster tiles
(.png) are generated on the fly from the vector tiles and cached by
Varnish. That rasterization process is cheap, so some level of cache
miss are not an issue. That being said, please don't try to download
all our tiles by iterating over them!
Ok.
We don't provide dumps of our vector tiles
either, as this is not that
helpful, since you would still have to build the rendering
infrastructure. At that point, you're probably better off building the
full stack.
What do you use to build the vector tiles ?
We are using our own stack (Kartotherian [1]), which is based on
mapnik [2]. There are a number of other alternatives possible.
Do you directly start for OSM dumps ?
Yes, we start from OSM dumps, that we load in a postgres database. We
update that dump daily.
What we are looking for is something pre-generated so
that we don't
have to extract the whole OSM dump (we don't have enough computing
power to parse that huge file). It would be ok, for us, to have an updated
mbtiles that will be rendered and cached on-the-fly (we already have tried
that with success, but we are unable to find an updated mbtiles dump
and, as wrote above, we are unable to generate that dumps on own own)
Options that
might work for you:
* Have a proxy on your side that would cache our tiles. Depending on
your access patterns and the cache duration, this might be enough.
We also tried that by using cloudflare (keep the following private):
$ curl -I
https://thumbnails.take2me.it/w_256/h_256/url/https://maps.wikimedia.org/os…
HTTP/2 200
date: Sun, 11 Nov 2018 08:32:10 GMT
content-type: image/jpeg
content-length: 15128
set-cookie: __cfduid=d44f565b88a285b2334ea1699eddca8461541925130;
expires=Mon, 11-Nov-19 08:32:10 GMT; path=/; domain=.take2me.it;
HttpOnly
cache-control: max-age=31536000, public
expires: Mon, 11 Nov 2019 08:31:49 GMT
last-modified: Sun, 11 Nov 2018 08:31:49 GMT
cf-cache-status: HIT
expect-ct: max-age=604800,
report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct…
server: cloudflare
cf-ray: 477f68222d554334-MXP
as you can see, it's a cache hit (1 year) from cloudflare, so that
tile would be fetched from cloudflare and not from your server.
Probably, by respecting your actual cache expire (1day) we can have an
always updated map.
How much time do you need to update your maps with updated data from
OSM ? 1 day ? 1 Month? Just to know...
At the moment, we are updating every 2 days. We'd like to increase
this frequency, but we need some work on our side to make that happen.
By caching your tiles has some advantages (almost-zero
infrastructure
on our side) but also some disadvantage: we can't change the style.
Any suggestion on how to generate raster tiles (are much faster to
elaborate than vectors, on some smartphones) without having to
generate the mbtiles from raw OSM data ? Are wrote above, we can't
handle that huge file and the only MBTiles I have found is 1 year old,
in example:
https://openmaptiles.com/downloads/tileset/osm/europe/albania/?usage=educat…
So, to recap: we are able to cache your tiles on our side, but we
would like to generate our own raster tiles. We are able to generate
raster tiles on our own, starting from a pre-computed mbtiles, but
that file is outdated. How can we generate (or download) an updated
mbtiles
without using the whole OSM dump?
One of the problem is that we already apply some style based decisions
when generating our vector tiles. For example, we already filter out
features that will not be rendered at some zoom levels. So basing a
style on our vector tiles will limit you in what you can render.
Overall, those vector tiles are an implementation detail, coupled to
our style and not meant to be reused. The format is likely to change
without warning, which would break whatever integration you might
have.
If you need your own style, are limited on resources and don't want to
rely on our raster tiles, I'm not sure what the solution could be.
There are a few maps provider that will take care of the
infrastructure for you and let you provide your own style. But that's
not really our mission. We provide maps with the goal of having them
integrate into the various Wikipedia projects. You are very welcomed
to reuse those maps if they work for you. But we're not a general
purpose maps provider.
Another thing to keep private:
we are using your maps on our procution site. If I tell you the
referral, could you please set that as whitelist
(or at least, warn me before any suspensions ? We don't want to see
our website down because you dropped our
access to your maps)
As long as you follow the Maps Terms of Use [3], we should not have
any reason to block you. We always reserve the right to block any
traffic that we see as abusive or that might be a risk to the
stability of our infrastructure. But we will make reasonable efforts
to contact you in case of trouble.
Thank you in advance
Alessandro.
_______________________________________________
Maps-l mailing list
Maps-l(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/maps-l
[1]
https://github.com/kartotherian/kartotherian
[2]
https://mapnik.org/
[3]
https://foundation.wikimedia.org/wiki/Maps_Terms_of_Use
--
Guillaume Lederrey
Operations Engineer, Search Platform
Wikimedia Foundation
UTC+1 / CET
