Hi folks! I am the Engineering Community Manager at the Wikimedia
Foundation, which means I'm a liaison for people who want to improve the
technical capabilities of our projects. I spoke with a few of you
offlist (including at Wikimania) about maps and Wikimedia. While I'm
still getting up to speed on the proposals and tools people have made, I
wanted to give you an update regarding what's happening with Wikimedia's
planned OpenStreetMap tileserver.
We know that we will set up a production tileserver. That is, Wikimedia
plans to have an OpenStreetMap tileserver, in production (that is, fully
supported by our Operations department and with uptime expectations
similar to those for our main sites). We need this not only for the
tools community but also for use in our official mobile apps. Brandon
Black is the contact for this. He is a member of the Operations team at
WMF. In order to set up a production-level tileserver, we will also need
a production OSM database, so he is working on figuring out how best to
grab that data from the official OSM database regularly (I am
oversimplifying here).
Brandon has already started on the project of setting up the production
tileserver. I know that he will be posting regular updates at
https://wikitech.wikimedia.org/wiki/OSM_Tileserver but he is still in
the investigation and architecture phase so he does not have a timeline
yet. I'm sorry, I know that is frustrating.
Very soon after we have the production tileserver running, we plan to
replicate that OSM database to Wikimedia Labs for use on Tool Labs, or
do similar SQL import magic. People using Labs will be able to draw
data directly from that replica, and will be able to grab tiles from the
production tileserver. I'm the contact on this but since Coren
(Marc-Andre Pelletier) is the domain expert on this particular bit of
database replication, please go ahead and include both of us on any
questions.
Thanks, all.
--
Sumana Harihareswara
Engineering Community Manager
Wikimedia Foundation
hi,
Thought this might have been of interest to this list.
warm regards,
Pradeep
Pradeep Mohandas
How Pradeep uses email? - http://goo.gl/6v1I9
On Friday, 18 October 2013 7:45 PM, Arun Ganesh <arun.planemad(a)gmail.com> wrote:
Dear community,
User:Yug[1] from the Graphic Labs and I, User:Planemad[2] from the Indian community, have put together a comprehensive IEG proposal to improve all the base maps used on WIkimedia projects with updated cartographic conventions and accuracy based on research.
The output will be a large set of editable vector maps, research documentation and a comprehensive map creation workflow that anyone can reuse.
The total grant request is for an amount of USD 10500, which will allow both of us to work on this project full time for a period of 3 months. The grant also includes a budget to hire external consultants where required to accomplish the stated goals.
You can view the proposal and provide your valuable feedback/endorsement here:
https://meta.wikimedia.org/wiki/Grants:IEG/Wikimaps_Atlas
PS: This is the first step of the much larger Wikimaps project that aims to be the open source map engine that brings together Wikidata, Commons and Openstreetmap.
[1] https://en.wikipedia.org/wiki/User:Yug
[2] https://en.wikipedia.org/wiki/User:Planemad
[3] https://meta.wikimedia.org/wiki/Wikimaps
--
Arun Ganesh
(planemad)
_______________________________________________
Wikimediaindia-l mailing list
Wikimediaindia-l(a)lists.wikimedia.org
To unsubscribe from the list / change mailing preferences visit https://lists.wikimedia.org/mailman/listinfo/wikimediaindia-l
Hello,
in the last days Toolserver experienced outages of web pages which were caused by too many queries from only a few hosts.
They are using OSM images and - please dont ask me why - single IPs tend to query about 40-50 pictures per second - for minutes or hours, peaks can be worse.
At some points our web server give up then.
Yes sorry ;). I can proudly say that only today about 11.7 millions web queries were answered somehow.
I tried to mitigate the problem of "too many requests per IP" via blocking but it is not an option.
One problem is that users of at least one portal then complain and another is that the IP addresses seem random - coming even from dial up ranges.
There might be something badly wrong with cache-control headers for the images (or probably we can tweak at that point) or - I dont know what it could be.
To make the long story short - I rate limited the OSM tile delivery to 40 images per second per IP - allowed burst is 55.
Users will then get a 503 error if the rate exceeds until it decreases - but delivery isnt stopped completely.
It seems to work since I have some notices which IPs were throtteled and these are IPs that have heavy usage.
I used this here to throttle: http://nginx.org/en/docs/http/ngx_http_limit_req_module.html
I dont want to have this option configured forever - I rather hope we can do something about caching or give the pictures they need to the projects themselves (I doubt we have to deliver hill shading pictures for everyone - this is Toolserver)
If anyone has an idea what to do / questions - please let me know.
Cheers
Marlen/nosy