Hello all!
The maps master server in our codfw datacenter has run out of disk
space [1]. While we have added more disks into that server lately, we
have not changed our partitioning configuration to make use of those
new disks (yes, you are allowed to laugh at me). This lead to this
server to run out of disk space.
We have enough redundancy in this cluster that this does not have
immediate impact on our tile service [2]. We are working on reimaging
this server with the correct disk configuration. During this work, the
tile generation will lag, and the latest updates to OSM will not be
applied in a timely manner.
Thanks for your comprehension and patience!
Guillaume
[1] https://phabricator.wikimedia.org/T224395
[2] https://maps.wikimedia.org/
--
Guillaume Lederrey
Operations Engineer, Search Platform
Wikimedia Foundation
UTC+1 / CET
This is just another small reminder that, because the servers which host
tiles.wmflabs.org and wma.wmflabs.org (wikiminiatlas) and overpass-wiki
run on a version of the OS (Ubuntu Trusty) that is no longer supported (and
hasn't been available for new instances since november 2017).
These services need maintainers and support by community members in order
to keep them alive after dec 18th (after which wmflabs will phase out those
versions) and before the EOL of early 2019 of the OS. Unfortunately it
seems no one is stepping up so far to convert these machines.
This issue is tracked at https://phabricator.wikimedia.org/T204506
As I was curious, I looked around on the tile server a bit and used what I
could find to update
https://wikitech.wikimedia.org/wiki/OSM_Tileserver#Technology_stack
This is all the information that I could gather, but i'm FAR from sure if
that is complete information and if I would break anything with a rebuild
basing myself on that info, so any information on missing elements etc.
would be appreciated. I've not gotten around to looking at wikiminiatlas.
If the services are not rebuild then likely they will just disappear at
some point for all layer variants. This includes the mapnik, black and
white, hill shading, hike bike layers. As I have no idea how many users of
these services there are, it is hard to say what the effect of that would
be.
DJ
FYI, with apologies for cross-posting...!
--
deb tankersley
program manager, engineering
Wikimedia Foundation
---------- Forwarded message ---------
From: Léa Lacroix <lea.lacroix(a)wikimedia.de>
Date: Mon, Dec 10, 2018 at 8:23 AM
Subject: [Wikidata] Miniature map for coordinate properties
To: Discussion list for the Wikidata project. <wikidata(a)lists.wikimedia.org>
Hello all,
After we deployed the Commons miniature pictures for the image properties,
some editors suggested to do something similar with maps. From Wednesday,
December 12th, we will also have a miniature map displayed for the
properties containing coordinates, like coordinate location (P625)
<https://www.wikidata.org/wiki/Property:P625>. This feature uses
mw:Extension:Kartographer
<https://www.mediawiki.org/wiki/Extension:Kartographer> and OpenStreetMap.
Viewing the map directly in the item will help the editors to check quickly
if the coordinates seem correct, and therefore improve the data quality.
-
<https://commons.wikimedia.org/wiki/File:Screenshot_Wikidata_map_feature_2.p…>
View in read mode
-
<https://commons.wikimedia.org/wiki/File:Screenshot_Wikidata_map_feature_3.p…>
View in edit mode
-
<https://commons.wikimedia.org/wiki/File:Screenshot_Wikidata_map_feature_1.p…>
Full screen with list of external maps
Some useful features:
- Double-click on the miniature map: display it in full screen
- On full screen view: click on “external maps” to display different map
services and links to many external maps
- In edit mode: when adding or editing coordinates in the field, the
maps is automatically updated to fit the coordinates you entered
- The coordinates are still displayed under the map and link to Geohack
If you have any questions or issues with this feature, let me know. You can
also see the related ticket <https://phabricator.wikimedia.org/T184933>.
Cheers,
--
Léa Lacroix
Project Manager Community Communication for Wikidata
Wikimedia Deutschland e.V.
Tempelhofer Ufer 23-24
10963 Berlin
www.wikimedia.de
Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.
Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/029/42207.
_______________________________________________
Wikidata mailing list
Wikidata(a)lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata
Hi,
thanks for providing this brilliant map service. I'd like to use it in a
third-party application (a website), in accordance with the terms of use. I
am sorry if this has been asked before, I tried to check the last couple
months of the mailing list, but couldn't find anything related.
For privacy reasons, the website uses the no-referrer policy. While I think
sharing a referrer with Wikimedia is alright, my website allows browsing
OSM data, including links to the websites set in OSM, and I don't trust
these websites with a referrer. Unfortunately, the referrer policy does not
allow fine-grained configuration, it's a global on/off switch. Changing the
user agent in a browser is considered a hack, and while it might work, it
can break other third-party tools and could break with every browser update
or when using an uncommon browser.
So I was wondering if you would consider a third method of authorization,
like a query parameter that can be added to the url like `?referrer=
example.com&contact=mail(a)example.com`? I think this could help people to
use your service in accordance to the terms of use.
Thanks in advance,
Robin
Hi to all.
We are currently using, for a project, your maps tiles (we like your
color schema)
As we would like to put them in our CDN to offload your servers, do
you have a safe
method to download pre-rendered map tiles or should I download the whole OSM
data file and generate all tiles manually ?
If possible, we prefere to not generate all tiles manually, due to
lack of resources
on our side.
Any idea ?
Thank you
For some time I’ve been working as a contractor developing a new style
and vector tile schema for the Wikimedia Foundation. It’s been completed
but not deployed for several months. As my contract finishes this month
and Wikimedia Foundation leadership has decided to not deploy the new
map styles, I’m writing up a technical lessons learned from my
experiences on the style. I’m not going to be discussing the
organizational factors that lead to the decision, but looking at how I’d
code things differently if starting over.
*Overview*
A complete map style consists of three parts: The database loading
rules, the feature selection rules, and the styling rules. For a style
written in the languages used by the WMF stack, these are expressed in
osm2pgsql instructions, a tm2source project with SQL, and CartoCSS. The
first tells you how to get the data into the database, the second
defines what data goes into the vector tiles, and the third is how to
draw features in the vector tiles. What goes in the vector tiles is also
known as the “schema” and can be expressed in terms of what features
appear and when, e.g. secondary roads first appear on zoom 12. For
increased confusion, the database also has a “schema”, both of which are
distinct from a PostgreSQL “SCHEMA.”
In the current style, the parts are the osm2pgsql C transforms,
osm-bright.tm2source, and WMF’s fork of osm-bright.tm2. In the new
style, the parts are ClearTables + osm2pgsql, meddo, and brighmed.
The goal with the style changes was to improve the representation of
disputed borders, switch to a vector tile schema without a legal cloud
over it, and make some styling improvements. In this it succeeded.
*Database schema*
The decision was made early on to go with ClearTables. This is an
alternative set of rules for osm2pgsql which loads the data into many
more tables for greater performance, easier style rules, and a bigger
layer of abstraction between raw OSM tags and the SQL you need to write.
It was started by me before my work at WMF and only a few features were
added.
ClearTables does what it is designed to do, yet it was a mistake for
this project. I still believe it is technically a better solution, yet
the advantages are not worth the costs of doing something different.
The two most common database schemas are the built-in osm2pgsql “C
transforms” and the OpenStreetMap Carto. They aren’t any better code -
with ClearTable’s test suite, it’s probably got fewer bugs, but there
are many guides on how to set them up, and it requires fewer components.
Setting up the database isn’t an issue for WMF production servers, but
is considered one of the more difficult steps for potential contributors
to any style. Minimizing differences from other setups here helps
greatly. A second issue is that many potential users of the style
already have a database. I have heard from multiple people who would
like to run the style if it could be used with their existing databases.
*Static data*
Map styles need some forms of “static” data loaded such as oceans,
low-zoom data, and borders. Normally this is done on an ad-hoc basis
with a long complicated shp2pgsql or ogr2ogr command, but I wrote a
python script that downloads the data and loads it with ogr2ogr, as well
as handling all the SQL needed to update the data without a service
interruption.
This script is useful enough that I have reused it for other projects,
which was made easy because I didn’t hard-code the files used into the
script, but used another file to define them.
*Borders*
One of the drivers of the work was to better display disputed borders.
To do this a pre-processing step was considered necessary, and I wrote a
necessary program in C++ with libosmium. This worked, but I should have
made more of an effort to get it packed by Debian GIS and run on Jochen
Toph’s OpenStreetMapData.com servers so others could use the work to
encourage more developers to participate in maintenance. I should also
have given pyosmium a more detailed look.
Vector tile schema
One of the reasons for switching to a new schema was legal threats
against people using the Mapbox Streets schema. This meant
osm2vectortiles also had to switch schemas at the same time. There was
an effort to work with them to use a common schema, but it never
happened because we had different needs. In retrospect, we should have
either gone with a common schema and tm2source project, or done nothing
in common. Either choice is valid, and it’s a balance of coordination
work against a common development direction.
It was useful to have someone external to discuss ideas with, but this
wouldn’t have been required with other people on the team.
*Style*
The original plan was to largely stick with the cartography of
osm-bright. This changed once we got into implementation and we realized
how insane some parts of the osm-bright cartography were, and efforts
were made towards redoing the style.
The road colours selected were from ColorBrewer2 OrRd6, with casing
colours done by adjusting the Lch lightness and chroma. It would have
been better to pick endpoints and generate colours using a script,
similar to osm-carto. This would have allowed easier changes and sped up
development by reducing the number of variables that need to be manually
set.
*Overall*
The style was completed successfully in time, and none of the changes
would have significantly changed that. They would have mainly made it
easier to attract external contributors if an effort were put into that.
As attracting external contributors wasn’t a priority, they didn’t matter.
