Hi
Now that we have a full Histoy Planet Dump (Thank you Lars, Matt and all who helped with that!), we can again start talking about a History Server and a History Api.
What I'm thinking of is an extension of the 0.6 API, that includes calls like this:
GET /api/0.6/[way|relation]/#id/#version/full GET /api/0.6/node/#id/#version/ways GET /api/0.6/node/#id/ways?time=#time GET /api/0.6/[node|way|relation]/#id?time=#time GET /api/0.6/[node|way|relation]/#id/full?time=#time GET /api/0.6/map?time=#time
and maybe others like GET /api/0.6/[nodes|ways|relations]/#id1,#id2,...,#idN GET /api/0.6/[nodes|ways|relations]/#id1/#ver1/.../#idN/#verN
This would make it easy for a client to fetch the state of a city at any given time. It would also enable a client to get the complete geometry of a way or a relation in any given version or at any given time.
This would make it easy to create animations like the "year of edits" for a town or maybe also a country on demand and with real mapnik styles.
The version-based calls (especially GET /api/0.6/node/#id/#version/ways) would make it possible to visualize the changes made in a given changeset: if a way was changed, it is contained with each node and each node's version, so a client can fetch all those nodes from api and re-build the way's exact geometry before and after the change
if a node is changed, the ways depending on it are not included in the changeset and with the current api it's not possible to fetch the geometry of the ways depending on this node as they were before the change
I know that there are implications on how versions are counted (one version of a way could stand for more than one geometry, as its nodes could have been moved without the version of the way being incremented) but they can be solved by simply defining sth. like GET /api/0.6/way/#id/#version/full returns the full geometry of the way #id, as it was when version #version of was created.
Still all possible geometries of the way could be accessed via the time predicates.
When taking this idea further, a historic api could even implement "minor versions" on ways/relations, thus enabling a client to address any change in the geometry of a way (should read: any change on one of it's nodes) explicitly.
Such an api could create heavy load on the underlying database, what is AFAIR the main reason why such calls aren't implemented in the current api. But unlike on the main api, queries to an such a history api don't need to be answered in miliseconds. When a /map query for a city on 1st jan 2007 takes 15 minutes, well that's ok. I's still faster than to download the corresponding planet dump, import int into postgis and catch up with the diffs, so what?
My questions to you are - do you think such an server/service/api would be of use for a greater audience? - who would be willing to develop, and who would be able to host such a service? - do you think these calls can be implemented without having 10 queries crash the database?
I think ptolemy, the new-new Wikimedia Toolserver has enough space (2336 GB of raw HD space -- physical, bot logical!) and enough power to run such an api -- that's why I CCed Maps-L -- maybe Ævar can say if he thinks this would be possible.
Thank you for reading and commenting, Peter
Something like this is well within the capabilities of XAPI and I had already downloaded a copy of the full history and was just starting to think about what should be done with it.
80n
On Wed, Dec 9, 2009 at 8:27 PM, Peter Körner osm-lists@mazdermind.dewrote:
Hi
Now that we have a full Histoy Planet Dump (Thank you Lars, Matt and all who helped with that!), we can again start talking about a History Server and a History Api.
What I'm thinking of is an extension of the 0.6 API, that includes calls like this:
GET /api/0.6/[way|relation]/#id/#version/full GET /api/0.6/node/#id/#version/ways GET /api/0.6/node/#id/ways?time=#time GET /api/0.6/[node|way|relation]/#id?time=#time GET /api/0.6/[node|way|relation]/#id/full?time=#time GET /api/0.6/map?time=#time
and maybe others like GET /api/0.6/[nodes|ways|relations]/#id1,#id2,...,#idN GET /api/0.6/[nodes|ways|relations]/#id1/#ver1/.../#idN/#verN
This would make it easy for a client to fetch the state of a city at any given time. It would also enable a client to get the complete geometry of a way or a relation in any given version or at any given time.
This would make it easy to create animations like the "year of edits" for a town or maybe also a country on demand and with real mapnik styles.
The version-based calls (especially GET /api/0.6/node/#id/#version/ways) would make it possible to visualize the changes made in a given changeset: if a way was changed, it is contained with each node and each node's version, so a client can fetch all those nodes from api and re-build the way's exact geometry before and after the change
if a node is changed, the ways depending on it are not included in the changeset and with the current api it's not possible to fetch the geometry of the ways depending on this node as they were before the change
I know that there are implications on how versions are counted (one version of a way could stand for more than one geometry, as its nodes could have been moved without the version of the way being incremented) but they can be solved by simply defining sth. like GET /api/0.6/way/#id/#version/full returns the full geometry of the way #id, as it was when version #version of was created.
Still all possible geometries of the way could be accessed via the time predicates.
When taking this idea further, a historic api could even implement "minor versions" on ways/relations, thus enabling a client to address any change in the geometry of a way (should read: any change on one of it's nodes) explicitly.
Such an api could create heavy load on the underlying database, what is AFAIR the main reason why such calls aren't implemented in the current api. But unlike on the main api, queries to an such a history api don't need to be answered in miliseconds. When a /map query for a city on 1st jan 2007 takes 15 minutes, well that's ok. I's still faster than to download the corresponding planet dump, import int into postgis and catch up with the diffs, so what?
My questions to you are
- do you think such an server/service/api would be of use for a greater audience?
- who would be willing to develop, and who would be able to host such a service?
- do you think these calls can be implemented without having 10 queries crash the database?
I think ptolemy, the new-new Wikimedia Toolserver has enough space (2336 GB of raw HD space -- physical, bot logical!) and enough power to run such an api -- that's why I CCed Maps-L -- maybe Ævar can say if he thinks this would be possible.
Thank you for reading and commenting, Peter
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
On this topic, note that there are now full history daily diffs available that can be used to keep such a database up to date. http://planet.openstreetmap.org/history/
When up to date information is not critical, they're a little easier to deal with than the http://planet.openstreetmap.org/minute-replicate/ diffs because they are exactly time aligned, and in daily chunks.
In theory it is possible to replay these history diffs sequentially from day one to the current time and end up with a hopefully similar result to the full history dump.
On Thu, Dec 10, 2009 at 7:52 AM, 80n 80n80n@gmail.com wrote:
Something like this is well within the capabilities of XAPI and I had already downloaded a copy of the full history and was just starting to think about what should be done with it.
80n
On Wed, Dec 9, 2009 at 8:27 PM, Peter Körner osm-lists@mazdermind.dewrote:
Hi
Now that we have a full Histoy Planet Dump (Thank you Lars, Matt and all who helped with that!), we can again start talking about a History Server and a History Api.
What I'm thinking of is an extension of the 0.6 API, that includes calls like this:
GET /api/0.6/[way|relation]/#id/#version/full GET /api/0.6/node/#id/#version/ways GET /api/0.6/node/#id/ways?time=#time GET /api/0.6/[node|way|relation]/#id?time=#time GET /api/0.6/[node|way|relation]/#id/full?time=#time GET /api/0.6/map?time=#time
and maybe others like GET /api/0.6/[nodes|ways|relations]/#id1,#id2,...,#idN GET /api/0.6/[nodes|ways|relations]/#id1/#ver1/.../#idN/#verN
This would make it easy for a client to fetch the state of a city at any given time. It would also enable a client to get the complete geometry of a way or a relation in any given version or at any given time.
This would make it easy to create animations like the "year of edits" for a town or maybe also a country on demand and with real mapnik styles.
The version-based calls (especially GET /api/0.6/node/#id/#version/ways) would make it possible to visualize the changes made in a given changeset: if a way was changed, it is contained with each node and each node's version, so a client can fetch all those nodes from api and re-build the way's exact geometry before and after the change
if a node is changed, the ways depending on it are not included in the changeset and with the current api it's not possible to fetch the geometry of the ways depending on this node as they were before the change
I know that there are implications on how versions are counted (one version of a way could stand for more than one geometry, as its nodes could have been moved without the version of the way being incremented) but they can be solved by simply defining sth. like GET /api/0.6/way/#id/#version/full returns the full geometry of the way #id, as it was when version #version of was created.
Still all possible geometries of the way could be accessed via the time predicates.
When taking this idea further, a historic api could even implement "minor versions" on ways/relations, thus enabling a client to address any change in the geometry of a way (should read: any change on one of it's nodes) explicitly.
Such an api could create heavy load on the underlying database, what is AFAIR the main reason why such calls aren't implemented in the current api. But unlike on the main api, queries to an such a history api don't need to be answered in miliseconds. When a /map query for a city on 1st jan 2007 takes 15 minutes, well that's ok. I's still faster than to download the corresponding planet dump, import int into postgis and catch up with the diffs, so what?
My questions to you are
- do you think such an server/service/api would be of use for a greater audience?
- who would be willing to develop, and who would be able to host such a service?
- do you think these calls can be implemented without having 10 queries crash the database?
I think ptolemy, the new-new Wikimedia Toolserver has enough space (2336 GB of raw HD space -- physical, bot logical!) and enough power to run such an api -- that's why I CCed Maps-L -- maybe Ævar can say if he thinks this would be possible.
Thank you for reading and commenting, Peter
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
dev mailing list dev@openstreetmap.org http://lists.openstreetmap.org/listinfo/dev
On 09/12/09 20:27, Peter Körner wrote:
Such an api could create heavy load on the underlying database, what is AFAIR the main reason why such calls aren't implemented in the current api. But unlike on the main api, queries to an such a history api don't need to be answered in miliseconds. When a /map query for a city on 1st jan 2007 takes 15 minutes, well that's ok. I's still faster than to download the corresponding planet dump, import int into postgis and catch up with the diffs, so what?
You may not mind that it takes 15 minutes to answer your query but the problem is that long query times like that probably won't scale because if an average query takes 15 minutes then the server (assuming that we're talking an I/O bound process) can only handle four queries per hour before it starts running into the ground.
Even if it's CPU bound you'll only manage 4 queries/CPU/hour.
Tom
Hi,
Tom Hughes wrote:
You may not mind that it takes 15 minutes to answer your query but the problem is that long query times like that probably won't scale because if an average query takes 15 minutes then the server (assuming that we're talking an I/O bound process) can only handle four queries per hour before it starts running into the ground.
One could always try to make the user interface so awkward that only 4 people per hour manage to put in a request ;-)
Bye Frederik
Peter Here's an implementation of the first item on your list (ways not relations): http://osmxapi.hypercube.telascience.org/api/0.6/way/#id/#version/full
The following example, for Pharaoh's Island in the River Thames shows a series of edits with different versions of nodes depending on which version of the way you look at:
http://osmxapi.hypercube.telascience.org/api/0.6/way/8136261/8/full http://osmxapi.hypercube.telascience.org/api/0.6/way/8136261/9/full http://osmxapi.hypercube.telascience.org/api/0.6/way/8136261/10/full http://osmxapi.hypercube.telascience.org/api/0.6/way/8136261/11/full
This should work for any versions of any way since yesterday, but as the full history is not yet loaded older revisions are not yet available.
80n
On Wed, Dec 9, 2009 at 8:27 PM, Peter Körner osm-lists@mazdermind.dewrote:
Hi
Now that we have a full Histoy Planet Dump (Thank you Lars, Matt and all who helped with that!), we can again start talking about a History Server and a History Api.
What I'm thinking of is an extension of the 0.6 API, that includes calls like this:
GET /api/0.6/[way|relation]/#id/#version/full GET /api/0.6/node/#id/#version/ways GET /api/0.6/node/#id/ways?time=#time GET /api/0.6/[node|way|relation]/#id?time=#time GET /api/0.6/[node|way|relation]/#id/full?time=#time GET /api/0.6/map?time=#time
and maybe others like GET /api/0.6/[nodes|ways|relations]/#id1,#id2,...,#idN GET /api/0.6/[nodes|ways|relations]/#id1/#ver1/.../#idN/#verN
This would make it easy for a client to fetch the state of a city at any given time. It would also enable a client to get the complete geometry of a way or a relation in any given version or at any given time.
This would make it easy to create animations like the "year of edits" for a town or maybe also a country on demand and with real mapnik styles.
The version-based calls (especially GET /api/0.6/node/#id/#version/ways) would make it possible to visualize the changes made in a given changeset: if a way was changed, it is contained with each node and each node's version, so a client can fetch all those nodes from api and re-build the way's exact geometry before and after the change
if a node is changed, the ways depending on it are not included in the changeset and with the current api it's not possible to fetch the geometry of the ways depending on this node as they were before the change
I know that there are implications on how versions are counted (one version of a way could stand for more than one geometry, as its nodes could have been moved without the version of the way being incremented) but they can be solved by simply defining sth. like GET /api/0.6/way/#id/#version/full returns the full geometry of the way #id, as it was when version #version of was created.
Still all possible geometries of the way could be accessed via the time predicates.
When taking this idea further, a historic api could even implement "minor versions" on ways/relations, thus enabling a client to address any change in the geometry of a way (should read: any change on one of it's nodes) explicitly.
Such an api could create heavy load on the underlying database, what is AFAIR the main reason why such calls aren't implemented in the current api. But unlike on the main api, queries to an such a history api don't need to be answered in miliseconds. When a /map query for a city on 1st jan 2007 takes 15 minutes, well that's ok. I's still faster than to download the corresponding planet dump, import int into postgis and catch up with the diffs, so what?
My questions to you are
- do you think such an server/service/api would be of use for a greater audience?
- who would be willing to develop, and who would be able to host such a service?
- do you think these calls can be implemented without having 10 queries crash the database?
I think ptolemy, the new-new Wikimedia Toolserver has enough space (2336 GB of raw HD space -- physical, bot logical!) and enough power to run such an api -- that's why I CCed Maps-L -- maybe Ævar can say if he thinks this would be possible.
Thank you for reading and commenting, Peter
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l
http://osmxapi.hypercube.telascience.org/api/0.6/way/8136261/8/full http://osmxapi.hypercube.telascience.org/api/0.6/way/8136261/9/full http://osmxapi.hypercube.telascience.org/api/0.6/way/8136261/10/full http://osmxapi.hypercube.telascience.org/api/0.6/way/8136261/11/full
This should work for any versions of any way since yesterday, but as the full history is not yet loaded older revisions are not yet available.
Very cool thing.. Maybe I'll be able to implement some visualization from this, at least i think it would be cool just to try it.
If we'd have GET /api/0.6/node/#id/#version/ways we'd have all api calls together, needed for a very cool changeset-changes viewer and a revert tool *yeehaww*
Peter
2009/12/10 Peter Körner osm-lists@mazdermind.de:
http://osmxapi.hypercube.telascience.org/api/0.6/way/8136261/8/full http://osmxapi.hypercube.telascience.org/api/0.6/way/8136261/9/full http://osmxapi.hypercube.telascience.org/api/0.6/way/8136261/10/full http://osmxapi.hypercube.telascience.org/api/0.6/way/8136261/11/full
This should work for any versions of any way since yesterday, but as the full history is not yet loaded older revisions are not yet available.
Very cool thing.. Maybe I'll be able to implement some visualization from this, at least i think it would be cool just to try it.
If we'd have GET /api/0.6/node/#id/#version/ways we'd have all api calls together, needed for a very cool changeset-changes viewer and a revert tool *yeehaww*
What about deleted nodes/ways/relations, but instead of giving an ID how about a bounding box so you can undo a delete?
What about deleted nodes/ways/relations, but instead of giving an ID how about a bounding box so you can undo a delete?
For the moment I'm thinking about a changeset revert. In a changeset you'll get deleted objects marked as such. Finding a deleted node (or better: the changeset in which a node was deleted) in an area is another thing.
But one step after the another..
Peter
On Thu, Dec 10, 2009 at 11:44 AM, Peter Körner osm-lists@mazdermind.dewrote:
http://osmxapi.hypercube.telascience.org/api/0.6/way/8136261/8/full http://osmxapi.hypercube.telascience.org/api/0.6/way/8136261/9/full http://osmxapi.hypercube.telascience.org/api/0.6/way/8136261/10/full http://osmxapi.hypercube.telascience.org/api/0.6/way/8136261/11/full
This should work for any versions of any way since yesterday, but as the full history is not yet loaded older revisions are not yet available.
Very cool thing.. Maybe I'll be able to implement some visualization from this, at least i think it would be cool just to try it.
If we'd have GET /api/0.6/node/#id/#version/ways we'd have all api calls together, needed for a very cool changeset-changes viewer and a revert tool *yeehaww*
The instance of XAPI on hypercube now implements the following:
GET http://osmxapi.hypercube.telascience.org/api/0.6/node/#id GET http://osmxapi.hypercube.telascience.org/api/0.6/node/#id/#version *GET http://osmxapi.hypercube.telascience.org/api/0.6/node/#id/#version/ways * GET http://osmxapi.hypercube.telascience.org/api/0.6/way/#id GET http://osmxapi.hypercube.telascience.org/api/0.6/way/#id/full GET http://osmxapi.hypercube.telascience.org/api/0.6/way/#id/#version GET http://osmxapi.hypercube.telascience.org/api/0.6/way/#id/#version/full
GET http://osmxapi.hypercube.telascience.org/api/0.6/relation/#id GET http://osmxapi.hypercube.telascience.org/api/0.6/relation/#id/full GET http://osmxapi.hypercube.telascience.org/api/0.6/relation/#id/#version GET http://osmxapi.hypercube.telascience.org/api/0.6/relation/#id/#version/full
The full history hasn't been loaded so there is only version data for new versions created in the last two or three days. Until the full history is loaded (which may not happen this year) any output from these queries should be considered to be experimental and may be incomplete.
Feedback and comments would be welcome and much appreciated. 80n
The instance of XAPI on hypercube now implements the following:
GET http://osmxapi.hypercube.telascience.org/api/0.6/node/#id GET http://osmxapi.hypercube.telascience.org/api/0.6/node/#id/#version *GET http://osmxapi.hypercube.telascience.org/api/0.6/node/#id/#version/ways
GET http://osmxapi.hypercube.telascience.org/api/0.6/way/#id GET http://osmxapi.hypercube.telascience.org/api/0.6/way/#id/full GET http://osmxapi.hypercube.telascience.org/api/0.6/way/#id/#version GET http://osmxapi.hypercube.telascience.org/api/0.6/way/#id/#version/full
GET http://osmxapi.hypercube.telascience.org/api/0.6/relation/#id GET http://osmxapi.hypercube.telascience.org/api/0.6/relation/#id/full GET http://osmxapi.hypercube.telascience.org/api/0.6/relation/#id/#version GET http://osmxapi.hypercube.telascience.org/api/0.6/relation/#id/#version/full
The full history hasn't been loaded so there is only version data for new versions created in the last two or three days. Until the full history is loaded (which may not happen this year) any output from these queries should be considered to be experimental and may be incomplete.
Feedback and comments would be welcome and much appreciated. 80n
Oh whow, this is sooooooo damn cool! This opens up the way for real smart diff- and revert tools, which will greatly improve our response-times on vandalism.
I really need to get some more time to work on this.. I think I got to take some holidays in January..
Thank you so much!
Peter
2009/12/15 80n 80n80n@gmail.com:
Feedback and comments would be welcome and much appreciated. 80n
Is it possible to supply a parameter to not just get a particular version, but to get all versions?
On Tue, Dec 15, 2009 at 2:19 AM, John Smith deltafoxtrot256@gmail.comwrote:
2009/12/15 80n 80n80n@gmail.com:
Feedback and comments would be welcome and much appreciated. 80n
Is it possible to supply a parameter to not just get a particular version, but to get all versions?
Like this: GET http://osmxapi.hypercube.telascience.org/api/0.6/node/#id/history GET http://osmxapi.hypercube.telascience.org/api/0.6/way/#id/history GET http://osmxapi.hypercube.telascience.org/api/0.6/relation/#id/history
80n
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Peter Körner:
I think ptolemy, the new-new Wikimedia Toolserver has enough space (2336 GB of raw HD space -- physical, bot logical!)
ptolemy has 820GB of usable space, of which 100GB are taken up by the current copy of the mapnik database. we may be able to add more space later if there's a use for it.
- river.
XAPI currently requires about 350Gb. I don't know yet how much will be required with full history loaded, but I'd guess it might be about 200Gb more.
80n
On Thu, Dec 10, 2009 at 8:21 AM, River Tarnell < river@loreley.flyingparchment.org.uk> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Peter Körner:
I think ptolemy, the new-new Wikimedia Toolserver has enough space (2336 GB of raw HD space -- physical, bot logical!)
ptolemy has 820GB of usable space, of which 100GB are taken up by the current copy of the mapnik database. we may be able to add more space later if there's a use for it.
- river.
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (HP-UX)
iEYEARECAAYFAksgr5QACgkQIXd7fCuc5vI/1QCgre5R+17qQKqoWl14JtyG8VmM x1YAnRppdxUR/irmV1EEB4YGIGu9DVG7 =oID8 -----END PGP SIGNATURE-----
Maps-l mailing list Maps-l@lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/maps-l