Hello, On 2012-12-18 15:29, Denny Vrandečić wrote: Sometimes we do not have exact dates for an events, because they are in future, like the opening of the airport Berlin/Brandenburg ;-) So in this case it would be great to have property values like N/A for "not available" for some parts. A date could then look like May, N/A 1435. Expressions like 2nd to 56h century could easily be expressed by using tolerances. This would then look like 300 AD ± 200 years, which will is (max+min)/2 ± |(max-(max+min)/2)|. Altitudes are for sure very important. Maybe not in Germany but for Austria I know several altitudes. It's very interested for passes, mountains, cities, villages, wells of brooks, etc. How will you create a list of the highest mountains without an altitude of their highest peaks (Phase 3)? Or even one for the lowest passes for crossing the alps. Look at the articles for living creatures. Often there is mentioned from which to which altitude they can be found. There we also have it twice. But here we do not have geolocations. IMHO it would be make sense to have something hybrid. The datatype for geolocation should accept something like a NAN-value for optional altitudes. But it should also be possible to use altitudes without longitude and latitude. So we can store something like http://www.grassyknolltv.com/2012/tour-de-france/resources/maps/ETAPE%2007%… http://www.grassyknolltv.com/2012/tour-de-france/resources/profiles/profile… which is both about the 7th stage of the Tour de France 2012, just in different views. Providing the possibility to store seperate altitudes allows us to store properties like "grows at altitudes from 1800 to 2200 meters above sea level". Separate altitudes is possibly very similar to lengths of snakes which is also measured in meters and has also to value just in another dimension. IMHO it would make sense to use the [[International System of Units]] for internal storage. It is not consequently used in other realms, not even in the German spoken countries (PS vs. kW for cars). Maybe it would be possible to use small scripts (such as WP-templates) to transcalc values, which can easily be developed by the community. Cheers Marco

On 2012-12-18 16:52, Denny Vrandečić wrote: NP I'd use the same unit as tha base – its 300 years AD so the uncertainty is also measured in years. A list of the highest mountains with their peaks within a specific area? The answer is below. Peaks of mountains e.g. are geolocated with an altitude. Also passes, some events, places where something scientific was found or took place. E.g. preventing having one geolocation and multiple altitudes. A list of point on one section of the Tour de France should contain all point where each one should contain all three components, because they belong together. The place is always mentioned with the altitude. +1 Cheers Marco

On 2012-12-18 17:49, Gregor Hagedorn wrote: But floating point numbers are handled by the implementation of [[IEEE floating-point standard]]. Displaying the numbers is another question. There I have to agree that it always makes sense to also store a typical used unit for that type of data. Distances for towns are usually displayed in km the High of mountains is displayd in metres if we're talking of the metric system. So I assume that this is user or language preference also called locale-set. Cheers Marco

On 19/12/12 08:53, Gregor Hagedorn wrote: Perhaps both should be stored. 1.6e-8 is necessary for sorting and comparison. But 10.6 nm is how the user entered it, presumably how it was written in the source that the user used, how is it preferably used in the given field, and how other users would want to see it and edit it. As an example, human height is commonly given in centimetres, while building height is commonly given in metres. So, users will probably prefer to edit the tallest person as 282cm and the lowest building as 2.1m even though the absolute values are similar.

On 19.12.2012 11:56, Friedrich Röhrs wrote: We are talking about searching and sorting millions of data entries - doing that in PHP would be extremely slow and would take far more memory than we have. It has to be done natively in the database. So we have to use a data representation that can be natively compared and sorted by the database (at the very least by MySQL, but ideally, by many different database systems). -- daniel -- Daniel Kinzler, Softwarearchitekt Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.

Hi, Sorry for my ignorance, if this is common knowledge: What is the use case for sorting millions of different measures from different objects? The only use case i can think of where the sorting would be necessary is lists like "cities in France by size" or "list of cars by fuel/per 100km". Those list do not have millions of entries, and even there you would need some further logic, since the size is not just one single value, there could be multiple values for the size (or fuel consumption) with different sources and for different times. For cars there could be entries by the manufacturer, by some car-testing magazine, etc. I don't see how this could be adequatly represented/sorted by a database only query. If however this is necessary, i still don't understand why it must affect the datavalue structure. If a index is necessary it could be done over a serialized representation of the value. This needs to be done anyway, since the values are saved at a specific unit (which is just a wikidata item). To compare them on a database level they must all be saved at the same unit, or some sort of procedure must be used to compare them (or am i missing something again?). Friedrich On Wed, Dec 19, 2012 at 12:25 PM, Gregor Hagedorn <g.m.hagedorn(a)gmail.com>wrote;wrote:

Hey wikidatians, occasionally checking threads in this list like the current one, I get a mixed feeling: on one hand, it is sad to see the efforts and resources waisted as Wikidata tries to reinvent RDF, and now also triplestore design as well as XSD datatypes. What's next, WikiQL instead of SPARQL? On the other hand, it feels reassuring as I was right to predict this: http://www.mail-archive.com/wikidata-l@lists.wikimedia.org/msg00056.html http://www.mail-archive.com/wikidata-l@lists.wikimedia.org/msg00750.html Best, Martynas graphity.org On Wed, Dec 19, 2012 at 4:11 PM, Daniel Kinzler <daniel.kinzler(a)wikimedia.de> wrote:

Martynas, could you please let me know where RDF or any of the W3C standards covers topics like units, uncertainty, and their conversion. I would be very much interested in that. Cheers, Denny 2012/12/19 Martynas Jusevičius <martynas(a)graphity.org> -- Project director Wikidata Wikimedia Deutschland e.V. | Obentrautstr. 72 | 10963 Berlin Tel. +49-30-219 158 26-0 | http://wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985.

The NIST ontology defines 4 basic classes that are great: _qudt:QuantityKind [11]_, _qudt:Quantity [12]_, _qudt:QuantityValue [13]_, _qudt:Unit [14]_ but the properties set leaves me a bit thirsty. Take "Area" as an example. I'd like to reference properties named .ft2 and .m2 so that, for instance, an annotation might be [[Leasable area.ft2::12345]]. To state the precision applicable to that measurement, might be [[Leasable area.ft2:fractionDigits :: 0]] to indicate say, rounding. However, in the NIST ontology, there is no "ft2" property at all -- this is an SI unit though, so it seems identifying first the system of measurement units, and then the specific measurement unit is not a great idea because these notations are then divorced from the property name itself, a scenario guaranteed to cause more user errors & omissions I think. Someone's mentioned uncertainty facets, so I suggest these from the qudt ontology: Property: .anyType:relativeStandardUncertainty Property: .anyType:standardUncertainty Other facets noted might include Property: .anyType:abbreviation Property: .anyType:description Property: .anyType:symbol -john On 19.12.2012 08:10, Herman Bruyninckx wrote: Martynas, could you please let me know where RDF or any of the W3C standards covers topics like units, uncertainty, and their conversion. I would be very much interested in that. in OWL: "QUDT - Quantities, Units, Dimensions and Data Types in OWL and XML": share Martynas' concerns: most of the problems that are being discussed in this thread (and that are very relevant and interesting) should not be solved with an "object oriented" approach (that is, via properties of objects, and "inheritance") but by semantic modelling (that base on who context are the "Topic Map" opinion: Denny 16 328056 Vice-President Research euRobotics <http://www.eu-robotics.net [7]> Open RObot COntrol Software <http://www.orocos.org [8]> Associate Editor JOSER <http://www.joser.org [9]>, IJRR <http://www.ijrr.org [10] mailing list > Wikidata-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-l [1] Links: ------ [1] https://lists.wikimedia.org/mailman/listinfo/wikidata-l [2] http://www.mail-archive.com/wikidata-l@lists.wikimedia.org/msg00056.html [3] http://www.mail-archive.com/wikidata-l@lists.wikimedia.org/msg00750.html [4] http://wikimedia.de [5] http://www.qudt.org/qudt/owl/1.0.0/index.html [6] http://en.wikipedia.org/wiki/Topic_map [7] http://www.eu-robotics.net [8] http://www.orocos.org [9] http://www.joser.org [10] http://www.ijrr.org [11] http://www.qudt.org/qudt/owl/1.0.0/qudt/index.html#QuantityKind [12] http://www.qudt.org/qudt/owl/1.0.0/qudt/index.html#Quantity [13] http://www.qudt.org/qudt/owl/1.0.0/qudt/index.html#QuantityValue [14] http://www.qudt.org/qudt/owl/1.0.0/qudt/index.html#Unit

Martynas, I think you misinterpret the thread. There is no discussion not to build on the datatypes defined in http://www.w3.org/TR/xmlschema-2/ What we are doing is discussing compositions of elements, all typed to xml datatypes, that shall be able to express scientific and engineering requirements as to statistics, signficant digits (except perhaps for duration, none of the data types in http://www.w3.org/TR/xmlschema-2/ supports that), as well as means to express uncertainty and confidence intervals. Many existing xml schemata define such compositions, all squarely built on http://www.w3.org/TR/xmlschema-2/ - wikidata is certainly not unique in this effort. If you can point the team to further well reviewed solutions, this would be very useful. Gregor

On 19 December 2012 20:01, &lt;jmcclure(a)hypergrove.com&gt; wrote: It is about semantics, not formatting. In science and engineering, the number of significant digits is not used to right align numbers, but to semantically indicate the order of magnitude of the accuracy and/or precision of a measurement or quantity. Thus, the weight of a machine can be given as 1.2 t (exact to +/- 50 kg), 1200 kg (+/- 1 kg), or 1200.000 g. This is not part of IEEE floating point numbers, which always have the type dependent same precision or number of significant digits, regardless whether this is semantically justified or not. IEEE 754 standard double always has about 16 decimal significant digits, i.e. the value 1.2 tons will always be given as 1.200000000000000 tons. This is good for calculations, but lacks the information for final rounding. Gregor

they don't. They allow to define derived datatypes and thus apply to the datatype, not the measurement. Different measurements of the same datatype may be of different precision. --gregor

On 20 December 2012 02:20, &lt;jmcclure(a)hypergrove.com&gt; wrote: Yes, that would be one way of modeling it. And I agree with you that, although the xsd attributes originally are devised for datatypes, there is nothing wrong with re-using it for quantities and measurements. So one way of expressing a measurement with significant digits is: (Proposal 1) * normalizedValue * totalDigits * fractionDigits * originalUnit * normalizedUnit To recover the original information (e.g. that the original value was in feet with a given number of significant digits) the software must convert normalizedUnit to originalUnit, scale to totalDigits with fractionDigits, calculate the remaining powers of ten, and use some information that must be stored together with each unit whether this then should be expressed using an SI unit prefix (the Exa, Tera, Giga, Mega, kilo, hekto, deka, centi, etc.). Some units use them, others not, and some units use only some. Hektoliter is common, hektometer would be very odd. This is slightly complicated by the fact that for some units prefix usage in lay topics differs from scientific use. If all numbers were expressed ONLY as total digits with fraction digits and unit-prefix, i.e. no power-of-ten exponential, the above would be sufficiently complete. However, without additional information it does not allow to recover the entry: 100,230 * 10^3 tons (value 1.0023e8, 6 total, 3 fractional digits, original unit tons, normalized unit gram) I had therefore made (on the wiki) the proposal to express it as: (Proposal 2) * normalizedValue * significantDigits (= and I am happy with totalDigits instead) * originalUnit * originalUnitPrefix * normalizedUnit However I see now that the analysis was wrong, indeed it needs fractionDigits in addition to totalDigits, else a similar problem may occur, i.e. the distribution of the total order of magnitude of the number between non-fractional digits, fractional digits, powers of 10 and powers-of-10-expressed through SI units is still not unambigous. So the minimal representation seems to be: (Proposal 3) * normalizedValue (xsd:double or xsd:decimal) * totalDigits (xsd:smallint) * fractionDigits (xsd:smallint) * originalUnit (a wikidata item) * originalUnitPrefix (a wikidata item) * normalizedUnit (a wikidata item) Adding the originalUnitPrefix has the advantage that it gathers knowledge from users and data creators or resources about which unit prefix is appropriate in a given context. I see the current wikidata plan to solve this problem by heuristics very critical, I do not see the data set that sufficiently tests the heuristics yet. Gathering information from data entered and creating a formatting heuristics modules over the coming years (instead of weeks) will be valuable for reformatting. The Proposal 3 allows to gather this information. Gregor Note 1: The question of other means to express accuracy or precision, e.g. by error margins, statistical measures of spread such as variance, confidence intervals, percentiles, min/max etc. is not yet covered. Given the present discussion, this should probably be separately agreed upon. Note 2: Wikipedia Infoboxes may desire to override it, this is for data entering, review, curation, and a default display where no other is defined

I am still trying to catch up with the whole discussion and to distill the results, both here and on the wiki. In the meanwhile, I have tried to create a prototype of how a complex model can still be entered in a simple fashion. A simple demo can be found here: <http://simia.net/valueparser/> The prototype is not i18n. The user has to enter only the value, in a hopefully intuitive way (try it out), and the full interpretation is displayed here (that, alas, is not intuitive, admittedly). Cheers, Denny 2012/12/20 &lt;jmcclure(a)hypergrove.com&gt; -- Project director Wikidata Wikimedia Deutschland e.V. | Obentrautstr. 72 | 10963 Berlin Tel. +49-30-219 158 26-0 | http://wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985.

I believe there are a lot of dangerous assumptions on http://simia.net/valueparser/ First: there is no indication in a number that it is _not_ endlessly precise. Apostles = 12 has no uncertainty, representing it as 12 ± 1 is wrong, but also 12 ± 0.5 is wrong. The same applies to a number like 12.2. The data source and author MAY desire to express significant digits, but we simply don't know. Wikidata should keep this at the don't know level and not force-convert a number of unknown measurement precision to a number with explicitly stated (but potentially totally wrong) precision or accuracy limits. For example, in science it is quite common to give light microscopic measurement to one decimals behind the micrometer, even though the precision is 0.2 µm. The latter is simply known and therefore not constantly repeated, unless specific circumstances justify this. As discussed above: plus minus 1 s.d. does not give you a confidence interval for the mean, it gives you a measure of dispersion. --------- My proposal: make the default: plus-minus values unknown, only significant digits known. The interpretation of significant digits is not machine-available unless qualifiers say so. It can however be used to result in an estimate of significant digits after conversion. Make the interval-points an option. If explicitly entered: excellent information. If not: don't try to create (false) knowledge from void. Gregor

Yes, I would like too. What I argue is that the problem is that you simply in 99.99999 % (not a researched of number of course) of cases simply don't know more than that there is a given number of digits base 10. Whether that is meaningful or just sloppy or even a wilfull simplification (probably the vast majority of quantities in current Wikipedia belong to the latter category) is unknown. that will usually be the case. Unless you know which kind of margin the numbers reflect, you cannot use it for answering anyways. What do you do with the two examples: 100 +/- 50 and 100 +/- 0.1 that are the results of the same dataset and precisely reflect the same quantity? If you know that the first is a 95% measure of dispersion, and the second a 95% CI for the mean, you can ask people whether they look for the mean (best estimate) or for a single observation. The default has to reflect reality. If you make it a complication to enter the actual default situation, and automatically add a margin of error/dispersion/tolerance whatever then people will simply allow it to happen, start ignoring it, don't understand it, and in the end Wikidata will be known as a bunch of unreliable encoded information. Which will force all re-users to understand this and to throw away these values prior to any analysis... Why so complicated? Gregor

On 20.12.2012 20:31, Friedrich Röhrs wrote: That's strange. When I enter 324m, it correctly suggests 324m+/-1 for me. -- daniel -- Daniel Kinzler, Softwarearchitekt Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.

Hi all, wow! Thanks for all the input. I read it all through, and am trying to digest it currently into a new draft of the data model for the discussed data values. I will try to adress some questions here. Please be kind if I refer the wrong person at one place or the other. Whenever I refer to the "current model", I mean the version as it was during this discussion < http://meta.wikimedia.org/w/index.php?title=Wikidata/Development/Representi… The term "updated model" refers to the new one, which is not published yet. I hope I can do that soon. == General comments == I want to remind everyone of the Wikidata requirements: < http://meta.wikimedia.org/wiki/Wikidata/Notes/Requirements Here especially: * The expressiveness of Wikidata will be limited. There will always be examples of knowledge that Wikidata will not be able to convey. We hope that this expressiveness can increase over time. * The first goal of Wikidata is to serve actual use cases in Wikipedia, not to enable some form of hypothetical perfection in knowledge representation. * Wikidata has to balance ease of use and expressiveness of statements. The user interface should not get complicated to merely cover a few exceptional edge cases. * What is an exceptional case, and what is not, will be defined by how often they appear in Wikipedia. Instead of anecdotal evidence or hypothetical examples we will analyse Wikipedia and see how frequent specific cases are. In general this means that we cannot express everything that is expressible. A statement should not be intended to reflect the source as close as possible, but rather to be *supported* by the source. I.e. if the source says "He died during the early days of 1876" this would also support a statement like "died in - 19th century". It does not have to be more exact than that. Martynas, there is no mention here of XSD etc. because it is not relevant on this level of discussion. For exporting the data we will obviously use XSD datatypes. This is so obvious that I didn't think it needed to be explicitly stated. Tom, thanks for the links to EDTF and the Freebase work, this was certainly very enlightening. Friedrich, the term "query answering" simply means the ability to answer queries against the database in Phase 3, e.g. the list of cities located in Ghana with a population over 25,000 ordered by population. A query system that deals well with intervals -- I would need a pointer for that. For now I was always assuming to use a single value internally to answer such queries. If the values is 90+-20 then the query >100? would not contain that result. Sucks, but I don't know of any better system. We do not anywhere rely on floats (besides in internal representations), but always use decimals. Floats have some inherent problems in representing some numbers that could be interesting for us. == Time == Marco suggested to N/A some values of dates. This is partially the idea of the "precision" attribute in the current data. Anything below the precision would be N/A. It would not be possible to N/A the year when the month or day is known though, as Friedrich suggested. Friedrich also suggested to use a value like April-July 1567 for uncertain time instead of the current precision model. I prefer his suggestion to the current one and will include that in the updated model. The accuracy though has to be in the unit given by the precision, we cannot just take seconds, since there is no well-defined number of seconds in a month or a year, or, almost anything, actually. Note though that the intervals that Sven mentioned -- useful for e.g. reigns or office periods -- are different beasts and should have uncertainty entries both for the start and end date. We have intervals in the data model, and plan to implement them later -- it is just that they are not such a high priority (dates appear 2.5 Million times in infoboxes, intervals only 80,000 times). I am completely unsure what to do with a value like "about 1850" if not to interpret it at as something like 1850 +- 50, but Sven seems to dislike that. == Location == After the discussion, I decided to drop altitude / elevation from the Geolocation. It can still be expressed through a property, and have all the flexibility of a normal property (including qualifiers etc.) In a Geolocation, neither the lat nor the long is optional (sorry Nikola). The Geolocation as a whole can be optional, though (i.e. unknown), but not only one of them. For the geolocations uncertainty I would like to use the same uncertainty model as for Quantity values and now for time. I know that "meters" have been suggested instead of degrees, but that would be kind of ugh considering that the biggest reason why we need the uncertainty is for converting units, in this case from decimals to degree-minute-seconds. == Quantity values == Sorry to disagree with Daniel here, but we will definitively store a quantity value in the unit that the editor used for input. We will then internally normalize it for indexing etc., but the editor won't be bothered with that as long as they do not ask for a conversion. Storing it with the original unit is important for a number of reasons, most of which Gregor already alluded to. I very much like Gregor's suggestion: rename the lower uncertainty and upper uncertainty to something with less semantic baggage. What about upper and lower bound? Or just upper and lower? And then leave the interpretation to others. Gregor, an infinitively precise number (the number of apostles, e.g.) would be handled trivially by +- 0. Also I am taking the hint from Avenue and others and drop confidence. I don't think it is useful to have it so deeply embedded in the data model, and should properly be handled through qualifiers. Regarding the height of the Eiffel tower: 324 m +- 1m is exactly what I would like to see here if the source states 324 meter. I know the source doesn't say +-1m, but this is certainly supported by the source. Think about why we need this +-1m: it is simply so we can give a useful transformation into feet. Otherwise we cannot convert units. The +-1m would not be displayed usually. == Units == I sense consensus that we should allow declaration of units in the wiki, and not to have it hardcoded in the software. Having discussed the various options and in light of the discussion here, the current suggestion would be to create a page for every quantity unit including the appropriate factors (for linear translations). This is similar to the way Freebase does it, as sent around by Tom, and what John McClure suggested. Then on a given property, the property points to a quantity unit and furthermore lists the "usual units" for the given property (pointing to the given items), which is used for display. Internally, for indexing, sorting, and query answering, we would always transform the input to the quantity unit so they are comparable. But this is usually neither exposed nor a useful number (e.g. it might have too many significant digits etc.) This would allow to use historic units like Li or historic miles even though we do not know how to translate them to other units (but not by the same property). This would also allow for other units, like Avenue has pointed out. Those are important. Nikola, we will not have special handling for money for now. This would require a whole different spec I am afraid. Currency happen 200,000 times in Wikipedia -- it is often, but not so often to be high priority. I hope that I managed to digest the whole discussion and bring it together. Cheers, Denny

On 21 December 2012 18:14, Denny Vrandečić &lt;denny.vrandecic(a)wikimedia.de&gt; wrote: I did like these but it has a hard semantics as well (which I were not acquainted with, so I did like it until reading: http://en.wikipedia.org/wiki/Upper_and_lower_bounds ) I like plus and minus better then. For the limited application that this value will have (i.e. not expressing statistics, min/max, ranges, etc.) it should not be a full value, but an addition/subtraction value. Making it a full value will throw the quantity type into the muddy waters of being abused as a (so far missing) "range-interval with optional central value" type. I think this should be avoided. By your own priorities, I am somewhat confused why you want to cover this case at all. Can you show the Wikipedia occurrence statistics for values for which a margin of error is given? I can remember plenty of cases where an interval type is needed, but I cannot recall more than 1 or 2 times I have encountered a plus/minus number. I propose to drop it altogether, and use the xsd based total digits + fractional digits attributes instead, as per the proposals int he discussion. I understand that, my problem is the default behaviour. By default (unless a wikidata editor reads this thread first :-P) you make it "11 to 13" If Wikidata converts all data entered into something that is supported by the data, but much less informative, the value will be drastically diminished. Your argument would hold for "I know the source doesn't say +-100 m, but this is certainly supported by the source." as well. The margin of error is as valuable information as the value itself. I see no reason nor justification to fabricate it. Yes, you can add an additional property "fabricated margin of error" (a somewhat derogative, but I think fair rendering of "autouncertainty"). Why does Wikidata want to introduce this level of complexity? It seems you intend to use it for dealing with conversions and return converted values with meaningful significant digits, is that correct? According to my analysis, the error introduced in applying the digits after conversion is at most 0.5 in the original unit. This is less than the error introduced in the unsupported assumption made when the software invents somewhat arbitrary margins of error (like the +/- 1 m for Eiffel tower) and identical to the error introduced by the more conventional bracketing of 0/- 0.5: 1.234 mm -> 0.001234 m = significant digits are lossless. 1.234 m -> 4.04856 ft -> round to significant digits: -> 4.049 ft 1.234 ft -> 0.376123 m -> round to significant digits: -> 0.3761 m Gregor

Hi, "Denny Vrandečić" &lt;denny.vrandecic(a)wikimedia.de&gt; schrieb: So your test UI, had it been a proposal or just anything to have to type in? As a user I'd like to enter values as a whole with its unit and tolerance, absolute as well as relative and in range notation. So everything with just one <input type="text" />. With this it is easier to C+P values from sources. I cannot understand this decision, but yes, it can be expressed outside. Why not using Javascript, Lua or anything else? So one could also provide non-linear transcalculation like decibel for example. Here the only thing that could help, would be a bot updating the exchange rates. BTW, this is also kind of statistic data. So one could query those on WM as well. But sometimes transcalculations are only wanted once, so such values would also require a timestamp. Cheers Marco

Not quite on topic but on the subject of uncertainty around dates I've worked with a couple of data sets where birth and death dates were unknown but activity periods [1] were known. These have either had a separate flag called is_flourished (or similar) used to modify born / died or separate flourished dates from birth / death Some flourished dates here: http://en.wikipedia.org/wiki/List_of_British_architects http://en.wikipedia.org/wiki/Template:Clan_Maclean_Chiefs [1] http://en.wikipedia.org/wiki/Floruit ________________________________ From: wikidata-l-bounces(a)lists.wikimedia.org [wikidata-l-bounces(a)lists.wikimedia.org] on behalf of Avenue [avenue42(a)gmail.com] Sent: 20 December 2012 19:59 To: Discussion list for the Wikidata project. Subject: Re: [Wikidata-l] Data values Thanks, the prototype helps make some this more concrete. I am increasingly wondering if "uncertainty" will be overloaded here. People seem to want to use it for various types of measurement uncertainty (e.g. the standard error), ranges with no defined central value, and distributional summaries (e.g. max and min), as well as for the precision with which a value is entered (as in the "auto-certainty" value in the prototype). These are all quite different beasts, and conflating them will probably lead to problems - particularly for precision versus the rest. Which do we choose, if both apply? How will we know which is meant? Maybe marking "auto-certainty" values somehow would mitigate the latter problem, at least. Avenue On Thu, Dec 20, 2012 at 4:10 PM, Denny Vrandečić <denny.vrandecic@wikimedia.de<mailto:denny.vrandecic@wikimedia.de>> wrote: I am still trying to catch up with the whole discussion and to distill the results, both here and on the wiki. In the meanwhile, I have tried to create a prototype of how a complex model can still be entered in a simple fashion. A simple demo can be found here: <http://simia.net/valueparser/> The prototype is not i18n. The user has to enter only the value, in a hopefully intuitive way (try it out), and the full interpretation is displayed here (that, alas, is not intuitive, admittedly). Cheers, Denny 2012/12/20 <jmcclure@hypergrove.com<mailto:jmcclure@hypergrove.com>> (Proposal 3, modified) * value (xsd:double or xsd:decimal) * unit (a wikidata item) * totalDigits (xsd:smallint) * fractionDigits (xsd:smallint) * originalUnit (a wikidata item) * originalUnitPrefix (a wikidata item) JMc: I rearranged the list a bit and suggested simpler naming JMc: Is not originalUnitPrefix directly derived from originalUnit? JMc: May be more efficient to store not reconstruct the original value. May even be better to store the original value somewhere else entirely, earlier in the process, eg within the context that you indicate would be worthwhile to capture, because I wouldnt expect alot of retrievals, but you anticipate usage patterns certainly better than I. How about just: Datatype: .number (Proposal 4) ----------------------------------------- :value (xsd:double or xsd:decimal) :unit (a wikidata item) :totalDigits (xsd:smallint) :fractionDigits (xsd:smallint) :original (a wikidata item that is a number object) On 20.12.2012 03<tel:20.12.2012%2003>:08, Gregor Hagedorn wrote: On 20 December 2012 02:20, <jmcclure@hypergrove.com<mailto:jmcclure@hypergrove.com>> wrote: For me the question is how to name the precision information. Do not the XSD facets "totalDigits" and "fractionDigits" work well enough? I mean Yes, that would be one way of modeling it. And I agree with you that, although the xsd attributes originally are devised for datatypes, there is nothing wrong with re-using it for quantities and measurements. So one way of expressing a measurement with significant digits is: (Proposal 1) * normalizedValue * totalDigits * fractionDigits * originalUnit * normalizedUnit To recover the original information (e.g. that the original value was in feet with a given number of significant digits) the software must convert normalizedUnit to originalUnit, scale to totalDigits with fractionDigits, calculate the remaining powers of ten, and use some information that must be stored together with each unit whether this then should be expressed using an SI unit prefix (the Exa, Tera, Giga, Mega, kilo, hekto, deka, centi, etc.). Some units use them, others not, and some units use only some. Hektoliter is common, hektometer would be very odd. This is slightly complicated by the fact that for some units prefix usage in lay topics differs from scientific use. If all numbers were expressed ONLY as total digits with fraction digits and unit-prefix, i.e. no power-of-ten exponential, the above would be sufficiently complete. However, without additional information it does not allow to recover the entry: 100,230 * 10^3 tons (value 1.0023e8, 6 total, 3 fractional digits, original unit tons, normalized unit gram) I had therefore made (on the wiki) the proposal to express it as: (Proposal 2) * normalizedValue * significantDigits (= and I am happy with totalDigits instead) * originalUnit * originalUnitPrefix * normalizedUnit However I see now that the analysis was wrong, indeed it needs fractionDigits in addition to totalDigits, else a similar problem may occur, i.e. the distribution of the total order of magnitude of the number between non-fractional digits, fractional digits, powers of 10 and powers-of-10-expressed through SI units is still not unambigous. So the minimal representation seems to be: (Proposal 3) * normalizedValue (xsd:double or xsd:decimal) * totalDigits (xsd:smallint) * fractionDigits (xsd:smallint) * originalUnit (a wikidata item) * originalUnitPrefix (a wikidata item) * normalizedUnit (a wikidata item) Adding the originalUnitPrefix has the advantage that it gathers knowledge from users and data creators or resources about which unit prefix is appropriate in a given context. I see the current wikidata plan to solve this problem by heuristics very critical, I do not see the data set that sufficiently tests the heuristics yet. Gathering information from data entered and creating a formatting heuristics modules over the coming years (instead of weeks) will be valuable for reformatting. The Proposal 3 allows to gather this information. Gregor Note 1: The question of other means to express accuracy or precision, e.g. by error margins, statistical measures of spread such as variance, confidence intervals, percentiles, min/max etc. is not yet covered. Given the present discussion, this should probably be separately agreed upon. Note 2: Wikipedia Infoboxes may desire to override it, this is for data entering, review, curation, and a default display where no other is defined _______________________________________________ Wikidata-l mailing list Wikidata-l@lists.wikimedia.org<mailto:Wikidata-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wikidata-l _______________________________________________ Wikidata-l mailing list Wikidata-l@lists.wikimedia.org<mailto:Wikidata-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wikidata-l -- Project director Wikidata Wikimedia Deutschland e.V. | Obentrautstr. 72 | 10963 Berlin Tel. +49-30-219 158 26-0<tel:%2B49-30-219%20158%2026-0> | http://wikimedia.de Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e.V. Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter der Nummer 23855 B. Als gemeinnützig anerkannt durch das Finanzamt für Körperschaften I Berlin, Steuernummer 27/681/51985<tel:27%2F681%2F51985>. _______________________________________________ Wikidata-l mailing list Wikidata-l@lists.wikimedia.org<mailto:Wikidata-l@lists.wikimedia.org> https://lists.wikimedia.org/mailman/listinfo/wikidata-l ---------------------------- http://www.bbc.co.uk This e-mail (and any attachments) is confidential and may contain personal views which are not the views of the BBC unless specifically stated. If you have received it in error, please delete it from your system. Do not use, copy or disclose the information in any way nor act in reliance on it and notify the sender immediately. Please note that the BBC monitors e-mails sent or received. Further communication will signify your consent to this. ---------------------

On 20.12.2012 20:59, Avenue wrote: The idea is to allow for the detailed information as qualifiers, while using the "conflated" uncertainty for query answering. Ideally, for queries, we need a single value. The current solution would be use a single value plus a range of uncertainty for answering queries, so "Primates taller than 1m" will include a species said to be 90+/-20cm or something. -- daniel -- Daniel Kinzler, Softwarearchitekt Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.

We can happily drop "Normalized". I like it better with, but that is simply a matter of taste. No, for kilo the original Unit would be gram, original Unit Prefix "k" kilo. But yes, if we rather want to give each combination of SI prefix and unit a wikidata page, we could do that. Happily. I just try to minimize the number of attributes. Slight correction: Possibly yes. As said above, this implies that all combinations of units and SI prefixes get their own Wikidata item. This is possible to create and could even reflect some of the knowledge about applicable prefixes. Only drawback is that for the normalizedMeasurement fractionDigits is redundant, it will always be totalDigits minus 1, since normalized. But that is a small price for a nicer model. Gregor

I suspect what Martynas is driving at is that XMLS defines **FACETS** for its datatypes - accepting those as a baseline, and then extending them to your requirements, is a reasonable, community-oriented procss. However, wrapping oneself in the flag of "open development" is to me unresponsive to a simple plea to stand on the shoulders of giants gone before, to act in a responsible manner cognizant of the interests of the broader community. And personally I have to say I don't like the word "clinging" -- clearly a red flag meant to inflame if not insult. This is no place for that! On 19.12.2012 09:47, Sven Manguard wrote: Wikidata and Wikidata's needs. If people want to reuse our content, and the choices we've made make existing tools unworkable, they can build new tools themselves. We should not be clinging to "what's been done already" if it gets in the way of "what will make Wikidata better". Everything that we make and do is open, including the software we're going to operate the database on. Every WMF project has done things differently from the standards of the time, and people have developed tools to use our content before. Wikidata will be no different in that regard. Jusevičius &lt;martynas(a)graphity.org&gt; wrote: sidestepping the main issue here -- every sensible architecture should build on as much previous standards as possible, and build own custom solution only if a *very* compelling reason is found to do so instead of finding a compromise between the requirements and the standard. Wikidata seems to be constantly doing the opposite -- building a custom solution with whatever reason, or even without it. This drives the compatibility and reuse towards zero. originally discussed datatypes for values such as numbers, their intervals -- semantics for all of those are defined in Schema Datatypes: http://www.w3.org/TR/xmlschema-2/ [1] and RDF tools are compatible with XSD, however I don't even a single mention of it in this thread? What makes special that its datatypes cannot build on XSD? And this of the issues, I've pointed out others earlier. >> >> Martynas graphity.org [2] Vrandečić >> &lt;denny.vrandecic(a)wikimedia.de&gt; wrote: >> > Martynas, > >> > could you please let me know where RDF or any of the W3C standards covers conversion. I would be very much Cheers, Jusevičius &lt;martynas(a)graphity.org&gt; >> >> >> Hey wikidatians, >> >> occasionally checking threads in this list like the current one, I get and also WikiQL >> >> instead of SPARQL? >> >> >> On the other hand, it feels reassuring as I was right to predict this: >> http://www.mail-archive.com/wikidata-l@lists.wikimedia.org/msg00056.html [3] >> http://www.mail-archive.com/wikidata-l@lists.wikimedia.org/msg00750.html [4] >> >> >> Best, >> >> >> Martynas >> >> graphity.org [2] >> >> On Wed, Dec 19, 2012 at 4:11 PM, Daniel Kinzler >> &lt;daniel.kinzler(a)wikimedia.de&gt; wrote: Friedrich Röhrs wrote: >> >> >> Hi, >> >> >> >> >> Sorry for my ignorance, if this is common knowledge: What is the use for objects? inhabitants requires the values for the property "population" (or even all >> >> > properties >> > with countable values, depending on implementation an query planning), those with a greater value. To speed sorted by this value would be needed. could be entries by the manufacturer, by some magazine, etc. I don't see how this could be adequatly >> >> represented/sorted by a database only query. cannot be done adequatly on the database level, then it cannot be done task is to come up this. >> >> > (One way to allow "scripted" queries like this to run efficiently is to way, using a map/reduce framework. But that's >> >> > also not trivial, and would require a whole new server infrastructure). >> > >> >> >> If however this is necessary, i still don't understand why it must necessary it could be done over a >> >> >> serialized >> >> representation of the value. >> >> > "Serialized" can mean a lot of things, but an index on some data blob is for exact matches, it can not be used for greater/lesser queries. >> > We need >> >> > to map our values to scalar data types the database can understand >> >> > directly, and >> >> > use for indexing. >> > >> >> >> This needs to be done anyway, since the values are >> >> saved at a specific unit (which is just a wikidata item). To compare >> >> them on a >> >> >> database level they must all be saved at the same unit, or some sort of them (or am i missing something >> >> >> again?). >> >> > If they measure the same dimension, they should be saved using the same >> > unit >> >> > (probably the SI base unit for that dimension). Saving values using run efficient queries against these defying one of the major reasons for Wikidata's existance. I don't see a >> >> > way around this. >> >> > -- daniel >> > >> >> > -- >> >> > Daniel Kinzler, Softwarearchitekt >> >> > Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V. >> > >> >> > _______________________________________________ Wikidata-l mailing list >> >> > Wikidata-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-l [5] >> >> _______________________________________________ list >> >> Wikidata-l(a)lists.wikimedia.org >> https://lists.wikimedia.org/mailman/listinfo/wikidata-l [5] > >> > >> > -- >> > Project director Wikidata >> > Wikimedia Deutschland e.V. | Obentrautstr. 72 | 10963 Berlin http://wikimedia.de [6] zur Förderung Freien Wissens e.V. des Amtsgerichts Berlin-Charlottenburg unter Als gemeinnützig anerkannt durch das Finanzamt für Berlin, Steuernummer 27/681/51985. _______________________________________________ list https://lists.wikimedia.org/mailman/listinfo/wikidata-l [5] >> > >> _______________________________________________ list >> Wikidata-l(a)lists.wikimedia.org https://lists.wikimedia.org/mailman/listinfo/wikidata-l [5] _______________________________________________ list https://lists.wikimedia.org/mailman/listinfo/wikidata-l [5] Links: ------ [1] http://www.w3.org/TR/xmlschema-2/ [2] http://graphity.org [3] http://www.mail-archive.com/wikidata-l@lists.wikimedia.org/msg00056.html [4] http://www.mail-archive.com/wikidata-l@lists.wikimedia.org/msg00750.html [5] https://lists.wikimedia.org/mailman/listinfo/wikidata-l [6] http://wikimedia.de

Here's a suggestion. Property names for numeric information seem to be on the table -- these should be viewed systematically not haphazardly. If all text properties had a "dotted" lower-case name, life would be simpler in SMW land all around and maybe Wikidata land too. All page names have an initial capital as a consequence of requiring all text properties to be named with an initial "period" followed by a lower-case letter. The SMW tool mandates the properties from which all derive: .text, .string and .number are basic (along with others like .page). Then, strings have language-based subproperties and number expression subproperties, and numbers have XSD datatype subpropertiess, which in turn have SI unit type subproperties, and so on. Here's a "Consolidated Listing of ISO 639, ISO 4217, SI Measurement Symbols, and World Time Zones [1]" [1] to illustrate that it is possible to create a unified string- & numeric-type property name dictionary across a wide swath of the standards world. The document lists a few overlapping symbols then re-assigned to another symbol. Adopting a "dotted name" text-property naming convention, can segue to easier user interfaces too for query forms at least plus impacts exploited by an SMW query engine. What is meant by these expressions seems pretty natural to most people: Property: Height - the value is a wiki pagename or objectname for a "height" numeric object Property: .text - (on Height) the value is text markup associated with the Height object Property: .string - (on Height) the value is text non-markup data for the Height object Property: .ft - (on Height) the value is number of feet associated with the Height object Property: Height.text - the value is text markup associated with an anonymous Height object Property: Height.string - the value is a "string" property of an anonymous Height object Property: Height.ft - the value is a "feet" property of an anonymous Height object [1] http://www.hypergrove.com/Publications/Symbols.html Links: ------ [1] http://www.hypergrove.com/Publications/Symbols.html

I think that Tom Morris tragically misunderstood my point, although that was likely helped by the fact that, as I've insinuated already, the many standards and acronyms being thrown about are largely lost on me. My point is not "We can just throw everything out because we're big and awesome and have name brand power". My point was "We're going to reach a point where some of the existing standards and tools just don't work because when they were built things like Wikidata weren't envisioned. We need to have the mindset that developing new pieces that work for us is better than trying to force a square peg into a round hole just because something is already widely used. If what exists doesn't work, we're going to do more harm than good if we have to start cutting corners or cutting features to try and get it to work. We have an infrestructure that would allow third parties to come along later and build tools that allow there to be a bridge between whatever we create and whatever exists already". Sven On Wed, Dec 19, 2012 at 2:40 PM, Tom Morris &lt;tfmorris(a)gmail.com&gt; wrote:

On 2012-12-19 15:11, Daniel Kinzler wrote: To be added by multiple simultaneous sorting operations. Software developers are not allowed to just think of the status quo they also have to think of use case the solution might gonna be used. There is e.g. the idea of pushing the monuments lists into wikidata. Only in Austria there are 36.000-37.000 of those. Germany is much bigger but has a similar history with probably an equal number per square kilometers. Sorting this by distance to a specific place needs to be done by the database. Everything else will be too ineffective. +1 IMHO this should be part of a model. E.g. Altitudes are usually measured in metres or feet, never in km or yards. Distances have the same SI base unit but are measured also measured in km, depending of the use case. Maybe we should make a difference between internal usage and visualization. Comparing meters with kilometers and feet is quite difficult, transcaling everything on visualization not. Cheers Marco

Yes, but the first is an altitude, the second more precisely termed an elevation. Good to have examples for both... Altitude may be above Mean Sea Level (MSL) or above local ground level (Above Ground Level, or AGL) or even further definitions exist. Elevation is always Ground Level above Mean Sea Level. I believe Wikidata should only deal with the latter. Alternatively, if sticking with Altitude a required qualifier would be need to distinguish which kind of altitude is meant. Else a geolocation of a radiotower: with altitude = 400 m could mean that the ground is 400 m above MSL or that the top is 400 m about GL. Gregor

On 19 December 2012 15:11, Daniel Kinzler &lt;daniel.kinzler(a)wikimedia.de&gt; wrote: Daniel confirms (in separate mail) that Wikidata indeed intends to convert any derived SI units to a common formula of base units. Example: a quantity like "1013 hektopascal", the common unit for meterological barometric pressure (this used to be millibar), would be stored and re-displayed as 1.013 10^5 kg⋅m−1⋅s−2 I see several problems with this approach: 1. Many base units are little known. "kg⋅m2⋅s−3⋅A−2" for Ohm... It breaks communication with humans curating data on wikidata. It will make it very difficult to compare data entered into wikidata for correctness, because the data displayed after saving will have little relation with the data entered. This makes Wikidata inherently unsuitable for an effort like Wikipedia with many authors and the reliance on fact checking. 2. Even for standard base units, there is often a 1:n relation. e,g, both gray and sievert have the same base unit. The base unit for lumen is candela (because the steradians is not a unit, but part of the derived unit applicability definition) Gregor

On 2012-12-19 16:56, Daniel Kinzler wrote: Actually we have 3 different use cases of values: 1. Internally in the data base 2. On wikidata.org 3. On other projects like WP and also WM external projects SI shall be used internally (1.) On 2. the user can decide what he wants. On 3. either some standard setting of the MW-project says, what is desired or the article's author. Via API (also 3.) you should be able to choose: * precision * displaying of accuracy * unit -- Marco

On 19.12.2012 08:53, Gregor Hagedorn wrote: Yes, absolutely. I think neither is sufficient: we need a representation that allows for arbitrary (or at least very great) precision, and can still be indexed and compared natively by (different!) database systems. Fixed length strings can easily do that, if they are long enough. That's pretty inefficient, though. IEEE floats work natively, but don't guarantee enough precision (well, maybe 128 bit floats come close?). The SQL, "decimal" might be sufficient: in MySQL, it allows 30 decimal digits before the decimal point, and up to 64 after. But that's still not enough to measure the extent of the universe in Plancks. I see no point in storing the unit used for input. That's how Denny proposed to calculate the default accuracy. If the accuracy is given by a complex model (e.g. a gamma distribution), then it might be handy to have a simple value that tells us the significant digits. Hm... perhaps it's best to always express accuracy as "+/-n", and allow for more detailed information (standard deviation, whatever) as *additional* information about the accuracy (could be modelled as a qualifier internally). No, it would return using whatever system of measurement the user has selected in their preferences. We'll need heuristics to pick the correct secondary unit (e.g. nm or km). The general rule could be to pick a unit so that the actual value is between 1 and 10, with some additional rules for dealing with cultural specialities (decimeter is rarely used, hectoliter however is pretty common. The decagram is commonly used in Austria only, etc). Note that for rendering of values in infoboxes, the desired unit and precision can always be given explicitly. Note "precision" vs "accuracy" here: the precision controls how many digits are shown, while the accuracy indicates how exact our knowledge is. The Precision can be derived from the accuracy and vice versa, using appropriate heuristics. But they are not the same. IMHO, the accuracy should always be stored with the value, the precision never. -- daniel -- Daniel Kinzler, Softwarearchitekt Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.

On 19.12.2012 15:12, Gregor Hagedorn wrote: Yes. Otherwise, they would not be comparable in a database query. It's generally a good idea to normalize all values of a given dimension to use the same unit, for the samerason that it's best to convert dates in a database th UTC, instead of storing time zone info for each entry. This would be done based on the accuracy. If the accuracy is ~1000m, the heuristic would decide that km is probably a good unit, with no digits after the decimal point. Besides that, the desired unit and precision can be specified when rendering values on a wiki page. Yes, that *is* the accuracy value i mean. Ok, there's some terminology confusion here. I'm using "accuracy" to refer to the accuracy of measurement (e.g. standard deviation), and "precision" to refer to the precision of presentation (e.g. significant digits). We need these two things at least, and words for them. I don't care much which words we use. Yes, all this should be handled by the component responsible for parsing user input for quantity values. True, though I think that we can lump them together for the sake of comparison, picking appropriate units, etc. The finer points can be handled by qualifiers (of the value, or even of the accuracy). I have lost the information which unit was used when originally entering the data. That's it. I made up the term, yes :) Whatever you call "km", if "m" is the real unit and "k" is the prefix. It can be stored as an auxilliary data point, that is, as a qualifier ("measured in feet"). It should not IMHO be part of the data value as such, because that would make it extremely hard to use the values in a database. Sorry, I didn't mean to say that you can calculate one from the other, but that you can make a decent guess that is useful for parsing and display. e.g. if a value is given as 5km, we can assume that it wasn't measured with millimeter precision. And if something is declared to have millimeter accuracy, it probably makes sense to use an precision of 3 or 4 digits after the decimal point when converting to feet. This is just heuristics, and yes, we shouldn't try to be too smart here. Just smart enough to be useful. Ok, so my use of precision is not precise :) We are stumbling over words here. What would you call the level of detail used for presentation, if not "precision"? -- daniel -- Daniel Kinzler, Softwarearchitekt Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.

I would call them significant digits, Wikipedia seems to prefer the lemma http://en.wikipedia.org/wiki/Significant_figures -- both will do and have the same concept. But accuracy and precision are interpretations of this, and there are more flexible ways to express these other than through significant digits/figures like an upper or lower value. But the two things are not the same. I think it is not an overcomplication to provide both for significant digits and for precise upper and lower interval. Gregor

On 19.12.2012 16:41, Marco Fleckinger wrote: I'd have thought that we'd have one such table per dimension (such as "length" or "weight"). It may make sense to override that on a per-property basis, so 2300m elevation isn't shown as 2.3km. Or that can be done in the template that renders the value. What is saved would depend on unit conversion, the value actually stored in the database would be in a base unit. In addition, the input'S precision would be usewd to derive the value'S accuracy: entering 4.10m will make the accuracy default to 10cm (+/- 5cm). I'd suggest to store such additional information in a Qualifier instead of the Data Value itself. Indeed we do: https://wikidata.org/wiki/Wikidata:Glossary I use "precision" exactly like that: significant digits when rendering output or parsing intput. It can be used to *guess* at the values accuracy, but is not the same. -- daniel -- Daniel Kinzler, Softwarearchitekt Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.

On 19.12.2012 18:00, Gregor Hagedorn wrote: I was trying to avoid storing the original input value and unit, but perhaps that was a bad idea. Wikidata will have to store such values twice anyway: once as structured data in the primary data record, and once as an index value. The index value has to be normalized and "dumb" - we can't put structure there, it's going to be a single value, possibly augmented by some kind of accuracy indicator. The index value needs to have a form that can be easily sorted, compared, indexed and queried by a wide variety of database systems. The value we store in the primary data record however can be as complex as we need it, and can be in any unit we like. The rendering of a value will be based on the primary data record, to the conversion and rendering logic has access to all the additional information it may want to use. -- daniel -- Daniel Kinzler, Softwarearchitekt Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.

On 19.12.2012 18:13, Gregor Hagedorn wrote: Oh, I didn't mean to imply it's there. But we do have a glossary, and that's where any definition should go, once we have them. I'd prefer a concept that is independent of representation, the number of significant digits depends on the unit and also a base-10 representation. I was using "accuracy" to describe how sure we are about a given value, and "precision" to describe how exactly we represent (render) the value. I now see that this choice of terminology may be confusing, and that there are also multiple aspects of certainty. So, please suggest terms to use for at least these two things: 1) value certainty (ideally, not using "digits", but something that is independent of unit and rendering) 2) output exactness (here, the number of digits is actually what we want to talk about) Perhaps we need additional distinctions, but I'm sure we at least need these two concepts. -- daniel -- Daniel Kinzler, Softwarearchitekt Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.

Here we want to talk about something that the true value is with a certain probability within a given interval, something like: "2.3 +/-0.2 µm" I am not too sure here myself. Different terms exist whether you talk about an inherent measurement error of a single individual with a single true value, or whether you speak of statistical measures or estimates. Marco gives yet another example: "We want to specify the "limits of (possible) variation" of a value, which would be Engineering tolerance. E.g. the value of electrical resistances, capacitors, etc. are measured in Ω ± % or F ± %. We could also either use/allow/display absolute or relative values." -- In this case, it is actually not a uncertainty of the actual sample of resistors, but a design specification, i.e. the specification that resistors must be (all or only 95%?) within _at least_ these limits. So what to do here? List the different use cases of a value plus-minus other values? * measurement-method limited precision range of single measurements (e.g.small structures in light microscope, limited by resolution capability of blue light, approx. 0.2 µm) * measurement-method limited accuracy range (or accuracy plus precision) * Confidence interval for mean (or other statistical parameters: mode, variance, etc.) of the population as estimated based on a sample * one of potentially several percentiles (incl. +- s.d.) measuring spread, but giving no information about the probability that the true mean is between these values * engineering design specifications that a given (unknown) fraction of individuals must be within these limits I believe for the moment you don't want to go into certainty in the sense that a number is an estimate of a All these different concepts have rightly so different names. There can be: * precision +/- 0.2 * accuracy +/- 0.2 * tolerance +/- 0.2 * error margin +/- 0.2 * +/- 1 or 2 s.d. +/- 0.2 * 95% confidence interval (CI) +/- 0.2 * 10 to 90% percentile +/- 0.2 * uncertainty (of what?) +/- 0.2 (ASIDE: the +/1 2 s.d. defines roughly a 95% probability that the next value from a random sample is in the interval, the 95% CI that the true value of the mean is in that interval. These are completely different things -- for the same measurements you can report validly 100 +/- 50 for the first and 100 +-0.001 for the second. That is, with probability 95% the next randomly sampled measurement will be between 50 and 150, and with probability 95% it is known that the true mean is between 99.999 and 100.001. Semantic matters, not only the "pattern" of plus-minus a value.) Because of the widely varying use cases listed above, I believe we need very neutral labels for the plus-minus values if the data type shall simple provide two "variables" in a generic sense, the true semantics of which are then provided by qualifier information. I could think of something: * lower range (lowerRange) and upper range (upperRange). * lower/upper interval value/endpoint but I don't very much like this because it would force people to abandon the plus/minus notation and calculate actual values. Better may be something like: * upwardsAbsolute * downwardsAbsolute * upwardsPercent * downwardsPercent or * plusValueAbsolute * minusValueAbsolute * plusValuePercent * minusValuePercent * as neutral terms - but I would be glad if someone comes up with other neutral terms. However, I hope we start realizing that all of us seem to look at this primarily from only one of the use cases listed above (me included, I usually have cases with variance spread or CI of mean). We should stop using terms that are specific to one but not the other of the cases. The assumption "these things are all more or less the same" is not true. A confidence interval is neither a manufacturing tolerance nor a measurement precision. And precision is not accuracy, etc. xsd:totalDigits or Wikipedia: significantDigits or significantFigures that is one way to express value exactness, albeit a course on. Marco writes: "Everywhere in the realm of software development precision is used for this. Therefore also here the suggestion of precision was not that bad." -> In software development, the term is about the precision of the numeric data type, i.e. the precision of the storage mechanism. The term precision is correctly applied here. However, we talk about the actually significant digits of a measurement, which are part of the potential information on precision and accuracy of the value. The measured value with e.g. 6 digits may be stored in a data type which has a precision of 16 digits. I think applying "precision" to significant digits is and produces a fundamental misunderstand of what precision is, see the Wikipedia topic on precision and accuracy. Gregor

On 21.12.2012 11:29, Marco Fleckinger wrote: Well, like you said, the output format includes a lot of things, like the decimal separator, the thousands separator, location of the negative sign, whether to put a zero before the decimal point, and digits after the decimal point. We still need a name for the latter :) -- daniel -- Daniel Kinzler, Softwarearchitekt Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.

The xsd:minInclusive, xsd:maxInclusive, xsd:minExclusive and xsd:maxExclusive facets are absolute expressions not relative +/- expressions, in order to accommodate fast queries. These four facets permit specification of ranges with an unspecified median and ranges with a specified mode, inclusie or exclusive of endpoints, a six-fer. For these reasons I believe the XSD approach is superior for specifying value set when compared to storing the dispersion factors themselves, eg the "3" of +/- 3.

I detect a need to characterize the range expression - most important of which is whether the range is complete, or whether it excludes (equal) tails on each end. XSD presumes a complete range is being specified, not a subset, is the issue you're raising? Could an additional facet for "percentage-tails-excluded" effectively communicate this estimate? On 21.12.2012 10:41, Gregor Hagedorn wrote: December 2012 19:36, &lt;jmcclure(a)hypergrove.com&gt; wrote: xsd:minInclusive, xsd:maxInclusive, xsd:minExclusive and xsd:maxExclusive facets are absolute expressions not relative +/- expressions, in order to accommodate fast queries. These four facets permit specification of ranges with an unspecified median and ranges with a specified mode, inclusie or exclusive of endpoints, a six-fer. For these reasons I believe the XSD approach is superior for specifying value set when compared to storing the dispersion factors themselves, eg the "3" of +/- 3. semantics of min. and the semantics of the unknown, their use is something is plus/minus 2 percentile ... I again can use system. But not without. _______________________________________________ list https://lists.wikimedia.org/mailman/listinfo/wikidata-l [1] Links: ------ [1] https://lists.wikimedia.org/mailman/listinfo/wikidata-l

On 19.12.2012 08:34, Nikola Smolenski wrote: I'd argue that the prime meridian does not have a location. And I think it would be bad to make longitude or latitude optional in a geo-coordinate data type. There is however nothing that would keep you from defining "longitude" and "latitude" as separate properties, and assigning them a 1D scalar value. But that's unrelated to the geo-coordinate data type. I don't think we can sensibly support historical units with unknown conversions, because they cannot be compared directly to SI units. So, they couldn't be used to answer queries, can't be converted for display, etc - they arn't units in any sense the software can understand. This is a solvable problem, but would add a tremendous amount of complexity. I think we should always store a values accuracy (which, if not given explicitly, can be derived from the input's unit and number of digits). I think it would be a bad idea to store the original input unit, though. In order to make values comparable (and thus usable in queries), they need to be converted to internal units, with very good precision; for output, they need to be converted to whatever the user prefers. I see no useful place for the original input units. -- daniel -- Daniel Kinzler, Softwarearchitekt Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.

On 19.12.2012 15:26, Avenue wrote: I'm sure standard coordinate systems have a convention for representing them. These all pose the same problems, correct. At the moment, I'm very unsure about how to accommodate these at all. Maybe we can have them as "custom units", which are fixed for a given property, and can not be converted. -- daniel -- Daniel Kinzler, Softwarearchitekt Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.

On 19/12/12 12:23, Daniel Kinzler wrote: Ah, but they could still be meaningfully compared to each other. And if approximate conversion is known, this could be still be used to make the conversion so that the measure is converted and its uncertainty increased. Just throwing more info here: there might also be cases where we could have multiple competing conversions. Somewhat similar to units, something that I would very much like to see is comparison of various monetary values, adjusted for inflation or exchange rate. But then you would have various estimates of inflation by various bodies and you might want to compare by either of them (or a combination of them?).

I get the feeling that I might be the only person on this thread that doesn't have a maths/sciences/computers background here. I'm going to be frank here: We need to snap out of the mindset that all of the data we're collecting is going to be easily expressible using modern scientific units and methodologies. If we try and cram everything into a small number of common units, without giving the users some method of expressing non-standard/uncommon/non-scientific values, we're going to have a massive database that is going to at best be cumbersome and at worst be useless for a great deal of information. Traditional Chinese units of measurement [1] have changed their actual value over time. A "li" in one century is not as long as it is in another century, and while there is a li to SI conversion, it's artificial and when we try to use the modern li to measure something, we get a different value for that thing than the historically documented li value states it should be. There is a balance. The more flexible the parameters, the easier it is to put data in, but the harder it is for computers to make useful connections with it. I'm not sure how to handle this, but I am sure that we can't just keep pretending that all of the data we're going to collect falls nicely into the metric system. Reality just doesn't work that way, and for Wikidata to be useful, we can't discount data that doesn't fit in the mold of modern units. Sven [1] http://en.wikipedia.org/wiki/Chinese_units_of_measurement

On 20/12/12 11:26, Daniel Kinzler wrote: I don't think old units are used a lot in Wikipedia infoboxes. But one use case is digitalization and importing of various datasets, and in some of them old units are used and will be needed.

Hi, * Time: Would it make sense to use time periods instead of partial datetimes with lower precision levels? Instead of using May 1918 as birth date it would be something like "birth date in the interval 01.05.1918 - 31.05.1918". This does not necessarly need to be reflected in the UI of course, it could still allow the "leave the field you dont know blank" way. This would allow the value to be 2nd-5th century AD (01.01.100 - 31.12.400). Going with this idea all the way, datetimes at the highest precision level could be handled as periods too, just as zero length periods.. Another question that popped is how to represent (or if it should be represented) times where for example the hour is known, but the day isn't. If it is known someone was killed at Noon, but not the specific day. Friedrich On Tue, Dec 18, 2012 at 3:29 PM, Denny Vrandečić < denny.vrandecic(a)wikimedia.de&gt; wrote:

How about this: - Values default to a non-range value - You can click a checkbox that says "range" to turn the input into a range value instead - An entry can only be represented by either a non-range or a range number, not both This relieves our issue with query answering: Query: When was XXX invented? Non-range answer: June 1988 Range answer: sometime between May 1988 and October 1989 Does that work? Sven M On Tue, Dec 18, 2012 at 10:56 AM, Denny Vrandečić < denny.vrandecic(a)wikimedia.de&gt; wrote:

On 18.12.2012 17:08, Sven Manguard wrote: Please be careful not to mix ranges as actual data values with ranges as accuracies: There could be an "in-office" property which takes a range. The beginning and end of the range may each have an accuracy (for example, for some medieval king, we may have an exact date of death (and end of office), but only a vague idea of the coronation date. This is conceptually different from a property that has a specific date/time as a value, which is only known with some given level of accuracy, which may be expressed as a range. Now, I don't think we need or want ranges as a data type at all (better have separate properties for the beginning and end). But I still think the above distinction is an important one. -- daniel -- Daniel Kinzler, Softwarearchitekt Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.

Denny, Thanks for this. Are there ways to structure this geolocation data now to anticipate more 'fluid' uses of it, say 5 or 10 years from now, or beyond, in representing water, or astronomical processes, in something like interactive, realistic models of the earth or the universe, which would also be useful to Wikipedia's developing goals / mission? Scott On Tue, Dec 18, 2012 at 8:57 AM, Gregor Hagedorn &lt;g.m.hagedorn(a)gmail.com&gt;wrote;wrote: -- Scott MacLeod Founder & President http://scottmacleod.com -- World University and School (like Wikipedia with MIT Open Course Ware) http://worlduniversityandschool.blogspot.com/ http://worlduniversity.wikia.com/wiki/World_University P.O. Box 442, (86 Ridgecrest Road), Canyon, CA 94516 415 480 4577 worlduniversityandschool(a)gmail.com Skype: scottm100 Google + main, WUaS pages: https://plus.google.com/u/0/115890623333932577910/posts https://plus.google.com/u/0/b/108179352492243955816/108179352492243955816/p… Please contribute, and invite friends to contribute, tax deductibly, via PayPal and credit card: http://scottmacleod.com/worlduniversityandschool.htm World University and School is a 501 (c) (3) tax-exempt educational organization. World University and School is sending you this because of your interest in free, online, higher education. If you don't want to receive these, please reply with 'remove' in the subject line. Thank you.

Sven, To complement your thought, and for an historical perspective influencing this discussion, UNIX has many plusses and minuses informed by the computer science of the 1950s and 1960s, and it would be great to anticipate as many 'plusses' as possible, and as sophisticatedly as possible, in the development of Wikidata et al. in the 2010s, especially in terms of values such as geolocation, - for example, with both virtual worlds and brainwave headsets (so, neural interfaces ... e.g. see Tan Le's TED Talk), but in many other ways, as I see it - which is why Denny is asking. Scott On Tue, Dec 18, 2012 at 9:24 AM, Sven Manguard &lt;svenmanguard(a)gmail.com&gt;wrote;wrote: -- Scott MacLeod Founder & President http://scottmacleod.com -- World University and School (like Wikipedia with MIT Open Course Ware) http://worlduniversityandschool.blogspot.com/ http://worlduniversity.wikia.com/wiki/World_University P.O. Box 442, (86 Ridgecrest Road), Canyon, CA 94516 415 480 4577 worlduniversityandschool(a)gmail.com Skype: scottm100 Google + main, WUaS pages: https://plus.google.com/u/0/115890623333932577910/posts https://plus.google.com/u/0/b/108179352492243955816/108179352492243955816/p… Please contribute, and invite friends to contribute, tax deductibly, via PayPal and credit card: http://scottmacleod.com/worlduniversityandschool.htm World University and School is a 501 (c) (3) tax-exempt educational organization. World University and School is sending you this because of your interest in free, online, higher education. If you don't want to receive these, please reply with 'remove' in the subject line. Thank you.

I believe it would be more profitable to build a query system which always queries for the range. This would work for interval-only values (see my comment on the wiki page) as well as for value with interval. I don't see this as a big overhead. It is more a problem for ordering, but internally, wikidata could store a "midpoint" value for intervals where no explicit central value is given, and use these for ordering purposes. I think it would be great if the system is consistent for quantities, dates, geographical longitude/latitude, etc. Gregor

Can this conceptually distinguish between a meaningful midpoint value, and one that is useful for ordering, but has no meaning and should not be displayed as a result value? See the examples on https://meta.wikimedia.org/wiki/Talk:Wikidata/Development/Representing_valu… Gregor PS: With accuracy you introduce a new concept here which was not in the representing values paper (see http://en.wikipedia.org/wiki/Accuracy_and_precision). This is different from confidence interval ("uncertainty") where it is not yet decided whether the value indicates accuracy or dispersion. Confidence interval is a measure of Accuracy only if the sample measurements are normally distributed and if no systematic bias exist. --- I believe it is important that wikidata is flexible enought so it can capture both, especially because in many cases dispersion is used as a rough estimate for otherwise unknown accuracy, and since in many cases there is no "true single value" and the dispersion is systematic (see e.g. car model length example).

my 2¢ On 18.12.2012 15:29, Denny Vrandečić wrote: We need a system for expressing accuracy anyway, for empirical values (measurements). I suggest to apply the same concepts (maybe even the same code) to dates. We'll need to be able to express accuracy in several ways, e.g. as standard deviation, +/-% range, gamma distribution, etc. Simply specifying a range as "+/- 3600 seconds" to indicate a precision level in the hour range would fit in, I think. I'd include altitude, but make it optional (while long/lat are mandatory). Expressing accuracy here can and should work similarly to "normal" (1D) measurements and dates, usually given as a +/- range. I don't like the idea of defining units on the property pages at all. For one thing, it means a lot of overhead introducing new units, or at least standardizing the names of units (should that be "ton" or "metric ton"?...). This makes it hard to compare values of different properties (say, comparing average life expectancy to maximum life span or some such). At least the base units (SI units) should be implemented in software. Derived units could be defined on the wiki, but on their own pages, not for each property. A far more tricky conversion would be between different reference globes for coordinates. But that wouldn't be handled as a "unit", I suppose. -- daniel -- Daniel Kinzler, Softwarearchitekt Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.