Latimer Core General

[magpiedin]

This discussion thread is meant as a starting spot for questions relating to Latimer Core's overall structure, purpose, usage, etc.

The Latimer Core browser and the following wiki pages can provide a jumping-off point for further exploration and discoveries:

• 1. Overview of Latimer Core
  • 1.6 Mapping using SKOS & SSSOM
• 2. Glossary

[baskaufs]

This is a technical issue that can be handled when the List of Terms is regenerated.

We have been trying to update the RFC 2119 keywords statement to the most up-to-date version on all TDWG standards pages that use it. The appropriate text can be seen on the DwC List of terms page, section 1.2 (raw text available here).

[mswoodburn]

Thanks, have created an issue for this (#61)

[baskaufs]

Another technical issue.

I think that in order to support internationalization, we have tried to use "IRI" (Internationalized Resource Identifier) instead of "URI" (Uniform Resource Identifier). In reviewing DwC and AC documents, I can see that we have not been consistent about this. But given that this is a new standard, I would recommend that we use IRI consistently here (and try to update the other TDWG docs to do the same).

For those not familiar with this issue, IRI is effectively a superset of URI (all URIs are IRIs) that allows for some use of non-Latin characters. We are not likely to use non-Latin characters in our IRIs, but to be consistent with use in other standards (e.g. W3C), we should use the currently preferred form, IRI.

[mswoodburn]

Thanks, have created an issue for this (#62)

[baskaufs]

Section 2 of the list of terms doc says:

Note: URIs for terms in most of these namespaces do not dereference to anything. The authoritative documentation can be obtained by clicking on the vocabulary names in the table.

As far as I know, all of the terms in these namespace dereference. I just did a test with dcterms:, schema:, and the TDWG namespaces and they all were dereferencing as of 2023-12-20.

I would recommend changing the wording to

Note: URIs for terms in some namespaces may not dereference. In such cases, the authoritative documentation can be obtained by clicking on the vocabulary names in the table.

[mswoodburn]

Hi Steve, I think this stemmed from an earlier time when we had another standard in the list where links didn't dereference, which has since been removed. Since everything now does dereference, might it be clearer just to remove that statement?

[baskaufs]

Form of controlled value strings in examples
I noticed that in your examples, you often have English prose phrases as your example controlled value strings. I would recommend that you consider not doing that. There are two reasons:

1. You imply to people that if they don't speak English, they should write a phrase in their own language.
2. You increase the number of possible permutations that need to be disambiguated. For example, under the contactDetailCategory examples, you have "Phone (mobile)". You are going to get things like "phone (mobile)", "Phone (Mobile)", "Phone: mobile", "Phone - mobile", "mobile phone", etc.

When the first official controlled vocabularies were ratified for Darwin Core, the somewhat arbitrary decision was made to use lowerCamelCase for phrases. Using that precedent, the controlled value would be "phoneMobile". By having such a rule-based system you eliminate most possible permutations. The other thing that this makes possible is to make clear the distinction between controlled value string and their labels. There is one controlled value string: "phoneMobile" but a label for each language: "Phone (mobile)" [en], "teléfono móvil" [es], etc. People should get the difference and not use a phrase in their own language that someone later will have to disambiguate.

I would recommend getting a grip on this from the start because if you start getting into the business of formally defining controlled vocabularies later, you will have to deal with the fact that people have already been publishing data based on your examples. They may even have written applications with those examples in pick lists. We had this problem with examples in the Audiovisual Core variant CV (http://rs.tdwg.org/ac/doc/variant/) where people had been using the earlier examples that did not conform to the lowerCamelCase convention. For the purposes of stability, we continued to use the earlier (non-lowerCamelCase) values. But now that controlled vocabulary is (currently) the only controlled vocabulary that doesn't follow the lowerCamelCase convention that we would like people to expect.

[baskaufs]

This would generally apply to any term where you say "best practice is to use a controlled vocabulary".

[baskaufs]

Hmmm. Having said this, I see that you have quite a few cases like ltc:ecoregion where you say "It is recommended to use controlled vocabularies such as those adopted by the World Wildlife Fund (WWF) / Global 200 (Olson et at. 1998)." In this case, I suppose it makes sense to use a non-lowerCamelCase phrase. However, if you do recommend that, I think you need to provide a URL telling people where to go to find those strings. Otherwise, it's not really controlled and people will put whatever they want.

[jbstatgen]

Thanks a lot, @baskaufs, for looking more closely at the controlled vocabularies and sharing your experiences. Your advice on existing best practices in TDWG is very much appreciated.

Can I use the opportunity to ask if TDWG could please set up a dedicated page/home for all things controlled vocabularies (CVs)? There is a wonderful, very informative Github thread on CV decisions and best practices by you. Though, I struggle to find it when I have questions and want to check back with what you explained.

The situation of controlled vocabularies (CVs) doesn't stop at what you are pointing out here for LtC. I would add that all CV terms should be fully documented, ie. have a definition, datatypes, RDF type, alternative and related labels (eg. abbreviations), comprehensiveness of CV term sets and moderation, etc. Furthermore, there are questions about disaggregating different dimensions of meaning and separating them out into their own sets, associated with different properties. During implementation also questions of hierarchical relationships between CVs and thus the nature of their terms arise and how to handle that. Finally, there is the wide field of how to present CVs, that is UI and UX that will underlie acceptance and use.

It's a big topic, really a standard in its own right. Your foundational work with the CVs for AC shows that.

The question here is, how to deal with it right now for the ratification and release of LtC?

1. Put the ratification of LtC on hold while we develop and update the CVs and their terms for some years? A major part of this will be to develop knowledge among stakeholders about CVs (names?, CVS?, labels?, moderated lists?), reach understanding and agreement and thus gain acceptance, confidence in handling CVs and actual usage.
2. Remove all examples and recommendations that have something to do with CV terms?
3. Accept the situation as it is, well aware of its shortcomings?

At this point, I would choose door number 3. One reason is that changing all CV terms into lowerCamelCase or snake_case will suggest a development background and stage that doesn't exist. Another reason is that based on experience so far I don't think that the question of correctly approaching CVs is primarily a technical question. Because I agree with you that the current state isn't really a satisfactory solution, I will continue to ask about TDWG's capacity development strategy and the role it wants/needs to fill in this changing world.

[baskaufs]

@jbstatgen I would definitely NOT chose option 1. LtC needs to be done and available to use soon.

As for setting up a page on all things controlled vocabularies, that's a great idea, but somebody would have to create it. At this point, the current practice can be seen pretty clearly by looking at the example of the three Darwin Core controlled vocabularies available from the "terms" dropdown menu on any of the Darwin Core web pages. The Audiovisual Core controlled vocabularies for subjectPart http://rs.tdwg.org/ac/doc/part/ and subjectOrientation http://rs.tdwg.org/ac/doc/orient/ also follow the current design pattern for controlled vocabularies with respect to the controlled values that are derived from short phrases.

I guess my feeling is that if you can't provide good guidance in the examples where you say "best practice is to use a controlled vocabulary, you should leave the examples out. It is a simple matter to add the examples later. It is not a simple example to change what people are doing once a precedent has been set. The change process detailed in the Vocabulary Maintenance Specification is not strict for examples since they are not normative. It should be possible for the LtC Maintenance Group to easily add the examples at any time once they have been thought through carefully. But it would be difficult to "put the genie back in the bottle" once inappropriate examples have been given.

Here is my opinion.

1. In each case where you suggest using some external source (e.g. the WWF names for bioimes, ecoregions, etc.), see if you can find a stable URL to which users can be directed, and state clearly what they should use (i.e. "For the controlled value, provide the English label for the ecoregion as listed at https://example.org/list."). If it is not possible to do that, then you should not provide examples, but rather should create a TDWG-generated controlled vocabulary that uses the WWF English labels acquired from some source. There is already a precedent for being cautious in this circumstance -- in the DwC proposals for some terms (environmentalMaterial, see New term - environmentalMaterial dwc#40 (comment) and biome, see New term - biome dwc#38 (comment)), it was suggested that labels from an OBO ontology be used. However, upon investigation, those labels weren't stable nor readily available. The same thing happened for a term in the Humboldt extension. If people can't find the exact strings that they should be using, then there isn't really any controlled vocabulary and we need to hold off on suggesting values until we create one.
2. In cases where you are just using your own short phrases or words, just convert them to lowerCamelCase. For better or worse this is the precedent that we have. Doing so would not prevent creating a formal controlled vocabulary later. It would probably take less than an hour to do that for all of the examples that fall into this category.
3. For terms where you recommend booleans, I would insert the text: "Best practice is to use the TDWG Boolean Controlled Vocabulary (http://rs.tdwg.org/tag/doc/boolean/)." For some reason this URL is not dereferencing, but I will fix that. It should redirect to https://tag.tdwg.org/boolean/ . The reason the TAG made this vocabulary was for exactly this situation, so it can simply be used by reference. You can keep your true and false examples, as they are consistent with the recommendations in this CV.

As I said, this is just my opinion. I would recommend that @ben-norton bring this up at the next TAG meeting. If the TAG thinks it is fine for people to just go their own way on the design pattern for controlled values, I won't stand in the way. But if they feel that maintaining consistent patterns among TDWG controlled vocabularies is important, then you should probably fix this prior to ratification.

[ben-norton]

I have a plan for controlled vocabularies in my head and scattered among notebooks. It has much broader implications that LtC. I think its alsi a competely separate issue all together. I don't think LtC should wait until the controlled vocab details are worked out.
A citation with a url to the controlled vocab is sufficient. If you can't locate a stable url / citation then I don't think it should be recommended.

[ben-norton]

JSON-Ld-ish record for a 'definedtermset'
"@type": "DefinedTermSet",
"@id": "https://data.naturalsciences.org/api/v2/vocabularies?guid=8099d18a-d9e5-4542-8430-277ad21b0cc7",
"name": "Mammal Diversity Database Data Dictionary",
"guid": "8099d18a-d9e5-4542-8430-277ad21b0cc7",
"description": "Data dictionary for the Mammal Diversity Database as provided in the metadata file by the publisher at https://zenodo.org/record/4679816",
"identifier": "mdd-data-dictionary",
"citation": "Mammal Diversity Database. (2021). Mammal Diversity Database (1.4) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.4679816",
"version": "1.8",
"publisher": null,
"relatedResource": {
"guid": "2e8a1866-3103-11ec-bd0a-80e82c0fcef6",
"title": "ASM Mammal Diversity Database",
"resourceType": "Data Product",
"relation": "https://data.naturalsciences.org/api/v2/data-products?guid=2e8a1866-3103-11ec-bd0a-80e82c0fcef6"
},
"classification": {
"resourceClass": {
"name": "Vocabulary",
"description": "Sets of defined terms for the purposes of describing, interpreting, and reusing data.",
"entryPoint": "https://data.naturalsciences.org/api/v2/vocabularies"
},
"resourceType": {
"name": "Data Dictionary",
"description": "Linked collections of terms and definitions that describe response body parameters in endpoints belonging to the NCMNS API.",
"entryPoint": "https://data.naturalsciences.org/api/v2/vocabularies/data-dictionaries"
}
}
},

[baskaufs]

Please refer to Section 4.1.2, which designates controlled value terms as skos:Concept and Section 4.5.4 of the Standards Documentation Specification which describes the required metadata fields for TDWG controlled vocabularies.

Please note that the examples, which predate any of the existing TDWG CVs do not follow the currently used design pattern of opaque IRIs and controlled value terms in lowerCamelCase.

[baskaufs]

Here's what the machine-readable metadata looks like for the term list of a TDWG CV: http://rs.tdwg.org/dwcem/values.ttl

The SKOS Concept Scheme is considered a distinct entity from the term list. An example of machine-readable metadata for the corresponding concept scheme for the above list of terms is at http://rs.tdwg.org/dwcem/values/e.ttl

The individual terms are linked to the list of terms and concept schemes like this: http://rs.tdwg.org/dwcem/values/e004.ttl

This machine-readable metadata all gets generated automatically as part of the automated build process described here: https://github.com/tdwg/rs.tdwg.org/blob/master/process/create-vocabulary.md
and here:
https://github.com/tdwg/rs.tdwg.org/blob/master/process/process-vocabulary.md

The overall model for TDWG vocabularies is describe here: https://tdwg.github.io/rs.tdwg.org/
The term lists and vocabulary terms (i.e. the controlled value terms) fit into this model.

[ben-norton]

@baskaufs Awesome Steve. Thanks.
Note regarding controlled value lists. They go by a lot of names. Enumerated Domains, Value Domains, Controlled Value Lists, Picklists, Controlled Vocabularies. Just to avoid confusion, I think we're talking about standardized . Here's an excellent resource: https://www.niso.org/publications/ansiniso-z3919-2005-r2010

I think we need to extend the controlled values schema and separate it as an independent formal process for publishing controlled value lists. Here's my expanded version of the current tdwg set. I'm going to start publishing buckets of controlled value lists associated with the geosciences domain. For now, we need to make sure to include a source URI for the value list.
Example: https://resource.geosciml.org/classifier/cgi/geologicunittype
or
MODIS Land Cover Classification
https://modis.gsfc.nasa.gov/data/dataprod/mod12.php

[baskaufs]

Thanks for the link to the NISO pub. The one issue there is that it is guidance for construction of monolingual controlled vocabularies. One of the major reasons for basing the TDWG controlled vocabularies on SKOS is that SKOS was build from the ground up to handle multilingual labels for concepts. That is really pretty non-negotiable for an international organization like TDWG.

I had some issues with the two links you gave. The first one gave me an "unable to connect" response. The second one was not very clear about where one goes to get a list of controlled values. There were 503s for the links in the "Product Name Combined Aqua & Terra Product ID" column of the table. I did look at the User Guide Download and ATBD links, but there were no obvious places to go for controlled values. We need a link directly to a table or list of strings for this kind of stuff and not links to a long textual description or pages with a whole bunch of additional links to be explored.

[ben-norton]

You can easily extend the ANSI guide to multiple languages using @lang and json-ld. I'm not sure why they included monolingual in the title except to keep the scope targeted.
You make a good point regarding the URI.
Here's a better url for the first
http://inspire.ec.europa.eu/codelist/GeologicUnitTypeValue

As for the second, I've come across a lot of controlled value lists in a very loose structure. The pdf has the definitions, but there's no URI, just a more generic source link. Is that sufficient for current purposes? I think so, but in a different field like sourceURI.
For many controlled value lists, definitions are required, but a URI needs to be constructed. There's an opportunity here, an exchange. I've offered the following: If you allow me to publish your vocabulary, I'll create the infrastructure to do so properly. Then, it's a matter of building the records. It's worked if I can get to it.
Across the landscape, practices vary. I've come across too many lists with just values. See https://www.geosamples.org/vocabularies
It's disappointing because they're completely useless without definitions and arguably lead to more problems.

[baskaufs]

Well, the best practice would seem to be that if the external organization does not provide an easily accessible list of controlled value strings with definitions, we create our own vocabulary based on theirs and include all of the necessary metadata. This has been done at least three times already. The AC subjectPart and subjectOrientation CVs borrow definitions from OBO ontologies. We link out to the source definitions semantically via sawsdlrdf:modelReference (see http://rs.tdwg.org/acpart/values/p0003.rdf for an example). The AC subtype CV links its values to a corresponding term in the Getty AAT (see http://rs.tdwg.org/acsubtype/values/DesignPlan for an example). In both of these cases, the link out is to a specific term, because they provide links to them. But the links could be to an entire page or pdf

In both of these cases, the source vocabularies don't really provide clear, single controlled values. The just provide labels, which may be unstable or in multiple languages. So on our TDWG controlled vocabulary page, we specific what that controlled value is in the "Controlled Value" field of the page.

So we already have a mechanism and precedent in place for creating controlled vocabularies in cases where the desired source does not provide enough information. All that needs to happen is to build them. I'm happy to talk about/work with the Latimer Core team to do this. But it would take longer than the amount of time you want to take for the ratification. So as I suggested before, just leave the examples out and say "best practice is to use a controlled vocabulary". Then build or refer to the appropriate CV later. We've done this before for DwC and AC, and are doing it for Humboldt Extension, so it's not a problem to defer.