MIDS levels as their own terms, what do you think?

[ManonGros]

I would like to get a sense of how widespread are MIDS level percentages when characterising collections. Would it make sense to have them defined as LtC Terms?

In the context of the GRSciColl roadmap (https://scientific-collections.gbif.org/road-map#2-support-structured-collection-descriptors), I want collections to be searchable by a number of fields. Right now in LtC, something like the percentage of specimens digitised at a particular MIDS level has to be formatted as measurmentsOrFacts. This makes sense but this isn't something that could easily be implemented for searching.

I would like a sense of how many institutions track MIDS level (and would potentially be able to provide those) and how interesting it would be for find collections based on the digitisation level of some groups of scientific names (for a given taxon for example).

I am tagging @mswoodburn since I think this is something that the MHN has. Maybe folks from the MIDs task group could give their input (https://github.com/tdwg/mids), @CatChapman?

[jbstatgen]

Hi @ManonGros ,

regarding the LtC part of your inquiry, should there be interest by the community in collection-level MIDS-summaries, it would make sense to me to create a LtC class dedicated to data quality. This class will build on ltc:MeasurementsOrFacts, though have dedicated schemes and vocabularies. Such an endeavor should be discussed with the MIDS and BDQ working groups and align with their developments. That's my intuitive 5c, happy for others to chime in.

Best, Jutta

[ben-norton]

I think MIDS and LtC would be better served as adjoining standards. We would need some sort of mechanism to facilitate that arrangement. I'll contact them and see about putting the topic on their agenda. They meet at the same time as the regular LtC meeting.

[mswoodburn]

@ManonGros the main uses of MIDS %s at collection level that I've seen/worked on so far have been DiSSCo-related collections surveys (the SYNTHESYS+ Collections Digitisation Dashboard, DiSSCo Flanders, DiSSCo UK).

So institutions contributing to those surveys (~100 in total) have calculated or estimated MIDS %s for the collections breakdowns you can see in the dashboards, but I'm not sure that's there's much further institutional use of those metrics as yet. The calculation of MIDS levels against specimen level records as e.g. part of the Digital Extended Specimen plans, plus linkage of specimen-level records to LtC object group records, are probably the keys in the longer to more widespread use of aggregated MIDS % against collections records term, and the capacity to keep the figures updated as digitisation progresses.

I can definitely see your point about the challenge of having MIDS %s as MeasurementsOrFacts with respect to search. However, it might be a valid (and practical) approach for implementations to flatten parts of the LtC model in their actual schema, particularly MeasurementOrFacts, for this reason (and also to simplify data entry forms, etc). So you might have object_group.mids_1_percentage, object_group.mids_2_percentage (and so on) fields in your database schema, but then have logic that refactors them into separate MeasurementOrFacts when you want to export or expose the data as fully LtC-compliant records. As long as your flattened schema doesn't actually break the rules in the LtC model, contains the few mandatory terms, and allows the reconstruction of the data as fully LtC-compliant, I would see that as legitimate and probably quite essential for many practical applications of the standard. Others may well have opinions though :-)

Final braindump: I can though envisage a useful model for LtC development where widespread community use of a particular type of MeasurementOrFact suggests that it should become a core LtC term (and so simpler to work with) and provides the supporting evidence for that, so I wouldn't rule it out for MIDS %s (although LtC will be locked for changes for a while now that it's been ratified). Worth bearing in mind though that the MeasurementOrFact class does come with some additional properties to help to describe and qualify the values, which might be lost if using a simple term instead.

@jbstatgen - interesting point! Could certainly see a conceptual argument for making a distinction between MeasurementOrFacts about the physical objects within the object group, and MeasurementOrFacts about the digital representations of the objects within the object group...

[jbstatgen]

@ben-norton How do you see both standards integrating in an implementation?

[jbstatgen]

@ManonGros & @mswoodburn : RE flattening
After months of struggling with finding a good balance and solution for the question of normalization vs. pragmatic implementation, this are my experiences and thoughts - would be happy to learn from your experiences and solutions.

As I understand it, one is developing/aiming for a normalized data model and its normalized implementation mainly for data quality and versatility reasons. Latimer Core provides a framework that supports such normalization and its resulting data quality and flexible use/applications. Thus, the decision of where to localize a solution in the continuum between normalization and flattening becomes one of purpose.

What is the main business objective of the infrastructure? For biodiversity data infrastructures I can see three, not necessarily exhaustive:

1. enable a social network: connect and engage biodiversity enthusiasts everywhere and enable them to communicate and share their knowledge, experiences, data and highlight their achievements. The availability of such a platform is crucial, eg. for biodiversity conservation and connecting people with nature and their environments. While data quality are important, for this purpose they are not key. As long as on average the data are fine, that's ok.
2. sell a product: here the biodiversity data platform functions predominantly as the equivalent of an online shop. The product is data and the payment is funding. The higher the amount of data in your repository and the larger the traffic and attribution that you generate, the more the infrastructure provider can show proof of their importance to funding bodies. In contrast to most online markets that rely on payments per transaction (transferring a physical product, gaining access to a software via a license key), user can't return the product and get their money back. Most biodiversity data infrastructure systems have statements that fitness for purpose is not guaranteed, the user gets the product as it is. Moreover, there is no certification body or regulating (standards) authority who are checking data and provide data quality certificates. Data cleaning and quality checks are outsourced to the user and third parties who provide data cleaning pipelines and work environments. This requires users to have expert knowledge and resources. Thus incurring hidden costs that make the data inaccessible to many potential users. Biodiversity data infrastructures are very resource-intense endeavours that all run and thrive on the immense engagement and volunteer efforts of their developers, staff and communities. Being able to show the amount of data in the repositories and databases, the traffic and downloads and received attribution via PIDs and/or publications is invaluable and indispensable for societies to understand the importance and value of these infrastructures and the data, information and knowledge they provide and mediate.
3. provide data for application and decision-making: these require reliable, high quality data. Due to the complexity and connectedness of all things biodiversity, much needed applications that contribute to eg. addressing the biodiversity crisis, OneHealth questions, FAO-related objectives, SDG goals, all require high quality data as input to advanced statistical analyses. Eg. the statistics of biodiversity monitoring are really exciting, though try to achieve reliably >95% accuracy or rather 99.9999% if livelihoods and lives are at stake, and you find yourself scrambling for information-rich, quality ascertained data and the newest mindboggling statistical approaches and algorithms. In addition, you never know what lies around the corner and is coming up next.

As soon as we are not talking about expert infrastructures used by highly trained and experienced experts, all three purposes need to be covered simultaneously. Fortunately, there is no hard, fundamental conflict between the purposes. This is not a "wicked" situation. It's more a question of the least common denominator. To cover bullet number 3, the infrastructure needs to offer at least some of its data in high quality and with quality assurance measures. Which is exactly what you are planning to do, Marie. In that case, flattening the model at the same time doesn't really seem to make sense.

Focusing on GBIF and considering what I know about your infrastructure system, I don't see the problem? Which simply shows that I'm not an expert, and should you and your colleagues be willing, I'm happy to learn. Here the (naive, simple-minded?) reasons from somebody who likely still has a much too theoretical approach to this:

Intuitive interface for user acceptance and satisfaction:
Users don't interact with the normalized version of the data, unless they really ask for it. They should see interfaces that they understand and that support their use cases. I thought that the GBIF infrastructure is based on a Postgres database system and thus, you can create views that show the data in the combinations (and thus UI/UX formats) that users need. Thus, I'm not quite sure why you would abandon the MeasurementsOrFact class, pull out certain properties and flatten them into the ObjectGroup class? If most users are intimidated by the clutter of quality assurance properties in the MoF class, then consider adding only the one key property to your view and enable users who want to know/enter more to select a button that opens/adds the rest of the fields as needed? What am I not getting?

The "flatten but keep aligned" approach proposed by you, Matt might seem like a simple solution, but seems to create even more complexity elsewhere. Mirroring a property/field by adding a copy elsewhere requires a set of defined validation rules, constraints, synchronization processes, etc. to assure data integrity and (thus) quality. Thus quality assurance is lifted from the data model level to an additional level in which rules and constraints need to be explicitly defined (instead of being in many ways implicitly implemented at the level of the normalized data structure). What I am seeing is that this level of validation rules is extensive and needs to be maintained, requires consideration whenever only a small update is made since large impacts might result and the question is how to find the source of an error once one become apparent. Wouldn't one rather avoid expanding that level even further?

Search
What I learned from GBIF, specifically after someone shared a figure showing its infrastructure system, is that the database, the user interface and the search environment are different systems that are and can be optimized independently. You are really the experts and I feel a bit silly talking to you about search options. Though, here is our current understanding/framework for RECODE (obviously, as I understand it).

Search isn't monolithic. Depending on use case, there are different approaches that are themselves optimized for different search questions and contexts. Thus, flattening a property might provide advantages in some search contexts, but disadvantages in others.

A set of complementary approaches seems promising:

1. Start the search with a term that returns the smallest result set. For this guide users via information and/or workflows start the search with the most promising keyword and/or the system detects this keyword based on available (composite) indices in the back. Thus, a user interested in information-rich aka MIDS level 2+ digital records of Silene acaulis (L.) Jacq. worldwide, might start for searching for S. acaulis entries and only then filter for records with higher MIDS values (they are not interested in MIDS quality measures associated with koala or any Animalia records).
2. Provide options to easily perform cascading searches in the form of workflows and processes that users understand: The previous user furthermore is looking for digitized collection records from Kilpisjärvi, specifically with determinations verified by a colleague with diacritic marks in their name and different name versions (latinized, birth, ennoblement).

For these kinds of searches it seems important that the entered keywords are provided exactly as recorded in the indices and that name variants, eg. for location and agent are easily searchable in one go. With names normalized this should be straightforward, flattened into different properties for birth name, latinized name and "primary" name it might be difficult to find all the fitting records (I seem to regularly have a hard time finding institutions by name in GRSciColl, both on the web frontend and the registry). Similar for MIDS-level even if it seems less of a problem: flattened out into level-specific properties it might be quite difficult to develop the correct search filter.

3. Finally, sometimes there is no narrowing down of the result set. You want to know about usage of MIDS measures and maybe even distribution of the different MIDS-levels worldwide. In that case, there is full-text search, which quite certainly comes in different flavours (vaguely waiving her hand). GBIF has a dedicated system for it, as far as I understand. Here, it doesn't really matter (does it?) if your data comes as normalized or flattened. Everything is searched. A variant that we are investigating in RECODE are full-text document indices of business objects in key:value pair format (eg. JSON). While in this search category normalization doesn't seem to provide any advantages, it doesn't seem to pose an obstacle either.

In sum, even after way too much thought and text, why would you, Marie & Matt, consider flattening the MIDS-properties? What parts of search do you consider difficult?