Merge pull request #28 from Infectious-Disease-Modeling-Hubs/misc-doc…

…-updates Misc documentation updates
hubverse-org · Jan 11, 2023 · db775f1 · db775f1
2 parents 3096bf7 + ae41d3f
commit db775f1
Show file tree

Hide file tree

Showing 10 changed files with 130 additions and 69 deletions.
diff --git a/docs/source/data.md b/docs/source/data.md
diff --git a/docs/source/format/hub-metadata.md b/docs/source/format/hub-metadata.md
@@ -1,4 +1,5 @@
-# Hub Metadata
+(hub-metadata)=
+# Hub configuration files
 
 ## Directory Structure
 The `hub-config` directory in a modeling hub is required to contain three files:
@@ -43,6 +44,7 @@ The administrative hub metadata file contains settings that are expected to rema
 * Something about scoring?
 * Something about report generation?
 
+(tasks_metadata)=
 ## Hub model task metadata (`tasks.json` file)
 The hub model task metadata file specifies the model tasks and model output formats for the hub. To reduce redundancy, hubs may optionally specify a 'defaults' entry with values that apply unless they are overridden by round-specific entries; this may be particularly useful for forecast hubs, which typically accept the same formats for all rounds.
 

diff --git a/docs/source/format/hub-structure.md b/docs/source/format/hub-structure.md
@@ -1,3 +1,4 @@
+(hub-structure)=
 # Structure of Hub repositories
 
 A Hub should be structured according to the following recommendations. 

diff --git a/docs/source/format/intro-data-formats.md b/docs/source/format/intro-data-formats.md
diff --git a/docs/source/format/model-abstracts.md b/docs/source/format/model-abstracts.md
@@ -14,4 +14,4 @@ There is one directory for each team and within that, possibly multiple files, o
 
 ## Purpose
 
-The `model-abstracts` directory is an optional feature of a hub that may be useful to keep track of round-specific updates to the metadata, e.g. describing modeling assumptions specific to one modeling round. Note: as part of a separate workflow, hubs may find it helpful to associate round-specific metadata with the corresponding model output files.
+The `model-abstracts` directory is an optional feature of a hub that may be useful to keep track of round-specific detailed narrative descriptions of modeling methods and results, e.g. describing modeling assumptions specific to one modeling round. Note: as part of a separate workflow, hubs may find it helpful to associate round-specific metadata with the corresponding model output files.
diff --git a/docs/source/format/model-metadata.md b/docs/source/format/model-metadata.md
@@ -1,3 +1,4 @@
+(model-metadata)=
 # Model metadata
 
 ## Directory structure

diff --git a/docs/source/format/model-output.md b/docs/source/format/model-output.md
@@ -1,3 +1,4 @@
+(model-output)=
 # Model output
 
 ## Directory structure

diff --git a/docs/source/format/target-data.md b/docs/source/format/target-data.md
@@ -1,4 +1,5 @@
 
+(target-data)=
 # Target data
 
 ## Purpose

diff --git a/docs/source/index.md b/docs/source/index.md
@@ -4,7 +4,38 @@
 This project is under active development.
 ```
 
-The Consortium of Infectious Disease Modeling Hubs is a collaboration of research teams that have built and maintained predictive modeling hubs for infectious disease applications. Working together, we have developed software to for groups that are running collaborative modeling hub efforts. This website documents the requirements for using the infrastructure that our collaborative group has set up.
+The Consortium of Infectious Disease Modeling Hubs is a collaboration of research teams that have built and maintained predictive modeling hubs for infectious disease applications. Working together, we have developed software to for groups that are running collaborative modeling hub efforts. This website documents the requirements for using the infrastructure that our collaborative group has set up. The following sections of this page provide an outline of the different resources created by this project.
+
+## Tools for building and hosting modeling hubs
+
+The following subsections provide pointers to resources developed by the Consortium to make designing, launching, and maintaining hubs easier.
+
+### Template hubs
+
+The [template hub repositories](https://github.com/Infectious-Disease-Modeling-Hubs?q=&type=template&language=&sort=) provided by the consortium may be cloned directly to start a new hub. Unlike the example hubs below, these repositories do not have any data in them, they just provide a skeletal structure of a hub. Currently, we only host a single [template hub](https://github.com/Infectious-Disease-Modeling-Hubs/hubTemplate).
+
+### Example hubs
+
+We have created some [example Hub repositories](https://github.com/Infectious-Disease-Modeling-Hubs?q=example&type=all&language=&sort=) that provide minimal working examples of hubs. These repositories could be used for ideas of how to set up configuration files for new projects. They are also used as use-cases for testing the software described below.
+
+- The [Simple Forecast Hub Example](https://github.com/Infectious-Disease-Modeling-Hubs/example-simple-forecast-hub) is designed to be similar to the [US CDC FluSight Hospitalization Forecasting exercise](https://github.com/cdcepi/Flusight-forecast-data) from 2022-2023.
+- The [Complex Forecast Hub Example](https://github.com/Infectious-Disease-Modeling-Hubs/example-complex-forecast-hub) is designed to be similar to the [US COVID-19 Forecast Hub](https://github.com/reichlab/covid19-forecast-hub) and the [European COVID-19 Forecast Hub](https://github.com/covid19-forecast-hub-europe/covid19-forecast-hub-europe).
+- The [Complex Scenario Hub Example](https://github.com/Infectious-Disease-Modeling-Hubs/example-complex-scenario-hub) is designed to be similar to the [US COVID-19 Scenario Modeling Hub](https://github.com/midas-network/covid19-scenario-modeling-hub)
+
+### Schema files for hub configuration
+
+To take advantage of the infrastructure designed by the Consortium, a repository must contain JSON configuration files in a [specific location and format](hub-metadata). The schemas that define the structure and formats of the configuration files live in their own [schemas repository](https://github.com/Infectious-Disease-Modeling-Hubs/schemas). The schemas are versioned, and every hub must point to a specific version of the schemas that they are using.
+
+## Software for modeling hubs
+
+The main benefit of setting up a hub using the structure outlined in this documentation is that it enables you to use a wide array of tools designed to support common modeling hub tasks, like loading model output data, plotting the model output data, building ensembles using the data, and in some cases evaluating the predictions made by different models. 
+
+- [hubUtils](https://infectious-disease-modeling-hubs.github.io/hubUtils/) is an R package with utility functions for working with data from modelings hubs.
+- [hubEnsembles](https://github.com/Infectious-Disease-Modeling-Hubs/hubEnsembles) is an R package with functionality to build simple ensembles of data from modeling hubs.
+
+
+
+
 
 
 
@@ -24,11 +55,10 @@ overview/definitions.md
 format/intro-data-formats.md
 format/hub-structure.md
 format/hub-metadata.md
-format/model-output.md
 format/model-metadata.md
-format/model-abstracts.md
+format/model-output.md
 format/target-data.md
-data.md
+format/model-abstracts.md
 ```
 
 ```{toctree}

diff --git a/docs/source/overview/definitions.md b/docs/source/overview/definitions.md
@@ -1,11 +1,19 @@
 # Key definitions
 
-```{admonition} Modeling hub
+```{admonition} Modeling Hub
 a consortium of research groups working together on a common set of modeling tasks, with the goal of developing ensemble models to provide stakeholders with a single model output representing uncertainty across different modeling assumptions and frameworks.
 ```
 
+```{admonition} Round
+a time-period for which a set of specific model outputs are solicited. Rounds define the "cadence" of submission for a modeling hub. For example, some hubs might accept daily submissions, where each day is considered a different round. Other hubs might have one round every month, with a submission period that may be open for multiple days.
+```
+
+```{admonition} Target
+a quantitative outcome of interest for a modeling Hub. For example, "incident case counts". Targets typically (and sometimes implicitly) refer to a value of an observable variable in a given window of time, a given location, and possibly other stratifications (such as age group).
+```
+
 ```{admonition} Task
-a definition of the goals of a modeling effort, possibly including conditions, assumptions, and targets. Some tasks are relatively fixed, as for forecast hubs that regularly predictions for a set time horizon in the near-term future. Other tasks may be more variable, as for scenario hubs that model differing hypothetical futures.
+a definition of the goals of a modeling effort, possibly including conditions, assumptions, and targets. Some tasks may be fixed across rounds, as for forecast hubs that regularly solicit predictions for a set time horizon in the near-term future. Other tasks may be more variable, as for scenario hubs that model hypothetical futures with differnt assumptions in different modeling rounds.
 ```
 
 ```{admonition} Team 
@@ -17,11 +25,11 @@ a statistical or mathematical formulation of a model.
 ```
 
 ```{admonition} Modeling software
-Code that implements the modeling strategy used by a team to generate model outputs. Each team may have multiple instances of software.
+code that implements the mathematical model formulation/structure used by a team to generate model outputs. Each team may have multiple instances of software.
 ```
 
 ```{admonition} Model output 
-A set of target results generated in response to some modeling task. A model might be the result of a single team’s response to the task, or it might be an ensemble of results representing outcomes of multiple efforts
+a set of target results generated in response to some modeling task for a specific round. A model might be the result of a single team’s response to the task, or it might be an ensemble of results representing outcomes of multiple efforts
 ```
 
 ```{admonition} Nowcast
Original file line number	Diff line number	Diff line change
Expand Up		@@ -14,4 +14,4 @@ There is one directory for each team and within that, possibly multiple files, o

		## Purpose

		The `model-abstracts` directory is an optional feature of a hub that may be useful to keep track of round-specific updates to the metadata, e.g. describing modeling assumptions specific to one modeling round. Note: as part of a separate workflow, hubs may find it helpful to associate round-specific metadata with the corresponding model output files.
		The `model-abstracts` directory is an optional feature of a hub that may be useful to keep track of round-specific detailed narrative descriptions of modeling methods and results, e.g. describing modeling assumptions specific to one modeling round. Note: as part of a separate workflow, hubs may find it helpful to associate round-specific metadata with the corresponding model output files.