Add a data dictionary #315
Conversation
| @@ -231,10 +231,13 @@ Model accuracy for each parameter combination is measured on a validation set us | |||
|
|
|||
| ### Features Used | |||
|
|
|||
| The residential model uses a variety of individual and aggregate features to determine a property's assessed value. We've tested a long list of possible features over time, including [walk score](https://gitlab.com/ccao-data-science---modeling/models/ccao_res_avm/-/blob/9407d1fae1986c5ce1f5434aa91d3f8cf06c8ea1/output/test_new_variables/county_walkscore.html), [crime rate](https://gitlab.com/ccao-data-science---modeling/models/ccao_res_avm/-/blob/9407d1fae1986c5ce1f5434aa91d3f8cf06c8ea1/output/test_new_variables/chicago_crimerate.html), [school districts](https://gitlab.com/ccao-data-science---modeling/models/ccao_res_avm/-/blob/9407d1fae1986c5ce1f5434aa91d3f8cf06c8ea1/output/test_new_variables/county_school_boundaries_mean_encoded.html), and many others. The features in the table below are the ones that made the cut. They're the right combination of easy to understand and impute, powerfully predictive, and well-behaved. Most of them are in use in the model as of `r Sys.Date()`. | |||
| The residential model uses a variety of individual and aggregate features to determine a property's assessed value. We've tested a long list of possible features over time, including [walk score](https://gitlab.com/ccao-data-science---modeling/models/ccao_res_avm/-/blob/9407d1fae1986c5ce1f5434aa91d3f8cf06c8ea1/output/test_new_variables/county_walkscore.html), [crime rate](https://gitlab.com/ccao-data-science---modeling/models/ccao_res_avm/-/blob/9407d1fae1986c5ce1f5434aa91d3f8cf06c8ea1/output/test_new_variables/chicago_crimerate.html), [school districts](https://gitlab.com/ccao-data-science---modeling/models/ccao_res_avm/-/blob/9407d1fae1986c5ce1f5434aa91d3f8cf06c8ea1/output/test_new_variables/county_school_boundaries_mean_encoded.html), and many others. The features in the table below are the ones that made the cut. They're the right combination of easy to understand and impute, powerfully predictive, and well-behaved. | |||
There was a problem hiding this comment.
The only change to this paragraph is removing this line:
Most of them are in use in the model as of
r Sys.Date().
The first half of this line seems inaccurate to me (all of these features are in use in the model) and not particularly helpful (this table represents the most recent version of the parameters, so the date is not useful). Happy to keep one or both of these pieces of info if you think there's a good reason for them, though.
| param_tbl_fmt <- param_tbl %>% | ||
| mutate(description = param_notes) %>% | ||
| left_join( | ||
| ccao::vars_dict, | ||
| by = c("value" = "var_name_model") |
There was a problem hiding this comment.
I switched this up so that the params are the left side of the join here, which feels more intuitive to me than vars_dict being the left side. We also use a left join so that we'll preserve all of the parameters, even if one happens to be misdocumented in vars_dict in the future.
| feature_name = var_name_pretty, | ||
| variable_name = value, | ||
| description, | ||
| category = var_type, | ||
| type = var_data_type, |
There was a problem hiding this comment.
I think it's clearer for the CSV to use lowercase and underscored column names, so we start with those and then reformat them when rendering the table to the README.
| # Write machine-readable version of the table to file | ||
| param_tbl_fmt %>% | ||
| write_csv("docs/data-dict.csv") |
There was a problem hiding this comment.
This seemed to me like the simplest way to keep the data dict up to date: Any time we render the README, we'll save the data dict to the file. If model parameters haven't changed, the data dict file won't change, and there won't be a diff; otherwise, there will be a diff and the code author will be prompted to commit it. Not the most airtight system, but I figure it's probably a good enough starting place. Let me know if you have other ideas!
Perhaps out of scope for now, but we could also consider adding a pre-commit check similar to readme-rmd-rendered that compares the params in params.yml to the params in this file to make sure they match. I'm happy to take a crack at that now if you think it's a good idea.
There was a problem hiding this comment.
The pre-commit hook was pretty straightforward so I went ahead and implemented it in 46c163b.
README.md
Outdated
| For a machine-readable version of this data dictionary, see | ||
| [`docs/data-dict.csv`](./docs/data-dict.csv). | ||
|
|
||
| | Feature Name | Variable Name | Description | Category | Possible Values | |
There was a problem hiding this comment.
Big diff here, mainly because I've rearranged the column order. I double-checked to make sure the parameter count matches params.yaml.
| 5. Run `renv::activate(profile = "default")` if you would like to | ||
| switch back to the default renv profile |
There was a problem hiding this comment.
This seems like an unrelated change due to autoformat.
|
This is awesome. Question for anyone - if someone wanted to grab training data and use it to feed a python model, would they have the info they need to recode features/apply variable labels currently? |
Yes, the development version of the Python |
…a-dictionary-in-readme
…a-dictionary-in-readme
| - id: check-data-dict | ||
| name: Data dictionary must be up to date with params file | ||
| entry: Rscript R/hooks/check-data-dict.R | ||
| files: (^|/)((params\.yaml)|(data-dict\.csv))$ | ||
| language: r | ||
| additional_dependencies: | ||
| - yaml |
There was a problem hiding this comment.
I also considered adding a hook to make sure that ccao::vars_dict contains every feature in the param file, and that the dbt DAG has a description for every feature. This would help guard against a situation where we neglect to update the table property, but it has the downside of enforcing a commit-level check that we could only resolve by updating external dependencies, which I expect would lower our velocity during modeling. We could instead think about making this a CI check that only runs on tags, but I've skipped that here since it feels out of scope.
|
Currently blocked by ccao-data/data-architecture#704. Once that comes in I'll do a final pass at rendering the README and data dict and re-request review. |
…oblems" This reverts commit 95db04c.
|
Now blocked by ccao-data/actions#36. |
This reverts commit 87ba713.
This PR turns the "Features Used" table in the README into a proper data dictionary by making two changes:
Variable Namethat lists the name of the variable as it appears in the model codedocs/data-dict.csvWe also rename the
Notescolumn toDescriptionfor clarity, and move it to the left in the table so that it's more prominent.Note that this PR does not create historical dictionaries for past models. My expectation is that we will keep
docs/data-dict.csvup to date with the most recent version of the parameter file, and then in the future we can back out the data dict that we used for past models by referencing the version ofdocs/data-dict.csvthat existed at the time of the yearly model tag.If this change looks good, I'll go ahead and copy it to the condo model to address ccao-data/model-condo-avm#72.
Closes #300.