Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update sources, DOI and copyright link in PUDL #3004

Merged
merged 3 commits into from
Nov 2, 2023
Merged

Conversation

e-belfer
Copy link
Member

@e-belfer e-belfer commented Nov 2, 2023

In response to comments on #3000 and #3001 and the Zenodo migration in pudl-archiver PR #184 this PR makes some minor modifications to the way we write and read datapackages and source data metadata on the PUDL side. Specifically:

  • Removes individual contributors from PUDL data sources, including only Catalyst Cooperative as a data source for our main raw archived datasets
  • Adds a DOI field with pydantic validation to the datapackage class, using regex from the pudl-archiver
  • Adds a new zenodo-role field to the contributors which defaults to project member, and encodes the list of possible roles in the new Zenodo API (a newly required field).
  • Updates the US copyright link

PR Checklist

  • Merge the most recent version of the branch you are merging into (probably dev).
  • All CI checks are passing. Run tests locally to debug failures
  • Do your own explanatory review of the PR to help the reviewer understand what's going on and identify issues preemptively.

@e-belfer e-belfer added datapkg Frictionless data package input, output, metadata, manipulation zenodo Issues having to do with Zenodo data archiving and retrieval. labels Nov 2, 2023
@e-belfer e-belfer self-assigned this Nov 2, 2023
@e-belfer e-belfer requested a review from zaneselvans November 2, 2023 14:39
Copy link
Member

@zaneselvans zaneselvans left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a good reason not to re-use our existing ZenodoDoi pydantic type here?

src/pudl/metadata/constants.py Outdated Show resolved Hide resolved
src/pudl/metadata/classes.py Outdated Show resolved Hide resolved
Copy link

codecov bot commented Nov 2, 2023

Codecov Report

Attention: 13 lines in your changes are missing coverage. Please review.

Comparison is base (b376141) 88.7% compared to head (4b028d2) 88.7%.
Report is 73 commits behind head on dev.

Additional details and impacted files
@@          Coverage Diff          @@
##             dev   #3004   +/-   ##
=====================================
  Coverage   88.7%   88.7%           
=====================================
  Files         91      91           
  Lines      11007   11010    +3     
=====================================
+ Hits        9766    9769    +3     
  Misses      1241    1241           
Files Coverage Δ
src/pudl/metadata/classes.py 86.5% <100.0%> (+<0.1%) ⬆️
src/pudl/metadata/constants.py 100.0% <ø> (ø)
src/pudl/metadata/fields.py 100.0% <ø> (ø)
src/pudl/metadata/sources.py 100.0% <ø> (ø)
src/pudl/transform/ferc714.py 100.0% <100.0%> (ø)
src/pudl/transform/params/ferc1.py 100.0% <ø> (ø)
src/pudl/workspace/datastore.py 77.6% <100.0%> (ø)
src/pudl/output/ferc1.py 88.2% <69.2%> (ø)
src/pudl/transform/ferc1.py 96.7% <94.8%> (ø)

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@e-belfer e-belfer requested a review from zaneselvans November 2, 2023 16:47
Copy link
Member

@zaneselvans zaneselvans left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a cut-and-paste leftover comment string but other than that go for it.

@@ -228,6 +228,7 @@ def __repr_args__(self) -> list[tuple[str, Any]]:

HttpUrl = pydantic.AnyHttpUrl
"""Http(s) URL."""
"""DOIs from Zenodo archives."""
Copy link
Member

@zaneselvans zaneselvans Nov 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this docstring is a leftover?

@e-belfer e-belfer merged commit e7aedda into dev Nov 2, 2023
6 checks passed
@e-belfer e-belfer deleted the datapackage-update branch November 2, 2023 19:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
datapkg Frictionless data package input, output, metadata, manipulation zenodo Issues having to do with Zenodo data archiving and retrieval.
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

2 participants