Create a public data lineage viewer using read only Dagster UI

[bendnorman]

I recently learned we can run the dagster UI in read-only mode. This made me think we could host a public data lineage viewer on Cloud Run. Users would be able to

• interact with DAG
• explore dependencies
• inspect table metadata: description, number of rows, date range
• inspect column metadata: description, types, constraints, summaries (min, max, average...)

This might not be worth the time to expose all of the intermediate assets and complexities of the data processing, given we want to move toward exposing just a handful of tables to users. I could imagine receiving a lot of questions:

• Why are some of the assets in the DAG not in the database?
• Should I use the dagster UI or the data dictionary as my main reference?
• Which assets should I be interacting with?

If we did want to do this we'd need to:

• Setup a cloud Run instance that gets updated on successful nightly build runs.
• Update the documentation to describe the data lineage viewer.
• Set @asset(description=...) parameter to table descriptions in metadata. Dagster automatically makes the doc string the description of the asset. The description of the table in the data dictionary does not always match the doc string of the asset that produces it.
• Create a methods in the metadata classes to convert the metadata to pandera schemas so assets have column information available in the UI.

Some pros for creating this:

• Gives something for first time users to interact with beyond the documentation.
• Allows users with more complex needs to explore data dependencies without setting up the python environment.