automate updating of "NNNN - State & Local" collections?

[rahulbot]

When we seeded the global country collections years ago we created a "NNNN - State & Local" collection for each country. That was created by combining all the specific province/state collections together. That collection isn't dynamic, so changes to state collections aren't propagated into the combined "State & Local" one, making them slowly worse over time.

A quick fix would be to create a script that automatically updates the "State & Local" collection for each country by creating a union of all that country's state-level collections. This could run once a week and would improve the existing situation significantly. Perhaps this should live in web-search eventually? Moved from mediacloud/web-search#600. Adding here to track as a data science task related to directory maintenance.

A sketch of it (relying on our naming conventions) could look like this:

for each country:
    find the "- National" collection
    find the "- State & Local" collection
    find all the other state collections
    list all the sources in the state collections
        (bonus: email any sources in more than one state collection to someone)
    add all those sources to the "- State & Local"
        (bonus: if any were added new, post a slack msg with the country and list of domains added)
    update the note on the collection to indicate when it was last updated and that it was automated