Task: smoothly replace script.py with main.py

[nlebovits]

Describe the task

The old ETL pipeline is set up as script.py. The new one is main.py, and relies on a set of new files in the new_etl subdirectory. This task involves smoothly replacing the old pipeline with the new. This is going to be a really messy ticket until I have a chance to walk through everything that needs to happen; if you're interested, add a comment and I'll come back with more context/info.

Along the way, there's some more cleanup that can be done, including (but not limited to):

• consolidating classes and imports
• getting FORCE_RELOAD to load data from postgres again (not working rn)
• moving vars to the env and/or config files where relevant
• moving loaders.py back into featurelayer.py and converting database.py to a class instead of a function (possibly)

[nlebovits]

@rmartinsen if you're interested in tackling another issue, this would be a huge help. I'm traveling this week and don't have bandwidth to make this happen--any chance you're interested?

[github-actions]

This issue has been marked as stale because it has been open for 30 days with no activity.