Dagster takes too long to start

[ravenac95]

What is it?

This might not be something we can entirely avoid, but dagster takes quite a while to load. This can lead to really slow start times locally (which is frustrating) and also long load times on production that can sometimes cause timeouts.

There's likely some things we can do to potentially precompile the list of assets and load that somewhere like a either our cluster redis and then only recompute that every X interval. This way start up is faster.

We would then need to come up with a way to handle loading from this file and loading the cache once it's been invalidated. We can likely achieve this if all resources use our early_resources_asset_factory and change the internals of it to either run the code or use the cached values. This will complicate things, however, as the early resource loading would then need to be aware of all of the assets' resource dependencies in order to provide the correct resolution to dagster.