test: Add script to test model loading below n_parameters threshold

[isaac-chung]

Fixes #1690

• first get the model meta from model registry and check for n_parameters, and omit API models.
• if it is below threshold (2B for now), then run mteb.get_model and output the exception (None or raw text)
• write these to a results.json file in the repo (in the scripts folder)
• ability to continue running from where the script left off (args to specify running missing models, unsuccessful models, or by model names)
• Trigger script to run when there are changes to the model files, then write the results out to the json file.

Checklist

• Run tests locally to make sure nothing is broken using make test.
• Run the formatter to format the code using make lint.

[Samoed]

I think we could create a file to record whether a model load was successful, to avoid repeated loading attempts.

[isaac-chung]

Right now the main issue is disk running out of space. Looking into how not to save the weights into cache folder when loading a model.

[Samoed]

Also, test should skip API based models

[KennethEnevoldsen]

Looking good so far. Just to clarify the intention is to move this into the tests suite?

[isaac-chung]

@KennethEnevoldsen thanks, and yes.

[isaac-chung]

@KennethEnevoldsen @Samoed just pushed the latest changes, and updated the description. Would you mind reviewing and seeing if anything is unclear?

I'll try to push a "test" commit to see how it works.

[KennethEnevoldsen]

Generally looks good, very happy to accept this with just the changes the dependecy installations.

[isaac-chung]

Thanks both for your input. Right now the test fails at the merge base step (empty list is returned). If you have some time, it would be great if you could help take a quick look there. Otherwise I plan to return to this later tomorrow.

[Samoed]

I think the GitHub runner automatically merges branches during CI, so you don’t need to do it manually.

[KennethEnevoldsen]

Great work! I think we are almost there

[Samoed]

Great work!

[isaac-chung]

Thanks again. Merging now. Note that the model loading "test" will likely require merging the target branch to get the accurate diffs.