Add evaluation, multimodal, native tests to run-readme-pr-mps.yml

[mikekgfb]

Add evaluation, multimodal, native tests to run-readme-pr-mps.yml

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchchat/1410

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 2 New Failures, 4 Unrelated Failures

As of commit 45ec014 with merge base cbc72a4 ():

NEW FAILURES - The following jobs have failed:

• pull / runner-et (macos-14-xlarge) (gh)
  Process completed with exit code 1.
• pull / test-build-runner-et-android / linux-job (gh)
  RuntimeError: Command docker exec -t 0df06c5f62158552b7d33a86c21953f60fa2de74b43086ff6495b3b3b49c3c0b /exec failed with exit code 1

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / runner-et (16-core-ubuntu) (gh) (trunk failure)
  ##[error]The operation was canceled.
• pull / test-tinystories-executorch (16-core-ubuntu) (gh) (trunk failure)
  ##[error]The operation was canceled.
• pull / test-tinystories-executorch (macos-14-xlarge) (gh) (trunk failure)
  Process completed with exit code 1.
• pull / test-torchao-experimental (macos-14-xlarge) (gh) (trunk failure)
  Process completed with exit code 1.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[Jack-Khuu]

Just realized that none of the test in this file are being ran:

Invalid workflow file: .github/workflows/run-readme-pr-mps.yml#L13
The workflow is not valid. .github/workflows/run-readme-pr-mps.yml (Line: 13, Col: 24): Invalid input, timeout-minutes is not defined in the referenced workflow.

[mikekgfb]

Just realized that none of the test in this file are being ran:

Invalid workflow file: .github/workflows/run-readme-pr-mps.yml#L13
The workflow is not valid. .github/workflows/run-readme-pr-mps.yml (Line: 13, Col: 24): Invalid input, timeout-minutes is not defined in the referenced workflow.

Should be fixed now. Please rerun tests?

[mikekgfb]

Just realized that none of the test in this file are being ran:

Invalid workflow file: .github/workflows/run-readme-pr-mps.yml#L13
The workflow is not valid. .github/workflows/run-readme-pr-mps.yml (Line: 13, Col: 24): Invalid input, timeout-minutes is not defined in the referenced workflow.

Should be fixed now. Please rerun tests?

Still timed out @ 50 minutes, increased timeout further.

Beyond the "let's extend timeout" bandaid, I'm wondering whether this may be due to us trying to update llvm+openmp with the command conda create -y -n test-readme-mps-macos python=3.10.11 llvm-openmp which may be adding significant runtime to the test? This might benefit from somebody looking at it who is familiar with pytorch ci/cd setup on github? (i.e., the pt releng team.

Also, MPS is the only set of doc tests using conda, which on the one hand is good because we're testing setup with conda as well, but otoh makes MPS the odd man platform out at the moment.

[malfet]

Nice typo fix, should we land it as separate PR?

[mikekgfb]

Nice typo fix, should we land it as separate PR?

#1460