The filter-test-dev CI script doesn't seem to be working

[fabianp]

see for example https://github.com/iclr-blogposts/2024/actions/runs/6591771757

I'm commenting out for now the offending lines since the upload script depends on the success of this script

[fabianp]

I suspect that's because of some security restrictions of github actions. It might be necessary to put some of those tests in a different workflow_run file as I've done in deploy-for-review.yml. See for example https://stackoverflow.com/a/71683208

[Velythyl]

Alright so I did a LOT of new commits to fix the scripts for when an external PR is made.

I found some issues:

1. When an external person makes a PR, we have to manually "Approve & Run" the actions for it. This is doable manually but very annoying. -> TODO
2. When the filters fail, the comment wouldn't get written at all. This would also cause the S3 upload to bail prematurely. -> [FIXED in both cases]

[Velythyl]

The new file for writing comments on filter failures is here: https://github.com/iclr-blogposts/2024/blob/main/.github/workflows/comment-on-error.yaml

Here is an example PR from an external account that. You can tell by the automated comments on that PR that at first, I tested a success case, and then I swapped to a failure case. #52

[fabianp]

Regarding your first point, I believe we have to manually approve it only if it's a new github account (I don't know how github determines exactly that an account is new)

[fabianp]

thanks BTW for the work on fixing the actions!

[Velythyl]

I believe we have to manually approve it only if it's a new github account

ah ok I see! Maybe we can add a small message in the PR template to mention that if their run doesn't start automatically, they have to @ us to approve it

[fabianp]

yes that seems like a good idea