revisit context.projects

[escapewindow]

Aiui, we:

• updated autoland's scm level
• started failing cot because context.projects is cached over the lifetime of a running scriptworker. We could await context.populate_projects(force=True), but we never call that from anywhere in the code.

One potential solution here is:

• add a timestamp for when we populate context.projects, e.g. context._projects_timestamp. We should update this on the projects setter.
• add a config item for how long to cache context.projects if force=False.
• if we're past the TTL of the cache, update context.projects even if force=False.

Also this:

09:28 < aki> are there any outstanding cot questions? or all resolved
09:30 < tomprince> aki: All resolved, I think. I think we should consider making scriptworker use ci-admin to get the repository level, rather than having it understand the config format directly.
09:30 < aki> ok
09:30 < tomprince> aki: https://hg.mozilla.org/ci/ci-configuration/rev/745b0dce17f8 is what caused the breakage, I think.
09:33 < jlorenzo> sounds like a good fix
09:34 < aki> it's a lot more heavy weight than downloading a file, but it's doable. could we have ci-{configuration,admin} automation generate an artifact that scriptworker consumes?
09:35 < aki> that would avoid having to clone ci-{configuration,admin} everywhere we want to verify cot. i imagine we'll want to add cot verification to all artifact downloads in the graph at some point

This is the first time I know of that we've hit this issue, so this might not be super urgent. Good to address though.

[tomprince]

* updated autoland's scm level

To be specific, we updated how the level was represented in ci-config. projects.yml has the access group of the repositories on hg.mozilla.org. The group for autoland is currently represented as scm_level_3, but the actual group is scm_autoland. The change that landed was making it match the actual group, with logic in ci-admin to map that to the correct level.

[escapewindow]

This may be less concerning, since we can easily refresh projects by rolling out new docker images, and we start new workers on a more frequent basis than puppet.

[escapewindow]

09:30 < tomprince> aki: All resolved, I think. I think we should consider making scriptworker use ci-admin to get the repository level, rather than having it understand the config format directly.

Today this would likely require us to teach scriptworker how to do hg clones, which is probably too much overhead.