builds.sr.ht: Split OpenBSD job causing resource usage concerns

[euantorano]

I wasn't aware of any complaints RE OpenBSD utilisation issues, but I
know that the original build I wrote got split into 3 separate builds
because the build was taking a long time on OpenBSD. Any chance you can
summarise the issues and I'll report them to Nim and hopefully find some
time to look into them?

Long-running builds are less taxing on our resources than running
several builds at once. Nim runs a lot of builds, and when every one
includes three redundant OpenBSD jobs, the resource usage is not really
appropriate.

Each VM has 2 CPUs, and you should make sure you utilize both, but
shouldn't have so many build jobs for what is ultimately the same
system. I have to cancel a lot of Nim builds in order to make room for
other users.

This was reported by Drew (maintainer of builds.sr.ht), the split build introduced here is causing resource usage headaches for the service and impacting other users.

We should revert back to a single build file.

Note that having multiple build files will cause other headaches, as the maximum number of builds allowed is 4 and once builds.sr.ht supports NetBSD (soon, hopefully) we'll be adding a build file for that platform too which will give is 5 build files. When there are more than 4 build files, the service will select 4 at random.

[timotheecour]

see #17108 for a start.

When the time comes for netbsd (or if 2 is still too much), we can revisit this using several strategies (see aslso: faster CI · Issue #600 · timotheecour/Nim)

• further reduce from 2 to 1 job on openbsd

which we could offset by:

• generate better metrics to automatically report CI bottlenecks, time breakdowns etc
• run some tests with probability x < 1 on openbsd
• improve the logic behind NIM_TESTAMENT_BATCH to ensure there is no (or less) redundancy among batches
• (hard) identify test bottlenecks and reduce key parameters
• (users would forget) support a [ci ckip openbsd] commit message that would skip openbsd in PRs
• (error prone) avoid running openbsd CI for PR's (only when there's a merge)
• faster bootstrap on openbsd by using same approach as ci: see if setup-nim is faster than choosenim fusion#70
• (big impact) improve the megatest approach so that a megatest is also used in place of slow things like testament pcat lib, testament pcat js, and so isJoinableSpec returns true in more cases (eg targets: "c cpp js" currently would disable megatest but instead should join 3 megatests (1 for c, cpp, js)
• (big impact) megatest for failing tests, via compilesWithContext (nim check is ill suited for that), refs compiles(expr) with context: let context = compilesWithContext(expr)` with context.{ok, warnMsg, errorMsg} to simplify tests #8803 megatest for failing tests timotheecour/Nim#176

Each VM has 2 CPUs, and you should make sure you utilize both

we do, testament uses pcat

[euantorano]

(users would forget) support a [ci ckip openbsd] commit message that would skip openbsd in PRs

Not supported by builds.sr.ht - instead you set an option. See here.

(error prone) avoid running openbsd CI for PR's (only when there's a merge)

Easily done with dispatch.sr.ht as there are actually two triggers: PR + commit

faster bootstrap on openbsd by using same approach as nim-lang/fusion#70

I believe there is a plan for caching in builds.sr.ht at some point, so we could cache the sources build for example.

[timotheecour]

Not supported by builds.sr.ht - instead you set an option. See here.

but we could handle this in .builds/freebsd.yml + friends, so that the job would complete almost immediately. This could be done by running git log --oneline --format=%B -n 1 HEAD and checking whether [ci ckip openbsd] is in it, for example; so that users don't need a different way to skip just for openbsd

I believe there is a plan for caching in builds.sr.ht at some point, so we could cache the sources build for example.

I suggest something simpler and that doesn't need to wait for them to support caching: given that this never changes:

    git clone --depth 1 -q https://github.com/nim-lang/csources.git
    gmake -C csources -j $(sysctl -n hw.ncpu)

we can replace it by fetching the resulting bin/nim from a fixed url pre-built for the arch we're testing (openbsd, freebsd)

csources rarely changes, so we can manually change this when needed.

[narimiran]

After the concerns raised in #17479, sr.ht stuff is now disabled for PRs, and it will only run for commits.

[timotheecour]

@euantorano do you have any idea how to run openbsd (or freebsd) CI in PRs when something BSD specific needs to be addressed?

refs #17801 (comment) in which i suggested an opt-in approach via either [ci bsd] or a .builds/openbsd.yml.disabled that PR authors can temporarily enable in their PR, but for reasons that I don't understand, even that seems to face push back.

It's hard to claim we support platform X in nim when PR authors can't test for platform X.

refs #16282 (comment)

Every change is related to bootstrapping on the BSDs. I presume there is a regression for the 1.0 line where it's pickier about 'std' than even 0.9.x is.

[euantorano]

The ideal solution IMO would be to only run the tests that test anything that has been changed in each given PR. For example, if there’s a change to the tables module only run it’s tests, not tests for something entirely unrelated like regex or something. I’m not sure exactly how that could be done, but it might be a good starting place.

…

On Tue, 20 Apr 2021, at 21:24, Timothee Cour wrote: @euantorano <https://github.com/euantorano> do you have any idea how to run openbsd (or freebsd) CI in PRs when something BSD specific needs to be addressed? refs #17801 (comment) <#17801 (comment)> in which i suggested an opt-in approach via either `[ci bsd]` or a `.builds/openbsd.yml.disabled` that PR authors can temporarily enable in their PR, but for reasons that I don't understand, even that seems to face push back. It's hard to claim we support platform X in nim when PR authors can't test for platform X. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#17107 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFW24NZDHHOCBLTI5P3VXDTJXPHDANCNFSM4X4QEQSA>.

[timotheecour]

that's a separate topic though, it's doable and worth it (speeding up CI) but would require care that dependencies are tracked correctly. But we're not facing a real problem with our other CI, only with BSD CI right now.

[euantorano]

Well the dispatch integration for builds is limited to only running either all PRs or only commits. When using git.sr.ht supports some special options that I don’t think the GitHub integration can support. The other option is looking at other services like BuildKite or self hosted sr.ht where the Nim project is responsible for the hardware running the test suite - that way only Nim is affected by resource usage (and I suspect we will get a nasty shock about how heavy the test suite is...)

…

On Tue, 20 Apr 2021, at 22:00, Timothee Cour wrote: that's a separate topic though, it's doable and worth it (speeding up CI) but would require care that dependencies are tracked correctly. But we're not facing a real problem with our other CI, only with BSD CI right now. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#17107 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFW24P422B27NH6BGM4JW3TJXTPBANCNFSM4X4QEQSA>.

[ringabout]

see also #17801 (comment)