How to specify platform?

[pboling]

In a normal docker scenario we can specify a platform for docker to build with:

docker build --platform=linux/amd64 ...

Is there something similar we can do with the pipeline runner?

In Bitbucket, the pipeline is almost certainly running with linux/amd64, and I'd like to be able to run it that way locally on my Mac, even though it is arm64 (Apple silicon). This is what we do when we build docker images on our Macs normally.

I am currently getting an error in the yarn install phase, where optipng-bin fails to build and the error is hard to understand, but something to do with Rosetta (Apple's Intel emulation/translation layer) and the platform.

[pboling]

Actually, I think this is the bug I'm running into:
imagemin/imagemin-optipng#36

[mathieu-lemay]

That's a good one. This is not something that is supported as of now unfortunately, but I can definitely see the need for it. And it should be quite easy to fix too.

I've just done a quick half assed solution if you want to try it. Install version 0.4.11a0, then export PIPELINE_RUNNER_DOCKER_PLATFORM=linux/amd64, and then run a pipeline. It should use the specified platform.

The real fix will use a --platform flag, like docker, but this was quicker to implement for you to try.

[pboling]

The fix in the issue I linked did work export CPPFLAGS="-DPNG_ARM_NEON_OPT=0", but it is ugly, and I had to add a platform specific script wrapper around it, because the pipeline runner will idealy be run by people on many different platforms across our org.

Also, just in general, running the pipeline on a different platform than it does in Bitbucket doesn't feel awesome. :)

So, I'm going to try your solution now.

[pboling]

It seems that it is working, and the failure has moved to a similar, but different library, jpegtran-bin, presumably because it is running on a different architecture.

What I don't undetstand is why this isn't happening on Bitbucket Pipelines actual. I also got a bunch of odd errors from a bash script that runs as an early smoke check in the pipeline. I don't see syntax errors in actual bitbucket, so now I'm wondering how the bash version might be different, and why would it only be different when I'm running the same platform, as I didn't see this until I specified the platform.

... Investigating.

[pboling]

Strange things are happening... 😆

Second run it was back to failing with the same optipng-bin error that I had when it was running Apple Silicon.

Third run I realized it was restoring an old yarn cache from back when it was running on Apple Silicon, so when changing platform I think I need to kill the cache.

Also I realized that I may not have set the platform ENV variable properly. It needs to be set before the pipeline-runner command is executed, rather than be in the .env file that the pipeline runner uses, right, @mathieu-lemay ?

[pboling]

Bumping the cache version fixed it.

(added -v2 below)

  caches:
    yarn-cache-v2:
      key:
        files: # Check these files for changes when deciding whether to use the cache, or download fresh dependencies.
          - yarn.lock
      path: node_modules

[pboling]

OK, figured out the other issues, except for the bash syntax thing... But I think it is good to go from the --platform perspective @mathieu-lemay

[pboling]

Setting platform to a tri-sectional platform string doesn't appear to work.

For example, when I set:

export PIPELINE_RUNNER_DOCKER_PLATFORM=linux/amd64/v8

I get (added new lines to make it readable)

docker.errors.ImageNotFound: 404 Client Error for
http+docker://localhost/v1.47/containers/create?name=command-center-uofkC51v-step-ruby-bundle-install&platform=linux%2Famd64%2Fv8:
Not Found ("No such image: ruby:3.3.6-bookworm")

oooh, it is because images specifically for linux/amd64/v8 are often not published... :( Nevermind. I think --platform is good to go.

[pboling]

Additionally, I have just found this (see last two comments):
https://forums.docker.com/t/docker-default-platform-should-be-more-obvious-to-apple-silicon-users-everyone/120805/3

So, while I think it is good, and important, for this project to support the --platform flag, I don't think it is a viable solution for my use case.

[pboling]

Additionally, it does appear that setting (e.g. in my .env)

export DOCKER_DEFAULT_PLATFORM=linux/whatever

May be all that is needed to control the platform that docker, and thus pipeline runner, use to build.

See: DOCKER_DEFAULT_PLATFORM here: https://docs.docker.com/reference/cli/docker/#environment-variables

Still, again, I think supporting --platform is a good idea.

[pboling]

@mathieu-lemay When using the latest pre-release with the platform feature, it has a bug where it doesn't work if I do not specify a platform. It should work either way.

2024-12-27 13:18:28.554 pipeline_runner.service: Starting service: memcached
2024-12-27 13:18:28.555 pipeline_runner.runner: Error during pipeline execution
Traceback (most recent call last):
  File "/Users/pboling/.local/pipx/venvs/bitbucket-pipeline-runner/lib/python3.13/site-packages/pipeline_runner/runner.py", line 204, in run
    services_manager.start_services(f"container:{container_runner.get_container_name()}")
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pboling/.local/pipx/venvs/bitbucket-pipeline-runner/lib/python3.13/site-packages/pipeline_runner/service.py", line 62, in start_services
    sr.start()
    ~~~~~~~~^^
  File "/Users/pboling/.local/pipx/venvs/bitbucket-pipeline-runner/lib/python3.13/site-packages/pipeline_runner/service.py", line 138, in start
    pull_image(self._client, self._service.image)
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: pull_image() missing 1 required positional argument: 'platform'