[Flaky Py SDK Snapshots]: Publish Beam SDK Snapshots is failing for Python distroless container

[ahmedabu98]

What happened?

Example run: https://github.com/apache/beam/actions/runs/12933159540/job/36071120448

Error:
ERROR: failed to solve: process "/dev/.buildkit_qemu_emulator /bin/sh -c apt-get update && apt-get install -y libsnappy-dev libyaml-dev ccache libgeos-dev && rm -rf /var/lib/apt/lists/* && pip install --upgrade pip setuptools wheel && pip install --no-deps -r /tmp/base_image_requirements.txt && rm -rf /tmp/base_image_requirements.txt && python -c \"import nltk; nltk.download('stopwords')\" && rm /root/nltk_data/corpora/stopwords.zip && python -c \"from google.protobuf.internal import api_implementation; assert api_implementation._implementation_type == 'upb'; print ('Verified fast protobuf used.')\" && mkdir -p /usr/local/gcloud && cd /usr/local/gcloud && curl -s -O https://dl.google.com/dl/cloudsdk/release/google-cloud-sdk.tar.gz && tar -xf google-cloud-sdk.tar.gz && /usr/local/gcloud/google-cloud-sdk/install.sh && rm -rf /usr/local/gcloud/google-cloud-sdk/.install/.backup && rm google-cloud-sdk.tar.gz && ln -s /usr/bin/ccache /usr/local/bin/gcc && ccache --set-config=sloppiness=file_macro && ccache --set-config=hash_dir=false && pip install --no-deps -v /opt/apache/beam/tars/apache-beam.tar.gz[gcp] && pip check || (echo \"Container does not include required Beam dependencies or has conflicting dependencies. If Beam dependencies have changed, you need to regenerate base_image_requirements.txt files. See: [https://s.apache.org/beam-python-requirements-generate\](https://s.apache.org/beam-python-requirements-generate/)" && exit 1) && pip freeze --all && rm -rf /root/.cache/pip" did not complete successfully: exit code: 1

Issue Failure

Failure: Test is Flaky

Issue Priority

Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[ahmedabu98]

Hey @damondouglas, I see #33703 was for Java. Is this Python failure related in any way?

[Abacn]

We have had problems on this workflow - #33563 then we moved to use github hosted runner, and now it's coming back again

[damondouglas]

Hey @damondouglas, I see #33703 was for Java. Is this Python failure related in any way?

The source of the error is with sdks/python/container/Dockerfile as sdks/python/container/distroless/common.gradle configures dockerPrepare.dependsOn ":sdks:python:container:py${pythonVersionSuffix}:docker".

The fact that switching GitHub hosted vs the self-hosted runner suggests that there might be something with the underlying architecture of the machine types. The fact that we see buildkit_qemu_emulator in the https://github.com/apache/beam/actions/runs/12933159540/job/36071120448 logs suggests that this might be the case.

Long term, I suggest we perform all of our container image builds using Google Cloud Build, triggered by GitHub actions for the following reasons:

1. Cloud Build triggers may be manually triggered outside the context of a GitHub action making this process easier to test and troubleshoot
2. We can assume a consistent underlying architecture within which container images are built.