sanity.external failures on xlinux on ibmcloud systems

[sxa]

Please set the title to indicate the test name and machine name where known.

To make it easy for the infrastructure team to repeat and diagnose, please
answer the following questions:

• test suite/name (e.g, BUILD_LIST, TARGET, CUSTOM_TARGET)? sanity.external (JDK11 specifically analysed)
• a link into recent Test_ job on https://ci.adoptium.net which showed the failure Most on https://ci.adoptium.net/job/Test_openjdk11_hs_sanity.external_x86-64_linux/589/
• Hyperlink to re-run in Grinder: [Link](https://ci.adoptium.net/job/Grinder/parambuild/?SDK_RESOURCE=upstream&TARGET=sanity.external&TEST_FLAG=&UPSTREAM_TEST_JOB_NAME=&DOCKER_REQUIRED=true&ACTIVE_NODE_TIMEOUT=0&EXTRA_DOCKER_ARGS=-v+%24%7BTEST_JDK_HOME%7D%3A%2Fopt%2Fjava%2Fopenjdk&VENDOR_TEST_DIRS=&TKG_OWNER_BRANCH=AdoptOpenJDK%3Amaster&PLATFORM=x86-64_linux&GENERATE_JOBS=false&KEEP_REPORTDIR=false&PERSONAL_BUILD=false&ADOPTOPENJDK_REPO=https%3A%2F%2Fgithub.com%2FAdoptOpenJDK%2Fopenjdk-tests.git&LABEL=&EXTRA_OPTIONS=&CUSTOMIZED_SDK_URL=&BUILD_IDENTIFIER=&ADOPTOPENJDK_BRANCH=master&LIGHT_WEIGHT_CHECKOUT=false&ARTIFACTORY_SERVER=&KEEP_WORKSPACE=false&JDK_VERSION=11&USER_CREDENTIALS_ID=&ITERATIONS=1&VENDOR_TEST_REPOS=&JDK_REPO=https%3A%2F%2Fgithub.com%2Fadoptium%2Fjdk11u&OPENJ9_BRANCH=master&OPENJ9_SHA=&JCK_GIT_REPO=&VENDOR_TEST_BRANCHES=&UPSTREAM_JOB_NAME=build-scripts%2Fjobs%2Fjdk11u%2Fjdk11u-linux-x64-temurin&OPENJ9_REPO=https%3A%2F%2Fgithub.com%2Feclipse%2Fopenj9.git&CUSTOM_TARGET=&VENDOR_TEST_SHAS=&JDK_BRANCH=dev&LABEL_ADDITION=&ARTIFACTORY_REPO=&ARTIFACTORY_ROOT_DIR=&UPSTREAM_TEST_JOB_NUMBER=&DOCKERIMAGE_TAG=nightly&JDK_IMPL=hotspot&SSH_AGENT_CREDENTIAL=&AUTO_DETECT=true&DYNAMIC_COMPILE=true&TKG_SHA=&CUSTOMIZED_SDK_URL_CREDENTIAL_ID=&NUM_MACHINES=&OPENJDK_SHA=&USE_TESTENV_PROPERTIES=false&BUILD_LIST=external&UPSTREAM_JOB_NUMBER=271&TIME_LIMIT=10&JVM_OPTIONS=&PARALLEL=None]
• Is there an existing issue elsewhere covering this? Not that I've spotted so far
• Which machine(s) does it work on? equinix-ubuntu2204] 591
• Which machine(s) does it fail on? ibmcloud-ubuntu1604-1 589 592 593 ibm-cloud-rhel7-1 590

Any other details: may be related to the OS - Ubuntu 16.04 is obviously quite old. RHEL7 may have other limitations as the docker host used for these images. Since I'm not sure this has been previously reported it may be worth checking if the underlying images used for this have been updated to later versions of OSs that might be incompatible with newer containers.

[sxa]

Needs further investigation to determine if the older docker version on those distributions may be the cause of the problem as opposed to it being ibmcloud specific.

[sxa]

Ubuntu 16.04 may be beyond hope:

jenkins@test-ibmcloud-ubuntu1604-x64-1:~$ docker run -it docker.io/library/eclipse-temurin:11-jdk
[0.007s][warning][os,thread] Failed to start thread "GC Thread#0" - pthread_create failed (EPERM) for attributes: stacksize: 1024k, guardsize: 4k, detached.
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Cannot create worker GC thread. Out of system resources.
# An error report file with more information is saved as:
# //hs_err_pid1.log
jenkins@test-ibmcloud-ubuntu1604-x64-1:~$ 

[sxa]

On test-azure-ubuntu2404-x64-1 we get this log:

14:49:18       [exec] The test in the build_image() function is openliberty-mp-tck
14:49:18       [exec] #####################################################
14:49:18       [exec] INFO: sudo podman build  --no-cache -t adoptopenjdk-openliberty-mp-tck-test:11-jdk-ubuntu-hotspot-full -f /home/jenkins/workspace/Grinder/jvmtest/external/openliberty-mp-tck/dockerfile/11/jdk/ubuntu/Dockerfile.hotspot.full /home/jenkins/workspace/Grinder/jvmtest/external/
14:49:18       [exec] #####################################################
14:49:18       [exec] sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
14:49:18       [exec] sudo: a password is required
14:49:18  
14:49:18  BUILD FAILED

sudo podman, if it is being executed on the host system, is not going to work. This system also only has docker - not podman - available

[sxa]

Same issue is experienced on the rhel7 machine, which also has docker and not podman:

14:56:30       [exec] INFO: sudo podman build  --no-cache -t adoptopenjdk-openliberty-mp-tck-test:11-jdk-ubuntu-hotspot-full -f /home/jenkins/workspace/Grinder/jvmtest/external/openliberty-mp-tck/dockerfile/11/jdk/ubuntu/Dockerfile.hotspot.full /home/jenkins/workspace/Grinder/jvmtest/external/
14:56:30       [exec] #####################################################
14:56:30       [exec] 
14:56:30       [exec] We trust you have received the usual lecture from the local System
14:56:30       [exec] Administrator. It usually boils down to these three things:
14:56:30       [exec] 
14:56:30       [exec]     #1) Respect the privacy of others.
14:56:30       [exec]     #2) Think before you type.
14:56:30       [exec]     #3) With great power comes great responsibility.
14:56:30       [exec] 
14:56:30       [exec] sudo: no tty present and no askpass program specified
14:56:30  
14:56:30  BUILD FAILED

[sxa]

@smlambert Are these tests currently expected to be in a good state?

[smlambert]

I am not sure they have been triaged since this PR landed adoptium/aqa-tests#5460 (which seems to prefer podman if both are found on machines).

[sxa]

@judovana Could your PR referenced in the previous comment have prevented the external tests from running when only docker is available? It may need some more testing based on what I'm seeing in the runs above.

[judovana]

hi! I was not touching the decision tree, whether the test should run or not. From my PR, the sudo should not have been used at all, unless user literally forced it. So I guess the original call before the PR was "sudo docker..." and was failing anyway (?). I thought I adapted all calls except criu test, but I could have missed some. In meantime, also the adoptium/aqa-tests#5553 landed, and that was even more intrusive change.

From, reading the PRs, I doubt I have "prevented the external tests from running when only docker is available" but I could have break them in unpredicted way.

I have never run whole external tests as one batch, because even before both adoptium/aqa-tests#5553 and adoptium/aqa-tests#5460 they were failing. I found several combinations which worked on most jdks (lucene, jacoco and few other) and tested on those. I'm going to map what external tests runs/pass/fails on various docker/podman/jdk8/11/17/21/ubuntu/fedora/rhels . it will take some time, but as result, should be healthy external subset (adoptium/aqa-tests#5575 (comment))

[sxa]

Thanks for the update. We can figure a look next week ... Time to relax for the weekend I feel!