System tests not running correctly on test-macstadium-macos1015-x64-1

[lumpfish]

sanity.system or extensed.system tests fail whenever test-macstadium-macos1015-x64-1 is selected as the test machine.
Invocation of the mail stf.pl script fails with message like:

01:24:48  Can't open perl script "/Users/jenkins/workspace/Test_openjdk11_j9_sanity.system_x86-64_mac/openjdk-tests/TKG/../../jvmtest/system/mauveLoadTest/..//stf/stf.core/scripts/stf.pl": No such file or directory

More output around this point of failure:

01:24:48  ===============================================
01:24:48  Running test MauveMultiThreadLoadTest_0 ...
01:24:48  ===============================================
01:24:48  MauveMultiThreadLoadTest_0 Start Time: Sun Dec  6 01:24:48 2020 Epoch Time (ms): 1607217888563
01:24:48  "/Users/jenkins/workspace/Test_openjdk11_j9_sanity.system_x86-64_mac/openjdkbinary/j2sdk-image/Contents/Home/bin/..//bin/java" -Xshareclasses:destroyAll; "/Users/jenkins/workspace/Test_openjdk11_j9_sanity.system_x86-64_mac/openjdkbinary/j2sdk-image/Contents/Home/bin/..//bin/java" -Xshareclasses:groupAccess,destroyAll; echo "cache cleanup done";
01:24:48  JVMSHRC005I No shared class caches available
01:24:48  JVMSHRC005I No shared class caches available
01:24:48  cache cleanup done
01:24:48  variation: NoOptions
01:24:48  JVM_OPTIONS:  
01:24:48  { itercnt=1; \
01:24:48  	mkdir -p "/Users/jenkins/workspace/Test_openjdk11_j9_sanity.system_x86-64_mac/openjdk-tests/TKG/../TKG/test_output_1607217859648/MauveMultiThreadLoadTest_0"; \
01:24:48  	cd "/Users/jenkins/workspace/Test_openjdk11_j9_sanity.system_x86-64_mac/openjdk-tests/TKG/../TKG/test_output_1607217859648/MauveMultiThreadLoadTest_0"; \
01:24:48  	perl /Users/jenkins/workspace/Test_openjdk11_j9_sanity.system_x86-64_mac/openjdk-tests/TKG/../../jvmtest/system/mauveLoadTest/..//stf/stf.core/scripts/stf.pl -test-root="/Users/jenkins/workspace/Test_openjdk11_j9_sanity.system_x86-64_mac/openjdk-tests/TKG/../../jvmtest/system/mauveLoadTest/..//stf;/Users/jenkins/workspace/Test_openjdk11_j9_sanity.system_x86-64_mac/openjdk-tests/TKG/../../jvmtest/system/mauveLoadTest/..//openjdk-systemtest;/Users/jenkins/workspace/Test_openjdk11_j9_sanity.system_x86-64_mac/openjdk-tests/TKG/../../jvmtest/system/mauveLoadTest/..//openj9-systemtest" -systemtest-prereqs="/Users/jenkins/workspace/Test_openjdk11_j9_sanity.system_x86-64_mac/openjdk-tests/TKG/../../jvmtest/system/mauveLoadTest/..//systemtest_prereqs" -java-args=' ' -results-root="/Users/jenkins/workspace/Test_openjdk11_j9_sanity.system_x86-64_mac/openjdk-tests/TKG/../TKG/test_output_1607217859648/MauveMultiThreadLoadTest_0" -test=MauveMultiThreadLoadTest; \
01:24:48  	if [ $? -eq 0 ] ; then echo ""; echo "MauveMultiThreadLoadTest_0""_PASSED"; echo ""; cd /Users/jenkins/workspace/Test_openjdk11_j9_sanity.system_x86-64_mac/openjdk-tests/TKG/..;  else echo ""; echo "MauveMultiThreadLoadTest_0""_FAILED"; echo ""; fi; } 2>&1 | tee -a "/Users/jenkins/workspace/Test_openjdk11_j9_sanity.system_x86-64_mac/openjdk-tests/TKG/../TKG/test_output_1607217859648/TestTargetResult";
01:24:48  Can't open perl script "/Users/jenkins/workspace/Test_openjdk11_j9_sanity.system_x86-64_mac/openjdk-tests/TKG/../../jvmtest/system/mauveLoadTest/..//stf/stf.core/scripts/stf.pl": No such file or directory
01:24:48  
01:24:48  MauveMultiThreadLoadTest_0_FAILED

The tests run fine on other mac tet machines - e.g.

Pass: test-macstadium-macos1013-x64-1
https://ci.adoptopenjdk.net/job/Test_openjdk11_j9_extended.system_x86-64_mac/417/

Pass: test-macstadium-macos1014-x64-3
https://ci.adoptopenjdk.net/job/Test_openjdk11_j9_extended.system_x86-64_mac/418/

Fail: test-macstadium-macos1015-x64-1
https://ci.adoptopenjdk.net/job/Test_openjdk11_j9_extended.system_x86-64_mac/415/
https://ci.adoptopenjdk.net/job/Test_openjdk11_j9_extended.system_x86-64_mac/416/

The error message implies that the stf.pl script does not exist. The test material build stage appears to run OK, but I do not see the common_dist target (which looks like it copies the build output to the /jvmtest directory being run where the tests fail.

This is the common_dist target output from a passing test:

00:48:03  common_dist:
00:48:03       [copy] Copying 330 files to /Users/jenkins/workspace/Test_openjdk11_j9_extended.system_x86-64_mac/jvmtest/system/stf
00:48:04       [copy] Copying 1285 files to /Users/jenkins/workspace/Test_openjdk11_j9_extended.system_x86-64_mac/jvmtest/system/openjdk-systemtest
00:48:05       [copy] Copying 292 files to /Users/jenkins/workspace/Test_openjdk11_j9_extended.system_x86-64_mac/jvmtest/system/openj9-systemtest
00:48:05       [copy] Copying 9 files to /Users/jenkins/workspace/Test_openjdk11_j9_extended.system_x86-64_mac/jvmtest/system/systemtest_prereqs

@Mesbah-Alam - any ideas why common_dist might not be being executed? Does that look like the underlying issue?

[Mesbah-Alam]

The first problem in this failing build seems to be the following compilation error:

20:27:56       [exec] gcc -m64 -dynamiclib -o /Users/jenkins/workspace/Test_openjdk11_j9_extended.system_x86-64_mac/openjdk-tests/system/openjdk-systemtest/openjdk.test.modularity/bin/tests/com.test.jlink/native/bin/osx_x86-64/libJniTest.dylib /Users/jenkins/workspace/Test_openjdk11_j9_extended.system_x86-64_mac/openjdk-tests/system/openjdk-systemtest/openjdk.test.modularity/bin/tests/com.test.jlink/native/lib/osx_x86-64/JniTest.o
20:27:56       [exec] ld: unsupported tapi file type '!tapi-tbd' in YAML file '/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/lib/libSystem.tbd' for architecture x86_64
20:27:56       [exec] clang: error: linker command failed with exit code 1 (use -v to see invocation)
20:27:56       [exec] make[2]: *** [/Users/jenkins/workspace/Test_openjdk11_j9_extended.system_x86-64_mac/openjdk-tests/system/openjdk-systemtest/openjdk.test.modularity/bin/tests/com.test.jlink/native/bin/osx_x86-64/libJniTest.dylib] Error 1

I can reproduce the issue in a Grinder run on the same machine too.

The same test material obviously compiles fine in the passing build.

Is this a machine issue in test-macstadium-macos1015-x64-1?

@sxa - could you please check if there is any difference between the GCC installed ontest-macstadium-macos1015-x64-1 and test-macstadium-macos1014-x64-3 (or test-macstadium-macos1013-x64-1)?

[sxa]

Not sure on the machine setup of that one as I wasn't involved in it, but for reference we have an infrastructure issue at adoptium/infrastructure#1673 for this too

[smlambert]

So there are 2 infra issues (adoptium/infrastructure#1777 and adoptium/infrastructure#1673) and 2 test issues (this one and #2133) tracking the same problem, I will close the test issues as it is specifically an infra issue on a single machine.