Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added feature to download boot jdk on demand #3725

Merged
merged 18 commits into from
May 9, 2024
Merged

Added feature to download boot jdk on demand #3725

Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
a392a92
Added feature to download boot jdk on demand
judovana Mar 23, 2024
2747543
removed unused variabble
judovana Mar 25, 2024
6e6bbad
Refactor also for windows
judovana Mar 26, 2024
a60a4f7
Changed mask for finding the jdk dir
judovana Mar 26, 2024
cb770b3
Added small readme to the downloader.sh
judovana Mar 27, 2024
92b5dcb
More quoting to make bash lint happy
judovana Apr 1, 2024
280a8eb
Update sbin/prepareWorkspace.sh
judovana Apr 1, 2024
58fc2f9
Finished wording of comment about downloaders.sh
judovana Apr 1, 2024
6c57490
Mentioned download on -J/--jdk-boot-dir man line
judovana Apr 1, 2024
10bc5b5
Update build-farm/platform-specific-configurations/downloaders.sh
judovana Apr 1, 2024
661f1bc
relying on $bootDir being set - shellcheck disable=SC2154
judovana Apr 3, 2024
763c02b
Fixed coyrigth header
judovana Apr 3, 2024
00b599c
No longer using global bootDir
judovana Apr 3, 2024
2963f89
Merge branch 'master' into downloadBootJdks
karianna Apr 5, 2024
60e60f2
Merge branch 'master' into downloadBootJdks
karianna Apr 23, 2024
78f6c94
Merge branch 'master' into downloadBootJdks
karianna May 7, 2024
47f9cf2
moved downloaders.sh to sbin/common/
judovana May 7, 2024
9482b54
Merge branch 'master' into downloadBootJdks
karianna May 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
153 changes: 153 additions & 0 deletions build-farm/platform-specific-configurations/downloaders.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,153 @@
#!/bin/bash
# shellcheck disable=SC1091

################################################################################
judovana marked this conversation as resolved.
Show resolved Hide resolved
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
################################################################################

####################################################################################
# This file is gathering all download boot jdk functions, so they can b reused later
# On long run, the methods - due theirs similarity - should converge to one
####################################################################################

# This function switches logic based on the (supported) os
# It is currently relying on $bootDir being set. This should be fixed later
function downloadBootJDK() {
judovana marked this conversation as resolved.
Show resolved Hide resolved
if uname -o | grep -i -e Linux ; then
downloadLinuxBootJDK "$@"
elif uname -o | grep -i -e Cygwin -e Windows ; then
downloadWindowsBootJDK "$@"
else
echo "Unsupported platfrom for direct download of boot jdk: $(uname -m) $(uname -o)"
exit 1
fi
}

function downloadLinuxBootJDK() {
ARCH=$1
VER=$2
export downloadArch
case "$ARCH" in
"riscv64") downloadArch="$NATIVE_API_ARCH";;
*) downloadArch="$ARCH";;
esac
releaseType="ga"
vendor="eclipse"
apiUrlTemplate="https://api.adoptium.net/v3/binary/latest/\${VER}/\${releaseType}/linux/\${downloadArch}/jdk/hotspot/normal/\${vendor}"
apiURL=$(eval echo "${apiUrlTemplate}")
echo "Downloading GA release of boot JDK version ${VER} from ${apiURL}"
# make-adopt-build-farm.sh has 'set -e'. We need to disable that for
# the fallback mechanism, as downloading of the GA binary might fail.
set +e
curl -L -o bootjdk.tar.gz "${apiURL}"
apiSigURL=$(curl -v "${apiURL}" 2>&1 | tr -d \\r | awk '/^< Location:/{print $3 ".sig"}')
if ! grep "No releases match the request" bootjdk.tar.gz; then
curl -L -o bootjdk.tar.gz.sig "${apiSigURL}"
gpg --keyserver keyserver.ubuntu.com --recv-keys 3B04D753C9050D9A5D343F39843C48A565F8F04B
echo -e "5\ny\n" | gpg --batch --command-fd 0 --expert --edit-key 3B04D753C9050D9A5D343F39843C48A565F8F04B trust;
gpg --verify bootjdk.tar.gz.sig bootjdk.tar.gz || exit 1
# shellcheck disable=SC2154
mkdir "$bootDir"
# shellcheck disable=SC2154
tar xpzf bootjdk.tar.gz --strip-components=1 -C "$bootDir"
set -e
else
# We must be a JDK HEAD build for which no boot JDK exists other than
# nightlies?
echo "Downloading GA release of boot JDK version ${VER} failed."
# shellcheck disable=SC2034
releaseType="ea"
# shellcheck disable=SC2034
vendor="adoptium"
apiURL=$(eval echo ${apiUrlTemplate})
echo "Attempting to download EA release of boot JDK version ${VER} from ${apiURL}"
set +e
curl -L -o bootjdk.tar.gz "${apiURL}"
if ! grep "No releases match the request" bootjdk.tar.gz; then
judovana marked this conversation as resolved.
Show resolved Hide resolved
apiSigURL=$(curl -v "${apiURL}" 2>&1 | tr -d \\r | awk '/^< Location:/{print $3 ".sig"}')
curl -L -o bootjdk.tar.gz.sig "${apiSigURL}"
gpg --keyserver keyserver.ubuntu.com --recv-keys 3B04D753C9050D9A5D343F39843C48A565F8F04B
echo -e "5\ny\n" | gpg --batch --command-fd 0 --expert --edit-key 3B04D753C9050D9A5D343F39843C48A565F8F04B trust;
gpg --verify bootjdk.tar.gz.sig bootjdk.tar.gz || exit 1
# shellcheck disable=SC2154
mkdir "$bootDir"
# shellcheck disable=SC2154
tar xpzf bootjdk.tar.gz --strip-components=1 -C "$bootDir"
else
# If no binaries are available then try from adoptopenjdk
echo "Downloading Temurin release of boot JDK version ${VER} failed."
# shellcheck disable=SC2034
releaseType="ga"
# shellcheck disable=SC2034
vendor="adoptium"
apiURL=$(eval echo ${apiUrlTemplate})
echo "Attempting to download GA release of boot JDK version ${VER} from ${apiURL}"
curl -L "${apiURL}" | tar xpzf - --strip-components=1 -C "$bootDir"
fi
fi
}

function downloadWindowsBootJDK() {
ARCHITECTURE="${1}"
VER=${2}
# This is needed to convert x86-32 to x32 which is what the API uses
export downloadArch
case "$ARCHITECTURE" in
"x86-32") downloadArch="x32";;
"aarch64") downloadArch="x64";;
*) downloadArch="$ARCHITECTURE";;
esac
releaseType="ga"
vendor="eclipse"
api="adoptium"
apiUrlTemplate="https://api.\${api}.net/v3/binary/latest/\${VER}/\${releaseType}/windows/\${downloadArch}/jdk/hotspot/normal/\${vendor}"
apiURL=$(eval echo ${apiUrlTemplate})
echo "Downloading GA release of boot JDK version ${VER} from ${apiURL}"
# make-adopt-build-farm.sh has 'set -e'. We need to disable that for
# the fallback mechanism, as downloading of the GA binary might fail
set +e
wget -q "${apiURL}" -O openjdk.zip
retVal=$?
set -e
if [ $retVal -ne 0 ]; then
# We must be a JDK HEAD build for which no boot JDK exists other than
# nightlies?
echo "Downloading GA release of boot JDK version ${VER} failed."
# shellcheck disable=SC2034
releaseType="ea"
# shellcheck disable=SC2034
vendor="adoptium"
apiURL=$(eval echo ${apiUrlTemplate})
echo "Attempting to download EA release of boot JDK version ${VER} from ${apiURL}"
set +e
wget -q "${apiURL}" -O openjdk.zip
retVal=$?
set -e
if [ $retVal -ne 0 ]; then
# If no binaries are available then try from adoptopenjdk
echo "Downloading Temurin release of boot JDK version ${VER} failed."
# shellcheck disable=SC2034
releaseType="ga"
# shellcheck disable=SC2034
vendor="adoptopenjdk"
# shellcheck disable=SC2034
api="adoptopenjdk"
apiURL=$(eval echo ${apiUrlTemplate})
echo "Attempting to download GA release of boot JDK version ${VER} from ${apiURL}"
wget -q "${apiURL}" -O openjdk.zip
fi
fi
unzip -q openjdk.zip
# shellcheck disable=SC2154
mv "$(ls -d jdk*"${VER}"*)" "$bootDir"
}
64 changes: 2 additions & 62 deletions build-farm/platform-specific-configurations/linux.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
# shellcheck source=sbin/common/constants.sh
source "$SCRIPT_DIR/../../sbin/common/constants.sh"
source "$SCRIPT_DIR/downloaders.sh"

if [[ "$JAVA_FEATURE_VERSION" -ge 21 ]]; then
# jdk-21+ uses "bundled" FreeType
Expand Down Expand Up @@ -143,67 +144,6 @@ function setCrossCompilerEnvironment()
fi
}

function downloadBootJDK()
{
ARCH=$1
VER=$2
export downloadArch
case "$ARCH" in
"riscv64") downloadArch="$NATIVE_API_ARCH";;
*) downloadArch="$ARCH";;
esac
releaseType="ga"
vendor="eclipse"
apiUrlTemplate="https://api.adoptium.net/v3/binary/latest/\${VER}/\${releaseType}/linux/\${downloadArch}/jdk/hotspot/normal/\${vendor}"
apiURL=$(eval echo "${apiUrlTemplate}")
echo "Downloading GA release of boot JDK version ${VER} from ${apiURL}"
# make-adopt-build-farm.sh has 'set -e'. We need to disable that for
# the fallback mechanism, as downloading of the GA binary might fail.
set +e
curl -L -o bootjdk.tar.gz "${apiURL}"
apiSigURL=$(curl -v "${apiURL}" 2>&1 | tr -d \\r | awk '/^< Location:/{print $3 ".sig"}')
if ! grep "No releases match the request" bootjdk.tar.gz; then
curl -L -o bootjdk.tar.gz.sig "${apiSigURL}"
gpg --keyserver keyserver.ubuntu.com --recv-keys 3B04D753C9050D9A5D343F39843C48A565F8F04B
echo -e "5\ny\n" | gpg --batch --command-fd 0 --expert --edit-key 3B04D753C9050D9A5D343F39843C48A565F8F04B trust;
gpg --verify bootjdk.tar.gz.sig bootjdk.tar.gz || exit 1
mkdir "$bootDir"
tar xpzf bootjdk.tar.gz --strip-components=1 -C "$bootDir"
set -e
else
# We must be a JDK HEAD build for which no boot JDK exists other than
# nightlies?
echo "Downloading GA release of boot JDK version ${VER} failed."
# shellcheck disable=SC2034
releaseType="ea"
# shellcheck disable=SC2034
vendor="adoptium"
apiURL=$(eval echo ${apiUrlTemplate})
echo "Attempting to download EA release of boot JDK version ${VER} from ${apiURL}"
set +e
curl -L -o bootjdk.tar.gz "${apiURL}"
if ! grep "No releases match the request" bootjdk.tar.gz; then
apiSigURL=$(curl -v "${apiURL}" 2>&1 | tr -d \\r | awk '/^< Location:/{print $3 ".sig"}')
curl -L -o bootjdk.tar.gz.sig "${apiSigURL}"
gpg --keyserver keyserver.ubuntu.com --recv-keys 3B04D753C9050D9A5D343F39843C48A565F8F04B
echo -e "5\ny\n" | gpg --batch --command-fd 0 --expert --edit-key 3B04D753C9050D9A5D343F39843C48A565F8F04B trust;
gpg --verify bootjdk.tar.gz.sig bootjdk.tar.gz || exit 1
mkdir "$bootDir"
tar xpzf bootjdk.tar.gz --strip-components=1 -C "$bootDir"
else
# If no binaries are available then try from adoptopenjdk
echo "Downloading Temurin release of boot JDK version ${VER} failed."
# shellcheck disable=SC2034
releaseType="ga"
# shellcheck disable=SC2034
vendor="adoptium"
apiURL=$(eval echo ${apiUrlTemplate})
echo "Attempting to download GA release of boot JDK version ${VER} from ${apiURL}"
curl -L "${apiURL}" | tar xpzf - --strip-components=1 -C "$bootDir"
fi
fi
}

if [ "${ARCHITECTURE}" == "x64" ]
then
export PATH=/opt/rh/devtoolset-2/root/usr/bin:$PATH
Expand Down Expand Up @@ -314,7 +254,7 @@ if [ ! -d "$(eval echo "\$$BOOT_JDK_VARIABLE")" ]; then
# shellcheck disable=SC2140
export "${BOOT_JDK_VARIABLE}"="/usr/lib/jvm/jdk-${JDK_BOOT_VERSION}"
elif [ "$JDK_BOOT_VERSION" -ge 8 ]; then # Adoptium has no build pre-8
downloadBootJDK "${ARCHITECTURE}" "${JDK_BOOT_VERSION}"
downloadLinuxBootJDK "${ARCHITECTURE}" "${JDK_BOOT_VERSION}"
fi
fi
fi
Expand Down
51 changes: 2 additions & 49 deletions build-farm/platform-specific-configurations/windows.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
# shellcheck source=sbin/common/constants.sh
source "$SCRIPT_DIR/../../sbin/common/constants.sh"
source "$SCRIPT_DIR/downloaders.sh"

export ANT_HOME=/cygdrive/C/Projects/OpenJDK/apache-ant-1.10.1
export DRAGONWELL8_BOOTSTRAP=/cygdrive/C/openjdk/dragonwell-bootstrap/jdk8u272-ga
Expand Down Expand Up @@ -43,55 +44,7 @@ if [ ! -d "$(eval echo "\$$BOOT_JDK_VARIABLE")" ]; then
# shellcheck disable=SC2140
export "${BOOT_JDK_VARIABLE}"="/cygdrive/c/openjdk/jdk-${JDK_BOOT_VERSION}"
elif [ "$JDK_BOOT_VERSION" -ge 8 ]; then # Adoptium has no build pre-8
# This is needed to convert x86-32 to x32 which is what the API uses
export downloadArch
case "$ARCHITECTURE" in
"x86-32") downloadArch="x32";;
"aarch64") downloadArch="x64";;
*) downloadArch="$ARCHITECTURE";;
esac
releaseType="ga"
vendor="eclipse"
api="adoptium"
apiUrlTemplate="https://api.\${api}.net/v3/binary/latest/\${JDK_BOOT_VERSION}/\${releaseType}/windows/\${downloadArch}/jdk/hotspot/normal/\${vendor}"
apiURL=$(eval echo ${apiUrlTemplate})
echo "Downloading GA release of boot JDK version ${JDK_BOOT_VERSION} from ${apiURL}"
# make-adopt-build-farm.sh has 'set -e'. We need to disable that for
# the fallback mechanism, as downloading of the GA binary might fail
set +e
wget -q "${apiURL}" -O openjdk.zip
retVal=$?
set -e
if [ $retVal -ne 0 ]; then
# We must be a JDK HEAD build for which no boot JDK exists other than
# nightlies?
echo "Downloading GA release of boot JDK version ${JDK_BOOT_VERSION} failed."
# shellcheck disable=SC2034
releaseType="ea"
# shellcheck disable=SC2034
vendor="adoptium"
apiURL=$(eval echo ${apiUrlTemplate})
echo "Attempting to download EA release of boot JDK version ${JDK_BOOT_VERSION} from ${apiURL}"
set +e
wget -q "${apiURL}" -O openjdk.zip
retVal=$?
set -e
if [ $retVal -ne 0 ]; then
# If no binaries are available then try from adoptopenjdk
echo "Downloading Temurin release of boot JDK version ${JDK_BOOT_VERSION} failed."
# shellcheck disable=SC2034
releaseType="ga"
# shellcheck disable=SC2034
vendor="adoptopenjdk"
# shellcheck disable=SC2034
api="adoptopenjdk"
apiURL=$(eval echo ${apiUrlTemplate})
echo "Attempting to download GA release of boot JDK version ${JDK_BOOT_VERSION} from ${apiURL}"
wget -q "${apiURL}" -O openjdk.zip
fi
fi
unzip -q openjdk.zip
mv "$(ls -d jdk-"${JDK_BOOT_VERSION}"*)" "$bootDir"
downloadWindowsBootJDK "${ARCHITECTURE}" "${JDK_BOOT_VERSION}"
fi
fi
fi
Expand Down
6 changes: 4 additions & 2 deletions makejdk-any-platform.1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,11 +113,13 @@ print this help.
.BR \-i ", " \-\-ignore-container
ignore the existing docker container if you have one already.
.TP
.BR \-J ", " \-\-jdk-boot-dir " " \fI<jdk_boot_dir>\fR
.BR \-J ", " \-\-jdk-boot-dir " " \fI<jdk_boot_dir>\fR "; " \-J ", " \-\-jdk-boot-dir " " \fIdownload\fR
specify the JDK boot dir.
For reference, OpenJDK needs the previous version of a JDK in order to build
itself. You should select the path to a JDK install that is N-1 versions below
the one you are trying to build.
the one you are trying to build. On some platforms, you can use magic keyword
of `download`, and boot jdk will be downloaded for you in best attempt. It will
judovana marked this conversation as resolved.
Show resolved Hide resolved
be reused as long as you keep workspace or clean libs.
.TP
.BR \-k ", " \-\-keep
if using docker, keep the container after the build.
Expand Down
21 changes: 19 additions & 2 deletions sbin/prepareWorkspace.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -623,15 +623,32 @@ downloadDevkit() {
fi
}

# Download all of the dependencies for OpenJDK (Alsa, FreeType etc.)
downloadBootJdkIfNeeded () {
if [[ "${BUILD_CONFIG[JDK_BOOT_DIR]}" == "download" ]]; then
# The bootDir is used by downloaders.sh methods; it would be nice to change it in future so the download method is unified
bootDir="${BUILD_CONFIG[WORKSPACE_DIR]}/downloaded-boot-jdk-${BUILD_CONFIG[OPENJDK_FEATURE_NUMBER]}"
if [ -e "$bootDir" ] ; then
echo "Reusing $bootDir"
else
source "$SCRIPT_DIR/../build-farm/platform-specific-configurations/downloaders.sh"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Architecturally I'm not a fan of this. The stuff in the root of this repository and sbin is supposed to be fully independent. The bits in build-farm is for the adoptium build processes and configures the environment for our setup and wraps around the other stuff. Calling out from sbin to platform-specific-configurations is therefore not ideal.

I like the idea of the option to specify a boot JDK to download, but I think this implementation will need some refinement first.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be quite strightforward to move downloaders.sh next to common.sh and refactor them so the $bootDir is not missused.

I will elaborate on the bootDir as I really do not like it. Do you have any prefferences on downloaders.sh location?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm undecided but marginally leaning towards sbin or sbin/common. WDYT @andrew-m-leonard ?

Copy link
Contributor Author

@judovana judovana Apr 3, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Architecturally I'm not a fan of this. The stuff in the root of this repository and sbin is supposed to be fully independent. The bits in build-farm is for the adoptium build processes and configures the environment for our

Actually there is an misunderstanding, caused by bootJdk variable name clash. I will rework it so it is no longer there.

The if [ -e "$bootDir" ] ; then is independent local check, if boot jdk was already downloaded, so it is not downloaded again.

Pleae advice on downloaders.sh location

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done 00b599c

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmmh. I would probably leave the file in build-farm. Just to be sure, that it is clear, where the logic is borrowed from.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm undecided but marginally leaning towards sbin or sbin/common. WDYT @andrew-m-leonard ?

@sxa yes I agree with you, looking at this again I feel uncomfortable that "sbin" code is calling out to "build-farm", as this could lead code-creep in the future architecturally in the wrong direction.
@judovana We should move the downloaders.sh to sbin/common, which is the common place that both sbin and build-farm pull "common" things from please.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, Thanx a lot, on it!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done. Tested on win x86, el8 aarch64 and f38 x64 aganst jdk8,17 and 21.

echo "Downloading to $bootDir"
downloadBootJDK "$(uname -m)" "${BUILD_CONFIG[OPENJDK_FEATURE_NUMBER]}"
fi
BUILD_CONFIG[JDK_BOOT_DIR]="${bootDir}"
fi
}

# Download all of the dependencies for OpenJDK (Alsa, FreeType, boot-jdk etc.)
downloadingRequiredDependencies() {
if [[ "${BUILD_CONFIG[CLEAN_LIBS]}" == "true" ]]; then
rm -rf "${BUILD_CONFIG[WORKSPACE_DIR]}/libs/freetype" || true

rm -rf "${BUILD_CONFIG[WORKSPACE_DIR]}/${BUILD_CONFIG[WORKING_DIR]}/installedalsa" || true
rm -rf "${BUILD_CONFIG[WORKSPACE_DIR]}/${BUILD_CONFIG[WORKING_DIR]}/installedfreetype" || true
rm -rf "${BUILD_CONFIG[WORKSPACE_DIR]}/downloaded-boot-jdk-${BUILD_CONFIG[OPENJDK_FEATURE_NUMBER]}" || true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm inclined to suggest leaving this in place as a cache for anyone using our scripts since the JDK is quite a big download, but I'm also ok with this.

I was going to comment on the fact that || true is unnecessary with rm -f but sine that's also in the previous three lines I'll ignore it ;-)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, and I'm keeping it. It will be removed only if if [[ "${BUILD_CONFIG[CLEAN_LIBS]}" == "true" ]]; then
oook? Yah, I was wondering on the double passig rf || true, but left in the style!

fi

downloadBootJdkIfNeeded

mkdir -p "${BUILD_CONFIG[WORKSPACE_DIR]}/libs/" || exit
cd "${BUILD_CONFIG[WORKSPACE_DIR]}/libs/" || exit

Expand Down
Loading