Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added feature to download boot jdk on demand #3725

Merged
merged 18 commits into from
May 9, 2024
Merged

Added feature to download boot jdk on demand #3725

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
a392a92
Added feature to download boot jdk on demand
judovana Mar 23, 2024
2747543
removed unused variabble
judovana Mar 25, 2024
6e6bbad
Refactor also for windows
judovana Mar 26, 2024
a60a4f7
Changed mask for finding the jdk dir
judovana Mar 26, 2024
cb770b3
Added small readme to the downloader.sh
judovana Mar 27, 2024
92b5dcb
More quoting to make bash lint happy
judovana Apr 1, 2024
280a8eb
Update sbin/prepareWorkspace.sh
judovana Apr 1, 2024
58fc2f9
Finished wording of comment about downloaders.sh
judovana Apr 1, 2024
6c57490
Mentioned download on -J/--jdk-boot-dir man line
judovana Apr 1, 2024
10bc5b5
Update build-farm/platform-specific-configurations/downloaders.sh
judovana Apr 1, 2024
661f1bc
relying on $bootDir being set - shellcheck disable=SC2154
judovana Apr 3, 2024
763c02b
Fixed coyrigth header
judovana Apr 3, 2024
00b599c
No longer using global bootDir
judovana Apr 3, 2024
2963f89
Merge branch 'master' into downloadBootJdks
karianna Apr 5, 2024
60e60f2
Merge branch 'master' into downloadBootJdks
karianna Apr 23, 2024
78f6c94
Merge branch 'master' into downloadBootJdks
karianna May 7, 2024
47f9cf2
moved downloaders.sh to sbin/common/
judovana May 7, 2024
9482b54
Merge branch 'master' into downloadBootJdks
karianna May 7, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 2 additions & 62 deletions build-farm/platform-specific-configurations/linux.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
# shellcheck source=sbin/common/constants.sh
source "$SCRIPT_DIR/../../sbin/common/constants.sh"
source "$SCRIPT_DIR/../../sbin/common/downloaders.sh"

if [[ "$JAVA_FEATURE_VERSION" -ge 21 ]]; then
# jdk-21+ uses "bundled" FreeType
Expand Down Expand Up @@ -173,67 +174,6 @@ function setCrossCompilerEnvironment()
fi
}

function downloadBootJDK()
{
ARCH=$1
VER=$2
export downloadArch
case "$ARCH" in
"riscv64") downloadArch="$NATIVE_API_ARCH";;
*) downloadArch="$ARCH";;
esac
releaseType="ga"
vendor="eclipse"
apiUrlTemplate="https://api.adoptium.net/v3/binary/latest/\${VER}/\${releaseType}/linux/\${downloadArch}/jdk/hotspot/normal/\${vendor}"
apiURL=$(eval echo "${apiUrlTemplate}")
echo "Downloading GA release of boot JDK version ${VER} from ${apiURL}"
# make-adopt-build-farm.sh has 'set -e'. We need to disable that for
# the fallback mechanism, as downloading of the GA binary might fail.
set +e
curl -L -o bootjdk.tar.gz "${apiURL}"
apiSigURL=$(curl -v "${apiURL}" 2>&1 | tr -d \\r | awk '/^< Location:/{print $3 ".sig"}')
if ! grep "No releases match the request" bootjdk.tar.gz; then
curl -L -o bootjdk.tar.gz.sig "${apiSigURL}"
gpg --keyserver keyserver.ubuntu.com --recv-keys 3B04D753C9050D9A5D343F39843C48A565F8F04B
echo -e "5\ny\n" | gpg --batch --command-fd 0 --expert --edit-key 3B04D753C9050D9A5D343F39843C48A565F8F04B trust;
gpg --verify bootjdk.tar.gz.sig bootjdk.tar.gz || exit 1
mkdir "$bootDir"
tar xpzf bootjdk.tar.gz --strip-components=1 -C "$bootDir"
set -e
else
# We must be a JDK HEAD build for which no boot JDK exists other than
# nightlies?
echo "Downloading GA release of boot JDK version ${VER} failed."
# shellcheck disable=SC2034
releaseType="ea"
# shellcheck disable=SC2034
vendor="adoptium"
apiURL=$(eval echo ${apiUrlTemplate})
echo "Attempting to download EA release of boot JDK version ${VER} from ${apiURL}"
set +e
curl -L -o bootjdk.tar.gz "${apiURL}"
if ! grep "No releases match the request" bootjdk.tar.gz; then
apiSigURL=$(curl -v "${apiURL}" 2>&1 | tr -d \\r | awk '/^< Location:/{print $3 ".sig"}')
curl -L -o bootjdk.tar.gz.sig "${apiSigURL}"
gpg --keyserver keyserver.ubuntu.com --recv-keys 3B04D753C9050D9A5D343F39843C48A565F8F04B
echo -e "5\ny\n" | gpg --batch --command-fd 0 --expert --edit-key 3B04D753C9050D9A5D343F39843C48A565F8F04B trust;
gpg --verify bootjdk.tar.gz.sig bootjdk.tar.gz || exit 1
mkdir "$bootDir"
tar xpzf bootjdk.tar.gz --strip-components=1 -C "$bootDir"
else
# If no binaries are available then try from adoptopenjdk
echo "Downloading Temurin release of boot JDK version ${VER} failed."
# shellcheck disable=SC2034
releaseType="ga"
# shellcheck disable=SC2034
vendor="adoptium"
apiURL=$(eval echo ${apiUrlTemplate})
echo "Attempting to download GA release of boot JDK version ${VER} from ${apiURL}"
curl -L "${apiURL}" | tar xpzf - --strip-components=1 -C "$bootDir"
fi
fi
}

if [ "${ARCHITECTURE}" == "x64" ]
then
export PATH=/opt/rh/devtoolset-2/root/usr/bin:$PATH
Expand Down Expand Up @@ -344,7 +284,7 @@ if [ ! -d "$(eval echo "\$$BOOT_JDK_VARIABLE")" ]; then
# shellcheck disable=SC2140
export "${BOOT_JDK_VARIABLE}"="/usr/lib/jvm/jdk-${JDK_BOOT_VERSION}"
elif [ "$JDK_BOOT_VERSION" -ge 8 ]; then # Adoptium has no build pre-8
downloadBootJDK "${ARCHITECTURE}" "${JDK_BOOT_VERSION}"
downloadLinuxBootJDK "${ARCHITECTURE}" "${JDK_BOOT_VERSION}" "$bootDir"
fi
fi
fi
Expand Down
51 changes: 2 additions & 49 deletions build-farm/platform-specific-configurations/windows.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
# shellcheck source=sbin/common/constants.sh
source "$SCRIPT_DIR/../../sbin/common/constants.sh"
source "$SCRIPT_DIR/../../sbin/common/downloaders.sh"

export ANT_HOME=/cygdrive/C/Projects/OpenJDK/apache-ant-1.10.1
export DRAGONWELL8_BOOTSTRAP=/cygdrive/C/openjdk/dragonwell-bootstrap/jdk8u272-ga
Expand Down Expand Up @@ -43,55 +44,7 @@ if [ ! -d "$(eval echo "\$$BOOT_JDK_VARIABLE")" ]; then
# shellcheck disable=SC2140
export "${BOOT_JDK_VARIABLE}"="/cygdrive/c/openjdk/jdk-${JDK_BOOT_VERSION}"
elif [ "$JDK_BOOT_VERSION" -ge 8 ]; then # Adoptium has no build pre-8
# This is needed to convert x86-32 to x32 which is what the API uses
export downloadArch
case "$ARCHITECTURE" in
"x86-32") downloadArch="x32";;
"aarch64") downloadArch="x64";;
*) downloadArch="$ARCHITECTURE";;
esac
releaseType="ga"
vendor="eclipse"
api="adoptium"
apiUrlTemplate="https://api.\${api}.net/v3/binary/latest/\${JDK_BOOT_VERSION}/\${releaseType}/windows/\${downloadArch}/jdk/hotspot/normal/\${vendor}"
apiURL=$(eval echo ${apiUrlTemplate})
echo "Downloading GA release of boot JDK version ${JDK_BOOT_VERSION} from ${apiURL}"
# make-adopt-build-farm.sh has 'set -e'. We need to disable that for
# the fallback mechanism, as downloading of the GA binary might fail
set +e
wget -q "${apiURL}" -O openjdk.zip
retVal=$?
set -e
if [ $retVal -ne 0 ]; then
# We must be a JDK HEAD build for which no boot JDK exists other than
# nightlies?
echo "Downloading GA release of boot JDK version ${JDK_BOOT_VERSION} failed."
# shellcheck disable=SC2034
releaseType="ea"
# shellcheck disable=SC2034
vendor="adoptium"
apiURL=$(eval echo ${apiUrlTemplate})
echo "Attempting to download EA release of boot JDK version ${JDK_BOOT_VERSION} from ${apiURL}"
set +e
wget -q "${apiURL}" -O openjdk.zip
retVal=$?
set -e
if [ $retVal -ne 0 ]; then
# If no binaries are available then try from adoptopenjdk
echo "Downloading Temurin release of boot JDK version ${JDK_BOOT_VERSION} failed."
# shellcheck disable=SC2034
releaseType="ga"
# shellcheck disable=SC2034
vendor="adoptopenjdk"
# shellcheck disable=SC2034
api="adoptopenjdk"
apiURL=$(eval echo ${apiUrlTemplate})
echo "Attempting to download GA release of boot JDK version ${JDK_BOOT_VERSION} from ${apiURL}"
wget -q "${apiURL}" -O openjdk.zip
fi
fi
unzip -q openjdk.zip
mv "$(ls -d jdk-"${JDK_BOOT_VERSION}"*)" "$bootDir"
downloadWindowsBootJDK "${ARCHITECTURE}" "${JDK_BOOT_VERSION}" "$bootDir"
fi
fi
fi
Expand Down
6 changes: 4 additions & 2 deletions makejdk-any-platform.1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -122,11 +122,13 @@ print this help.
.BR \-i ", " \-\-ignore-container
ignore the existing docker container if you have one already.
.TP
.BR \-J ", " \-\-jdk-boot-dir " " \fI<jdk_boot_dir>\fR
.BR \-J ", " \-\-jdk-boot-dir " " \fI<jdk_boot_dir>\fR "; " \-J ", " \-\-jdk-boot-dir " " \fIdownload\fR
specify the JDK boot dir.
For reference, OpenJDK needs the previous version of a JDK in order to build
itself. You should select the path to a JDK install that is N-1 versions below
the one you are trying to build.
the one you are trying to build. On some platforms, you can use magic keyword
of `download`, and boot jdk will be downloaded for you in best attempt. It will
judovana marked this conversation as resolved.
Show resolved Hide resolved
be reused as long as you keep workspace or clean libs.
.TP
.BR \-k ", " \-\-keep
if using docker, keep the container after the build.
Expand Down
148 changes: 148 additions & 0 deletions sbin/common/downloaders.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,148 @@
#!/bin/bash
# shellcheck disable=SC1091

# ********************************************************************************
# Copyright (c) 2018 Contributors to the Eclipse Foundation
#
# See the NOTICE file(s) with this work for additional
# information regarding copyright ownership.
#
# This program and the accompanying materials are made
# available under the terms of the Apache Software License 2.0
# which is available at https://www.apache.org/licenses/LICENSE-2.0.
#
# SPDX-License-Identifier: Apache-2.0
# ********************************************************************************

####################################################################################
# This file is gathering all download boot jdk functions, so they can b reused later
# On long run, the methods - due theirs similarity - should converge to one
####################################################################################

# This function switches logic based on the (supported) os
function downloadBootJDK() {
if uname -o | grep -i -e Linux ; then
downloadLinuxBootJDK "$@"
elif uname -o | grep -i -e Cygwin -e Windows ; then
downloadWindowsBootJDK "$@"
else
echo "Unsupported platform for direct download of boot jdk: $(uname -m) $(uname -o)"
exit 1
fi
}

function downloadLinuxBootJDK() {
local ARCH="${1}"
local VER="${2}"
local bootDir="${3}"
export downloadArch
case "$ARCH" in
"riscv64") downloadArch="$NATIVE_API_ARCH";;
*) downloadArch="$ARCH";;
esac
releaseType="ga"
vendor="eclipse"
apiUrlTemplate="https://api.adoptium.net/v3/binary/latest/\${VER}/\${releaseType}/linux/\${downloadArch}/jdk/hotspot/normal/\${vendor}"
apiURL=$(eval echo "${apiUrlTemplate}")
echo "Downloading GA release of boot JDK version ${VER} from ${apiURL}"
# make-adopt-build-farm.sh has 'set -e'. We need to disable that for
# the fallback mechanism, as downloading of the GA binary might fail.
set +e
curl -L -o bootjdk.tar.gz "${apiURL}"
apiSigURL=$(curl -v "${apiURL}" 2>&1 | tr -d \\r | awk '/^< Location:/{print $3 ".sig"}')
if ! grep "No releases match the request" bootjdk.tar.gz; then
curl -L -o bootjdk.tar.gz.sig "${apiSigURL}"
gpg --keyserver keyserver.ubuntu.com --recv-keys 3B04D753C9050D9A5D343F39843C48A565F8F04B
echo -e "5\ny\n" | gpg --batch --command-fd 0 --expert --edit-key 3B04D753C9050D9A5D343F39843C48A565F8F04B trust;
gpg --verify bootjdk.tar.gz.sig bootjdk.tar.gz || exit 1
mkdir "$bootDir"
tar xpzf bootjdk.tar.gz --strip-components=1 -C "$bootDir"
set -e
else
# We must be a JDK HEAD build for which no boot JDK exists other than
# nightlies?
echo "Downloading GA release of boot JDK version ${VER} failed."
# shellcheck disable=SC2034
releaseType="ea"
# shellcheck disable=SC2034
vendor="adoptium"
apiURL=$(eval echo ${apiUrlTemplate})
echo "Attempting to download EA release of boot JDK version ${VER} from ${apiURL}"
set +e
curl -L -o bootjdk.tar.gz "${apiURL}"
if ! grep "No releases match the request" bootjdk.tar.gz; then
apiSigURL=$(curl -v "${apiURL}" 2>&1 | tr -d \\r | awk '/^< Location:/{print $3 ".sig"}')
curl -L -o bootjdk.tar.gz.sig "${apiSigURL}"
gpg --keyserver keyserver.ubuntu.com --recv-keys 3B04D753C9050D9A5D343F39843C48A565F8F04B
echo -e "5\ny\n" | gpg --batch --command-fd 0 --expert --edit-key 3B04D753C9050D9A5D343F39843C48A565F8F04B trust;
gpg --verify bootjdk.tar.gz.sig bootjdk.tar.gz || exit 1
mkdir "$bootDir"
tar xpzf bootjdk.tar.gz --strip-components=1 -C "$bootDir"
else
# If no binaries are available then try from adoptopenjdk
echo "Downloading Temurin release of boot JDK version ${VER} failed."
# shellcheck disable=SC2034
releaseType="ga"
# shellcheck disable=SC2034
vendor="adoptium"
apiURL=$(eval echo ${apiUrlTemplate})
echo "Attempting to download GA release of boot JDK version ${VER} from ${apiURL}"
curl -L "${apiURL}" | tar xpzf - --strip-components=1 -C "$bootDir"
fi
fi
}

function downloadWindowsBootJDK() {
local ARCHITECTURE="${1}"
local VER="${2}"
local bootDir="${3}"
# This is needed to convert x86-32 to x32 which is what the API uses
export downloadArch
case "$ARCHITECTURE" in
"x86-32") downloadArch="x32";;
"aarch64") downloadArch="x64";;
*) downloadArch="$ARCHITECTURE";;
esac
releaseType="ga"
vendor="eclipse"
api="adoptium"
apiUrlTemplate="https://api.\${api}.net/v3/binary/latest/\${VER}/\${releaseType}/windows/\${downloadArch}/jdk/hotspot/normal/\${vendor}"
apiURL=$(eval echo ${apiUrlTemplate})
echo "Downloading GA release of boot JDK version ${VER} from ${apiURL}"
# make-adopt-build-farm.sh has 'set -e'. We need to disable that for
# the fallback mechanism, as downloading of the GA binary might fail
set +e
wget -q "${apiURL}" -O openjdk.zip
retVal=$?
set -e
if [ $retVal -ne 0 ]; then
# We must be a JDK HEAD build for which no boot JDK exists other than
# nightlies?
echo "Downloading GA release of boot JDK version ${VER} failed."
# shellcheck disable=SC2034
releaseType="ea"
# shellcheck disable=SC2034
vendor="adoptium"
apiURL=$(eval echo ${apiUrlTemplate})
echo "Attempting to download EA release of boot JDK version ${VER} from ${apiURL}"
set +e
wget -q "${apiURL}" -O openjdk.zip
retVal=$?
set -e
if [ $retVal -ne 0 ]; then
# If no binaries are available then try from adoptopenjdk
echo "Downloading Temurin release of boot JDK version ${VER} failed."
# shellcheck disable=SC2034
releaseType="ga"
# shellcheck disable=SC2034
vendor="adoptopenjdk"
# shellcheck disable=SC2034
api="adoptopenjdk"
apiURL=$(eval echo ${apiUrlTemplate})
echo "Attempting to download GA release of boot JDK version ${VER} from ${apiURL}"
wget -q "${apiURL}" -O openjdk.zip
fi
fi
unzip -q openjdk.zip
mv "$(ls -d jdk*"${VER}"*)" "$bootDir"
}
20 changes: 18 additions & 2 deletions sbin/prepareWorkspace.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -652,15 +652,31 @@ downloadDevkit() {
fi
}

# Download all of the dependencies for OpenJDK (Alsa, FreeType etc.)
downloadBootJdkIfNeeded () {
if [[ "${BUILD_CONFIG[JDK_BOOT_DIR]}" == "download" ]]; then
local futureBootDir="${BUILD_CONFIG[WORKSPACE_DIR]}/downloaded-boot-jdk-${BUILD_CONFIG[OPENJDK_FEATURE_NUMBER]}"
if [ -e "$futureBootDir" ] ; then
echo "Reusing $futureBootDir"
else
source "$SCRIPT_DIR/common/downloaders.sh"
echo "Downloading to $futureBootDir"
downloadBootJDK "$(uname -m)" "${BUILD_CONFIG[OPENJDK_FEATURE_NUMBER]}" "${futureBootDir}"
fi
BUILD_CONFIG[JDK_BOOT_DIR]="${futureBootDir}"
fi
}

# Download all of the dependencies for OpenJDK (Alsa, FreeType, boot-jdk etc.)
downloadingRequiredDependencies() {
if [[ "${BUILD_CONFIG[CLEAN_LIBS]}" == "true" ]]; then
rm -rf "${BUILD_CONFIG[WORKSPACE_DIR]}/libs/freetype" || true

rm -rf "${BUILD_CONFIG[WORKSPACE_DIR]}/${BUILD_CONFIG[WORKING_DIR]}/installedalsa" || true
rm -rf "${BUILD_CONFIG[WORKSPACE_DIR]}/${BUILD_CONFIG[WORKING_DIR]}/installedfreetype" || true
rm -rf "${BUILD_CONFIG[WORKSPACE_DIR]}/downloaded-boot-jdk-${BUILD_CONFIG[OPENJDK_FEATURE_NUMBER]}" || true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm inclined to suggest leaving this in place as a cache for anyone using our scripts since the JDK is quite a big download, but I'm also ok with this.

I was going to comment on the fact that || true is unnecessary with rm -f but sine that's also in the previous three lines I'll ignore it ;-)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, and I'm keeping it. It will be removed only if if [[ "${BUILD_CONFIG[CLEAN_LIBS]}" == "true" ]]; then
oook? Yah, I was wondering on the double passig rf || true, but left in the style!

fi

downloadBootJdkIfNeeded

mkdir -p "${BUILD_CONFIG[WORKSPACE_DIR]}/libs/" || exit
cd "${BUILD_CONFIG[WORKSPACE_DIR]}/libs/" || exit

Expand Down
Loading