Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable Hybrid test cases in premerge/nightly CIs [databricks] #11906

Draft
wants to merge 36 commits into
base: branch-25.02
Choose a base branch
from
Draft

Enable Hybrid test cases in premerge/nightly CIs [databricks] #11906

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
65de010
Merge C2C code to main
Nov 12, 2024
e6cede2
Update the dependencies in pom.xml
Nov 13, 2024
1e4fc13
revert BD velox hdfs code
Nov 15, 2024
46e19df
fit codes into the new HybridScan hierarchy
sperlingxx Nov 18, 2024
4f2a4d6
refine QueryPlan, RapidsMeta and test suites for HybridScan
sperlingxx Nov 20, 2024
4d52f90
Integrate Hybrid plugin; update IT
Nov 25, 2024
c82eb29
Make Hybrid jar provoided scope; Update shim to only applicable for S…
Dec 4, 2024
65b585a
Fix comments
Dec 4, 2024
d214739
Code comment update, a minor change
Dec 5, 2024
e0f1e3b
Fix shim logic
Dec 5, 2024
6331ab8
Fix shims: build for all shims, but report error when Spark is CDH or…
Dec 6, 2024
b1b8481
Remove useless shim code
Dec 9, 2024
dbae63f
IT: add tests for decimal types
Dec 9, 2024
5e972d6
Add checks for Java/Scala version, only supports Java 1.8 and Scala 2.12
Dec 9, 2024
c6fa249
Check datasource is v1
Dec 10, 2024
e95e7cc
Update test case: skip if runtime Spark is Databricks
Dec 11, 2024
092dab8
Update Hybrid Config doc: not all Spark versions are fully tested, on…
Dec 11, 2024
519f33c
Merge branch 'branch-25.02' into merge-c2c
Dec 11, 2024
b3b6f80
some refinement
sperlingxx Dec 13, 2024
f0921a4
fix tests && unsupported types
sperlingxx Dec 17, 2024
dd5d8f9
Add doc for Hybrid execution feature
Dec 23, 2024
6149589
Update doc
Dec 24, 2024
114b93a
Check Hybrid jar in executor
Dec 24, 2024
36d3cdf
Update hybrid-execution.md
winningsix Dec 25, 2024
5bfd763
Remove check for Java versions
Dec 25, 2024
275fa3d
Fix scala 2.13 check failure
Dec 25, 2024
cbb5609
Fix for Scala 2.13 building
Dec 30, 2024
c1df7c4
Fix: specify default value for loadBackend to avoid exception
Jan 13, 2025
99602c5
Update Copyright to add new year 2025
Jan 13, 2025
3d3b172
Fix Databricks building
Jan 14, 2025
d718d8c
Minor format change
Jan 14, 2025
4e79c2b
Merge branch 'branch-25.02' into merge-c2c
Jan 14, 2025
aae45b9
Add shim 354 for Hybrid feature
Jan 14, 2025
4fd5fdb
Fix Databricks building
Jan 14, 2025
fd2b513
Enable Hybrid cases in premerge/nightly CIs
Dec 25, 2024
7b87caa
Add doc for how to run Hybrid feature Python testg cases
Jan 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions jenkins/hybrid_execution.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
#!/bin/bash
Copy link
Collaborator

@pxLi pxLi Jan 15, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's better to make this script for preparing artifacts only and the actual caller place to run test command directly,
or at least separate the func of artifacts preparation from the actual test call func.

sth like

prepare() {
    # detect the os,JVM, etc
    echo "success" # or "skip/fail/or actual reason why this is discontinued" which is non-success to skip run_test

    # the func still errors out in case that it passes the env detection but fails to do the preparation works like download error
}

run_test() {
    # run the case
}

# from caller
# source the script
result=$(prepare)

if [[ "$result" == "success" ]]; then
    run_test
else
    echo "skipped test run blahblah: $result"
fi

#
# Copyright (c) 2025, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

set -ex

. jenkins/version-def.sh

test_hybrid_feature() {
echo "Run hybrid execution test cases..."

# parameters for Hybrid featrue
spark_prefix="${SPARK_VER:0:3}" # get prefix from SPARK_VER, e.g.: 3.2, 3.3 ... 3.5
GLUTEN_BUNDLE_JAR="gluten-velox-bundle-spark${spark_prefix}_2.12-ubuntu_${GLUTEN_FOR_OS}_x86_64-${GLUTEN_VERSION}.jar"
HYBRID_JAR="rapids-4-spark-hybrid_2.12-${PROJECT_TEST_VER}.jar"
GLUTEN_THIRD_PARTY_JAR="gluten-thirdparty-lib-${GLUTEN_VERSION}-ubuntu-${GLUTEN_FOR_OS}-x86_64.jar"

# download Gluten, Hybrid jars
mvn -B dependency:get -DgroupId=com.nvidia \
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: avoid using mvn dependency:get if possible, the IT run does not require mvn dep.
use wget/curl instead

-DartifactId=gluten-velox-bundle \
-Dversion=${GLUTEN_VERSION} \
-Dpackaging=jar \
-Dclassifier=spark${spark_prefix}_2.12-ubuntu_${GLUTEN_FOR_OS}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo ?

  • \ --> -Dclassifier=spark${spark_prefix}_2.12-ubuntu_${GLUTEN_FOR_OS} '

-Dtransitive=false \
-Ddest=/tmp/$GLUTEN_BUNDLE_JAR
mvn -B dependency:get -DgroupId=com.nvidia \
-DartifactId=rapids-4-spark-hybrid_2.12 \
-Dversion=${PROJECT_TEST_VER} \
-Dpackaging=jar \
-Dtransitive=false \
-Ddest=/tmp/$HYBRID_JAR
wget -O /tmp/${GLUTEN_THIRD_PARTY_JAR} ${MVN_URM_MIRROR}/com/nvidia/gluten-thirdparty-lib/${GLUTEN_VERSION}/${GLUTEN_THIRD_PARTY_JAR}

# run Hybrid Python tests
LOAD_HYBRID_BACKEND=1 \
HYBRID_BACKEND_JARS=/tmp/${HYBRID_JAR},/tmp/${GLUTEN_BUNDLE_JAR},/tmp/${GLUTEN_THIRD_PARTY_JAR} \
./integration_tests/run_pyspark_from_build.sh -m hybrid_test
}
6 changes: 5 additions & 1 deletion jenkins/spark-premerge-build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/bin/bash
#
# Copyright (c) 2020-2024, NVIDIA CORPORATION. All rights reserved.
# Copyright (c) 2020-2025, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand Down Expand Up @@ -109,6 +109,10 @@ mvn_verify() {
do
TZ=$tz ./integration_tests/run_pyspark_from_build.sh -m tz_sensitive_test
done

# test Hybrid feature
source "${WORKSPACE}/jenkins/hybrid_execution.sh"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how long will this test take? we may need some duration to determine if we should put it into in ci1 or ci2 stage for balancing the workloads

test_hybrid_feature
}

rapids_shuffle_smoke_test() {
Expand Down
8 changes: 7 additions & 1 deletion jenkins/spark-tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#!/bin/bash
#
# Copyright (c) 2019-2024, NVIDIA CORPORATION. All rights reserved.
# Copyright (c) 2019-2025, NVIDIA CORPORATION. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand Down Expand Up @@ -351,6 +351,12 @@ if [[ "$TEST_MODE" == "NON_UTC_TZ" ]]; then
run_non_utc_time_zone_tests
fi

# hybrid execution tests
if [[ "$TEST_MODE" == "HYBRID_EXECUTION" ]]; then
source "${WORKSPACE}/jenkins/hybrid_execution.sh"
test_hybrid_feature
fi

popd
stop-worker.sh
stop-master.sh
3 changes: 3 additions & 0 deletions jenkins/version-def.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,9 @@ for VAR in $OVERWRITE_PARAMS; do
done
IFS=$PRE_IFS

# configs for Hybrid feature
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Better use default value patten instead of hardcode, then we can overwrite these vars outside without changing this script

GLUTEN_VERSION=${GLUTEN_VERSION:-"1.2.0"}
GLUTEN_FOR_OS=${GLUTEN_FOR_OS:-"20.04"}

GLUTEN_VERSION=1.2.0
GLUTEN_FOR_OS=20.04

CUDA_CLASSIFIER=${CUDA_CLASSIFIER:-"cuda11"}
CLASSIFIER=${CLASSIFIER:-"$CUDA_CLASSIFIER"} # default as CUDA_CLASSIFIER for compatibility
Expand Down
Loading