Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[2.12][fix] the Verify Notebook Has Not Restarted upgrade test #2185

Merged
merged 1 commit into from
Jan 20, 2025
Merged

[2.12][fix] the Verify Notebook Has Not Restarted upgrade test #2185

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions ods_ci/tests/Resources/Page/ODH/JupyterHub/JupyterLabLauncher.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,10 +167,9 @@
Should Match "${ls_server}" "${EMPTY}"

Get User Notebook Pod Name
[Documentation] Returns notebook pod name for given username (e.g. for user ldap-admin1 it will be jupyterhub-nb-ldap-2dadmin1)
[Documentation] Returns notebook pod name for given username (e.g. for user ldap-admin1 it will be jupyter-nb-ldap-2dadmin1-0)
[Arguments] ${username}
${safe_username}= Get Safe Username ${username}
#${notebook_pod_name}= Set Variable jupyterhub-nb-${safe_username}
${notebook_pod_name}= Set Variable jupyter-nb-${safe_username}-0
RETURN ${notebook_pod_name}

Expand All @@ -179,7 +178,7 @@
[Arguments] ${username}
${safe_username}= Get Safe Username ${username}
${notebook_cr_name}= Set Variable jupyter-nb-${safe_username}
RETURN ${notebook_cr_name}

Check warning

Code scanning / Robocop

Line is too long ({{ line_length }}/{{ allowed_length }}) Warning test

Line is too long (131/120)

Get User Notebook Pod UID
[Documentation] Returns notebook pod UID for given username and ${namespace} (e.g. for user ldap-admin1 it will be jupyter-nb-ldap-2dadmin1)
Expand Down
100 changes: 74 additions & 26 deletions ods_ci/tests/Tests/0200__rhoai_upgrade/0201__pre_upgrade.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,28 +1,31 @@
*** Settings ***
Documentation Test Suite for Upgrade testing, to be run before the upgrade
Library OpenShiftLibrary
Resource ../../Resources/RHOSi.resource
Resource ../../Resources/ODS.robot
Resource ../../Resources/Page/ODH/ODHDashboard/ODHDashboard.resource
Resource ../../Resources/Page/ODH/ODHDashboard/ODHDashboardResources.resource
Resource ../../Resources/Page/ODH/ODHDashboard/ODHModelServing.resource
Resource ../../Resources/Page/ODH/ODHDashboard/ODHDataScienceProject/DataConnections.resource
Resource ../../Resources/Page/ODH/JupyterHub/HighAvailability.robot
Resource ../../Resources/Page/ODH/ODHDashboard/ODHDataScienceProject/Projects.resource
Resource ../../Resources/Page/ODH/ODHDashboard/ODHDataScienceProject/ModelServer.resource
Resource ../../Resources/Page/ODH/AiApps/Anaconda.resource
Resource ../../Resources/Page/LoginPage.robot
Resource ../../Resources/Page/OCPLogin/OCPLogin.robot
Resource ../../Resources/Common.robot
Resource ../../Resources/Page/OCPDashboard/Pods/Pods.robot
Resource ../../Resources/Page/OCPDashboard/Builds/Builds.robot
Resource ../../Resources/Page/HybridCloudConsole/OCM.robot
Resource ../../Resources/CLI/ModelServing/modelmesh.resource
Resource ../../Resources/Page/DistributedWorkloads/DistributedWorkloads.resource
Resource ../../Resources/Page/DistributedWorkloads/WorkloadMetricsUI.resource
Suite Setup Dashboard Suite Setup
Suite Teardown RHOSi Teardown
Test Tags PreUpgrade
Documentation Test Suite for Upgrade testing, to be run before the upgrade

Library OpenShiftLibrary
Resource ../../Resources/RHOSi.resource
Resource ../../Resources/ODS.robot
Resource ../../Resources/Page/ODH/ODHDashboard/ODHDashboard.resource
Resource ../../Resources/Page/ODH/ODHDashboard/ODHDashboardResources.resource
Resource ../../Resources/Page/ODH/ODHDashboard/ODHModelServing.resource
Resource ../../Resources/Page/ODH/ODHDashboard/ODHDataScienceProject/DataConnections.resource
Resource ../../Resources/Page/ODH/JupyterHub/HighAvailability.robot
Resource ../../Resources/Page/ODH/ODHDashboard/ODHDataScienceProject/Projects.resource
Resource ../../Resources/Page/ODH/ODHDashboard/ODHDataScienceProject/ModelServer.resource
Resource ../../Resources/Page/ODH/AiApps/Anaconda.resource
Resource ../../Resources/Page/LoginPage.robot
Resource ../../Resources/Page/OCPLogin/OCPLogin.robot
Resource ../../Resources/Common.robot
Resource ../../Resources/Page/OCPDashboard/Pods/Pods.robot
Resource ../../Resources/Page/OCPDashboard/Builds/Builds.robot
Resource ../../Resources/Page/HybridCloudConsole/OCM.robot
Resource ../../Resources/CLI/ModelServing/modelmesh.resource
Resource ../../Resources/Page/DistributedWorkloads/DistributedWorkloads.resource
Resource ../../Resources/Page/DistributedWorkloads/WorkloadMetricsUI.resource

Suite Setup Upgrade Suite Setup
Suite Teardown RHOSi Teardown

Test Tags PreUpgrade


*** Variables ***
Expand All @@ -38,6 +41,9 @@
${MODEL_CREATED}= ${FALSE}
${RUNTIME_NAME}= Model Serving Test
${DW_PROJECT_CREATED}= False
${CODE} while True: import time ; time.sleep(10); print ("Hello")

Check warning

Code scanning / Robocop

The assignment sign is not consistent inside the variables section. Expected '{{ expected_sign }}' but got '{{ actual_sign }}' instead Warning test

The assignment sign is not consistent inside the variables section. Expected '=' but got '' instead
${UPGRADE_NS} upgrade

Check warning

Code scanning / Robocop

The assignment sign is not consistent inside the variables section. Expected '{{ expected_sign }}' but got '{{ actual_sign }}' instead Warning test

The assignment sign is not consistent inside the variables section. Expected '=' but got '' instead
${UPGRADE_CONFIG_MAP} upgrade-config-map

Check warning

Code scanning / Robocop

The assignment sign is not consistent inside the variables section. Expected '{{ expected_sign }}' but got '{{ actual_sign }}' instead Warning test

The assignment sign is not consistent inside the variables section. Expected '=' but got '' instead


*** Test Cases ***
Expand Down Expand Up @@ -173,12 +179,54 @@
Run Training Operator ODH Upgrade Test TestSetupSleepPytorchjob
[Teardown] Teardown Training Operator E2E Upgrade Test Suite

Long Running Jupyter Notebook
[Documentation] Launch a long running notebook before the upgrade
[Tags] Upgrade
Launch Notebook
Add And Run JupyterLab Code Cell In Active Notebook ${CODE}

# Get the notebook pod creation timestamp
${notebook_pod_name}= Get User Notebook Pod Name ${TEST_USER2.USERNAME}

Check warning

Code scanning / Robocop

The assignment sign is not consistent within the file. Expected '{{ expected_sign }}' but got '{{ actual_sign }}' instead Warning test

The assignment sign is not consistent within the file. Expected ' =' but got '=' instead
${return_code} ${ntb_creation_timestamp} = Run And Return Rc And Output
... oc get pod -n ${NOTEBOOKS_NAMESPACE} ${notebook_pod_name} --no-headers --output='custom-columns=TIMESTAMP:.metadata.creationTimestamp' # robocop: disable: line-too-long
Should Be Equal As Integers ${return_code} 0 msg=${ntb_creation_timestamp}

# Save the timestamp to the OpenShift ConfigMap so it can be used in test in the next phase
${return_code} ${cmd_output} = Run And Return Rc And Output
... oc create configmap ${UPGRADE_CONFIG_MAP} -n ${UPGRADE_NS} --from-literal=ntb_creation_timestamp=${ntb_creation_timestamp} # robocop: disable: line-too-long
Should Be Equal As Integers ${return_code} 0 msg=${cmd_output}

Close Browser


*** Keywords ***
Dashboard Suite Setup
[Documentation] Basic suite setup
Launch Notebook
[Documentation] Launch notebook for the suite
[Arguments] ${notebook_image}=minimal-notebook
... ${username}=${TEST_USER2.USERNAME}
... ${password}=${TEST_USER2.PASSWORD}
... ${auth_type}=${TEST_USER2.AUTH_TYPE}
Begin Web Test username=${username} password=${password} auth_type=${auth_type}
Launch Jupyter From RHODS Dashboard Link
Spawn Notebook With Arguments
... image=${notebook_image}
... username=${username}
... password=${password}
... auth_type=${auth_type}

Upgrade Suite Setup
[Documentation] Basic suite setup
Set Library Search Order SeleniumLibrary
RHOSi Setup
# Prepare a namespace for storing values that should be shared between different upgrade test phases
# 1. if the namespace exists already, let's remove it
${return_code} ${cmd_output} = Run And Return Rc And Output
... oc delete namespace --wait --ignore-not-found ${UPGRADE_NS}
Should Be Equal As Integers ${return_code} 0 msg=${cmd_output}
# 2. create the namespace now
${return_code} ${cmd_output} = Run And Return Rc And Output
... oc create namespace ${UPGRADE_NS}
Should Be Equal As Integers ${return_code} 0 msg=${cmd_output}

Dashboard Test Teardown
[Documentation] Basic suite teardown
Expand Down
35 changes: 11 additions & 24 deletions ods_ci/tests/Tests/0200__rhoai_upgrade/0202__during_upgrade.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,21 +12,7 @@ Library JupyterLibrary
Test Tags DuringUpgrade


*** Variables ***
${CODE} while True: import time ; time.sleep(10); print ("Hello")


*** Test Cases ***
Long Running Jupyter Notebook
[Documentation] Launch a long running notebook before the upgrade
[Tags] Upgrade
Launch Notebook
Add And Run JupyterLab Code Cell In Active Notebook ${CODE}
${return_code} ${timestamp} Run And Return Rc And Output oc get pod -n ${NOTEBOOKS_NAMESPACE} jupyter-nb-ldap-2dadmin2-0 --no-headers --output='custom-columns=TIMESTAMP:.metadata.creationTimestamp' #robocop:disable
Should Be Equal As Integers ${return_code} 0
Set Global Variable ${timestamp} #robocop: disable
Close Browser

Upgrade RHODS
[Documentation] Approve the install plan for the upgrade and make sure that upgrade has completed
[Tags] ODS-1766
Expand Down Expand Up @@ -58,17 +44,18 @@ PyTorch Image Workload Test

*** Keywords ***
Launch Notebook
[Documentation] Launch notebook for the suite
[Arguments] ${notbook_image}=minimal-notebook ${username}=${TEST_USER2.USERNAME} ${password}=${TEST_USER2.PASSWORD} ${auth_type}=${TEST_USER2.AUTH_TYPE} #robocop: disable
Begin Web Test username=${username} password=${password} auth_type=${auth_type}
Login To RHODS Dashboard ${username} ${password} ${auth_type}
Wait For RHODS Dashboard To Load
[Documentation] Launch notebook for the suite
[Arguments] ${notebook_image}=minimal-notebook
... ${username}=${TEST_USER2.USERNAME}
... ${password}=${TEST_USER2.PASSWORD}
... ${auth_type}=${TEST_USER2.AUTH_TYPE}
Begin Web Test username=${username} password=${password} auth_type=${auth_type}
Launch Jupyter From RHODS Dashboard Link
Login To Jupyterhub ${username} ${password} ${auth_type}
${authorization_required} Is Service Account Authorization Required
IF ${authorization_required} Authorize Jupyterhub Service Account
Fix Spawner Status
Spawn Notebook With Arguments image=${notbook_image} username=${username} password=${password} auth_type=${auth_type} #robocop: disable
Spawn Notebook With Arguments
... image=${notebook_image}
... username=${username}
... password=${password}
... auth_type=${auth_type}

Upgrade Test Teardown
End Web Test
Expand Down
34 changes: 29 additions & 5 deletions ods_ci/tests/Tests/0200__rhoai_upgrade/0203__post_upgrade.robot
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@
${MODEL_CREATED}= ${FALSE}
${RUNTIME_NAME}= Model Serving Test
${DW_PROJECT_CREATED}= False
${UPGRADE_NS} upgrade

Check warning

Code scanning / Robocop

The assignment sign is not consistent inside the variables section. Expected '{{ expected_sign }}' but got '{{ actual_sign }}' instead Warning test

The assignment sign is not consistent inside the variables section. Expected '=' but got '' instead
${UPGRADE_CONFIG_MAP} upgrade-config-map

Check warning

Code scanning / Robocop

The assignment sign is not consistent inside the variables section. Expected '{{ expected_sign }}' but got '{{ actual_sign }}' instead Warning test

The assignment sign is not consistent inside the variables section. Expected '=' but got '' instead


*** Test Cases ***
Expand Down Expand Up @@ -69,11 +71,26 @@
END

Verify Notebook Has Not Restarted
[Documentation] Verify Notbook pod has not restarted after the upgrade
[Tags] Upgrade
${return_code} ${new_timestamp} Run And Return Rc And Output oc get pod -n ${NOTEBOOKS_NAMESPACE} jupyter-nb-ldap-2dadmin2-0 --no-headers --output='custom-columns=TIMESTAMP:.metadata.creationTimestamp' #robocop:disable
Should Be Equal As Integers ${return_code} 0
Should Be Equal ${timestamp} ${new_timestamp} msg=Running notebook pod has restarted
[Documentation] Verify Notebook pod has not restarted after the upgrade
[Tags] Upgrade
${notebook_name}= Get User CR Notebook Name ${TEST_USER2.USERNAME}
${notebook_pod_name}= Get User Notebook Pod Name ${TEST_USER2.USERNAME}

Check warning

Code scanning / Robocop

The assignment sign is not consistent within the file. Expected '{{ expected_sign }}' but got '{{ actual_sign }}' instead Warning test

The assignment sign is not consistent within the file. Expected '' but got '=' instead
# Get the running notebook creation timestamp

Check warning

Code scanning / Robocop

The assignment sign is not consistent within the file. Expected '{{ expected_sign }}' but got '{{ actual_sign }}' instead Warning test

The assignment sign is not consistent within the file. Expected '' but got '=' instead
${return_code} ${new_timestamp} Run And Return Rc And Output
... oc get pod -n ${NOTEBOOKS_NAMESPACE} ${notebook_pod_name} --no-headers --output='custom-columns=TIMESTAMP:.metadata.creationTimestamp' # robocop: disable: line-too-long
Should Be Equal As Integers ${return_code} 0 msg=${new_timestamp}

# Get the running notebook creation timestamp from the upgrade ConfigMap safed in the previous
# phase (before the actual RHOAI upgrade)
${return_code} ${ntb_creation_timestamp} Run And Return Rc And Output
... oc get configmap ${UPGRADE_CONFIG_MAP} -n ${UPGRADE_NS} -o jsonpath='{.data.ntb_creation_timestamp}'
Should Be Equal As Integers ${return_code} 0 msg=${ntb_creation_timestamp}

# The timestamps should be equal
Should Be Equal ${ntb_creation_timestamp} ${new_timestamp} msg=Running notebook pod has restarted

[Teardown] Terminate Running Notebook ${notebook_name}

Verify Custom Image Is Present
[Tags] Upgrade
Expand Down Expand Up @@ -241,6 +258,13 @@
IF not ${status} Fail Notebook image is deleted after the upgrade
Upgrade Test Teardown

Terminate Running Notebook
[Documentation] Terminates the running notebook instance
[Arguments] ${notebook_name}
${return_code} ${cmd_output} Run And Return Rc And Output
... oc delete Notebook.kubeflow.org -n ${NOTEBOOKS_NAMESPACE} ${notebook_name}
Should Be Equal As Integers ${return_code} 0 msg=${cmd_output}

Upgrade Test Teardown
Skip If RHODS Is Self-Managed
${expression} = Set Variable rhods_aggregate_availability&step=1
Expand Down
Loading