Skip to content

Commit

Permalink
[fix] the Verify Notebook Has Not Restarted upgrade test
Browse files Browse the repository at this point in the history 
This refactores this test - the first test part is moved to "pre_upgrade"
phase, because this phase is run on the pre-upgrade version with the
pre-upgrade code-base (branch). This is crucial since the further phases
will be run with the code-base of the upgraded product and we may hit
some issues in the differences in the product otherwise.

There are done other changes and fixes to the test to make it more
robust including the notebook deletion in the post upgrade phase.

Apart from that, to propagate the timestamp value between the phases,
this introduces a configmap in the "upgrade" namespace, where we can put
some context that should be passed to the followup phase.
  • Loading branch information
@jstourac
jstourac committed Jan 13, 2025
1 parent 58b325d commit b96252d
Show file tree
Hide file tree
Showing 3 changed files with 69 additions and 24 deletions.
39 changes: 39 additions & 0 deletions ods_ci/tests/Tests/0200__rhoai_upgrade/0201__pre_upgrade.robot
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@ Test Tags PreUpgrade
${CUSTOM_CULLER_TIMEOUT} 60000
${S_SIZE} 25
${DW_PROJECT_CREATED} False
${CODE} while True: import time ; time.sleep(10); print ("Hello")
${UPGRADE_NS} upgrade
${UPGRADE_CONFIG_MAP} upgrade-config-map


*** Test Cases ***
Expand Down Expand Up @@ -246,8 +249,44 @@ Model Registry Pre Upgrade Set Up
[Tags] Upgrade ModelRegistryUpgrade
Model Registry Pre Upgrade Scenario

Long Running Jupyter Notebook
[Documentation] Launch a long running notebook before the upgrade
[Tags] Upgrade
Launch Notebook
Add And Run JupyterLab Code Cell In Active Notebook ${CODE}

# Get the notebook pod creation timestamp
${notebook_pod_name}= Set Variable jupyter-nb-ldap-2dadmin2-0
${return_code} ${ntb_creation_timestamp} = Run And Return Rc And Output

Check warning

Code scanning / Robocop

The assignment sign is not consistent within the file. Expected '{{ expected_sign }}' but got '{{ actual_sign }}' instead Warning test

The assignment sign is not consistent within the file. Expected '=' but got ' =' instead
... oc get pod -n ${NOTEBOOKS_NAMESPACE} ${notebook_pod_name} --no-headers --output='custom-columns=TIMESTAMP:.metadata.creationTimestamp' # robocop: disable: line-too-long
Should Be Equal As Integers ${return_code} 0 msg=${ntb_creation_timestamp}

# Save the timestamp to the OpenShift ConfigMap so it can be used in test in the next phase
${return_code} ${cmd_output} = Run And Return Rc And Output

Check warning

Code scanning / Robocop

The assignment sign is not consistent within the file. Expected '{{ expected_sign }}' but got '{{ actual_sign }}' instead Warning test

The assignment sign is not consistent within the file. Expected '=' but got ' =' instead
... oc create namespace ${UPGRADE_NS}
Should Be Equal As Integers ${return_code} 0 msg=${cmd_output}
${return_code} ${cmd_output} = Run And Return Rc And Output

Check warning

Code scanning / Robocop

The assignment sign is not consistent within the file. Expected '{{ expected_sign }}' but got '{{ actual_sign }}' instead Warning test

The assignment sign is not consistent within the file. Expected '=' but got ' =' instead
... oc create configmap ${UPGRADE_CONFIG_MAP} -n ${UPGRADE_NS} --from-literal=ntb_creation_timestamp=${ntb_creation_timestamp} # robocop: disable: line-too-long
Should Be Equal As Integers ${return_code} 0 msg=${cmd_output}

Close Browser


*** Keywords ***
Launch Notebook
[Documentation] Launch notebook for the suite
[Arguments] ${notebook_image}=minimal-notebook
... ${username}=${TEST_USER2.USERNAME}
... ${password}=${TEST_USER2.PASSWORD}
... ${auth_type}=${TEST_USER2.AUTH_TYPE}
Begin Web Test username=${username} password=${password} auth_type=${auth_type}
Launch Jupyter From RHODS Dashboard Link
Spawn Notebook With Arguments
... image=${notebook_image}
... username=${username}
... password=${password}
... auth_type=${auth_type}

Dashboard Suite Setup
[Documentation] Basic suite setup
Set Library Search Order SeleniumLibrary
Expand Down
20 changes: 2 additions & 18 deletions ods_ci/tests/Tests/0200__rhoai_upgrade/0202__during_upgrade.robot
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,23 +14,7 @@ Library JupyterLibrary
Test Tags DuringUpgrade


*** Variables ***
${CODE} while True: import time ; time.sleep(10); print ("Hello")


*** Test Cases ***
Long Running Jupyter Notebook
[Documentation] Launch a long running notebook before the upgrade
[Tags] Upgrade
Launch Notebook
Add And Run JupyterLab Code Cell In Active Notebook ${CODE}
# robocop:disable
${return_code} ${timestamp} = Run And Return Rc And Output
... oc get pod -n ${NOTEBOOKS_NAMESPACE} jupyter-nb-ldap-2dadmin2-0 --no-headers --output='custom-columns=TIMESTAMP:.metadata.creationTimestamp'
Should Be Equal As Integers ${return_code} 0
Set Global Variable ${timestamp} # robocop: disable
Close Browser

Upgrade RHODS
[Documentation] Approve the install plan for the upgrade and make sure that upgrade has completed
[Tags] ODS-1766 Upgrade
Expand Down Expand Up @@ -79,7 +63,7 @@ PyTorch Image Workload Test
*** Keywords ***
Launch Notebook
[Documentation] Launch notebook for the suite
[Arguments] ${notbook_image}=minimal-notebook
[Arguments] ${notebook_image}=minimal-notebook
... ${username}=${TEST_USER2.USERNAME}
... ${password}=${TEST_USER2.PASSWORD}
... ${auth_type}=${TEST_USER2.AUTH_TYPE}
Expand All @@ -94,7 +78,7 @@ Launch Notebook
Fix Spawner Status
# robocop: disable
Spawn Notebook With Arguments
... image=${notbook_image}
... image=${notebook_image}
... username=${username}
... password=${password}
... auth_type=${auth_type}
Expand Down
34 changes: 28 additions & 6 deletions ods_ci/tests/Tests/0200__rhoai_upgrade/0203__post_upgrade.robot
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@ Test Tags PostUpgrade
*** Variables ***
${S_SIZE} 25
${DW_PROJECT_CREATED} False
${UPGRADE_NS} upgrade
${UPGRADE_CONFIG_MAP} upgrade-config-map


*** Test Cases ***
Expand Down Expand Up @@ -71,13 +73,26 @@ Verify Culler is Enabled
END

Verify Notebook Has Not Restarted
[Documentation] Verify Notbook pod has not restarted after the upgrade
[Tags] Upgrade AutomationBug RHOAIENG-14840
# robocop:disable
[Documentation] Verify Notebook pod has not restarted after the upgrade
[Tags] Upgrade
${notebook_name}= Set Variable jupyter-nb-ldap-2dadmin2

Check warning

Code scanning / Robocop

The assignment sign is not consistent within the file. Expected '{{ expected_sign }}' but got '{{ actual_sign }}' instead Warning test

The assignment sign is not consistent within the file. Expected '' but got '=' instead
${notebook_pod_name}= Set Variable ${notebook_name}-0

Check warning

Code scanning / Robocop

The assignment sign is not consistent within the file. Expected '{{ expected_sign }}' but got '{{ actual_sign }}' instead Warning test

The assignment sign is not consistent within the file. Expected '' but got '=' instead

# Get the running notebook creation timestamp
${return_code} ${new_timestamp} Run And Return Rc And Output
... oc get pod -n ${NOTEBOOKS_NAMESPACE} jupyter-nb-ldap-2dadmin2-0 --no-headers --output='custom-columns=TIMESTAMP:.metadata.creationTimestamp'
Should Be Equal As Integers ${return_code} 0
Should Be Equal ${timestamp} ${new_timestamp} msg=Running notebook pod has restarted
... oc get pod -n ${NOTEBOOKS_NAMESPACE} ${notebook_pod_name} --no-headers --output='custom-columns=TIMESTAMP:.metadata.creationTimestamp' # robocop: disable: line-too-long
Should Be Equal As Integers ${return_code} 0 msg=${new_timestamp}

# Get the running notebook creation timestamp from the upgrade ConfigMap safed in the previous
# phase (before the actual RHOAI upgrade)
${return_code} ${ntb_creation_timestamp} Run And Return Rc And Output
... oc get configmap ${UPGRADE_CONFIG_MAP} -n ${UPGRADE_NS} -o jsonpath='{.data.ntb_creation_timestamp}'
Should Be Equal As Integers ${return_code} 0 msg=${ntb_creation_timestamp}

# The timestamps should be equal
Should Be Equal ${ntb_creation_timestamp} ${new_timestamp} msg=Running notebook pod has restarted

[Teardown] Terminate Running Notebook ${notebook_name}

Verify Custom Image Is Present
[Documentation] Verify Custom Noteboook is not deleted after the upgrade
Expand Down Expand Up @@ -325,6 +340,13 @@ Delete OOTB Image
IF not ${status} Fail Notebook image is deleted after the upgrade
IF not ${IS_SELF_MANAGED} Managed RHOAI Upgrade Test Teardown

Terminate Running Notebook
[Documentation] Terminates the running notebook instance
[Arguments] ${notebook_name}
${return_code} ${cmd_output} Run And Return Rc And Output
... oc delete Notebook.kubeflow.org -n ${NOTEBOOKS_NAMESPACE} ${notebook_name}
Should Be Equal As Integers ${return_code} 0 msg=${cmd_output}

Managed RHOAI Upgrade Test Teardown
# robocop: off=too-many-calls-in-keyword
[Documentation] Check rhods_aggregate_availability metric when RHOAI is installed as managed
Expand Down

0 comments on commit b96252d

Please sign in to comment.