PlanningSceneMonitorTest.TestPersistentScene test sporadically fails in Humble CI

[TSNoble]

Description

As the title says, e.g.

https://github.com/moveit/moveit2/actions/runs/11959136854/job/33340257097
https://github.com/moveit/moveit2/actions/runs/11856401984/job/33042608098
https://github.com/moveit/moveit2/actions/runs/11723728898/job/32656021443
https://github.com/moveit/moveit2/actions/runs/11579145965/job/32234657733

The main difference between passing and failing runs seems to be that sometimes the /controller_manager service is unavailable

ROS Distro

Humble

OS and version

Humble CI Job

Source or binary build?

Source

If binary, which release version?

N/A

If source, which branch?

humble

Which RMW are you using?

None

Steps to Reproduce

Unclear how to reproduce consistently. It may be possible to run the test locally a large number of times until the failure occurs.

Expected behavior

This test should not fail sporadically, as this makes it difficult to determine whether a job is failing due to changes made in a PR

Actual behavior

Currently, this test does fail sporadically

Backtrace or Console output

Failing Run:

$ ( source /home/runner/work/moveit2/moveit2/.work/upstream_ws/install/setup.bash && cd /home/runner/work/moveit2/moveit2/.work/target_ws && colcon test-result --verbose; )
build/moveit_ros_planning/Testing/20241115-1352/Test.xml: 7 tests, 0 errors, 1 failure, 0 skipped
- planning_scene_monitor_test_launch_planning_scene_monitor.test.py
  <<< failure message
    -- run_test.py: invoking following command in '/home/runner/work/moveit2/moveit2/.work/target_ws/build/moveit_ros_planning/planning_scene_monitor':
     - ros2 test /home/runner/work/moveit2/moveit2/.work/target_ws/src/moveit2/moveit_ros/planning/planning_scene_monitor/test/launch/planning_scene_monitor.test.py test_binary_dir:=/home/runner/work/moveit2/moveit2/.work/target_ws/build/moveit_ros_planning/planning_scene_monitor --junit-xml=/home/runner/work/moveit2/moveit2/.work/target_ws/build/moveit_ros_planning/test_results/moveit_ros_planning/planning_scene_monitor_test_launch_planning_scene_monitor.test.py.xunit.xml --package-name=moveit_ros_planning
    Running with ROS_DOMAIN_ID 1
    ROS_DOMAIN_ID 1
    [INFO] [launch]: All log files can be found below /root/.ros/log/2024-11-15-13-52-54-849801-d461d375157d-31423
    [INFO] [launch]: Default logging verbosity is set to INFO
    test_gtest_run_complete (moveit_ros_planning.TestGTestWaitForCompletion) ... Warning: Automatically adding attributes like self.proc_info to the test class will be deprecated in a future release.  Instead, add proc_info to the test method argument list to access the test object you need
    [INFO] [ros2_control_node-1]: process started with pid [31425]
    [ros2_control_node-1] [INFO] [1731678777.017561036] [controller_manager]: Subscribing to '~/robot_description' topic for robot description file.
    [ros2_control_node-1] [INFO] [1731678777.018223763] [controller_manager]: update rate is 100 Hz
    [ros2_control_node-1] [INFO] [1731678777.018371777] [controller_manager]: RT kernel is recommended for better performance
    [INFO] [spawner-2]: process started with pid [31440]
    [INFO] [spawner-3]: process started with pid [31451]
    [spawner-2] [INFO] [1731678781.649892991] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available
    [spawner-3] [INFO] [1731678783.196703664] [spawner_panda_arm_controller]: Waiting for '/controller_manager' services to be available
    [spawner-2] [INFO] [1731678783.660546004] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available
    [INFO] [planning_scene_monitor_test-4]: process started with pid [31462]
    [planning_scene_monitor_test-4] [==========] Running 2 tests from 1 test suite.
    [planning_scene_monitor_test-4] [----------] Global test environment set-up.
    [planning_scene_monitor_test-4] [----------] 2 tests from PlanningSceneMonitorTest
    [planning_scene_monitor_test-4] [ RUN      ] PlanningSceneMonitorTest.TestPersistentScene
    [planning_scene_monitor_test-4] [INFO] [1731678784.748189748] [moveit_rdf_loader.rdf_loader]: Loaded robot model in 0.00462217 seconds
    [planning_scene_monitor_test-4] [INFO] [1731678784.748255882] [moveit_robot_model.robot_model]: Loading robot model 'panda'...
Error: anning_scene_monitor_test-4] [ERROR] [1731678784.757801175] [kinematics_plugin_loader]: The kinematics plugin (panda_arm) failed to load. Error: According to the loaded plugin descriptions the class kdl_kinematics_plugin/KDLKinematicsPlugin with base class type kinematics::KinematicsBase does not exist. Declared types are 
Error: anning_scene_monitor_test-4] [ERROR] [1731678784.757826242] [moveit_ros.robot_model_loader]: Kinematics solver could not be instantiated for joint group panda_arm.
    [planning_scene_monitor_test-4] [INFO] [1731678784.785387677] [moveit_ros.planning_scene_monitor.planning_scene_monitor]: Publishing maintained planning scene on 'monitored_planning_scene'
Error: anning_scene_monitor_test-4] [ERROR] [1731678784.785934690] [moveit_robot_state.conversions]: Found empty JointState message
    [spawner-3] [INFO] [1731678785.207073061] [spawner_panda_arm_controller]: Waiting for '/controller_manager' services to be available
    [spawner-2] [INFO] [1731678785.670762383] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available
    [spawner-3] [INFO] [1731678787.217687087] [spawner_panda_arm_controller]: Waiting for '/controller_manager' services to be available
    [spawner-2] [INFO] [1731678787.681018938] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available
    [spawner-3] [INFO] [1731678789.227985924] [spawner_panda_arm_controller]: Waiting for '/controller_manager' services to be available
    [spawner-2] [INFO] [1731678789.691057593] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available
Error: awner-3] [ERROR] [1731678791.238210538] [spawner_panda_arm_controller]: Controller manager not available
Error: ROR] [spawner-3]: process has died [pid 31451, exit code 1, cmd '/opt/ros/humble/lib/controller_manager/spawner panda_arm_controller -c /controller_manager --ros-args'].
    [spawner-2] [INFO] [1731678791.701209164] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available
    [spawner-2] [INFO] [1731678793.711465668] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available
    [spawner-2] [INFO] [1731678795.721483460] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available
    [spawner-2] [INFO] [1731678797.731539934] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available
    [spawner-2] [INFO] [1731678799.742223918] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available
    [spawner-2] [INFO] [1731678801.752916668] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available
    [spawner-2] [INFO] [1731678803.762920834] [spawner_joint_state_broadcaster]: Waiting for '/controller_manager' services to be available
  >>>
build/moveit_ros_planning/test_results/moveit_ros_planning/planning_scene_monitor_test_launch_planning_scene_monitor.test.py.xunit.xml: 1 test, 1 error, 0 failures, 0 skipped
- moveit_ros_planning planning_scene_monitor_test_launch_planning_scene_monitor.test.py.xunit.missing_result
  <<< error message
    The test did not generate a result file.
  >>>

Summary: 1366 tests, 1 error, 1 failure, 687 skipped

Passing Run:

  7: [planning_scene_monitor_test-4] [ RUN      ] PlanningSceneMonitorTest.TestPersistentScene
  7: [planning_scene_monitor_test-4] [INFO] [1731752174.725806886] [moveit_rdf_loader.rdf_loader]: Loaded robot model in 0.00545313 seconds
  7: [planning_scene_monitor_test-4] [INFO] [1731752174.725900100] [moveit_robot_model.robot_model]: Loading robot model 'panda'...
  Error: nning_scene_monitor_test-4] [ERROR] [1731752174.735197890] [kinematics_plugin_loader]: The kinematics plugin (panda_arm) failed to load. Error: According to the loaded plugin descriptions the class kdl_kinematics_plugin/KDLKinematicsPlugin with base class type kinematics::KinematicsBase does not exist. Declared types are 
  Error: nning_scene_monitor_test-4] [ERROR] [1731752174.735224069] [moveit_ros.robot_model_loader]: Kinematics solver could not be instantiated for joint group panda_arm.
  7: [planning_scene_monitor_test-4] [INFO] [1731752174.762992562] [moveit_ros.planning_scene_monitor.planning_scene_monitor]: Publishing maintained planning scene on 'monitored_planning_scene'
  Error: nning_scene_monitor_test-4] [ERROR] [1731752174.763490405] [moveit_robot_state.conversions]: Found empty JointState message
  7: [planning_scene_monitor_test-4] [INFO] [1731752175.013995396] [moveit_ros.planning_scene_monitor.planning_scene_monitor]: Stopped publishing maintained planning scene.
  7: [planning_scene_monitor_test-4] [       OK ] PlanningSceneMonitorTest.TestPersistentScene (984 ms)

[sea-bass]

Duplicate of #2821, except this was disabled on main so it does not show up there.