Test results differ on different environment/laptop

[silabs-theophilel]

I have experienced few times now, in my team, that some tests that are passing on my local laptop but not on someone else's or even sometimes passing in ci but not in my laptop. This concerns few test cases.

I would like to know if some of you already enctountered such issue and what could be the root cause?
We have checked , on same branch , same commit hash with same ENV variable. And some test always FAIL while passing on other laptop.

I tried to define a common build configuration with :tools: keywork but it does not seems to be taken in consideration for some reasons. I suspect problem comes from the build tool differences (mac vs linux).

Did any of you experienced the same behavior?

Here is my project.yml file

Ceedling v1.0.0
ruby --version
ruby 3.4.1 (2024-12-25 revision 48d4efcb85) +PRISM [arm64-darwin23]

---

# Notes:
# Sample project C code is not presently written to produce a release artifact.
# As such, release build options are disabled.
# This sample, therefore, only demonstrates running a collection of unit tests.

:project:
  :use_mocks: TRUE
  :compile_threads: :auto
  :test_threads: :auto
  :use_exceptions: FALSE
  :use_test_preprocessor: :all
  :use_auxiliary_dependencies: TRUE
  :use_backtrace: :simple
  :build_root: build
#  :release_build: TRUE
  :test_file_prefix: test_
  :which_ceedling: gem
  :ceedling_version: 1.0.0
  :default_tasks:
    - test:all
  :use_decorators: :none

:test_build:
  :use_assembly: TRUE

#:release_build:
#  :output: MyApp.out
#  :use_assembly: FALSE

:environment:

:extension:
  :executable: .out

:paths:
  :test:
    - +:test/**
    - -:test/support
  :source:
    - ../../src/**
  :include:
    ## Z-WAVE
    - "#{ENV['ZWAVE_DIR']}/PAL/inc/**"
    - "#{ENV['ZWAVE_DIR']}/mocks/ThirdParty/Freertos/include/**"
    - "#{ENV['ZWAVE_DIR']}/ThirdParty/FreeRTOS-Kernel/include/**"
    - "#{ENV['ZWAVE_DIR']}/config/FreeRTOS/**"
    - "#{ENV['ZWAVE_DIR']}/platform/SiliconLabs/PAL/inc/**"
    ## GSDK
    - "#{ENV['GSDK_DIR']}/platform/emlib/inc/**"
    - "#{ENV['GSDK_DIR']}/platform/Device/SiliconLabs/ZGM23/Include/**"
    - "#{ENV['GSDK_DIR']}/platform/CMSIS/Core/Include/**"
    - "#{ENV['GSDK_DIR']}/platform/common/inc/**"
    - "#{ENV['GSDK_DIR']}/platform/service/sleeptimer/inc/**"
    - "#{ENV['GSDK_DIR']}/platform/service/power_manager/inc/**"
  :support:
    - test/support
  :libraries: []

:defines:
  # in order to add common defines:
  #  1) remove the trailing [] from the :common: section
  #  2) add entries to the :common: section (e.g. :test: has TEST defined)
  :common: &common_defines []
  :test:
    - *common_defines
    - TEST
    - ZGM230SB27HGN
  :test_preprocess:
    - *common_defines
    - TEST

:cmock:
  :mock_prefix: mock_
  :when_no_prototypes: :warn
  :enforce_strict_ordering: TRUE
  :plugins:
    - :ignore
    - :ignore_arg
    - :return_thru_ptr
    - :callback
    - :expect
    - :expect_any_args
  :includes:
    - sl_sleeptimer.h

# Add -gcov to the plugins list to make sure of the gcov plugin
# You will need to have gcov and gcovr both installed to make it work.
# For more information on these options, see docs in plugins/gcov
:gcov:
  :reports:
    - HtmlDetailed
  :gcovr:
    :html_medium_threshold: 75
    :html_high_threshold: 90

#:tools:
# Ceedling defaults to using gcc for compiling, linking, etc.
# As [:tools] is blank, gcc will be used (so long as it's in your system path)
# See documentation to configure a given toolchain for use
:tools:
  :test_file_preprocessor:
    :executable: gcc
    :name: default_test_file_preprocessor
    :stderr_redirect: :none
    :background_exec: :none
    :optional: false
    :arguments:
    - ''
    - ''
    - "-E"
    - -I"$": COLLECTION_PATHS_TEST_SUPPORT_SOURCE_INCLUDE_VENDOR
    - -I"$": COLLECTION_PATHS_TEST_TOOLCHAIN_INCLUDE
    - "-D$": COLLECTION_DEFINES_TEST_AND_VENDOR
    - "-D$": DEFINES_TEST_PREPROCESS
    - "-DGNU_COMPILER"
    - "-include FreeRTOS.h"
    - '"${1}"'
    - -o "${2}"
  :test_file_preprocessor_directives:
    :executable: gcc
    :name: default_test_file_preprocessor_directives
    :stderr_redirect: :none
    :background_exec: :none
    :optional: false
    :arguments:
    - "-E"
    - -I"$": COLLECTION_PATHS_TEST_SUPPORT_SOURCE_INCLUDE_VENDOR
    - -I"$": COLLECTION_PATHS_TEST_TOOLCHAIN_INCLUDE
    - "-D$": COLLECTION_DEFINES_TEST_AND_VENDOR
    - "-D$": DEFINES_TEST_PREPROCESS
    - "-DGNU_COMPILER"
    - "-include FreeRTOS.h"
    - "-fdirectives-only"
    - '"${1}"'
    - -o "${2}"
  :test_includes_preprocessor:
    :executable: gcc
    :name: default_test_includes_preprocessor
    :stderr_redirect: :none
    :background_exec: :none
    :optional: false
    :arguments:
    - ''
    - ''
    - "-E"
    - "-MM"
    - "-MG"
    - -I"$": COLLECTION_PATHS_TEST_SUPPORT_SOURCE_INCLUDE_VENDOR
    - -I"$": COLLECTION_PATHS_TEST_TOOLCHAIN_INCLUDE
    - "-D$": COLLECTION_DEFINES_TEST_AND_VENDOR
    - "-D$": DEFINES_TEST_PREPROCESS
    - "-DGNU_COMPILER"
    - "-include FreeRTOS.h"
    - '"${1}"'
  :test_dependencies_generator:
    :executable: gcc
    :name: default_test_dependencies_generator
    :stderr_redirect: :none
    :background_exec: :none
    :optional: false
    :arguments:
    - ''
    - ''
    - "-E"
    - -I"$": COLLECTION_PATHS_TEST_SUPPORT_SOURCE_INCLUDE_VENDOR
    - -I"$": COLLECTION_PATHS_TEST_TOOLCHAIN_INCLUDE
    - "-D$": COLLECTION_DEFINES_TEST_AND_VENDOR
    - "-D$": DEFINES_TEST_PREPROCESS
    - "-DGNU_COMPILER"
    - "-include FreeRTOS.h"
    - -MT "${3}"
    - "-MM"
    - "-MD"
    - "-MG"
    - -MF "${2}"
    - -c "${1}"

# LIBRARIES
# These libraries are automatically injected into the build process. Those specified as
# common will be used in all types of builds. Otherwise, libraries can be injected in just
# tests or releases. These options are MERGED with the options in supplemental yaml files.
:libraries:
  :placement: :end
  :flag: "-l${1}"
  :path_flag: "-L ${1}"
  :system: []    # for example, you might list 'm' to grab the math library
  :test: []
  :release: []

:plugins:
  :load_paths: []
  :enabled:
    - report_tests_pretty_stdout
    - module_generator
...

[mvandervoord]

Hi!

Thanks for the good question. Without seeing testing details, I'm just making guesses here... but there are some common sources of this:

• By far the most common source is that you have some tests that rely on undefined behavior... often variables that aren't initialized. Depending on the defaults of a particular gcc (and yes, these features vary by platform), it might zero all uninitialized memory, or it might just leave it with whatever random data was there. These are going to lead to very different results.
• Another common issue is differences in data sizes. If you're using generic types like int instead of things like int64_t, the differences in size can creep into all sorts of places.
• Same with packing differences for structs. If you're not telling both compilers to pack all your structs, then any tests which are using the fallback memory comparison are in jeapardy of running into undefined behavior (because the padding is usually not touched, but IS compared in a memory comparison).

These are some that spring to mind. If you share more details on what's going on, we might be able to narrow it down more.

[silabs-theophilel]

Ok so I figured out that I had some code (CMock call expect ignore etc...) refactored under the test/utility folder. This code when moved to the original test file, the problem disappears. I wonder why is that... I did not find any known issue about that. For now I will sitck to put my test code only in one file, even if it meens code duplication for other tests...