Added preprocessing hooks to plugin handling

- plugin & plugin_manager updated with pre/post hooks for optional preprocessing before mock generation & test runner generation
- preprocessing refactored to support new hooks
- pre_test hook moved to appropriate position in new test build pipeline
- command_hooks plugin:
  - Updated to handle new hooks
  - Improved logging
  - Documentation improvements
- Ongoing revisions and additions to CeedlingPacket & ReleaseNotes
- Improved debug logging, especially in scenarios related to YAML validation before tasks are run
- Improved new search path handling to conform to previous decisions and documentation on path prioritization
- Began introducing hash use for methods with lengthy argument lists

docs/ReleaseNotes.md

@@ -12,7 +12,9 @@ This Ceedling release is probably the most significant since the project was fir
 
 Ceedling now runs in Ruby 3. Builds can now run much faster than previous versions because of parallelized tasks. For test suites, header file search paths, code defines, and tool run flags are now customizable per test executable.
 
-🏴‍☠️ **_Ahoy!_** There be **[breaking changes](#-Breaking-Changes)** ahead, mateys! Arrr…
+### Avast, Breaking Changes, Ye Scallywags! 🏴‍☠️
+
+**_Ahoy!_** There be **[breaking changes](#-Breaking-Changes)** ahead, mateys! Arrr…
 
 ### Big Deal Highlights 🏅
 
@@ -40,6 +42,15 @@ The following new features (discussed in later sections) contribute to this new
 - `[:defines]` handling. `#define`s are now specified for the compilation of all modules comprising a test executable. Matching is only against test file names but now includes wildcard and regular expression options.
 - `[:flags]` handling. Flags (e.g. `-std=c99`) are now specified for the build steps—preprocessing, compilation, and linking—of all modules comprising a test executable. Matching is only against test file names and now includes more sensible and robust wildcard and regular expression options.
 
+### Important Changes in Behavior to Be Aware Of 🚨
+
+- **Test suite build order 🔢.** Ceedling no longer builds each test executable one at a time. From the tasks you provide at the command line, Ceedling now collects up and batches all preprocessing steps, all mock generation, all test runner generation, all compilation, etc. Previously you would see each of these done for a single test executable and then repeated for the next executable and so on. Now, each build step happens to completion for all specified tests before moving on to the next build step. 
+- **Logging output order 🔢.** When multi-threaded builds are enabled, logging output may not be what you expect. Progress statements may be all batched together or interleaved in ways that are misleading. The steps are happening in the correct order. How you are informed of them may be somewhat out of order.
+- **Files generated multiple times 🔀.** Now that each test is essentially a self-contained mini-project, some output may be generated multiple times. For instance, if the same mock is required by multiple tests, it will be generated multiple times. The same holds for compilation of source files into object files. A coming version of Ceedling will concentrate on optimizations to reuse any output that is truly identical across tests.
+- **Test suite plugin runs 🏃🏻.** Because build steps are run to completion across all the tests you specify at the command line (e.g. all the mocks for your tests are generated at one time) you may need to adjust how you depend on build steps.
+
+Together, these changes may cause you to think that Ceedling is running steps out of order or duplicating work. While bugs are always possible, more than likely, the output you see and the build ordering is expected.
+
 ### Medium Deal Highlights 🥈
 
 #### `TEST_SOURCE_FILE(...)`

lib/ceedling/configurator.rb

@@ -69,6 +69,16 @@ def reset_defaults(config)
   end
 
 
+  # Set up essential flattened config
+  # (In case YAML validation failure prevents flattening of config into configurator accessors)
+  def set_debug(config)
+    if config[:project][:debug]
+      eval("def project_debug() return true end", binding())
+      eval("def project_verbosity() return Verbosity::DEBUG end", binding())
+    end
+  end
+
+
   # The default values defined in defaults.rb (eg. DEFAULT_TOOLS_TEST) are populated
   # into @param config
   def populate_defaults(config)
@@ -144,13 +154,17 @@ def get_cmock_config
   end
 
 
-  # grab tool names from yaml and insert into tool structures so available for error messages
-  # set up default values
+  # Grab tool names from yaml and insert into tool structures so available for error messages.
+  # Set up default values.
   def tools_setup(config)
     config[:tools].each_key do |name|
       tool = config[:tools][name]
 
-      # populate name if not given
+      if not tool.is_a?(Hash)
+        raise CeedlingException.new("ERROR: Expected configuration for tool :#{name} is a Hash but found #{tool.class}")
+      end
+
+      # Populate name if not given
       tool[:name] = name.to_s if (tool[:name].nil?)
 
       # handle inline ruby string substitution in executable

lib/ceedling/defaults.rb

@@ -306,6 +306,7 @@
       :options_paths => [],
       :release_build => false,
       :use_backtrace_gdb_reporter => false,
+      :debug => false
     },
 
     :release_build => {

lib/ceedling/include_pathinator.rb

@@ -49,4 +49,15 @@ def lookup_test_directive_include_paths(filepath)
     return @extractor.lookup_include_paths_list(filepath)
   end
 
+  # Gather together [:paths][:test] that actually contain .h files
+  def collect_test_include_paths
+    paths = []
+    @configurator.collection_paths_test.each do |path|
+      headers = @file_wrapper.directory_listing( File.join( path, '*' + @configurator.extension_header ) )
+      paths << path if headers.length > 0
+    end
+
+    return paths
+  end
+
 end

lib/ceedling/objects.yml

@@ -275,6 +275,7 @@ preprocessinator:
     - file_path_utils
     - file_wrapper
     - yaml_wrapper
+    - plugin_manager
     - project_config_manager
     - configurator
     - test_context_extractor

lib/ceedling/plugin.rb

@@ -43,35 +43,43 @@ def initialize(system_objects, name)
 
   def setup; end
 
-  # mock generation
+  # Preprocessing (before / after each and every header file preprocessing operation before mocking)
+  def pre_mock_preprocess(arg_hash); end
+  def post_mock_preprocess(arg_hash); end
+
+  # Preprocessing (before / after each and every test preprocessing operation before runner generation)
+  def pre_test_preprocess(arg_hash); end
+  def post_test_preprocess(arg_hash); end
+
+  # Mock generation (before / after each and every mock)
   def pre_mock_generate(arg_hash); end
   def post_mock_generate(arg_hash); end
 
-  # test runner generation
+  # Test runner generation (before / after each and every test runner)
   def pre_runner_generate(arg_hash); end
   def post_runner_generate(arg_hash); end
 
-  # compilation (test or source)
+  # Compilation (before / after each and test or source file compilation)
   def pre_compile_execute(arg_hash); end
   def post_compile_execute(arg_hash); end
 
-  # linking (test or source)
+  # Linking (before / after each and every test executable or release artifact)
   def pre_link_execute(arg_hash); end
   def post_link_execute(arg_hash); end
 
-  # test fixture execution
+  # Test fixture execution (before / after each and every test fixture executable)
   def pre_test_fixture_execute(arg_hash); end
   def post_test_fixture_execute(arg_hash); end
 
-  # test task
+  # Test task (before / after each test executable build)
   def pre_test(test); end
   def post_test(test); end
 
-  # release task
+  # Release task (before / after a release build)
   def pre_release; end
   def post_release; end
 
-  # whole shebang (any use of Ceedling)
+  # Whole shebang (any use of Ceedling)
   def pre_build; end
   def post_build; end
 

lib/ceedling/plugin_manager.rb

@@ -56,6 +56,12 @@ def register_build_failure(message)
 
   #### execute all plugin methods ####
 
+  def pre_mock_preprocess(arg_hash); execute_plugins(:pre_mock_preprocess, arg_hash); end
+  def post_mock_preprocess(arg_hash); execute_plugins(:post_mock_preprocess, arg_hash); end
+
+  def pre_test_preprocess(arg_hash); execute_plugins(:pre_test_preprocess, arg_hash); end
+  def post_test_preprocess(arg_hash); execute_plugins(:post_test_preprocess, arg_hash); end
+
   def pre_mock_generate(arg_hash); execute_plugins(:pre_mock_generate, arg_hash); end
   def post_mock_generate(arg_hash); execute_plugins(:post_mock_generate, arg_hash); end
 
@@ -98,7 +104,7 @@ def execute_plugins(method, *args)
       begin
         plugin.send(method, *args) if plugin.respond_to?(method)
       rescue
-        puts "Exception raised in plugin: #{plugin.name}, in method #{method}"
+        @streaminator.stderr_puts("Exception raised in plugin: #{plugin.name}, in method #{method}")
         raise
       end
     end

lib/ceedling/preprocessinator.rb

@@ -8,6 +8,7 @@ class Preprocessinator
               :file_path_utils,
               :file_wrapper,
               :yaml_wrapper,
+              :plugin_manager,
               :project_config_manager,
               :configurator,
               :test_context_extractor,
@@ -37,12 +38,15 @@ def extract_testing_context(filepath:, test:, flags:, include_paths:, defines:)
       @test_context_extractor.collect_includes( filepath )
     else
       # Run test file through preprocessor to parse out include statements and then collect header files, mocks, etc.
-      includes = preprocess_includes(
+      arg_hash = {
         filepath:      filepath,
         test:          test,
         flags:         flags,
         include_paths: include_paths,
-        defines:       defines)
+        defines:       defines
+      }
+
+      includes = preprocess_includes(**arg_hash)
 
       msg = @reportinator.generate_progress( "Processing #include statements for #{File.basename(filepath)}" )
       @streaminator.stdout_puts( msg, Verbosity::NORMAL )
@@ -51,46 +55,92 @@ def extract_testing_context(filepath:, test:, flags:, include_paths:, defines:)
     end
   end
 
-  def preprocess_header_file(filepath:, test:, flags:, include_paths:, defines:)
-    # Extract shallow includes & print status message
-    includes = preprocess_file_common(
-      filepath:      filepath,
-      test:          test,
-      flags:         flags,
-      include_paths: include_paths,
-      defines:       defines
-      )
+  def preprocess_mockable_header_file(filepath:, test:, flags:, include_paths:, defines:)
+    preprocessed_filepath = @file_path_utils.form_preprocessed_file_filepath( filepath, test )
+
+    plugin_arg_hash = {
+      header_file:              filepath,
+      preprocessed_header_file: preprocessed_filepath,
+      test:                     test,
+      flags:                    flags,
+      include_paths:            include_paths,
+      defines:                  defines      
+    }
+
+    # Trigger pre_mock_preprocessing plugin hook
+    @plugin_manager.pre_mock_preprocess( plugin_arg_hash )
+
+    arg_hash = {
+      filepath:       filepath,
+      test:           test,
+      flags:          flags,
+      include_paths:  include_paths,
+      defines:        defines      
+    }
+
+    # Extract shallow includes & print status message    
+    includes = preprocess_file_common(**arg_hash)
+
+    arg_hash = {
+      source_filepath:       filepath,
+      preprocessed_filepath: preprocessed_filepath,
+      includes:              includes,
+      flags:                 flags,
+      include_paths:         include_paths,
+      defines:               defines      
+    }
 
     # Run file through preprocessor & further process result
-    return @file_handler.preprocess_header_file(
-      filepath:      filepath,
-      subdir:        test,
-      includes:      includes,
-      flags:         flags,
-      include_paths: include_paths,
-      defines:       defines
-      )
+    @file_handler.preprocess_header_file(**arg_hash)
+
+    # Trigger post_mock_preprocessing plugin hook
+    @plugin_manager.post_mock_preprocess( plugin_arg_hash )
+
+    return preprocessed_filepath
   end
 
   def preprocess_test_file(filepath:, test:, flags:, include_paths:, defines:)
-    # Extract shallow includes & print status message
-    includes = preprocess_file_common(
+    preprocessed_filepath = @file_path_utils.form_preprocessed_file_filepath( filepath, test )
+
+    plugin_arg_hash = {
+      test_file:              filepath,
+      preprocessed_test_file: preprocessed_filepath,
+      test:                   test,
+      flags:                  flags,
+      include_paths:          include_paths,
+      defines:                defines      
+    }
+
+    # Trigger pre_mock_preprocessing plugin hook
+    @plugin_manager.pre_test_preprocess( plugin_arg_hash )
+
+    arg_hash = {
       filepath:      filepath,
       test:          test,
       flags:         flags,
       include_paths: include_paths,
-      defines:       defines
-      )
+      defines:       defines      
+    }
+
+    # Extract shallow includes & print status message
+    includes = preprocess_file_common(**arg_hash)
+
+    arg_hash = {
+      source_filepath:       filepath,
+      preprocessed_filepath: preprocessed_filepath,
+      includes:              includes,
+      flags:                 flags,
+      include_paths:         include_paths,
+      defines:               defines      
+    }
 
     # Run file through preprocessor & further process result
-    return @file_handler.preprocess_test_file(
-      filepath:      filepath,
-      subdir:        test,
-      includes:      includes,
-      flags:         flags,
-      include_paths: include_paths,
-      defines:       defines
-      )
+    @file_handler.preprocess_test_file(**arg_hash)
+
+    # Trigger pre_mock_preprocessing plugin hook
+    @plugin_manager.post_test_preprocess( plugin_arg_hash )
+
+    return preprocessed_filepath
   end
 
   def preprocess_file_directives(filepath)

lib/ceedling/preprocessinator_file_handler.rb

@@ -3,15 +3,13 @@ class PreprocessinatorFileHandler
 
   constructor :preprocessinator_extractor, :configurator, :flaginator, :tool_executor, :file_path_utils, :file_wrapper, :streaminator
 
-  def preprocess_header_file(filepath:, subdir:, includes:, flags:, include_paths:, defines:)
-    preprocessed_filepath = @file_path_utils.form_preprocessed_file_filepath( filepath, subdir )
-
-    filename = File.basename(filepath)
+  def preprocess_header_file(source_filepath:, preprocessed_filepath:, includes:, flags:, include_paths:, defines:)
+    filename = File.basename(source_filepath)
 
     command = @tool_executor.build_command_line(
       @configurator.tools_test_file_preprocessor,
       flags,
-      filepath,
+      source_filepath,
       preprocessed_filepath,
       defines,
       include_paths
@@ -58,17 +56,13 @@ def preprocess_header_file(filepath:, subdir:, includes:, flags:, include_paths:
     contents.gsub!( /(\h*\n){3,}/, "\n\n" )
 
     @file_wrapper.write( preprocessed_filepath, contents )
-
-    return preprocessed_filepath
   end
 
-  def preprocess_test_file(filepath:, subdir:, includes:, flags:, include_paths:, defines:)
-    preprocessed_filepath = @file_path_utils.form_preprocessed_file_filepath( filepath, subdir )
-
+  def preprocess_test_file(source_filepath:, preprocessed_filepath:, includes:, flags:, include_paths:, defines:)
     command = @tool_executor.build_command_line(
       @configurator.tools_test_file_preprocessor,
       flags,
-      filepath,
+      source_filepath,
       preprocessed_filepath,
       defines,
       include_paths
@@ -102,8 +96,6 @@ def preprocess_test_file(filepath:, subdir:, includes:, flags:, include_paths:,
     contents.gsub!( /(\h*\n){3,}/, "\n\n" ) # Collapse repeated blank lines
 
     @file_wrapper.write( preprocessed_filepath, contents )
-
-    return preprocessed_filepath
   end
 
 

lib/ceedling/rakefile.rb

@@ -48,7 +48,7 @@
   @ceedling[:setupinator].do_setup( project_config )
 
   # Configure Ruby's default reporting for Thread exceptions.
-  unless @ceedling[:configurator].project_verbosity == Verbosity::DEBUG
+  unless @ceedling[:configurator].project_debug
     # In Ceedling's case thread scenarios will fall into these buckets:
     #  1. Jobs shut down cleanly
     #  2. Jobs shut down at garbage collected after a build step terminates with an error
@@ -71,7 +71,12 @@
   # load rakefile component files (*.rake)
   PROJECT_RAKEFILE_COMPONENT_FILES.each { |component| load(component) }
 rescue StandardError => e
-  $stderr.puts(e)
+  $stderr.puts("#{e.class} ==> #{e.message}")
+  if @ceedling[:configurator].project_debug
+    $stderr.puts("Backtrace ==>")
+    $stderr.puts(e.backtrace)
+  end
+
   abort # Rake's abort
 end
 

lib/ceedling/setupinator.rb

@@ -20,6 +20,7 @@ def do_setup(config_hash)
     # load up all the constants and accessors our rake files, objects, & external scripts will need;
     # note: configurator modifies the cmock section of the hash with a couple defaults to tie 
     #       project together - the modified hash is used to build cmock object
+    @ceedling[:configurator].set_debug( config_hash )
     @ceedling[:configurator].populate_defaults( config_hash )
     @ceedling[:configurator].populate_unity_defaults( config_hash )
     @ceedling[:configurator].populate_cmock_defaults( config_hash )