Merge pull request #2400 from stfc/1508_disable_backend_checks

(closes #1508) allow PSyIR backend checks to be disabled

changelog

@@ -84,6 +84,8 @@
 	31) PR #2388 for #2323. Rename the array reduction 2code transformations
 	to use the 2loop postfix and improve their implementation. 
 
+	32) PR #2400 for #1508. Add flag/config option to disable backend checks.
+
 release 2.4.0 29th of September 2023
 
 	1) PR #1758 for #1741. Splits the PSyData read functionality into a

doc/developer_guide/psyir_backends.rst

@@ -1,7 +1,7 @@
 .. -----------------------------------------------------------------------------
    BSD 3-Clause License
 
-   Copyright (c) 2017-2022, Science and Technology Facilities Council.
+   Copyright (c) 2017-2023, Science and Technology Facilities Council.
    All rights reserved.
 
    Redistribution and use in source and binary forms, with or without
@@ -34,6 +34,7 @@
    Authors: R. W. Ford, A. R. Porter, S. Siso and N. Nobre, STFC Daresbury Lab
 
 
+.. _psyir-backends:
 
 PSyIR Back-ends
 ###############

doc/user_guide/configuration.rst

@@ -1,7 +1,7 @@
 .. -----------------------------------------------------------------------------
 .. BSD 3-Clause License
 ..
-.. Copyright (c) 2018-2022, Science and Technology Facilities Council
+.. Copyright (c) 2018-2023, Science and Technology Facilities Council
 .. All rights reserved.
 ..
 .. Redistribution and use in source and binary forms, with or without
@@ -49,7 +49,7 @@ is in the ``PSyclone/config`` directory of the PSyclone
 distribution.
 
 At execution-time, the user can specify a custom configuration file to
-be used. This can either be done with the ``--config`` command line
+be used. This can either be done with the ``--config`` command-line
 option, or by specifying the (full path to the) configuration file
 to use via the ``PSYCLONE_CONFIG`` environment variable. If the specified
 configuration file is not found then PSyclone will fall back to
@@ -141,18 +141,20 @@ including "true/false", "yes/no" and "1/0". See
 https://docs.python.org/3/library/configparser.html#supported-datatypes
 for more details.
 
+.. _config-default-section:
+
 ``DEFAULT`` Section
 ^^^^^^^^^^^^^^^^^^^
 
 This section contains entries that are, in principle, applicable to all APIs
 supported by PSyclone.
 
-.. tabularcolumns:: |l|L|
+.. tabularcolumns:: |l|L|l|
 
-======================= =======================================================
-Entry                   Description
-======================= =======================================================
-DEFAULTAPI              The API that PSyclone assumes an Algorithm/Kernel
+======================= ======================================================= ===========
+Entry                   Description                                             Type
+======================= ======================================================= ===========
+DEFAULTAPI              The API that PSyclone assumes an Algorithm/Kernel       str
                         conforms to if no API is specified. Must be one of the
                         APIs supported by PSyclone ("dynamo0.3", "gocean1.0"
                         and "nemo"). If there is no
@@ -162,26 +164,30 @@ DEFAULTAPI              The API that PSyclone assumes an Algorithm/Kernel
                         line option '-api'. If there is no API entry in the
                         config file, and '-api' is not specified on the 
                         command line, "dynamo0.3" is used as default.
-DEFAULTSTUBAPI          The API that the kernel-stub generator assumes by
+DEFAULTSTUBAPI          The API that the kernel-stub generator assumes by       str
                         default. Must be one of the stub-APIs supported by
                         PSyclone ("dynamo0.3" only at this stage).
-DISTRIBUTED_MEMORY      Whether or not to generate code for distributed-memory
+DISTRIBUTED_MEMORY      Whether or not to generate code for distributed-memory  bool
                         parallelism by default.  Note that this is currently
                         only supported for the LFRic (Dynamo 0.3) API.
-REPRODUCIBLE_REDUCTIONS Whether or not to generate code for reproducible OpenMP
+REPRODUCIBLE_REDUCTIONS Whether or not to generate code for reproducible OpenMP bool
                         reductions (see :ref:`openmp-reductions`) by default.
-REPROD_PAD_SIZE         If generating code for reproducible OpenMP reductions,
+REPROD_PAD_SIZE         If generating code for reproducible OpenMP reductions,  int
                         this setting controls the amount of padding used
                         between elements of the array in which each thread
                         accumulates its local reduction. (This prevents false
                         sharing of cache lines by different threads.)
-PSYIR_ROOT_NAME         The root for generated PSyIR symbol names if one is not
+PSYIR_ROOT_NAME         The root for generated PSyIR symbol names if one is not str
                         supplied when creating a symbol. Defaults to
                         "psyir_tmp".
-VALID_PSY_DATA_PREFIXES Which class prefixes are permitted in any
+VALID_PSY_DATA_PREFIXES Which class prefixes are permitted in any               list of str
                         PSyData-related transformations. See :ref:`psy_data`
                         for details.
-======================= =======================================================
+BACKEND_CHECKS_ENABLED  Optional (defaults to True). Whether or not the PSyIR   bool
+                        backend should validate the tree that it is passed.
+                        Can be overridden by the ``--backend`` command-line
+                        flag (see :ref:`backend-options`).
+======================= ======================================================= ===========
 
 Common Sections
 ^^^^^^^^^^^^^^^

doc/user_guide/psyclone_command.rst

@@ -65,7 +65,9 @@ by the command:
   usage: psyclone [-h] [-oalg OALG] [-opsy OPSY] [-okern OKERN] [-api API]
                   [-s SCRIPT] [-d DIRECTORY] [-I INCLUDE] [-l {off,all,output}]
                   [-dm] [-nodm] [--kernel-renaming {multiple,single}]
-                  [--profile {invokes,kernels}] [--config CONFIG] [--version]
+                  [--profile {invokes,kernels}]
+		  [--backend {enable-validation,disable-validation}]
+		  [--config CONFIG] [--version]
                   filename
 
   Run the PSyclone code generator on a particular file
@@ -101,6 +103,10 @@ by the command:
                           kernels
     --profile {invokes,kernels}, -p {invokes,kernels}
                           Add profiling hooks for either 'kernels' or 'invokes'
+    --backend {dis,en}able-validation
+                          Options to control the PSyIR backend used for code
+                          generation. Use 'disable-validation' to disable the
+                          validation checks that are performed by default.
     --config CONFIG       Config file with PSyclone specific options.
     --version, -v         Display version information (\ |release|\ )
 
@@ -379,6 +385,30 @@ by the ``-I``/``--include`` flags. (Currently this search assumes that a
 module named e.g. "my_mod" will be in a file named "my_mod.*90" - see issue
 #1895.)
 
+.. _backend-options:
+
+Backend Options
+---------------
+
+The final code generated by PSyclone is created by passing the PSyIR
+tree to one of the 'backends' (see :ref:`dev_guide:psyir-backends` in
+the Developer Guide for more details). The ``--backend`` flag permits
+a user to tune the behaviour of this code generation. Currently, the
+only option is ``{en,dis}able-validation`` which turns on/off the
+validation checks performed when doing code generation. By default,
+such validation is enabled as it is only at code-generation time that
+certain constraints can be checked (since PSyclone does not mandate
+the order in which code transformations are applied).  Occasionally,
+these validation checks may raise false positives (due to incomplete
+implementations), at which point it is useful to be able to disable
+them.  The default behaviour may be changed by adding the
+``BACKEND_CHECKS_ENABLED`` entry to the
+:ref:`configuration file <config-default-section>`. Any
+command-line setting always takes precendence though. It is
+recommended that validation only be disabled as a last resort and for
+as few input source files as possible.
+
+
 C Pre-processor #include Files
 ------------------------------
 

doc/user_guide/psyir.rst

@@ -67,13 +67,12 @@ collectively as 'PSyIR nodes'.
 
 At the present time PSyIR classes can be essentially split into two
 types. PSy-layer classes and Kernel-layer classes. PSy-layer classes
-make use of a ``gen_code()`` or an ``update()`` method to create
-Fortran code whereas Kernel-layer classes make use of PSyIR backends
-to create code.
+make use of a ``gen_code()`` method to create Fortran code whereas
+Kernel-layer classes make use of PSyIR backends to create code.
 
 .. note:: This separation will be removed in the future and eventually
 	  all PSyIR classes will make use of backends with the
-	  expectation that ``gen_code()`` and ``update()`` methods
+	  expectation that ``gen_code()`` methods
 	  will be removed. Further this separation will be superseded
 	  by a separation between ``language-level PSyIR`` and
 	  ``domain-specific PSyIR``.
@@ -229,7 +228,7 @@ Following the `parent` and `children` terminology, we define a node's `siblings`
 as the children of its parent. Note that this definition implies that all nodes
 are their own siblings.
 
-.. automethod:: psyclone.psyir.nodes.Node.siblings
+.. autoproperty:: psyclone.psyir.nodes.Node.siblings
 
 We can check whether two nodes are siblings which immediately precede or follow
 one another using the following methods:

src/psyclone/configuration.py

@@ -223,6 +223,11 @@ def __init__(self):
         # Number of OpenCL devices per node
         self._ocl_devices_per_node = 1
 
+        # By default, a PSyIR backend performs validation checks as it
+        # traverses the tree. Setting this option to False disables those
+        # checks which can be useful in the case of unimplemented features.
+        self._backend_checks_enabled = True
+
     # -------------------------------------------------------------------------
     def load(self, config_file=None):
         '''Loads a configuration file.
@@ -358,6 +363,17 @@ def load(self, config_file=None):
                     f"prefix must be valid for use as the start of a Fortran "
                     f"variable name.", config=self)
 
+        # Whether validation is performed in the PSyIR backends.
+        if 'BACKEND_CHECKS_ENABLED' in self._config['DEFAULT']:
+            try:
+                self._backend_checks_enabled = (
+                    self._config['DEFAULT'].getboolean(
+                        'BACKEND_CHECKS_ENABLED'))
+            except ValueError as err:
+                raise ConfigurationError(
+                    f"Error while parsing BACKEND_CHECKS_ENABLED: {err}",
+                    config=self) from err
+
         # Now we deal with the API-specific sections of the config file. We
         # create a dictionary to hold the API-specific Config objects.
         self._api_conf = {}
@@ -547,6 +563,31 @@ def default_stub_api(self):
         '''
         return self._default_stub_api
 
+    @property
+    def backend_checks_enabled(self):
+        '''
+        :returns: whether the validity checks in the PSyIR backend should be
+                  disabled.
+        :rtype: bool
+        '''
+        return self._backend_checks_enabled
+
+    @backend_checks_enabled.setter
+    def backend_checks_enabled(self, value):
+        '''
+        Setter for whether or not the PSyIR backend is to perform validation
+        checks.
+
+        :param bool value: whether or not to perform validation.
+
+        :raises TypeError: if `value` is not a boolean.
+
+        '''
+        if not isinstance(value, bool):
+            raise TypeError(f"Config.backend_checks_enabled must be a boolean "
+                            f"but got '{type(value).__name__}'")
+        self._backend_checks_enabled = value
+
     @property
     def supported_stub_apis(self):
         '''

src/psyclone/f2pygen.py

@@ -1,7 +1,7 @@
 # -----------------------------------------------------------------------------
 # BSD 3-Clause License
 #
-# Copyright (c) 2017-2022 and Technology Facilities Council.
+# Copyright (c) 2017-2023 and Technology Facilities Council.
 # All rights reserved.
 #
 # Redistribution and use in source and binary forms, with or without
@@ -37,7 +37,6 @@
 ''' Fortran code-generation library. This wraps the f2py fortran parser to
     provide routines which can be used to generate fortran code. '''
 
-from __future__ import absolute_import, print_function
 import abc
 from fparser.common.readfortran import FortranStringReader
 from fparser.common.sourceinfo import FortranFormat
@@ -48,6 +47,7 @@
 # cannot be used for imports (as that involves looking for the
 # specified name in sys.modules).
 from fparser import one as fparser1
+from psyclone.configuration import Config
 from psyclone.errors import InternalError
 
 # Module-wide utility methods
@@ -547,11 +547,15 @@ def __init__(self, parent, content):
         # Import FortranWriter here to avoid circular-dependency
         # pylint: disable=import-outside-toplevel
         from psyclone.psyir.backend.fortran import FortranWriter
+        # We need the Config object in order to see whether or not to disable
+        # the validation performed in the PSyIR backend.
+        config = Config.get()
 
         # Use the PSyIR Fortran backend to generate Fortran code of the
         # supplied PSyIR tree and pass the resulting code to the fparser1
         # Fortran parser.
-        fortran_writer = FortranWriter()
+        fortran_writer = FortranWriter(
+            check_global_constraints=config.backend_checks_enabled)
         reader = FortranStringReader(fortran_writer(content),
                                      ignore_comments=False)
         # Set reader as free form, strict

src/psyclone/generator.py

@@ -476,6 +476,12 @@ def main(args):
     parser.add_argument(
         '--profile', '-p', action="append", choices=Profiler.SUPPORTED_OPTIONS,
         help="Add profiling hooks for either 'kernels' or 'invokes'")
+    parser.add_argument(
+        '--backend', dest='backend',
+        choices=['enable-validation', 'disable-validation'],
+        help=("Options to control the PSyIR backend used for code generation. "
+              "Use 'disable-validation' to disable the validation checks that "
+              "are performed by default."))
     parser.set_defaults(dist_mem=Config.get().distributed_memory)
 
     parser.add_argument("--config", help="Config file with "
@@ -526,6 +532,12 @@ def main(args):
         api = args.api
         Config.get().api = api
 
+    if args.backend:
+        # A command-line flag overrides the setting in the Config file (if
+        # any).
+        Config.get().backend_checks_enabled = (
+            str(args.backend) == "enable-validation")
+
     # The Configuration manager checks that the supplied path(s) is/are
     # valid so protect with a try
     try:

src/psyclone/nemo.py

@@ -152,7 +152,8 @@ def gen(self):
         :rtype: str
 
         '''
-        fwriter = FortranWriter()
+        enable_checks = Config.get().backend_checks_enabled
+        fwriter = FortranWriter(check_global_constraints=enable_checks)
         return fwriter(self._container)
 
 

src/psyclone/psyGen.py

@@ -41,6 +41,7 @@
     and generation. The classes in this method need to be specialised for a
     particular API and implementation. '''
 
+import os
 from collections import OrderedDict
 import abc
 
@@ -1662,9 +1663,10 @@ def rename_and_write(self):
                                      is also flagged for module-inlining.
 
         '''
-        import os
         from psyclone.line_length import FortLineLength
 
+        config = Config.get()
+
         # If this kernel has not been transformed we do nothing, also if the
         # kernel has been module-inlined, the routine already exist in the
         # PSyIR and we don't need to generate a new file with it.
@@ -1696,12 +1698,12 @@ def rename_and_write(self):
                 # Atomically attempt to open the new kernel file (in case
                 # this is part of a parallel build)
                 fdesc = os.open(
-                    os.path.join(Config.get().kernel_output_dir, new_name),
+                    os.path.join(config.kernel_output_dir, new_name),
                     os.O_CREAT | os.O_WRONLY | os.O_EXCL)
             except (OSError, IOError):
                 # The os.O_CREATE and os.O_EXCL flags in combination mean
                 # that open() raises an error if the file exists
-                if Config.get().kernel_naming == "single":
+                if config.kernel_naming == "single":
                     # If the kernel-renaming scheme is such that we only ever
                     # create one copy of a transformed kernel then we're done
                     break
@@ -1718,7 +1720,8 @@ def rename_and_write(self):
         # file using a PSyIR back-end. At the moment there is no way to choose
         # which back-end to use, so simply use the Fortran one (and limit the
         # line length).
-        fortran_writer = FortranWriter()
+        fortran_writer = FortranWriter(
+            check_global_constraints=config.backend_checks_enabled)
         # Start from the root of the schedule as we want to output
         # any module information surrounding the kernel subroutine
         # as well as the subroutine itself.
@@ -1731,18 +1734,18 @@ def rename_and_write(self):
             # because the file already exists and the kernel-naming scheme
             # ("single") means we're not creating a new one.
             # Check that what we've got is the same as what's in the file
-            with open(os.path.join(Config.get().kernel_output_dir,
+            with open(os.path.join(config.kernel_output_dir,
                                    new_name), "r") as ffile:
                 kern_code = ffile.read()
                 if kern_code != new_kern_code:
                     raise GenerationError(
                         f"A transformed version of this Kernel "
                         f"'{self._module_name + '''.f90'''}' already exists "
                         f"in the kernel-output directory "
-                        f"({Config.get().kernel_output_dir}) but is not the "
+                        f"({config.kernel_output_dir}) but is not the "
                         f"same as the current, transformed kernel and the "
                         f"kernel-renaming scheme is set to "
-                        f"'{Config.get().kernel_naming}'. (If you wish to"
+                        f"'{config.kernel_naming}'. (If you wish to"
                         f" generate a new, unique kernel for every kernel "
                         f"that is transformed then use "
                         f"'--kernel-renaming multiple'.)")