Pass the catalog as a node argument

[echang1802]

Hi all!

I'm using Kedros version 0.16.6 and trying to pass the catalog as a argument to some nodes, is this posible?

I sucefully define the pipeline but when it try to run return:

ValueError: Pipeline input(s) {'catalog'} not found in the DataCatalog

Tracking the error to the runner.py file and I think the problem is their is in this validation:

unsatisfied = pipeline.inputs() - set(catalog.list())
if unsatisfied:
    raise ValueError(
        f"Pipeline input(s) {unsatisfied} not found in the DataCatalog"
    )

There is no "catalog" object inside the catalog, rigth?

Greetings.

[mzjp2]

Indeed, you aren’t expected to use the catalog within the node - rather, you should specify which entries of the catalog you want available inside your node as python objects and Kedro’s catalog will handle loading them in for you.

Is there a specific problem you’re trying to tackle by getting access to the entire catalog inside the node?

[echang1802]

Hi Zain!

Yes, I got several nodes that do exacly the same, just like

def default_node(option, catalog):
    module_1()
    module_2(option, catalog)
    module_3()
    ...

The module_2 is the one who actually read the datasets, but the datasets will vary for each value of option (also differ in the number of datasets), which are several of them and I want to avoid using a large number of if statements, my module_2 is:

def module_2(option, catalog):
    option_function = import_option_module(option)
    df = option_function(catalog)    
    ....

def import_option_module(option):
    from importlib import import_module as imp_mod
    pkg = f"aa_engine_pkg.pipelines.modules.module_for_{option}"
    module = imp_mod(pkg)
    return getattr(module, f"module_for_{option}")

So each module_for_{option} function read the needed dataset from the catalog directly, to avoid:

def module_2(option, dataset_1 = None, dataset_2= None, ....):
    if option == 1:
         df = module_for_1(dataset_1)
   elif option == 2:
         df = module_for_2(dataset_2, dataset_3)    
    ....

My final nodes will be:

def node_1(catalog):
    default_node(1, catalog)

def node_2(catalog):
    default_node(2, catalog)
....

This happen in more than one of this modules of the nodes.

[datajoely]

@echang1802 is this not somewhere a hook could be helpful?

You have the catalog available to you before_node_run for example:

[echang1802]

@datajoely thanks! I think this could help!

[echang1802]

Hi all!

I solved it using kwarg.

First defining the nodes as usually:

def node_1(dataset_1):
    default_node(1, dataset_1 = dataset_1)

def node_2(dataset_2, dataset_3):
    default_node(1, dataset_2 = dataset_2, dataset_3 = dataset_3)

Then, the module_2 function as:

def module_2(option, **kwarg):
    option_function = import_option_module(option)
    df = option_function(kwarg)    
    ...

Finally using the kwarg as the catalog inside the module_for_{option} functions:

def module_for_1(catalog):
    dataset_1 = catalog["dataset_1"]
    ...

Thanks you all for your help.

Regards.