[Bug]: File written with MatNWB fails to open with PyNWB

[arnodelorme]

What happened?

This file saved using the NwbFile object

https://nemar-dev.ucsd.edu/dataexplorer/download?filepath=/expanse/projects/nemar/openneuro/nwb_example_files/sub-005_ieeg.nwb

Can be reimported in MATLAB using the same object. However, it fails to be reimported in Python. @dungscout96, you used the two different Python libraries right? Can you explain further?

Steps to Reproduce

Try importing the file in Python.

Error Message

Operating System

macOS

Matlab Version

2024b

Code of Conduct

• I agree to follow this project's Code of Conduct
• Have you ensured this bug was not already reported?

[ehennestad]

Hi,

I have tried to read the file with both MatNWB and PyNWB (v2.8.1), and both worked. What version of PyNWB do you use?
(Use import pynwb; pynwb.__version__ to check)

[arnodelorme]

@dungscout96 can you comment?

[arnodelorme]

Oh this is actually this file https://testspeedeegdash.s3.us-east-2.amazonaws.com/sub-002_task-FaceRecognition_eeg.nwb -- I have just imported it in MATLAB and it was fine (it was exported using MATLAB tools). However, it fails in Python.

[dungscout96]

Here's the error I'm seeing using pynwb version 2.8.3:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py:1369, in ObjectMapper.construct(self, **kwargs)
   1368 try:
-> 1369     obj = self.__new_container__(cls, builder.source, parent, builder.attributes.get(self.__spec.id_key()),
   1370                                  **kwargs)
   1371 except Exception as ex:

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py:1382, in ObjectMapper.__new_container__(self, cls, container_source, parent, object_id, **kwargs)
   1380 # obj has been created and is in construction mode, indicating that the object is being constructed by
   1381 # the automatic construct process during read, rather than by the user
-> 1382 obj.__init__(**kwargs)
   1383 obj._in_construct_mode = False  # reset to False to indicate that the construction of the object is complete

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/utils.py:667, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    666 def func_call(*args, **kwargs):
--> 667     pargs = _check_args(args, kwargs)
    668     return func(args[0], **pargs)

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/utils.py:660, in docval.<locals>.dec.<locals>._check_args(args, kwargs)
    659         msg = '%s: %s' % (func.__qualname__, ', '.join(parse_err))
--> 660         raise ExceptionType(msg)
    662 return parsed['args']

TypeError: ElectrodeGroup.__init__: incorrect type for 'location' (got 'ndarray', expected 'str')

The above exception was the direct cause of the following exception:

ConstructError                            Traceback (most recent call last)
Cell In[1], line 3
      1 from pynwb import NWBHDF5IO
      2 with NWBHDF5IO('/Users/dtyoung/Downloads/sub-002_task-FaceRecognition_eeg.nwb', 'r') as io:
----> 3     nwbfile = io.read()

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/utils.py:668, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    666 def func_call(*args, **kwargs):
    667     pargs = _check_args(args, kwargs)
--> 668     return func(args[0], **pargs)

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/pynwb/__init__.py:458, in NWBHDF5IO.read(self, **kwargs)
    455         raise TypeError("NWB version %s not supported. PyNWB supports NWB files version 2 and above." %
    456                         str(file_version_str))
    457 # read the file
--> 458 file = super().read(**kwargs)
    459 return file

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/backends/hdf5/h5tools.py:501, in HDF5IO.read(self, **kwargs)
    498     raise UnsupportedOperation("Cannot read from file %s in mode '%s'. Please use mode 'r', 'r+', or 'a'."
    499                                % (self.source, self.__mode))
    500 try:
--> 501     return super().read(**kwargs)
    502 except UnsupportedOperation as e:
    503     if str(e) == 'Cannot build data. There are no values.':  # pragma: no cover

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/utils.py:668, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    666 def func_call(*args, **kwargs):
    667     pargs = _check_args(args, kwargs)
--> 668     return func(args[0], **pargs)

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/backends/io.py:60, in HDMFIO.read(self, **kwargs)
     57 if all(len(v) == 0 for v in f_builder.values()):
     58     # TODO also check that the keys are appropriate. print a better error message
     59     raise UnsupportedOperation('Cannot build data. There are no values.')
---> 60 container = self.__manager.construct(f_builder)
     61 container.read_io = self
     62 if self.herd_path is not None:

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/utils.py:668, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    666 def func_call(*args, **kwargs):
    667     pargs = _check_args(args, kwargs)
--> 668     return func(args[0], **pargs)

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/manager.py:286, in BuildManager.construct(self, **kwargs)
    282     result = self.__type_map.construct(builder, self, parent)
    283 else:
    284     # we are at the top of the hierarchy,
    285     # so it must be time to resolve parents
--> 286     result = self.__type_map.construct(builder, self, None)
    287     self.__resolve_parents(result)
    288 self.prebuilt(result, builder)

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/utils.py:668, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    666 def func_call(*args, **kwargs):
    667     pargs = _check_args(args, kwargs)
--> 668     return func(args[0], **pargs)

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/manager.py:827, in TypeMap.construct(self, **kwargs)
    825     raise ValueError('No ObjectMapper found for builder of type %s' % dt)
    826 else:
--> 827     return obj_mapper.construct(builder, build_manager, parent)

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/utils.py:668, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    666 def func_call(*args, **kwargs):
    667     pargs = _check_args(args, kwargs)
--> 668     return func(args[0], **pargs)

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py:1339, in ObjectMapper.construct(self, **kwargs)
   1337 cls = manager.get_cls(builder)
   1338 # gather all subspecs
-> 1339 subspecs = self.__get_subspec_values(builder, self.spec, manager)
   1340 # get the constructor argument that each specification corresponds to
   1341 const_args = dict()

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py:1268, in ObjectMapper.__get_subspec_values(self, builder, spec, manager)
   1266                 ret[subspec] = self.__flatten(sub_builder, subspec, manager)
   1267     # now process groups and datasets
-> 1268     self.__get_sub_builders(groups, spec.groups, manager, ret)
   1269     self.__get_sub_builders(datasets, spec.datasets, manager, ret)
   1270 elif isinstance(spec, DatasetSpec):

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py:1319, in ObjectMapper.__get_sub_builders(self, sub_builders, subspecs, manager, ret)
   1316     continue
   1317 if dt is None:
   1318     # recurse
-> 1319     ret.update(self.__get_subspec_values(sub_builder, subspec, manager))
   1320 else:
   1321     ret[subspec] = manager.construct(sub_builder)

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py:1268, in ObjectMapper.__get_subspec_values(self, builder, spec, manager)
   1266                 ret[subspec] = self.__flatten(sub_builder, subspec, manager)
   1267     # now process groups and datasets
-> 1268     self.__get_sub_builders(groups, spec.groups, manager, ret)
   1269     self.__get_sub_builders(datasets, spec.datasets, manager, ret)
   1270 elif isinstance(spec, DatasetSpec):

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py:1319, in ObjectMapper.__get_sub_builders(self, sub_builders, subspecs, manager, ret)
   1316     continue
   1317 if dt is None:
   1318     # recurse
-> 1319     ret.update(self.__get_subspec_values(sub_builder, subspec, manager))
   1320 else:
   1321     ret[subspec] = manager.construct(sub_builder)

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py:1268, in ObjectMapper.__get_subspec_values(self, builder, spec, manager)
   1266                 ret[subspec] = self.__flatten(sub_builder, subspec, manager)
   1267     # now process groups and datasets
-> 1268     self.__get_sub_builders(groups, spec.groups, manager, ret)
   1269     self.__get_sub_builders(datasets, spec.datasets, manager, ret)
   1270 elif isinstance(spec, DatasetSpec):

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py:1311, in ObjectMapper.__get_sub_builders(self, sub_builders, subspecs, manager, ret)
   1309     sub_builder = builder_dt.get(dt)
   1310     if sub_builder is not None:
-> 1311         sub_builder = self.__flatten(sub_builder, subspec, manager)
   1312         ret[subspec] = sub_builder
   1313 else:

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py:1324, in ObjectMapper.__flatten(self, sub_builder, subspec, manager)
   1323 def __flatten(self, sub_builder, subspec, manager):
-> 1324     tmp = [manager.construct(b) for b in sub_builder]
   1325     if len(tmp) == 1 and not subspec.is_many():
   1326         tmp = tmp[0]

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py:1324, in <listcomp>(.0)
   1323 def __flatten(self, sub_builder, subspec, manager):
-> 1324     tmp = [manager.construct(b) for b in sub_builder]
   1325     if len(tmp) == 1 and not subspec.is_many():
   1326         tmp = tmp[0]

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/utils.py:668, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    666 def func_call(*args, **kwargs):
    667     pargs = _check_args(args, kwargs)
--> 668     return func(args[0], **pargs)

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/manager.py:282, in BuildManager.construct(self, **kwargs)
    280 if parent_builder is not None:
    281     parent = self._get_proxy_builder(parent_builder)
--> 282     result = self.__type_map.construct(builder, self, parent)
    283 else:
    284     # we are at the top of the hierarchy,
    285     # so it must be time to resolve parents
    286     result = self.__type_map.construct(builder, self, None)

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/utils.py:668, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    666 def func_call(*args, **kwargs):
    667     pargs = _check_args(args, kwargs)
--> 668     return func(args[0], **pargs)

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/manager.py:827, in TypeMap.construct(self, **kwargs)
    825     raise ValueError('No ObjectMapper found for builder of type %s' % dt)
    826 else:
--> 827     return obj_mapper.construct(builder, build_manager, parent)

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/utils.py:668, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    666 def func_call(*args, **kwargs):
    667     pargs = _check_args(args, kwargs)
--> 668     return func(args[0], **pargs)

File ~/Documents/EEGLAB/EEG-Dash-Data/.venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py:1373, in ObjectMapper.construct(self, **kwargs)
   1371 except Exception as ex:
   1372     msg = 'Could not construct %s object due to: %s' % (cls.__name__, ex)
-> 1373     raise ConstructError(builder, msg) from ex
   1374 return obj

ConstructError: (root/general/extracellular_ephys/all_electrodes GroupBuilder {'attributes': {'description': 'all electrodes', 'location': array([], dtype=object), 'namespace': 'core', 'neurodata_type': 'ElectrodeGroup', 'object_id': '2a87fe4b-0cc3-458d-988b-34b55a2826d9'}, 'groups': {}, 'datasets': {}, 'links': {'device': root/general/extracellular_ephys/all_electrodes/device LinkBuilder {'builder': root/general/devices/array GroupBuilder {'attributes': {'namespace': 'core', 'neurodata_type': 'Device', 'object_id': '5f38dc95-4ad3-4700-a70a-58f803b99c82'}, 'groups': {}, 'datasets': {}, 'links': {}}}}}, "Could not construct ElectrodeGroup object due to: ElectrodeGroup.__init__: incorrect type for 'location' (got 'ndarray', expected 'str')")

Here's the code I'm running:

from pynwb import NWBHDF5IO
with NWBHDF5IO('/Users/dtyoung/Downloads/sub-002_task-FaceRecognition_eeg.nwb', 'r') as io:
    nwbfile = io.read()

[ehennestad]

Thanks for providing the file and the error trace. I can reproduce this now!

@stephprince , Can you weigh in:

There is one ElectrodeGroup in the h5 location "root/general/extracellular_ephys/all_electrodes".

The location property of the ElectrodeGroup is a string in MatNWB, but it is empty which should be fine. The corresponding dataset attribute in the HDF5 file also appears to be a string type (see screenshot below), but the PyNWB error says that it gets an ndarray value:

ElectrodeGroup.__init__: incorrect type for 'location' (got 'ndarray', expected 'str')")

[oruebel]

The corresponding dataset attribute in the HDF5 file also appears to be a string type (see screenshot below), but the PyNWB error says that it gets an ndarray value:

Location should be a Scalar attribute of type string. However, according to the screenshot you shared it appears to be an array of strings with 0 elements instead. If the string is empty then it should be e an empty scalar string not an empty array.