Can't open session .nwb files on Matlab or Python

eabhorrocks · February 29, 2020, 10:58pm

I’m trying to open visual-coding neuropixel .nwb files using both Python and Matlab using pynwb and matnwb. When I try and open the session .nwb files I get the following errors:

Python
I’m new to using .nwb files but have managed to write and read an example .nwb file in python.
However, when I try and read in an example neuropixles session using the following code

import numpy as np
from pynwb import NWBHDF5IO

io = NWBHDF5IO('session_767871931.nwb', 'r')
nwbfile_in = io.read()

I get the following error:
ValueError: Column name ‘name’ is not allowed because it is already an attribute

Matlab
After running the generateCore() function I run:

nwb = nwbRead('D:\AllenSDK\session_766640955\session_766640955.nwb')

And get the following error:

Error using types.util.checkUnset (line 13)
Properties {help} are not valid property names.

Error in types.core.TimeSeries (line 82)
types.util.checkUnset(obj, unique(varargin(1:2:end)));

Error in io.parseGroup (line 78)
parsed = eval([typename ‘(kwargs{:})’]);

Error in io.parseGroup (line 38)
subg = io.parseGroup(filename, group, Blacklist);

Error in nwbRead (line 33)
nwb = io.parseGroup(filename, h5info(filename), Blacklist);

From this it looks like there is an issue with the formatting of the session .nwb files that prevents them from being opened to me. Is there anything obvious I’m missing?

Thanks

eabhorrocks · March 2, 2020, 4:56pm

To give an update, in python I have also tried:

from allensdk.core.nwb_data_set import NwbDataSet

file_name = 'D:\AllenSDK\session_767871931\session_767871931.nwb'
data_set = NwbDataSet(file_name)

sweep_numbers = data_set.get_sweep_numbers()
sweep_number = sweep_numbers[0]

But I get the following error

Traceback (most recent call last):
  File "C:/Users/edward.horrocks/PycharmProjects/AllenSDK/venv/loadsessiontest.py", line 7, in <module>
sweep_numbers = data_set.get_sweep_numbers()
  File "C:\Users\edward.horrocks\PycharmProjects\AllenSDK\venv\lib\site-packages\allensdk\core\nwb_data_set.py", line 306, in get_sweep_numbers
for e in f['epochs'].keys() if e.startswith('Sweep_')]
  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "C:\Users\edward.horrocks\PycharmProjects\AllenSDK\venv\lib\site-packages\h5py\_hl\group.py", line 264, in __getitem__
oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py\h5o.pyx", line 190, in h5py.h5o.open
KeyError: "Unable to open object (object 'epochs' doesn't exist)"

nileg · March 2, 2020, 9:46pm

Hi eabhorrocks,

Welcome to the forums!

I think you are running afoul of a different issue in each of your posts. These are:

These NWB files were written using pynwb 1.0.2. Some newer versions of pynwb (issue here) cannot open these files. As a workaround, please try installing 1.0.2 using the allensdk requirements file.
The NwbDataSet in allensdk/core is for intracellular ephys (and NWB1) - you want to instead use EcephysSession from allensdk.brain_observatory.ecephys.ecephys_session. For more information and code examples, see the neuropixels landing page.

The problem you are running into when loading these files via MATLAB is unfamiliar to me. We don’t support MATLAB, but I wonder if you just need to load the extension files from here.

Thank you,
Nile

eabhorrocks · March 3, 2020, 5:00pm

Thanks for the information.

I did not check to see that the ncessary pynwb was already installed with pip install allensdk and had installed the latest version. Thanks very much for pointing out this was where this issue was.

In terms of Matlab, I tried loading those extension files but to no avail. It seems like a similar issue to python whereby file conventions have changed since the ecephys nwb files were generated. Perhaps an older version matnwb would also solve this issue, similarly to python.

Thanks for such a quick response, looking forward to using these datasets.

Edd

JorritMontijn · June 19, 2020, 11:19am

I had a similar problem, where NWB wouldn’t recognize “unit”. I got the following error:

Error using types.util.checkUnset
Properties {unit} are not valid property names.

After much pain and debugging someone else’s uncommented code, I found a work-around. If you add the following code to line 12 of \matnwb+types+util\checkUnset.m it will ignore the extra property of mean_waveform that causes the error:

if any(contains(argin,‘unit’))
anonNames = [anonNames;argin{contains(argin,‘unit’)}];
end

In case you’re looking for a quick fix and don’t want to dive into the code: I also made a fork of the 2.2.2 NWB branch here: https://github.com/JorritMontijn/matnwb. Note that the above does NOT solve the problem, it merely ignores it. I also won’t be updating the above fork, as I assume either the Allen Brain Institute or the NWB people (or both) will eventually solve the problem.

Also note that I haven’t done any tests wrt the actual data you retrieve. The only thing I can say with certainty is that nwbRead() now returns an object.

sarruedi · April 22, 2021, 5:09pm

I run into the same problem when trying to open ecephys files from AllenSDK.

nwb_test = nwbRead(‘C:\local1\ecephys_cache_dir\session_732592105\session_732592105.nwb’)
Error using types.util.checkUnset (line 13)
Unexpected properties {unit}.

Your schema version may be incompatible with the file. Consider checking the schema version of the file with
util.getSchemaVersion(filename) and comparing with the YAML namespace version present in nwb-schema/core/nwb.namespace.yaml

Error in types.hdmf_common.VectorData (line 26)
types.util.checkUnset(obj, unique(varargin(1:2:end)));

Error in io.parseDataset (line 72)
parsed = eval([Type.typename ‘(kwargs{:})’]);

Error in io.parseGroup (line 22)
dataset = io.parseDataset(filename, datasetInfo, fullPath, Blacklist);

Error in io.parseGroup (line 38)
subg = io.parseGroup(filename, group, Blacklist);

Error in nwbRead (line 33)
nwb = io.parseGroup(filename, h5info(filename), Blacklist);

I checked the YAML namespace version and is version 2.2.2. Accordingly I downgraded the version of thenwb-schema\core to the files of the version 2.2.2 but the error this remains after I regenerate the core using
generateCore();
generateExtension(‘R:\AllenSDK-master\allensdk\brain_observatory\ecephys\nwb\ndx-aibs-ecephys.namespace.yaml’);

Did anybody figure out how to solve this loading problem?

wayne · April 22, 2021, 7:05pm

Hello @sarruedi

I notice that you mention AllenSDK but you are getting an error using a pynwb function to read the file. Are you able to load the file using AllenSDK functionality? Here is what I did to load the file (based on this quickstart example notebook)

# using AllenSDK 2.10.2 on Python 3.7.6
from allensdk.brain_observatory.ecephys.ecephys_project_cache import EcephysProjectCache
cache = EcephysProjectCache.from_warehouse()
session_id = 732592105
session = cache.get_session_data(session_id)
session.metadata

This documentation page also links to other documentation and notebooks that may be helpful: Visual Coding – Neuropixels — Allen SDK dev documentation

Best regards,
Wayne

Sarruedi1 · April 22, 2021, 7:18pm

I am running into the same error when trying to load an nwb file using Matlab (downloaded from AllenSDK) and can’t open the ecephys files.

nwb = nwbRead(‘C:\local1\ecephys_cache_dir\session_732592105\session_732592105.nwb’)
Error using types.util.checkUnset (line 13)
Unexpected properties {unit}.

Your schema version may be incompatible with the file. Consider checking the schema version of the file with
util.getSchemaVersion(filename) and comparing with the YAML namespace version present in nwb-schema/core/nwb.namespace.yaml

The YAML namespace version of the nwb file is 2.2.2

I can load and process the data using the AllenSDK but I would like to load the data into Matlab instead.

I tried to fix the issue with the workaround from Jorrit (adding the lines of code he suggested).
Using this modification that ignores ‘units’ I am at least able to open a file but that somewhat defeats the purpose as the ‘unit’ information is what I would like to open.

NwbFile with properties:

                            nwb_version: '2.2.2'
                            acquisition: [3×1 types.untyped.Set]
                               analysis: [0×1 types.untyped.Set]
                       file_create_date: [1×1 types.untyped.DataStub]
                                general: [0×1 types.untyped.Set]
                general_data_collection: []
                        general_devices: [5×1 types.untyped.Set]
         general_experiment_description: []
                   general_experimenter: []
            general_extracellular_ephys: [5×1 types.untyped.Set]
 general_extracellular_ephys_electrodes: [1×1 types.hdmf_common.DynamicTable]
                    general_institution: 'Allen Institute for Brain Science'
            general_intracellular_ephys: [0×1 types.untyped.Set]
  general_intracellular_ephys_filtering: []
general_intracellular_ephys_sweep_table: []
                       general_keywords: []
                            general_lab: []
                          general_notes: []
                   general_optogenetics: [0×1 types.untyped.Set]
                 general_optophysiology: [0×1 types.untyped.Set]
                   general_pharmacology: []
                       general_protocol: []
           general_related_publications: []
                     general_session_id: '732592105'
                         general_slices: []
                  general_source_script: []
        general_source_script_file_name: []
                       general_stimulus: 'brain_observatory_1.1'
                        general_subject: [1×1 types.ndx_aibs_ecephys.EcephysSpecimen]
                        general_surgery: []
                          general_virus: []
                             identifier: '732592105'
                              intervals: [8×1 types.untyped.Set]
                       intervals_epochs: []
                intervals_invalid_times: [1×1 types.core.TimeIntervals]
                       intervals_trials: []
                             processing: [3×1 types.untyped.Set]
                                scratch: [0×1 types.untyped.Set]
                    session_description: 'Data and metadata for an Ecephys session'
                     session_start_time: 2019-01-08T16:26:20.000000-08:00
                  stimulus_presentation: [0×1 types.untyped.Set]
                     stimulus_templates: [0×1 types.untyped.Set]
              timestamps_reference_time: 2019-01-08T16:26:20.000000-08:00
                                  units: [1×1 types.core.Units]

wayne · April 22, 2021, 7:50pm

Hi @Sarruedi1 @sarruedi

I may not be able to help you with Matlab; I don’t even have a license for it.

Can you tell me what version of pynwb and hdmf are installed in your environment?

Thanks,
Wayne

wayne · April 22, 2021, 8:33pm

@Sarruedi1 @sarruedi With pynwb, there is a load option to use namespace. It is load_namespaces=True. This should solve the problem for pynwb. I don’t know if there is an equivalent for matnwb; you will need to contact that project to find out.

Thus, this works in Python:
import numpy as np
from pynwb import NWBHDF5IO
io = NWBHDF5IO(‘session_732592105.nwb’, mode=‘r’, load_namespaces=True)
nwbfile_in = io.read()

sarruedi · April 22, 2021, 10:38pm

Thanks a lot for the feedback Wayne,

I’ll use python to load the data for the time being. Still trying to figure out the issue of matnwb and the namespace.

Best
Sarah

Topic		Replies	Views
NWB Schema Version For Multimodal Characterization in Mouse Visual Cortex Technical nwb , allensdk , how-to , software	3	528	January 28, 2024
What are the features of the data and how to open? Technical how-to	1	404	August 1, 2022
AllenSDK 2.0 Release - Notice for Visual Coding Neuropixels version compatability Technical brain-observatory-visual-coding , nwb , allensdk	1	1051	June 10, 2020
Problem reading NWB files Technical nwb	2	465	March 1, 2023
Can't find NWB_adapter from swdb_2018_neuropixels.ephys_nwb_adapter Technical	5	294	July 26, 2023

Can't open session .nwb files on Matlab or Python

Related topics