Unable to read session.units and get lfp data

[chiyu1203]

Dear all

This might be a problem in the environment of my PC but I was hoping to get some ideas about how to fix these problems.

I was able to download the session .ned file from my PC with commands

session = cache.get_session_data(session_id)

after extending the timeouts of cache to 100 minutes, an issue I raised before https://community.brain-map.org/t/failed-to-get-session-data/503/3

Nevertheless, when I wanted to read session.units. I got an error message as below

The session.units I was hoping to read looks like this

On the other hand, when I tried to get lfp data from the session with commands
probe_id = session.probes.index.values[0]

lfp = session.get_lfp(probe_id)

I got an error message like this

It seems that both of them are about the way I worked with array but I do not know how to troubleshoot this. I install all the packages via pip install allensdk

Please feel free to let me know if you have any thought.
Thank you

Best Regards

Chi-Yu

[wayne]

Hi Chi-Yu

It is very difficult to understand your problem from the information that you have provided. Can you please provide the minimal code to reproduce the problem and the full stack trace? Please paste them in as code-formatted text - no screenshots please.

Can you please also include basic information about your hardware and software? Operating system and version? amount of RAM? version numbers of relevant software?

Best regards,
Wayne

[chiyu1203]

Dear Wayne

Thank you for your reply. I am sorry my question was not clear.
This is the code when I tried to read session.units

import os
from allensdk.brain_observatory.ecephys.ecephys_project_cache import EcephysProjectCache
from allensdk.brain_observatory.ecephys.ecephys_project_api import EcephysProjectWarehouseApi
from allensdk.brain_observatory.ecephys.ecephys_project_api.rma_engine import RmaEngine
sessionNumber = 778240327
data_directory= 'C:/Users/chi-yu.lee/Documents/test'
manifest_path=os.path.join(data_directory, "manifest.json")
cache = EcephysProjectCache(manifest=manifest_path, fetch_api=EcephysProjectWarehouseApi(RmaEngine(scheme="http",host="[api.brain-map.org (http://api.brain-map.org/)",timeout= 200* 60)))
session_id = sessionNumber #778240327
session = cache.get_session_data(session_id)
unitInfo = session.units

The following is the error message I got:

Traceback (most recent call last):
  File "C:\Anaconda3\envs\allen_data\lib\site-packages\IPython\core\interactiveshell.py", line 3331, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-10-1dae4a0b9e7a>", line 1, in <module>
    unitInfo = session.units
  File "C:\Anaconda3\envs\allen_data\lib\site-packages\allensdk\core\lazy_property\lazy_property_mixin.py", line 13, in __getattribute__
    curr_attr = super(LazyPropertyMixin, self).__getattribute__(name)
  File "C:\Anaconda3\envs\allen_data\lib\site-packages\allensdk\brain_observatory\ecephys\ecephys_session.py", line 207, in units
    return self._units.drop(columns=['width_rf', 'height_rf',
  File "C:\Anaconda3\envs\allen_data\lib\site-packages\allensdk\core\lazy_property\lazy_property_mixin.py", line 15, in __getattribute__
    return curr_attr.__get__(curr_attr)
  File "C:\Anaconda3\envs\allen_data\lib\site-packages\allensdk\core\lazy_property\lazy_property.py", line 16, in __get__
    self.value = self.calculate()
  File "C:\Anaconda3\envs\allen_data\lib\site-packages\allensdk\core\lazy_property\lazy_property.py", line 23, in calculate
    result = self.api_method(*self.args, **self.kwargs)
  File "C:\Anaconda3\envs\allen_data\lib\site-packages\allensdk\brain_observatory\ecephys\ecephys_session_api\ecephys_nwb_session_api.py", line 143, in get_units
    units = self._get_full_units_table()
  File "C:\Anaconda3\envs\allen_data\lib\site-packages\allensdk\brain_observatory\ecephys\ecephys_session_api\ecephys_nwb_session_api.py", line 304, in _get_full_units_table
    units = self.nwbfile.units.to_dataframe()
  File "C:\Anaconda3\envs\allen_data\lib\site-packages\pynwb\core.py", line 1264, in to_dataframe
    data[name] = col[:]
  File "C:\Anaconda3\envs\allen_data\lib\site-packages\pynwb\core.py", line 413, in __getitem__
    ret.append(self.__getitem_helper(i))
  File "C:\Anaconda3\envs\allen_data\lib\site-packages\pynwb\core.py", line 406, in __getitem_helper
    return self.target[start:end]
  File "C:\Anaconda3\envs\allen_data\lib\site-packages\pynwb\core.py", line 318, in __getitem__
    return self.data[args]
  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "C:\Anaconda3\envs\allen_data\lib\site-packages\h5py\_hl\dataset.py", line 562, in __getitem__
    arr = numpy.ndarray(mshape, new_dtype, order='C')
MemoryError: Unable to allocate 1.00 MiB for an array with shape (131722,) and data type float64
Traceback (most recent call last):
  File "C:\Users\chi-yu.lee\AppData\Local\JetBrains\PyCharm Community Edition 2019.3.4\plugins\python-ce\helpers\pydev\_pydev_comm\pydev_server.py", line 34, in handle
    self.processor.process(iprot, oprot)
  File "C:\Users\chi-yu.lee\AppData\Local\JetBrains\PyCharm Community Edition 2019.3.4\plugins\python-ce\helpers\third_party\thriftpy\_shaded_thriftpy\thrift.py", line 266, in process
    self.handle_exception(e, result)
  File "C:\Users\chi-yu.lee\AppData\Local\JetBrains\PyCharm Community Edition 2019.3.4\plugins\python-ce\helpers\third_party\thriftpy\_shaded_thriftpy\thrift.py", line 254, in handle_exception
    raise e
  File "C:\Users\chi-yu.lee\AppData\Local\JetBrains\PyCharm Community Edition 2019.3.4\plugins\python-ce\helpers\third_party\thriftpy\_shaded_thriftpy\thrift.py", line 263, in process
    result.success = call()
  File "C:\Users\chi-yu.lee\AppData\Local\JetBrains\PyCharm Community Edition 2019.3.4\plugins\python-ce\helpers\third_party\thriftpy\_shaded_thriftpy\thrift.py", line 228, in call
    return f(*(args.__dict__[k] for k in api_args))
  File "C:\Users\chi-yu.lee\AppData\Local\JetBrains\PyCharm Community Edition 2019.3.4\plugins\python-ce\helpers\pydev\_pydev_bundle\pydev_console_utils.py", line 306, in loadFullValue
    t.start()
  File "C:\Anaconda3\envs\allen_data\lib\threading.py", line 852, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

Then the following is the code I used to get lfp data:

import os
from allensdk.brain_observatory.ecephys.ecephys_project_cache import EcephysProjectCache
from allensdk.brain_observatory.ecephys.ecephys_project_api import EcephysProjectWarehouseApi
from allensdk.brain_observatory.ecephys.ecephys_project_api.rma_engine import RmaEngine
sessionNumber = 778240327
data_directory= 'C:/Users/chi-yu.lee/Documents/test'
manifest_path=os.path.join(data_directory, "manifest.json")
cache = EcephysProjectCache(manifest=manifest_path, fetch_api=EcephysProjectWarehouseApi(RmaEngine(scheme="http",host="[api.brain-map.org (http://api.brain-map.org/)",timeout= 200* 60)))
session_id = sessionNumber
session = cache.get_session_data(session_id)
probe_id = session.probes.index.values[0]
lfp = session.get_lfp(probe_id)

This is the error message I got

Traceback (most recent call last):
  File "C:\Anaconda3\envs\allen_data\lib\site-packages\IPython\core\interactiveshell.py", line 3331, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-3-9b876dc2c6be>", line 8, in <module>
    lfp = session.get_lfp(probe_id)
  File "C:\Anaconda3\envs\allen_data\lib\site-packages\allensdk\brain_observatory\ecephys\ecephys_session.py", line 349, in get_lfp
    lfp = self.api.get_lfp(probe_id)
  File "C:\Anaconda3\envs\allen_data\lib\site-packages\allensdk\brain_observatory\ecephys\ecephys_session_api\ecephys_nwb_session_api.py", line 161, in get_lfp
    data = series.data[:]
  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "C:\Anaconda3\envs\allen_data\lib\site-packages\h5py\_hl\dataset.py", line 562, in __getitem__
    arr = numpy.ndarray(mshape, new_dtype, order='C')
ValueError: array is too big; `arr.size * arr.dtype.itemsize` is larger than the maximum possible size

The following is the information about my hardware and software. I remote-in this PC from home.
Microsoft Windows 10 Enterprise (10.0.17763)
Intel® Core™ i7-9700 CPU @3.00 GHz, 3000 Mhz, 8 Core(s), 8 logical Processor(s)
Installed Physical Memory (RAM) 16.0GB
Available Physical Memory 5.30 GB
Total Virtual Memory 21.8 GB
Available Virtual Memory 6.74 GB

Pycharm 2019.3.4 (version: 193.6911.25)
Project Interpreter in Pycharm: Python 3.7, installed via Anaconda3 2020.02 (Python 3.7.6 32-bit)

[wayne]

Any chance you could run Python as a 64-bit application instead of 32-bit?

[chiyu1203]

Dear Wayne

I am sorry for my delay in reply. I got some problem installing Anaconda 64-bit in my computer.
After I ran Python 64-bit application for the above mentioned code.

WARNING:root:downloading a 1.66e+03mb file from http://api.brain-map.org//api/v2/well_known_file_download/1026123696
C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\allensdk\api\caching_utilities.py:123: UserWarning: retrying fetch (1 tries remaining)
  warnings.warn(retry_message)
WARNING:root:downloading a 1.66e+03mb file from http://api.brain-map.org//api/v2/well_known_file_download/1026123696
Traceback (most recent call last):
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\allensdk\api\caching_utilities.py", line 104, in call_caching
    return read()
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\allensdk\brain_observatory\ecephys\ecephys_project_cache.py", line 347, in read
    return EcephysSession(api=session_api, test=True)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\allensdk\brain_observatory\ecephys\ecephys_session.py", line 299, in __init__
    self.api.test()
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\allensdk\brain_observatory\ecephys\ecephys_session_api\ecephys_nwb_session_api.py", line 50, in test
    self.get_ecephys_session_id()
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\allensdk\brain_observatory\ecephys\ecephys_session_api\ecephys_nwb_session_api.py", line 278, in get_ecephys_session_id
    return int(self.nwbfile.identifier)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\allensdk\brain_observatory\nwb\nwb_api.py", line 23, in nwbfile
    io = pynwb.NWBHDF5IO(self.path, 'r')
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\pynwb\__init__.py", line 228, in __init__
    super(NWBHDF5IO, self).__init__(path, manager=manager, mode=mode, file=file_obj)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\backends\hdf5\h5tools.py", line 55, in __init__
    super(HDF5IO, self).__init__(manager, source=path)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\backends\io.py", line 18, in __init__
    self.open()
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\backends\hdf5\h5tools.py", line 400, in open
    self.__file = File(self.__path, open_flag, **kwargs)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\h5py\_hl\files.py", line 408, in __init__
    swmr=swmr)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\h5py\_hl\files.py", line 173, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
  File "h5py\_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py\_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py\h5f.pyx", line 88, in h5py.h5f.open
OSError: Unable to open file (truncated file: eof = 1617920, sblock->base_addr = 0, stored_eof = 1736516600)

During handling of the above exception, another exception occurred:


Traceback (most recent call last):
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\allensdk\api\caching_utilities.py", line 104, in call_caching
    return read()
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\allensdk\brain_observatory\ecephys\ecephys_project_cache.py", line 347, in read
    return EcephysSession(api=session_api, test=True)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\allensdk\brain_observatory\ecephys\ecephys_session.py", line 299, in __init__
    self.api.test()
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\allensdk\brain_observatory\ecephys\ecephys_session_api\ecephys_nwb_session_api.py", line 50, in test
    self.get_ecephys_session_id()
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\allensdk\brain_observatory\ecephys\ecephys_session_api\ecephys_nwb_session_api.py", line 278, in get_ecephys_session_id
    return int(self.nwbfile.identifier)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\allensdk\brain_observatory\nwb\nwb_api.py", line 24, in nwbfile
    return io.read()
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\backends\io.py", line 33, in read
    container = self.__manager.construct(f_builder)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\build\map.py", line 195, in construct
    result = self.__type_map.construct(builder, self)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\build\map.py", line 1579, in construct
    return attr_map.construct(builder, build_manager)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\build\map.py", line 1133, in construct
    subspecs = self.__get_subspec_values(builder, self.spec, manager)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\build\map.py", line 1077, in __get_subspec_values
    self.__get_sub_builders(groups, spec.groups, manager, ret)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\build\map.py", line 1115, in __get_sub_builders
    ret.update(self.__get_subspec_values(sub_builder, subspec, manager))
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\build\map.py", line 1077, in __get_subspec_values
    self.__get_sub_builders(groups, spec.groups, manager, ret)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\build\map.py", line 1107, in __get_sub_builders
    sub_builder = self.__flatten(sub_builder, subspec, manager)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\build\map.py", line 1120, in __flatten
    tmp = [manager.construct(b) for b in sub_builder]
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\build\map.py", line 1120, in <listcomp>
    tmp = [manager.construct(b) for b in sub_builder]
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\build\map.py", line 195, in construct
    result = self.__type_map.construct(builder, self)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\build\map.py", line 1579, in construct
    return attr_map.construct(builder, build_manager)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\build\map.py", line 1133, in construct
    subspecs = self.__get_subspec_values(builder, self.spec, manager)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\build\map.py", line 1077, in __get_subspec_values
    self.__get_sub_builders(groups, spec.groups, manager, ret)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\build\map.py", line 1093, in __get_sub_builders
    for parent_dt in manager.namespace_catalog.get_hierarchy(ns, dt):
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\spec\namespace.py", line 314, in get_hierarchy
    raise KeyError("'%s' not a namespace" % namespace)
KeyError: "'ndx-aibs-ecephys' not a namespace"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/chi-yu.lee/PycharmProjects/test_allen/first_test.py", line 32, in <module>
    session = cache.get_session_data(session_id)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\allensdk\brain_observatory\ecephys\ecephys_project_cache.py", line 354, in get_session_data
    num_tries=self.fetch_tries
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\allensdk\api\caching_utilities.py", line 183, in one_file_call_caching
    failure_message=failure_message,
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\allensdk\api\caching_utilities.py", line 133, in call_caching
    failure_message=failure_message,
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\allensdk\api\caching_utilities.py", line 133, in call_caching
    failure_message=failure_message,
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\allensdk\api\caching_utilities.py", line 104, in call_caching
    return read()
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\allensdk\brain_observatory\ecephys\ecephys_project_cache.py", line 347, in read
    return EcephysSession(api=session_api, test=True)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\allensdk\brain_observatory\ecephys\ecephys_session.py", line 299, in __init__
    self.api.test()
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\allensdk\brain_observatory\ecephys\ecephys_session_api\ecephys_nwb_session_api.py", line 50, in test
    self.get_ecephys_session_id()
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\allensdk\brain_observatory\ecephys\ecephys_session_api\ecephys_nwb_session_api.py", line 278, in get_ecephys_session_id
    return int(self.nwbfile.identifier)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\allensdk\brain_observatory\nwb\nwb_api.py", line 24, in nwbfile
    return io.read()
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\backends\io.py", line 33, in read
    container = self.__manager.construct(f_builder)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\build\map.py", line 195, in construct
    result = self.__type_map.construct(builder, self)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\build\map.py", line 1579, in construct
    return attr_map.construct(builder, build_manager)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\build\map.py", line 1133, in construct
    subspecs = self.__get_subspec_values(builder, self.spec, manager)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\build\map.py", line 1077, in __get_subspec_values
    self.__get_sub_builders(groups, spec.groups, manager, ret)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\build\map.py", line 1115, in __get_sub_builders
    ret.update(self.__get_subspec_values(sub_builder, subspec, manager))
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\build\map.py", line 1077, in __get_subspec_values
    self.__get_sub_builders(groups, spec.groups, manager, ret)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\build\map.py", line 1107, in __get_sub_builders
    sub_builder = self.__flatten(sub_builder, subspec, manager)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\build\map.py", line 1120, in __flatten
    tmp = [manager.construct(b) for b in sub_builder]
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\build\map.py", line 1120, in <listcomp>
    tmp = [manager.construct(b) for b in sub_builder]
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\build\map.py", line 195, in construct
    result = self.__type_map.construct(builder, self)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\build\map.py", line 1579, in construct
    return attr_map.construct(builder, build_manager)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\build\map.py", line 1133, in construct
    subspecs = self.__get_subspec_values(builder, self.spec, manager)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\build\map.py", line 1077, in __get_subspec_values
    self.__get_sub_builders(groups, spec.groups, manager, ret)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\build\map.py", line 1093, in __get_sub_builders
    for parent_dt in manager.namespace_catalog.get_hierarchy(ns, dt):
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "C:\Users\chi-yu.lee\Anaconda3\envs\test_allen\lib\site-packages\hdmf\spec\namespace.py", line 314, in get_hierarchy
    raise KeyError("'%s' not a namespace" % namespace)
KeyError: "'ndx-aibs-ecephys' not a namespace"

This new error message showed up

[wayne]

You have great timing

See this post about downloading and analyzing neuropixels data: AllenSDK 2.0 Release - Notice for Visual Coding Neuropixels version compatability

You will need to get the 2.0 version of AllenSDK (expected release date: tomorrow June 11, 2020)

[wayne]

allensdk 2.0 is now available. Please upgrade to that version and try your analysis again.

[chiyu1203]

Dear Wayne,
Thank you for the update about the new version.
I have upgraded Allensdk and run the analysis again (with the same code as mentioned above and remove the old manifest.json).
The following is the error I received.

C:\Users\chi-yu.lee\Anaconda3\envs\allen_conda\python.exe C:/Users/chi-yu.lee/PycharmProjects/allen_conda/allen_test1.py
WARNING:root:downloading a 2.71e+03mb file from http://api.brain-map.org//api/v2/well_known_file_download/1026124625
Traceback (most recent call last):
  File "C:\Users\chi-yu.lee\Anaconda3\envs\allen_conda\lib\site-packages\pandas\core\indexes\base.py", line 2897, in get_loc
    return self._engine.get_loc(key)
  File "pandas/_libs/index.pyx", line 107, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 131, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1607, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'pos'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/chi-yu.lee/PycharmProjects/allen_conda/allen_test1.py", line 42, in <module>
    stimtable["pos"].replace(to_replace="null", value="None", inplace=True)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\allen_conda\lib\site-packages\pandas\core\frame.py", line 2995, in __getitem__
    indexer = self.columns.get_loc(key)
  File "C:\Users\chi-yu.lee\Anaconda3\envs\allen_conda\lib\site-packages\pandas\core\indexes\base.py", line 2899, in get_loc
    return self._engine.get_loc(self._maybe_cast_indexer(key))
  File "pandas/_libs/index.pyx", line 107, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 131, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 1607, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 1614, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'pos'

Process finished with exit code 1

Best Regards

Chi-Yu

[njmei]

Hi @chiyu1203,

It looks like your code is generating a Pandas DataFrame called stimtable. The error you’ve shared is complaining that the stimtable DataFrame does not appear to have a column called pos. Can you link or share your full analysis code (looks to be named allen_test1.py)? Without more information it will be difficult to help you debug this.

Best,

Nick

[chiyu1203]

Dear njmei

Thank you for your reply. I have attached my analysis code as below

########## IMPORTS ##########
import os
import scipy.io as sio
import ast
import argparse

import numpy as np
import xarray as xr
import pandas as pd

from allensdk.brain_observatory.ecephys.ecephys_project_cache import EcephysProjectCache
from allensdk.brain_observatory.ecephys.ecephys_session import (
    EcephysSession,
    removed_unused_stimulus_presentation_columns
)

########## PARSE ARGS AND ACCESS SESSION ##########
ap = argparse.ArgumentParser()
ap.add_argument("-i", "--input", required=True,
type=int, help="session ID number")
ap.add_argument("-o", "--output", required=True,
help="path to output file")

args = ap.parse_args()

sessionNumber = args.input

# set path to maifest and data
manifest_path = os.path.join("D:\AllenSDK", "manifest.json")
cache = EcephysProjectCache.from_warehouse(manifest=manifest_path)

# load a specific session - best is to download before running
session_id = sessionNumber #778240327 # func connec
session = cache.get_session_data(session_id)

# get  basic session info
sessionInfo = {'genotype' : session.full_genotype
               }

########## PROCESS STIMULUS TABLE ##########
# get stimulus table and process required parameters
stimtable = session.get_stimulus_table();
stimtable.replace(to_replace=pd.np.nan, value="null", inplace=True)


# here we replace "null" values with None, ready to be evaled using ast.literal_eval
stimtable["pos"].replace(to_replace="null", value="None", inplace=True)
stimtable["phase"].replace(to_replace="null", value="None", inplace=True)
stimtable["size"].replace(to_replace="null", value="None", inplace=True)
stimtable["spatial_frequency"].replace(to_replace="null", value="None", inplace=True)

# convert these stimtable columns to list so we can eval them
pos = stimtable["pos"].tolist()
phase = stimtable["phase"].tolist()
size = stimtable["size"].tolist()
spatial_frequency = stimtable["spatial_frequency"].tolist()
length = len(pos)

# eval strings to become numeric, None become empty
for i in range(length):
    pos[i] = ast.literal_eval(pos[i])
    phase[i] = ast.literal_eval(phase[i])
    size[i] = ast.literal_eval(size[i])
    spatial_frequency[i] = ast.literal_eval(spatial_frequency[i])

# re-assign the lists to the stimtable dataframe
stimtable["pos"] = pos
stimtable["phase"] = phase
stimtable["size"] = size
stimtable["spatial_frequency"] = spatial_frequency

# replace any empty or "null" values with NaNs
stimtable.fillna(value=pd.np.nan, inplace=True)
stimtable.replace(to_replace="null", value=pd.np.nan, inplace=True)


########## PROCESS UNITS AND SPIKE TIMES ##########
# get generic unit info, e.g. unit location
unitInfo = session.units
unitIndex = np.array(unitInfo.index)

# dict of spike times for each unit, make matlab compatible
spikeTimes = session.spike_times
newSpikeTimes = {}
for key in spikeTimes.keys() :
    newSpikeTimes["c_" + str(key)] = [spikeTimes[key]]


########## PROCESS RUNNING AND PUPIL INFO ##########
# running speed info + pupil info
run_times = session.running_speed["start_time"] + \
    (session.running_speed["end_time"] - session.running_speed["start_time"]) / 2
run_speed = session.running_speed["velocity"]
run_times = np.array(run_times)
run_speed = np.array(run_speed)
runInfo = np.column_stack((run_times, run_speed))

pupilTable = session.get_pupil_data()
pupilTime = np.array(pupilTable.index)
pupilWidth = np.array(pupilTable["pupil_width"])
pupilHeight = np.array(pupilTable["pupil_height"])
pupil_x = np.array(pupilTable["pupil_center_x"])
pupil_y = np.array(pupilTable["pupil_center_y"])

# running speed info
pupilInfo = {'pupilTime' : pupilTime,
              'pupilWidth': pupilWidth,
              'pupilHeight' : pupilHeight,
              'pupil_x': pupil_x,
              'pupil_y' : pupil_y
             }


########## GENERATE OUTPUT VARS DICT AND SAVE ##########
outputVars = {'sessionInfo' : sessionInfo,
              'stim_table' : {name: col.values for name, col in stimtable.items()},
              'unitInformation' : {name: col.values for name, col in unitInfo.items()},
              'unitIndex': unitIndex,
              'spikeTimes' : newSpikeTimes,
              'runInfo' : runInfo,
              'pupilInfo' : pupilInfo
            }


sio.savemat(args.output, outputVars,  long_field_names=True)

And this is the hardware and software information.
Microsoft Windows 10 Enterprise (10.0.17763)
Intel® Core™ i7-9700 CPU @3.00 GHz, 3000 Mhz, 8 Core(s), 8 logical Processor(s)
Installed Physical Memory (RAM) 16.0GB
Available Physical Memory 5.30 GB
Total Virtual Memory 21.8 GB
Available Virtual Memory 6.74 GB
Project Interpreter in Pycharm: Python 3.7, installed via Anaconda3 2020.02 (Python 3.7.6 64-bit)

I had problem installing Python 64-bit in my PC earlier this week. Our school’s IT administrator installed Python 3.7 for me and I was able to use that Python version to run the script. Nevertheless, there was a security issue in the Python settings so I could not set up the Python environment on Pycharm/Spyder. In the end, I reinstalled Python 64-bit only for my windows account to set up my user path in my personal environmental variables. This time I could use a Python IDE to run the code but then I saw that error signals.

Therefore, I do not know whether this is still due to problems in the Python of my PC. I have been working with our school’s IT team to look into this issue.

Best Regards

Chi-Yu

[njmei]

Hi @chiyu1203,

The pos column has been removed from the stimulus_table returned by session.get_stimulus_table() due to being uninformative and being easily confused with the x_position and y_position columns which describe the x and y position of a presentation stimulus.

You can remove any processing on the pos column. The following works for me:

########## PROCESS STIMULUS TABLE ##########
# get stimulus table and process required parameters
stimtable = session.get_stimulus_table();
stimtable.replace(to_replace=pd.np.nan, value="null", inplace=True)
columns_to_transform = ["phase", "size", "spatial_frequency"]
# here we replace "null" values with None, ready to be evaled using ast.literal_eval
stimtable[columns_to_transform] = stimtable[columns_to_transform].replace(to_replace="null", value="None")
# apply ast.literal eval to columns that contains strings that can be 'eval'ed
stimtable[columns_to_transform] = stimtable[columns_to_transform].applymap(ast.literal_eval)
# replace any empty or "null" values with NaNs
stimtable.fillna(value=pd.np.nan, inplace=True)
stimtable.replace(to_replace="null", value=pd.np.nan, inplace=True)

Best,

Nick

[chiyu1203]

Dear Nick

Thank you for pointing out that for me.
It was such a shame that I didnt notice that bug before.
Although I still do not know why my python environment was not compatible with AllenSDK,
there was no such a problem after I started to use Python 64 bit with the latest edition of AllenSDK.

Thank you very much for your continual support again.

Best Regards

Chi-Yu