Unit ID number correspondence between session-specific get_units() and dataset-wide get_unit_table()

Technical
,
1

I am trying to understand the dataset-wide metadata obtained using get_unit_table and session-specific metadata using get_units().

For the former I do the following:

cache.load_latest_manifest()
units = cache.get_unit_table()

The pandas dataframe index column is “unit_id” and these correspond to unique identifiers for each unit in the dataset.

For the latter I do:

session_id = 1128719842
session = cache.get_ecephys_session(ecephys_session_id = session_id)
session_units = session.get_units(filter_by_validity=True)

Now, the session_units dataframe index column is “id” and also has unique identifiers. However, these values do not correspond to any “unit_id” index in the “units” dataframe of the dataset-wide metadata.

Have I fundamentally misunderstood something?

Thank you for your time.

2

Hi Rejwan,

You’re right that the index column for the session units table should return the same unique ids as the index column for the dataset-wide units table. We re-released this data on 12/1 and updated the SDK to fix a few bugs and add some requested features (see the changelog here: GETTING STARTED — Allen SDK dev documentation). I suspect that the problem you’re having comes from a mismatch in the latest metadata unit table and the local nwb files you have in your cache. If I create a new environment with allensdk 2.16.2, and make a fresh cache with the latest manifest (visual-behavior-neuropixels_project_manifest_v0.5.0.json), the two tables correspond as expected (see below):

units = cache.get_unit_table()
session_id = 1128719842
session = cache.get_ecephys_session(ecephys_session_id = session_id)
session_units = session.get_units(filter_by_validity=True)
assert(len(units.loc[session_units.index.values]) == len(session_units))

For that last assertion, both sides of the equality return a dataframe of length 1476.

Can you try making a fresh cache with the latest sdk and see if that resolves the problem (note that this will require re-downloading the nwb file)?

Cheers,
Corbett

3

Hi Corbett,

Thank you for the response. It turned out that I hadn’t fully downloaded the units.csv file. On deleting it and re-downloading it with

cache = VisualBehaviorNeuropixelsProjectCache.from_s3_cache(
            cache_dir=Path(output_dir))

the problem was fixed. Thanks again for clearing this up!

Kind regards,
Rejwan