# Slow loading of VBN data via AllenSDK

**URL:** <https://community.brain-map.org/t/slow-loading-of-vbn-data-via-allensdk/2090>\
**Category:** Technical\
**Tags:** allensdk\
**Created:** [January 19, 2023, 9:20pm UTC](https://community.brain-map.org/t/slow-loading-of-vbn-data-via-allensdk/2090 "2023-01-19T21:20:14Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![seankmartin](https://yyz1.discourse-cdn.com/flex027/user_avatar/community.brain-map.org/seankmartin/32/621_2.png) [@seankmartin](https://community.brain-map.org/u/seankmartin)\
**Post date:** [January 19, 2023, 9:20pm UTC](https://community.brain-map.org/t/slow-loading-of-vbn-data-via-allensdk/2090/1 "2023-01-19T21:20:15Z")

</div>

Hi,

The `cache.get_ecephys_session(id)` is quite slow for me on visual behavior neuropixels data that has already been downloaded (~1min), and takes about 20 times longer than a pynwb command to load the data via `NWBHDF5IO`.

I was wondering if there was something that might be causing this slowdown due to how I am using the function perhaps? Or is it simply the case that the AllenSDK performs a lot of background calculations after loading the dataset from NWB to add extra information that is time consuming?

## Further context

I have tried it on my own system with predownloaded data (Windows 10, python 3.8.10, allensdk 2.14.1, pynwb 2.2.0) - NWB took about 3seconds, AllenSDK about 1minute. I also tried on google colab with allensdk 2.14.1 on data that was mounted from my google drive - NWB took about 3seconds, AllenSDK about 30seconds.

I have included a code snippet that I used to test the times.

```python
from time import perf_counter
from pathlib import Path

from allensdk.brain_observatory.behavior.behavior_project_cache import (
    VisualBehaviorNeuropixelsProjectCache,
)
from pynwb import NWBHDF5IO

ALLEN_DIR = Path(r"D:\example-data\allen-data")
ALLEN_MANIFEST = "visual-behavior-neuropixels_project_manifest_v0.4.0.json"
EXAMPLE_SESSION = 1044385384

def timeit(name):
    def inner(func):
        def wrapper(*args, **kwargs):
            t1 = perf_counter()
            func(*args, **kwargs)
            t2 = perf_counter()

            print(f"{name} took {t2 - t1:.2f} seconds")

        return wrapper

    return inner

@timeit("Allen SDK loader")
def allen_example(cache):
    return cache.get_ecephys_session(EXAMPLE_SESSION)

@timeit("NWB loader")
def nwb_example(nwb_path):
    nwb_io = NWBHDF5IO(nwb_path, "r", load_namespaces=True)
    return nwb_io.read()

def main(cache_is_s3=True):
    if cache_is_s3:
        cache = VisualBehaviorNeuropixelsProjectCache.from_s3_cache(cache_dir=ALLEN_DIR)
    else:
        cache = VisualBehaviorNeuropixelsProjectCache.from_local_cache(
            cache_dir=ALLEN_DIR
        )
    cache.load_manifest(ALLEN_MANIFEST)

    nwb_path = (
        ALLEN_DIR
        / "visual-behavior-neuropixels-0.4.0"
        / "behavior_ecephys_sessions"
        / str(EXAMPLE_SESSION)
        / f"ecephys_session_{EXAMPLE_SESSION}.nwb"
    )

    nwb_example(nwb_path)
    allen_example(cache)

    allen_example(cache)
    nwb_example(nwb_path)

main(cache_is_s3=True)
main(cache_is_s3=False)

```

And the output of timing:

```auto
NWB loader took 2.76 seconds
Allen SDK loader took 56.84 seconds
Allen SDK loader took 50.09 seconds
NWB loader took 2.80 seconds
NWB loader took 2.36 seconds
Allen SDK loader took 51.56 seconds
Allen SDK loader took 56.49 seconds
NWB loader took 7.52 seconds

```

---

<div class="post-metadata">

**Author:** ![joshs](https://yyz1.discourse-cdn.com/flex027/user_avatar/community.brain-map.org/joshs/32/8_2.png) [@joshs](https://community.brain-map.org/u/joshs)\
**Post date:** [January 26, 2023, 12:32am UTC](https://community.brain-map.org/t/slow-loading-of-vbn-data-via-allensdk/2090/2 "2023-01-26T00:32:08Z")

</div>

Thanks for running this comparison, it’s very helpful! Do you mind re-posting this info [here](https://github.com/AllenInstitute/AllenSDK/issues)? That’s our preferred place to track software-related issues.

---

<div class="post-metadata">

**Author:** ![seankmartin](https://yyz1.discourse-cdn.com/flex027/user_avatar/community.brain-map.org/seankmartin/32/621_2.png) [@seankmartin](https://community.brain-map.org/u/seankmartin)\
**Post date:** [February 1, 2023, 11:14am UTC](https://community.brain-map.org/t/slow-loading-of-vbn-data-via-allensdk/2090/3 "2023-02-01T11:14:52Z")

</div>

Hi @joshs, sure thing, I’ll post it over on github now - thanks 🙂 Edit: posted [here](https://github.com/AllenInstitute/AllenSDK/issues/2657).