Slow download on NWB files from brain observatory Neuropixels data

I’m trying to download the session data for the virual coding experiments using allensdk, but getting really slow download speeds (regardless of internet connection). Here’s the code I’m using:

from allensdk.brain_observatory.ecephys.ecephys_project_cache import EcephysProjectCache
from allensdk.brain_observatory.ecephys.ecephys_project_api import EcephysProjectWarehouseApi
from allensdk.brain_observatory.ecephys.ecephys_project_api.rma_engine import RmaEngine

data_dirname = “./DATA/”
#!mkdir {data_dirname}
#!ls

data_directory = data_dirname + ‘ecephys_cache_dir’ # must be updated to a valid directory in your filesystem
manifest_path = os.path.join(data_directory, “manifest.json”)

#Timeout fix from Failed to get session data
cache = EcephysProjectCache(
manifest=manifest_path,
fetch_api=EcephysProjectWarehouseApi(RmaEngine(
scheme=“http”,
host=“api.brain-map.org”,
timeout=120 * 60 #set timeout to 60 minutes
))
)

sessions = cache.get_session_table()

#fetch a specific experiment, this is where the issue is
session = cache.get_session_data(sessions.index.values[20])

Screenshot from 2020-11-27 18-25-08

This screenshot is from my own machine - Notice the ETA which is >2hours for a single 2GB file. I’ve run the same thing on google colab thinking it was an issue with my internet connection, but same issue, slow download.

Is there a way to improve the download speed?

(there is a github issue on the allensdk repo regarding this: https://github.com/AllenInstitute/AllenSDK/issues/1505
No solution there either)

PS: I don’t yet have access to any AWS unit, so looking for alternatives

Hi @kinshuk

Our systems adminstration team looked into this during the week. They have implemented some configuration changes that may speed this up by about four times. There may also be a possibility for additional changes down the road that could speed it up further.

Best regards,
Wayne

1 Like

The speed improved to ~1MBps for me. Not the best but definitely a lot better than before. Thanks!

Some additional activity on this today. You may see some additional speed improvements.

1 Like