# Using the raw data **URL:** \ **Category:** General\ **Tags:** brain-observatory-visual-coding\ **Created:** [October 7, 2022, 6:13pm UTC](https://community.brain-map.org/t/using-the-raw-data/1898 "2022-10-07T18:13:35Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![babaq](https://yyz1.discourse-cdn.com/flex027/user_avatar/community.brain-map.org/babaq/32/697_2.png) [@babaq](https://community.brain-map.org/u/babaq)\ **Post date:** [October 7, 2022, 6:13pm UTC](https://community.brain-map.org/t/using-the-raw-data/1898/1 "2022-10-07T18:13:35Z")
Hi, I am trying to use the raw data to do CSD and spike sorting, and not sure of the data format and the time alignment with stimulus presentation. For the `lfp_band.dat` and `spike_band.dat` files, i assume it’s the binary data that can be memory mapped with a `Matrix{Int16}(nch,nsample), where nch=384, and nsample=filebytes/2/nch`. The `nsample`s i got were integers, so i assume i got it right. Then, to extract raw data for each of stimulus presentation in these binaries, i use `start_time` and `stop_time` in the dataframe returned by `session.get_stimulus_table(['flashes'])` for each stimulus presentation, and the 2.5kHz and 30kHz sampling rate to cut out the raw data for lfp\_band and spike\_band. However, the CSD i got were quite different from the one calculated from `csd = session.get_current_source_density(probe_id)`, I am wandering whether the timeline in stimulus presentation are the same for the raw data. I checked the overall duration of all stimuli in `session.get_stimulus_epochs()` and each of the binaries by `nsample/sampling_rate`, and they are almost the same length. I am not sure where goes wrong, would someone know the details clarify what i might missing? Thanks, Alex ---
**Author:** ![joshs](https://yyz1.discourse-cdn.com/flex027/user_avatar/community.brain-map.org/joshs/32/8_2.png) [@joshs](https://community.brain-map.org/u/joshs)\ **Post date:** [October 10, 2022, 5:54pm UTC](https://community.brain-map.org/t/using-the-raw-data/1898/2 "2022-10-10T17:54:02Z")
Hi Alex – you’re definitely on the right track in terms of loading the data. We just use `np.memmap` to read in the raw `.dat` files at 16-bit integers. A scaling factor of 0.195 is then used to convert the values to microvolts. To get the correct sample index for a particular stimulus presentation, you first need to find the index in the LFP data returned by the AllenSDK. This is an xarray `DataArray` with dimensions of time and channel. If you have a presentation start time (in seconds), you can find the corresponding index for the various arrays using the following method: ```auto index = np.searchsorted(lfp.time, start_index) raw_lfp_index = index * 2 # convert from 1250 Hz to 2500 Hz raw_spike_index = index * 24 # convert from 1250 Hz to 30000 Hz ``` Once you have the correct indices, there are a few additional transformations that need to happen to match the LFP data that is returned by the AllenSDK: - The channels are reordered according to [this mapping](https://github.com/AllenInstitute/AllenSDK/blob/56354b1ea48c2890af1ecdd67fb43306c593d625/allensdk/brain_observatory/ecephys/file_io/continuous_file.py#L119-L140) - The data is high-pass filtered to remove the DC offset - The median value of out-of-brain channels is subtracted to remove noise - The data is downsampled in space (4x) and time (2x) - Reference channels and channels far outside the brain are removed Let me know if that helps! ---
**Author:** ![babaq](https://yyz1.discourse-cdn.com/flex027/user_avatar/community.brain-map.org/babaq/32/697_2.png) [@babaq](https://community.brain-map.org/u/babaq)\ **Post date:** [October 12, 2022, 9:02pm UTC](https://community.brain-map.org/t/using-the-raw-data/1898/3 "2022-10-12T21:02:53Z")
Hi Joshs, I found a mistake in my estimation of total duration. I just checked the `lfp_band.dat` file and `spike_band.dat` file are almost same duration, but when i compare it with the `stat_time` and `stop_time` of the first stimulus and last stimulus, it seems the raw data recording continues over ~1-2 hours after all stimuli have been presented. Anyway, when i check the `lfp.time` returned by AllenSDK, that starting time is around 4 secs instead of 0, that explains why my CSD looks abnormal, because the recording start time t0 is not zero in stimulus timeline. For simplicity, I assume i can just subtract the t0 of raw data from the stimulus start\_time, and then map to the index using the corresponding sampling rates, where `t0 = lfp.time.data[0]`? I also checked the t0 for several probes of the same session, and they are seems the same, so can i assume they all equals, so i can just use one t0 instead of each t0 for each probe? The LFP AllenSDK returned seems out of order regarding the channels layout in the probe, as you mentioned they’ve been remapped, but if i assume column order memory mapping the binary data, the CSD looks similar to the one returned by SDK. (i assume samples of the same time saved together then next sample, so the 1D binary mapping should be reshaped in column-wise matrix(nch, nsample) and here the channel order are probe left column -\< right, and then tip → bottom) Thanks, Alex ---
**Author:** ![babaq](https://yyz1.discourse-cdn.com/flex027/user_avatar/community.brain-map.org/babaq/32/697_2.png) [@babaq](https://community.brain-map.org/u/babaq)\ **Post date:** [October 14, 2022, 7:33pm UTC](https://community.brain-map.org/t/using-the-raw-data/1898/4 "2022-10-14T19:33:22Z")
Hi Joshs, It seems the system blocked my previous reply, so i am writing this one. after checking the start time of `lfp.time` returned by AllenSDK, it seems Neuropixels recording were started ~4s in the master clock, where first stimulus presentation was ~8s. So for simplicity, I just use the `lfp.time.data[0]` as the `t0` of the rawdata. The stimulus times could be shifted to raw data times by subtract the `t0` from stimulus `start_time`, then i use the sampling rate to get indices of each presentation. I only checked several `t0` of probes in the same session, and they are almost the same, so can i assume the `t0`s of every probe in the same session all equals? For clarity, the memory mapped 1D binary file should be reshaped in column-wise matrix of nrow=384, ncol=nsample? The resulting CSD now looks similar to the one returned by AllenSDK. Thanks, Alex ---
**Author:** ![joshs](https://yyz1.discourse-cdn.com/flex027/user_avatar/community.brain-map.org/joshs/32/8_2.png) [@joshs](https://community.brain-map.org/u/joshs)\ **Post date:** [October 14, 2022, 7:51pm UTC](https://community.brain-map.org/t/using-the-raw-data/1898/5 "2022-10-14T19:51:55Z")
Hi Li – each probe may actually have a slightly different start time, as well as a different sample rate. So you should use the method I described to find the correct index for each probe individually. You’re right about the format of the binary files – there are always 384 rows. ---
**Author:** ![babaq](https://yyz1.discourse-cdn.com/flex027/user_avatar/community.brain-map.org/babaq/32/697_2.png) [@babaq](https://community.brain-map.org/u/babaq)\ **Post date:** [October 15, 2022, 11:55pm UTC](https://community.brain-map.org/t/using-the-raw-data/1898/6 "2022-10-15T23:55:25Z")
I assume the scaling factor of 0.195 you mentioned was for raw lfp\_band, is this also applys to raw spike\_band? ---
**Author:** ![joshs](https://yyz1.discourse-cdn.com/flex027/user_avatar/community.brain-map.org/joshs/32/8_2.png) [@joshs](https://community.brain-map.org/u/joshs)\ **Post date:** [October 16, 2022, 12:17am UTC](https://community.brain-map.org/t/using-the-raw-data/1898/7 "2022-10-16T00:17:16Z")
That’s right.