-
Notifications
You must be signed in to change notification settings - Fork 249
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
openephys legacy format : handle gaps more correctly #1387
openephys legacy format : handle gaps more correctly #1387
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a couple initial assert/error question comments for this.
neo/rawio/openephysrawio.py
Outdated
|
||
if channel_has_gaps: | ||
# protect against strange timestamp block like in file 'OpenEphys_SampleData_3' CH32 | ||
assert np.median(diff) == RECORD_SIZE, f"This file has strange block timestamp for channel {chan_id}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
strange is a bit hard to parse as an end user. What does strange mean? The previous not continuous would make the person think about the fact they paused/stopped and remember. Strange is not quite as descriptive. Is the issue that there are other problems than pauses/stops?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If it's the file I'm thinking of, the 'strangeness' came from the headstage getting unplugged mid recording
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Haha. That would make the recording a bit 'strange', but I think the old message still covered that. I can't think of a better single word then strange, but maybe the old verbosity would still be better than a difficult to parse word?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I will find a better message. The old message is outdated because we handle gaps now.
But in some case we have assymetric block size series instead of 1024 1024 1024 2048 1024 which is now correct we can have 700 524 700 524 700 524 which is very hard to handle and maybe is wrong.
Thank you @samuelgarcia, I can confirm this fixes the files that I have from the lab where this issue was discovered. The read time is definitely slow;
6-7 minutes to parse the header and
and I measured 5 minutes to retrieve the traces for the whole recording (~3 hours) with 32 channels. I think this is acceptable until we can guarantee that the timestamps are continuous and aligned with the data. |
That sounds good to me! |
Co-authored-by: Zach McKenzie <[email protected]>
Co-authored-by: Zach McKenzie <[email protected]>
@weiglszonja @bendichter @alejoe91 @CodyCBakerPhD
This PR should fix the gap problem in openephys legacy format.
The old an erroreneous
ignore_timestamps_errors
is deprecated.Now there is a gap checking when parsing the header. If gaps are detected then internally the reader go in a gap mode reading which is slower because it need np.seachrsorted at every get_analogsignal_chunk().
I propagate this on spikeinterface side here SpikeInterface/spikeinterface#2450