Universal path support (e.g. local, cloud) for bids datasets #11
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- parse bids directly from cloid - trying out coiled for parallelizing zarr processing
- remove local repo deps - tested with local, and gcs:// urls - also works with zarr.zip with latest zarrnii - note: need to pass the zarr_url directly to zarrnii, not use ome_zarr ZarrReader
|
just realized there are some other commits in here when I was playing around with coiled and blob detection -- I'll clean those out later.. |
replaces hardcoded stain names in config. note, this currently only reads stains from first subject --
note: will switch over to Karl's version later, this was just to make figures for short-term..
- now have an priority list of stains to use for registration - also a list of stains that will be used with blob detection - have included naming variations - uses custom snakebids to add a ignore option to generate inputs (to overcome what seems to be a bug in pybids (not parsing the default ignores properly when using a remote path.
for coarser ABAv3 labels (from Amr Eed)
was using ome-zarr, so was not compatible with ome.zarr.zip
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This makes use of custom (experimental) branches on pybids and snakebids that make use of fsspec and universal_pathlib to allow parsing of bids datasets that are not only local, but can also be on cloud storage (or the many other supported storage backends).
Note: since it depends on universal_pathlib, it doesn't have as wide compatibility as fsspec itself (e.g. it doesn't support
zip://).Since this requires a custom pybids, and then a custom snakebids on top of that, I am thinking we should first make PRs to pybids and then snakebids to add universal_pathlib compatibility there..
With this branch, you can now use a
gcs://...uri as the bids dataset. Note, you then need to consume the bids input data differently, e.g. either by usingstorage()in snakemake to have snakemake load and cache the file, or by using the url directly with dask or zarr. For the latter, you would need to make sure you pass it as a param not an input.