Skip to content

Commit

Permalink
Feat/concurrent members (#2519)
Browse files Browse the repository at this point in the history
* feat: add wrapperstore

* feat: add latencystore

* rename noisysetter -> noisygetter

* rename _wrapped to _store

* loggingstore inherits from wrapperstore

* initial commit

* working members traversal

* bolt concurrent members implementation onto async group

* update scratch file

* use metadata / node builders for v3 node creation

* fix key/name handling in recursion

* add latency-based test

* add latency-based concurrency tests for group.members

* improve comments for test

* add concurrency limit

* add test for concurrency limiting

* docstrings

* remove function that was only calling itself

* docstrings

* relax timing requirement for concurrency test

* Update src/zarr/core/group.py

Co-authored-by: Deepak Cherian <[email protected]>

* exists_ok -> overwrite

* simplify group_members_perf test, just require that the duration is less than the number of groups * latency

* update test docstring

* remove vestigial test

---------

Co-authored-by: Deepak Cherian <[email protected]>
  • Loading branch information
d-v-b and dcherian authored Jan 7, 2025
1 parent f9c2024 commit bc5877b
Show file tree
Hide file tree
Showing 5 changed files with 318 additions and 89 deletions.
1 change: 0 additions & 1 deletion src/zarr/api/asynchronous.py
Original file line number Diff line number Diff line change
Expand Up @@ -188,7 +188,6 @@ async def consolidate_metadata(
group.store_path.store._check_writable()

members_metadata = {k: v.metadata async for k, v in group.members(max_depth=None)}

# While consolidating, we want to be explicit about when child groups
# are empty by inserting an empty dict for consolidated_metadata.metadata
for k, v in members_metadata.items():
Expand Down
3 changes: 2 additions & 1 deletion src/zarr/core/array.py
Original file line number Diff line number Diff line change
Expand Up @@ -1995,10 +1995,11 @@ def path(self) -> str:

@property
def name(self) -> str:
"""Array name following h5py convention."""
return self._async_array.name

@property
def basename(self) -> str | None:
def basename(self) -> str:
"""Final component of name."""
return self._async_array.basename

Expand Down
Loading

0 comments on commit bc5877b

Please sign in to comment.