Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Piecewise version of loadExternalData #1494

Merged
merged 7 commits into from
Jan 24, 2025

Conversation

nselliott
Copy link
Contributor

Summary

The pre-existing usage pattern for external data is a three step process:

  1. Call IOManager::read() to execute parallel IO to read data from a file and load it into a Sidre Group.
  2. Directly set the addresses for every View in the hierarchy that has external data. The calling code is responsible to ensure that the addresses point to validly allocated data of the correct size.
  3. Call IOManager::loadExternalData() with the same root file and the same Group that was loaded in Step 1.

This added usage pattern changes steps 2 and 3. For step 2 the calling code sets the addresses for a subset of the Views with external data., then for step 3 it calls the new overloaded IOManager::loadExternalData on a Group somewhere in the hierarchy that contains the Views touched in step 3. Steps 2 and 3 can be repeated to set addresses and load data into external Views at different locations in the hierarchy.

@nselliott nselliott added enhancement New feature or request Sidre Issues related to Axom's 'sidre' component Spio Issues related to paralle I/O for Axom's Sidre component User Request Issues related to user requests labels Jan 23, 2025
* This is intended as an alternative way to load external data, in a
* piecewise manner rather than loading all external data of a Sidre
* hierarchy at once. load_group is a Group somewhere in the hierarchy
* under parent_group, and only the external data for views in the
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is parent group needed here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In order to find the needed data in the HDF file, the method needs the relative path from parent_group to load_group. The logic to figure this out is inside the method. The alternative would be to force the user to construct this relative path and pass it in.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it. Thank you for the explanation.

Copy link
Member

@rhornung67 rhornung67 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments about the comments....

@nselliott nselliott merged commit b68514e into develop Jan 24, 2025
13 checks passed
@nselliott nselliott deleted the feature/nselliott/piecewise-io branch January 24, 2025 23:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Sidre Issues related to Axom's 'sidre' component Spio Issues related to paralle I/O for Axom's Sidre component User Request Issues related to user requests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants