Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More Downloaders #510

Open
dkfellows opened this issue Jun 16, 2022 · 0 comments
Open

More Downloaders #510

dkfellows opened this issue Jun 16, 2022 · 0 comments
Assignees
Labels
help wanted Extra attention is needed question Further information is requested
Milestone

Comments

@dkfellows
Copy link
Member

The current fast data downloader is fine, but only handles recording channels. They're not the only things that are desired for parallel downloading, especially given @rowleya's megamodel!

There are also:

  1. General Regions (used for weights, provenance, etc.)
  2. IOBUFs

The downloading of general regions requires a data model update. In particular, we have to think of them as being snapshots of the state (instead of streams) and that requires additional information, especially what the current simulation timestep is. (In theory, it also requires the run number, but I'm not sure if that is not implicit in which database we're writing to.)

IOBUF downloading is mostly there, but I don't know if we need to extend what we've got. There's also the question of whether to pull back data into a pile of files (which are then easily examinable with text editors) or to instead put it into a database. I also don't know whether the code has kept pace with @Christian-B's changes to mapping dictionaries.

Feedback/design assistance would be welcome.

@dkfellows dkfellows added the question Further information is requested label Jun 16, 2022
@dkfellows dkfellows added the help wanted Extra attention is needed label Jun 28, 2022
@dkfellows dkfellows added this to the Bluesky milestone Mar 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed question Further information is requested
Projects
None yet
Development

No branches or pull requests

3 participants