Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ReadGridded should support cropping geographically #1370

Open
thorbjoernl opened this issue Oct 9, 2024 · 4 comments
Open

ReadGridded should support cropping geographically #1370

thorbjoernl opened this issue Oct 9, 2024 · 4 comments
Assignees
Labels
enhancement ✨ New feature or request SESAM Issues related to the SESAM project
Milestone

Comments

@thorbjoernl
Copy link
Collaborator

Is your feature request related to a problem? Please describe.
Reading the entire griddeddata is quite memory intensive and discarding unneeded data (or ideally not reading it in in the first place) would be useful.

Describe the solution you would like to see

  • Ideally, read calls would allow passing a bounding box and data outside of this bounding box is not read.

Additional context
Add any other context or screenshots about the feature request here.

@thorbjoernl thorbjoernl added the enhancement ✨ New feature or request label Oct 9, 2024
@thorbjoernl thorbjoernl added this to the m2024-11 milestone Oct 9, 2024
@thorbjoernl thorbjoernl self-assigned this Oct 9, 2024
@lewisblake
Copy link
Member

Note: this will not work out of the box for non-lon-lat grids. So this PR will have to be limited in scope to only implement what xarray offers (e.g., .sel on longitude and latitude)

@lewisblake
Copy link
Member

Another note: We want this to be implemented in such a way that it does not cause the data to be realized in memory earlier than the current implementation.

@heikoklein heikoklein modified the milestones: m2024-11, m2024-12 Nov 8, 2024
@heikoklein heikoklein modified the milestones: m2024-12, m2025-01 Nov 26, 2024
@lewisblake lewisblake modified the milestones: m2025-01, m2025-02 Jan 6, 2025
@heikoklein
Copy link
Member

@thorbjoernl Please specify the setup you are using and where you think extensive memory is used (in the log-files).

E.g. for collocation, we are never reading the complete gridded data, just a few time-slices of gridded data, and are then only keeping the lat/lon variables. Memory usage and time usage are tune to work well here.

@thorbjoernl
Copy link
Collaborator Author

I originally created this issue because I ran into memory issues with some of the work I did with David (which we worked around by using the workers). This isn't critical for my purposes anymore, and if what you say is correct it may not even be that relevant to reducing memory usage. So I'd say this issue may be closed.

That being said, I know @lewisblake also wanted this functionality, so maybe ask him as well.

@lewisblake lewisblake added the SESAM Issues related to the SESAM project label Jan 13, 2025
@lewisblake lewisblake modified the milestones: m2025-02, m2025-03 Jan 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement ✨ New feature or request SESAM Issues related to the SESAM project
Projects
None yet
Development

No branches or pull requests

3 participants