Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ICON XPP and NextGEMS dataset #28

Merged
merged 8 commits into from
Dec 14, 2024
Merged

ICON XPP and NextGEMS dataset #28

merged 8 commits into from
Dec 14, 2024

Conversation

SF-N
Copy link
Contributor

@SF-N SF-N commented Dec 9, 2024 •
edited
Loading

This replaces PR #26.
Adding 2 ICON datasets and improving other dataset notebooks.

TODOs:

  • Plotting of HEALPix data with earthkit-plots in 03-NextGEMS.ipynb (Issue 37)
  • Plotting data before 1678 and after 2262 (Issue 35)
  • Update READMEs
  • CORS warnings
  • add ICON-XPP licence information

@SF-N
Copy link
Contributor Author

SF-N commented Dec 9, 2024

Regarding your comment in PR #26:

https://object-store.os-api.cci1.ecmwf.int/ doesn't support CORS (https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS). While the CORS proxy of the online lab makes things work in any case, it now displays a warning. Perhaps you can poke someone to reconfigure the S3 bucket? If need be I can add some special cases to the ignore code so that URLs we already use won't spam users.

I am not sure which warning you are referring to, but we tried to configure the S3 bucket.
Please let me know if the warnings still appear.

@SF-N SF-N changed the title First version of ICON dataset notebooks and update other notebooks ICON XPP and NextGEMS dataset Dec 9, 2024
This was referenced Dec 9, 2024
Closed
Closed
@juntyr
Copy link
Member

juntyr commented Dec 9, 2024

Regarding your comment in PR #26:

https://object-store.os-api.cci1.ecmwf.int/ doesn't support CORS (https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS). While the CORS proxy of the online lab makes things work in any case, it now displays a warning. Perhaps you can poke someone to reconfigure the S3 bucket? If need be I can add some special cases to the ignore code so that URLs we already use won't spam users.

I am not sure which warning you are referring to, but we tried to configure the S3 bucket. Please let me know if the warnings still appear.

The warnings are on the main branch of the lab (if you use lab.climet.eu/main). They're not gone yet and it seems the S3 bucket still does not set the Access-Control-Allow-Origin: * header.

@juntyr
Copy link
Member

juntyr commented Dec 9, 2024

I've made some progress on adding healpy and pyicon but compiling healpy still fails (pyodide/pyodide#5250).

I'm tracking progress here: https://github.com/climet-eu/pyodide/tree/healpy

@SF-N
Copy link
Contributor Author

SF-N commented Dec 9, 2024

Regarding your comment in PR #26:

https://object-store.os-api.cci1.ecmwf.int/ doesn't support CORS (https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS). While the CORS proxy of the online lab makes things work in any case, it now displays a warning. Perhaps you can poke someone to reconfigure the S3 bucket? If need be I can add some special cases to the ignore code so that URLs we already use won't spam users.

I am not sure which warning you are referring to, but we tried to configure the S3 bucket. Please let me know if the warnings still appear.

The warnings are on the main branch of the lab (if you use lab.climet.eu/main). They're not gone yet and it seems the S3 bucket still does not set the Access-Control-Allow-Origin: * header.

After talking to the ECMWF user support, it seems like there is nothing that can be done on the S3 bucket side.
When debugging with curl, the correct headers seem to be there:

< access-control-allow-origin: *
< access-control-allow-methods: GET

The warning also only pops up during the first execution.
So I guess, ignoring this warning is probably the best solution currently.

@juntyr
Copy link
Member

juntyr commented Dec 9, 2024

Regarding your comment in PR #26:

https://object-store.os-api.cci1.ecmwf.int/ doesn't support CORS (https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS). While the CORS proxy of the online lab makes things work in any case, it now displays a warning. Perhaps you can poke someone to reconfigure the S3 bucket? If need be I can add some special cases to the ignore code so that URLs we already use won't spam users.

I am not sure which warning you are referring to, but we tried to configure the S3 bucket. Please let me know if the warnings still appear.

The warnings are on the main branch of the lab (if you use lab.climet.eu/main). They're not gone yet and it seems the S3 bucket still does not set the Access-Control-Allow-Origin: * header.

After talking to the ECMWF user support, it seems like there is nothing that can be done on the S3 bucket side. When debugging with curl, the correct headers seem to be there:

< access-control-allow-origin: *
< access-control-allow-methods: GET

The warning also only pops up during the first execution. So I guess, ignoring this warning is probably the best solution currently.

Huh, when I run

curl https://object-store.os-api.cci1.ecmwf.int/esiwacebucket/ICON-XPP/slo1802_atm_2d_ml_20200101T000000Z.nc.ref -D - -o /dev/null

I get the following

content-length: 473589
accept-ranges: bytes
last-modified: Fri, 06 Dec 2024 12:35:33 GMT
x-rgw-object-type: Normal
etag: "35c2574bd69afda9fb9daba53b2c8db2"
x-amz-meta-s3cmd-attrs: atime:1733487713/ctime:1733487713/gid:20/gname:staff/md5:35c2574bd69afda9fb9daba53b2c8db2/mode:33188/mtime:1733487713/uid:501/uname:ecm5945
x-amz-storage-class: STANDARD
x-amz-request-id: tx000001d12bb387d38674d-00675735fe-149ac72b-default
content-type: binary/octet-stream
date: Mon, 09 Dec 2024 18:25:02 GMT

where the CORS headers are missing.

@SF-N
Copy link
Contributor Author

SF-N commented Dec 10, 2024

Regarding your comment in PR #26:

https://object-store.os-api.cci1.ecmwf.int/ doesn't support CORS (https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS). While the CORS proxy of the online lab makes things work in any case, it now displays a warning. Perhaps you can poke someone to reconfigure the S3 bucket? If need be I can add some special cases to the ignore code so that URLs we already use won't spam users.

I am not sure which warning you are referring to, but we tried to configure the S3 bucket. Please let me know if the warnings still appear.

The warnings are on the main branch of the lab (if you use lab.climet.eu/main). They're not gone yet and it seems the S3 bucket still does not set the Access-Control-Allow-Origin: * header.

After talking to the ECMWF user support, it seems like there is nothing that can be done on the S3 bucket side. When debugging with curl, the correct headers seem to be there:

< access-control-allow-origin: *
< access-control-allow-methods: GET

The warning also only pops up during the first execution. So I guess, ignoring this warning is probably the best solution currently.

Huh, when I run

curl https://object-store.os-api.cci1.ecmwf.int/esiwacebucket/ICON-XPP/slo1802_atm_2d_ml_20200101T000000Z.nc.ref -D - -o /dev/null

I get the following

content-length: 473589
accept-ranges: bytes
last-modified: Fri, 06 Dec 2024 12:35:33 GMT
x-rgw-object-type: Normal
etag: "35c2574bd69afda9fb9daba53b2c8db2"
x-amz-meta-s3cmd-attrs: atime:1733487713/ctime:1733487713/gid:20/gname:staff/md5:35c2574bd69afda9fb9daba53b2c8db2/mode:33188/mtime:1733487713/uid:501/uname:ecm5945
x-amz-storage-class: STANDARD
x-amz-request-id: tx000001d12bb387d38674d-00675735fe-149ac72b-default
content-type: binary/octet-stream
date: Mon, 09 Dec 2024 18:25:02 GMT

where the CORS headers are missing.

The following command shows the headers:

curl --verbose -H 'Origin: http://example.com' "https://object-store.os-api.cci1.ecmwf.int/esiwacebucket/OpenIFS/HighResMIP_6h_reduced_pl_q.nc.ref" >/dev/null

Does that help?

@juntyr
Copy link
Member

juntyr commented Dec 10, 2024 via email

@SF-N
Copy link
Contributor Author

SF-N commented Dec 10, 2024

Would it be possible to add HEAD to the access-control-allow-methods as well?

Yes, this solved it. Now the warnings should not appear anymore.

@juntyr
Copy link
Member

juntyr commented Dec 10, 2024

Would it be possible to add HEAD to the access-control-allow-methods as well?

Yes, this solved it. Now the warnings should not appear anymore.

Yes, it's fixed now - thanks!

@juntyr
Copy link
Member

juntyr commented Dec 11, 2024

@SF-N healpy and pyicon now seem to work in the online lab (lab.climet.eu/main).

I saw some earthkit.plots examples using healpix data - so perhaps it could work for us as well?

https://earthkit-plots.readthedocs.io/en/latest/examples/gallery/grid-types/healpix-pixels.html

@SF-N
Copy link
Contributor Author

SF-N commented Dec 11, 2024

@SF-N healpy and pyicon now seem to work in the online lab (lab.climet.eu/main).

Great, thanks a lot!

I saw some earthkit.plots examples using healpix data - so perhaps it could work for us as well?

https://earthkit-plots.readthedocs.io/en/latest/examples/gallery/grid-types/healpix-pixels.html

James said that the issue is that earthkit-plots doesn't really support HEALPix from xarray at the moment - it largely depends on eccodes to understand HEALPix, and that only works well from GRIB. He wanted to create a new issue to support HEALPix from xarray.

@juntyr
Copy link
Member

juntyr commented Dec 11, 2024

Do you think we can get these two notebooks ready by the end of this week?

@SF-N
Copy link
Contributor Author

SF-N commented Dec 11, 2024 •
edited
Loading

Do you think we can get these two notebooks ready by the end of this week?

From my perspective, now that pyicon works, yes.
I am just not sure if plotting data before 1678 and after 2262 will be possible without errors by then.
The rest are just minor things.

@SF-N
Copy link
Contributor Author

SF-N commented Dec 13, 2024

I updated the notebooks and added the licence information (still missing for ICON-XPP).
It seems like it will take some more time until ecmwf/earthkit-plots#35, i.e. ecmwf/earthkit-plots#38 and ecmwf/earthkit-plots#37, i.e. ecmwf/earthkit-plots#39 will be solved.
We can wait a couple of more days, but then I suggest to release with the current workarounds.

@juntyr
Copy link
Member

juntyr commented Dec 14, 2024

I'll merge now rather than wait for a few more days. It's easier to fix a few lines in two notebooks on Monday than it is to do all of the notebook updates and merging then.

juntyr
juntyr approved these changes Dec 14, 2024
View reviewed changes
Copy link
Member

@juntyr juntyr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@juntyr juntyr merged commit 33f19b5 into main Dec 14, 2024
@juntyr juntyr deleted the ICON-datasets-new branch December 14, 2024 08:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@juntyr juntyr juntyr approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@SF-N @juntyr