zarr v3 sharding

[d-v-b]

This zarr v3 array does not render correctly in neuroglancer. No chunks get fetched. Looking at requests, I'm seeing "CORS failed" messages. Is it possible that the array metadata is somehow incompatible with neuroglancer?

For reference, this (a zarr v2 array containing the same values) is what I would expect to see.

[d-v-b]

I edited the title of the issue, because the problem seems to be specific to sharded arrays. An unsharded v3 array with the same values renders normally

[LDeakin]

I think this is just an issue with GitHub and range requests, rather than Neuroglancer. Your sharded array works fine when hosted on a supported HTTP server.

[d-v-b]

excellent! how is that server configured? when I tried hosting sharded arrays with my usual tools (the nodejs static file server serve, invoked with cors headers via serve --cors, s3, github) I got the same CORS-related errors.

[LDeakin]

That is just node.js http-server with --cors.

[joshmoore]

Thanks, @LDeakin! This leads me to wonder if we can specify that clients SHOULD check for server support for range-requests IF loading a shard.

[d-v-b]

maybe someone better at HTTP than me can figure out why, when I use a different static file server (in this case, serve, invoked with --cors), neuroglancer cannot read from the sharded array.

In the neuroglancer javascript console I see this error, which comes from this code:

Error retrieving chunk [object Object]:4,5,0: Error: Content-range header not provided with HTTP 206 response.  Check server CORS configuration.

The requests are succeeding with a status of 206 (partial), and invoking the request with curl gives me a request with a correct-looking header. Note the presence of the ContentRange that's missing when the same request is made in javascript.

curl 'http://localhost:3000/data/zarr-3/dtype-int8/nd-3/co-8_ci-8_ck-dot/c/0/0/3' -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br, zstd, identity' -H 'range: bytes=-132' -H 'Referer: http://neuroglancer-demo.appspot.com/' -H 'Origin: http://neuroglancer-demo.appspot.com' -H 'Connection: keep-alive' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: cross-site' -H 'Priority: u=4' -D -
HTTP/1.1 206 Partial Content
Access-Control-Allow-Origin: *
Access-Control-Allow-Headers: *
Access-Control-Allow-Credentials: true
Access-Control-Allow-Private-Network: true
Content-Length: 132
Content-Disposition: inline; filename="3"
Accept-Ranges: bytes
ETag: "a47d45a273c07de14051753cfda606199e4d701e"
Content-Range: bytes 64-195/196
Date: Mon, 08 Jul 2024 14:28:12 GMT
Connection: keep-alive
Keep-Alive: timeout=5

[d-v-b]

for comparison, here's the same request made against the http-server instance (i.e., this static server works for neuroglancer):

curl 'http://localhost:8080/data/zarr-3/dtype-int8/nd-3/co-8_ci-8_ck-dot/c/0/0/3' -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br, zstd, identity' -H 'range: bytes=-132' -H 'Referer: http://neuroglancer-demo.appspot.com/' -H 'Origin: http://neuroglancer-demo.appspot.com' -H 'Connection: keep-alive' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: cross-site' -H 'Priority: u=4' -D -
HTTP/1.1 416 Range Not Satisfiable
accept-ranges: bytes
Access-Control-Allow-Origin: *
access-control-allow-headers: Origin, X-Requested-With, Content-Type, Accept, Range
Date: Mon, 08 Jul 2024 14:49:36 GMT
Connection: keep-alive
Keep-Alive: timeout=5
Content-Length: 0

and here is the request neuroglancer actually makes to that server (note that it's not the same as the previous request, not sure if that's significant):

curl 'http://127.0.0.1:8080/data/zarr-3/dtype-int8/nd-3/co-8_ci-8_ck-dot/c/0/0/4' -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.0) Gecko/20100101 Firefox/127.0' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br, zstd, identity' -H 'range: bytes=48-55' -H 'Referer: http://neuroglancer-demo.appspot.com/' -H 'Origin: http://neuroglancer-demo.appspot.com' -H 'Connection: keep-alive' -H 'Sec-Fetch-Dest: empty' -H 'Sec-Fetch-Mode: cors' -H 'Sec-Fetch-Site: cross-site' -H 'Priority: u=4' -D -
HTTP/1.1 206 Partial Content
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
access-control-allow-headers: Origin, X-Requested-With, Content-Type, Accept, Range
Content-Range: bytes 48-55/196
Content-Length: 8
Content-Type: application/octet-stream
cache-control: max-age=3600
last-modified: Tue, 02 Jul 2024 14:58:18 GMT
etag: W/"80636898-196-2024-07-02T14:58:18.857Z"
Date: Mon, 08 Jul 2024 14:52:47 GMT
Connection: keep-alive
Keep-Alive: timeout=5

[d-v-b]

I could resolve the serve vs http-server differences in #611. With that patch, serve works. I believe the issue with github as a host may hinge on CORS preflight behavior, as it seems that github sends an HTTP status code of 403 for all cors preflights.