Passing ClientConfiguration via N5Factory to AmazonS3utils.createS3

[kgabor]

In our use case, using BigStitcher-Spark output in an EMR-serverless environment, S3 often fails with "reduce rate" error messages unless a retry/throttling policy is set up:

org.janelia.saalfeldlab.n5.N5Exception$N5IOException: Failed to write block [78, 11, 1] into dataset gabor.kovacs/exaSPIM_706301_2024-04-23_11-24-24_fusion_2024-05-21_00-00-01/fused.zarr/0/1/11/78
	at org.janelia.saalfeldlab.n5.zarr.ZarrKeyValueWriter.writeBlock(ZarrKeyValueWriter.java:545) ~[BigStitcher-Spark-0.0.2-OpenDAL_auth_2024-05-21.jar:?]
	at net.preibisch.bigstitcher.spark.blk.N5Helper.saveBlock(N5Helper.java:29) ~[BigStitcher-Spark-0.0.2-OpenDAL_auth_2024-05-21.jar:?]
	at net.preibisch.bigstitcher.spark.fusion.WriteSuperBlock.call(WriteSuperBlock.java:239) ~[BigStitcher-Spark-0.0.2-
...
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Please reduce your request rate. (Service: Amazon S3; Status Code: 503; Error Code: SlowDown; Request ID: 6XZ2S9JFW0820CS8; S3 Extended Request ID: RRg66prSANmMW6vUj5HUzhKuesUNPunbmjXAQ9E7QQ6YkmnzSz6m2L/y+gpfmbZHoeQ+FARkxGI=; Proxy: null)

The way to set this up is passing a ClientConfiguration with setting up a retry policy to the AmazonS3ClientBuilder, e.g.:

// Back up with the default increasing delays policy in case of reduce rate error
ClientConfiguration s3Conf = new ClientConfiguration().withMaxErrorRetry(32);  
final AmazonS3ClientBuilder builder = AmazonS3ClientBuilder.standard().withClientConfiguration(s3Conf);

Would you please add options to do such additional configuration via the N5Factory interface? The configuration should be applied when the AmazonS3 client is created here: org.janelia.saalfeldlab.n5.s3.AmazonS3Utils#createS3

[bogovicj]

#34

[bogovicj]

released in v4.2.0