synapse-s3-storage-provider cant connect to minio

[Fjox]

i checked the credentials and everything is properly set

  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/http/server.py", line 332, in _async_render_wrapper
    callback_return = await self._async_render(request)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/http/server.py", line 544, in _async_render
    callback_return = await raw_callback_return
                      ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/rest/client/media.py", line 194, in on_GET
    await remote_resp_function(
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/media/thumbnailer.py", line 496, in respond_remote_thumbnail
    media_info = await self.media_repo.get_remote_media_info(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/media/media_repository.py", line 590, in get_remote_media_info
    responder, media_info = await self._get_remote_media_impl(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/media/media_repository.py", line 660, in _get_remote_media_impl
    responder = await self.media_storage.fetch_media(file_info)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/media/media_storage.py", line 217, in fetch_media
    res: Any = await provider.fetch(path, file_info)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/logging/opentracing.py", line 922, in _wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/synapse/media/storage_provider.py", line 135, in fetch
    return await maybe_awaitable(self.backend.fetch(path, file_info))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/s3_storage_provider.py", line 218, in s3_download_task
    resp = s3_client.get_object(Bucket=bucket, Key=key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/botocore/client.py", line 569, in _api_call
    return self._make_api_call(operation_name, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venvs/matrix-synapse/lib/python3.11/site-packages/botocore/client.py", line 1023, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (SignatureDoesNotMatch) when calling the GetObject operation: The request signature we calculated does not match the signature you provided. Check your key and signing method.```

[halkeye]

did you make any progress on this? it used to work

[halkeye]

so it seems its an upstream issue - boto/boto3#4392

[superboum]

Same problem with Garage. It appears that switching from HTTPS to HTTP fixed the issue for us, possibly because boto uses a different logic for HTTP endpoints. (Note that we use HTTP with a local gateway so traffic is not going in cleartext on the network).

Edit: issue is tracked here for Garage now: https://git.deuxfleurs.fr/Deuxfleurs/garage/issues/824

[halkeye]

I just downgraded and pinned boto3 for now.

https://status.digitalocean.com/incidents/zbrpd3j7hrrd