Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

InvalidArgument when calling PutObject operation #4401

Closed
1 task
Zankroh opened this issue Jan 17, 2025 · 4 comments
Closed
1 task

InvalidArgument when calling PutObject operation #4401

Zankroh opened this issue Jan 17, 2025 · 4 comments
Assignees
@RyanFitzSimmonsAK
Labels
bug This issue is a confirmed bug. p2 This is a standard priority issue response-requested Waiting on additional information or feedback. s3

Comments

@Zankroh
Copy link

Zankroh commented Jan 17, 2025 •
edited
Loading

Describe the bug

Since 1.36 release, we're seeing upload errors with S3 client using upload_file method.
boto3.exceptions.S3UploadFailedError: Failed to upload test.txt to s3_bucket/test.txt: An error occurred (InvalidArgument) when calling the PutObject operation: Invalid Argument.

Regression Issue

  • Select this option if this issue appears to be a regression.

Expected Behavior

upload_file success without error.

Current Behavior

upload_file fails with boto3.exceptions.S3UploadFailedError exception.
boto3.exceptions.S3UploadFailedError: Failed to upload test.txt to s3_bucket/test.txt: An error occurred (InvalidArgument) when calling the PutObject operation: Invalid Argument.

Debug logs 
[2025-01-17 15:26:50][DEBUG] boto3.s3.transfer |> Opting out of CRT Transfer Manager. Preferred client: auto, CRT available: False, Instance Optimized: False.
[2025-01-17 15:26:50][DEBUG] boto3.s3.transfer |> Using default client. pid: 1632, thread: 26272
[2025-01-17 15:26:50][DEBUG] s3transfer.utils |> Acquiring 0
[2025-01-17 15:26:50][DEBUG] s3transfer.tasks |> UploadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x000001E5FC9F0520>}) about to wait for the following futures []
[2025-01-17 15:26:50][DEBUG] s3transfer.tasks |> UploadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x000001E5FC9F0520>}) done waiting for dependent futures
[2025-01-17 15:26:50][DEBUG] s3transfer.tasks |> Executing task UploadSubmissionTask(transfer_id=0, {'transfer_future': <s3transfer.futures.TransferFuture object at 0x000001E5FC9F0520>}) with kwargs {'client': <botocore.client.S3 object at 0x000001E5FC8C9160>, 'config': <boto3.s3.transfer.TransferConfig object at 0x000001E5FC8C9520>, 'osutil': <s3transfer.utils.OSUtils object at 0x000001E5FBDFE220>, 'request_executor': <s3transfer.futures.BoundedExecutor object at 0x000001E5FC9ED5E0>, 'transfer_future': <s3transfer.futures.TransferFuture object at 0x000001E5FC9F0520>}
[2025-01-17 15:26:50][DEBUG] s3transfer.futures |> Submitting task PutObjectTask(transfer_id=0, {'bucket': 's3_bucket', 'key': 'test.txt', 'extra_args': {'ChecksumAlgorithm': 'CRC32'}}) to executor <s3transfer.futures.BoundedExecutor object at 0x000001E5FC9ED5E0> for transfer request: 0.
[2025-01-17 15:26:50][DEBUG] s3transfer.utils |> Acquiring 0
[2025-01-17 15:26:50][DEBUG] s3transfer.tasks |> PutObjectTask(transfer_id=0, {'bucket': 's3_bucket', 'key': 'test.txt', 'extra_args': {'ChecksumAlgorithm': 'CRC32'}}) about to wait for the following futures []
[2025-01-17 15:26:50][DEBUG] s3transfer.utils |> Releasing acquire 0/None
[2025-01-17 15:26:50][DEBUG] s3transfer.tasks |> PutObjectTask(transfer_id=0, {'bucket': 's3_bucket', 'key': 'test.txt', 'extra_args': {'ChecksumAlgorithm': 'CRC32'}}) done waiting for dependent futures
[2025-01-17 15:26:50][DEBUG] s3transfer.tasks |> Executing task PutObjectTask(transfer_id=0, {'bucket': 's3_bucket', 'key': 'test.txt', 'extra_args': {'ChecksumAlgorithm': 'CRC32'}}) with kwargs {'client': <botocore.client.S3 object at 0x000001E5FC8C9160>, 'fileobj': <s3transfer.utils.ReadFileChunk object at 0x000001E5FC9F0B50>, 'bucket': 's3_bucket', 'key': 'test.txt', 'extra_args': {'ChecksumAlgorithm': 'CRC32'}}
[2025-01-17 15:26:50][DEBUG] botocore.hooks |> Event before-parameter-build.s3.PutObject: calling handler <function validate_ascii_metadata at 0x000001E5FB82B940>
[2025-01-17 15:26:50][DEBUG] botocore.hooks |> Event before-parameter-build.s3.PutObject: calling handler <function sse_md5 at 0x000001E5FB823D30>
[2025-01-17 15:26:50][DEBUG] botocore.hooks |> Event before-parameter-build.s3.PutObject: calling handler <function convert_body_to_file_like_object at 0x000001E5FB829280>
[2025-01-17 15:26:50][DEBUG] botocore.hooks |> Event before-parameter-build.s3.PutObject: calling handler <function validate_bucket_name at 0x000001E5FB823CA0>
[2025-01-17 15:26:50][DEBUG] botocore.hooks |> Event before-parameter-build.s3.PutObject: calling handler <function remove_bucket_from_url_paths_from_model at 0x000001E5FB829B80>
[2025-01-17 15:26:50][DEBUG] botocore.hooks |> Event before-parameter-build.s3.PutObject: calling handler <bound method S3RegionRedirectorv2.annotate_request_context of <botocore.utils.S3RegionRedirectorv2 object at 0x000001E5FC9ED4C0>>
[2025-01-17 15:26:50][DEBUG] botocore.hooks |> Event before-parameter-build.s3.PutObject: calling handler <bound method ClientCreator._inject_s3_input_parameters of <botocore.client.ClientCreator object at 0x000001E5FBE33B20>>
[2025-01-17 15:26:50][DEBUG] botocore.hooks |> Event before-parameter-build.s3.PutObject: calling handler <function generate_idempotent_uuid at 0x000001E5FB823AF0>
[2025-01-17 15:26:50][DEBUG] botocore.hooks |> Event before-parameter-build.s3.PutObject: calling handler <function _handle_request_validation_mode_member at 0x000001E5FB82A280>
[2025-01-17 15:26:50][DEBUG] botocore.hooks |> Event before-endpoint-resolution.s3: calling handler <function customize_endpoint_resolver_builtins at 0x000001E5FB829D30>
[2025-01-17 15:26:50][DEBUG] botocore.hooks |> Event before-endpoint-resolution.s3: calling handler <bound method S3RegionRedirectorv2.redirect_from_cache of <botocore.utils.S3RegionRedirectorv2 object at 0x000001E5FC9ED4C0>>
[2025-01-17 15:26:50][DEBUG] botocore.regions |> Calling endpoint provider with parameters: {'Bucket': 's3_bucket', 'Region': 'gra', 'UseFIPS': False, 'UseDualStack': False, 'Endpoint': 'https://s3_endpoint', 'ForcePathStyle': True, 'Accelerate': False, 'UseGlobalEndpoint': False, 'Key': 'test.txt', 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
[2025-01-17 15:26:50][DEBUG] botocore.regions |> Endpoint provider result: https://s3_endpoint/s3_bucket
[2025-01-17 15:26:50][DEBUG] botocore.regions |> Selecting from endpoint provider's list of auth schemes: "sigv4". User selected auth scheme is: "None"
[2025-01-17 15:26:50][DEBUG] botocore.regions |> Selected auth type "v4" as "v4" with signing context params: {'region': 'gra', 'signing_name': 's3', 'disableDoubleEncoding': True}
[2025-01-17 15:26:50][DEBUG] botocore.hooks |> Event before-call.s3.PutObject: calling handler <function add_expect_header at 0x000001E5FB82B040>
[2025-01-17 15:26:50][DEBUG] botocore.handlers |> Adding expect 100 continue header to request.
[2025-01-17 15:26:50][DEBUG] botocore.hooks |> Event before-call.s3.PutObject: calling handler <bound method S3ExpressIdentityResolver.apply_signing_cache_key of <botocore.utils.S3ExpressIdentityResolver object at 0x000001E5FC9ED490>>
[2025-01-17 15:26:50][DEBUG] botocore.hooks |> Event before-call.s3.PutObject: calling handler <function add_recursion_detection_header at 0x000001E5FB823700>
[2025-01-17 15:26:50][DEBUG] botocore.hooks |> Event before-call.s3.PutObject: calling handler <function add_query_compatibility_header at 0x000001E5FB82A1F0>
[2025-01-17 15:26:50][DEBUG] botocore.hooks |> Event before-call.s3.PutObject: calling handler <function inject_api_version_header_if_needed at 0x000001E5FB8293A0>
[2025-01-17 15:26:50][DEBUG] botocore.endpoint |> Making request for OperationModel(name=PutObject) with params: {'url_path': '/test.txt', 'query_string': {}, 'method': 'PUT', 'headers': {'x-amz-sdk-checksum-algorithm': 'CRC32', 'User-Agent': 'Boto3/1.36.1 md/Botocore#1.36.1 ua/2.0 os/windows#10 md/arch#amd64 lang/python#3.9.13 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.36.1', 'Expect': '100-continue', 'Transfer-Encoding': 'chunked', 'Content-Encoding': 'aws-chunked', 'X-Amz-Trailer': 'x-amz-checksum-crc32', 'X-Amz-Decoded-Content-Length': '12'}, 'body': <botocore.httpchecksum.AwsChunkedWrapper object at 0x000001E5FCA01700>, 'auth_path': '/s3_bucket/test.txt', 'url': 'https://s3_endpoint/s3_bucket/test.txt', 'context': {'client_region': 'gra', 'client_config': <botocore.config.Config object at 0x000001E5FC8C94C0>, 'has_streaming_input': True, 'auth_type': 'v4', 'unsigned_payload': None, 's3_redirect': {'redirected': False, 'bucket': 's3_bucket', 'params': {'Bucket': 's3_bucket', 'Key': 'test.txt', 'Body': <s3transfer.utils.ReadFileChunk object at 0x000001E5FC9F0B50>, 'ChecksumAlgorithm': 'CRC32'}}, 'input_params': {'Bucket': 's3_bucket', 'Key': 'test.txt'}, 'signing': {'region': 'gra', 'signing_name': 's3', 'disableDoubleEncoding': True}, 'endpoint_properties': {'authSchemes': [{'disableDoubleEncoding': True, 'name': 'sigv4', 'signingName': 's3', 'signingRegion': 'gra'}]}, 'checksum': {'request_algorithm': {'algorithm': 'crc32', 'in': 'trailer', 'name': 'x-amz-checksum-crc32'}}}}
[2025-01-17 15:26:50][DEBUG] botocore.hooks |> Event request-created.s3.PutObject: calling handler <function signal_not_transferring at 0x000001E5FC941CA0>
[2025-01-17 15:26:50][DEBUG] botocore.hooks |> Event request-created.s3.PutObject: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x000001E5FC8C9100>>
[2025-01-17 15:26:50][DEBUG] botocore.hooks |> Event choose-signer.s3.PutObject: calling handler <function set_operation_specific_signer at 0x000001E5FB823940>
[2025-01-17 15:26:50][DEBUG] botocore.hooks |> Event before-sign.s3.PutObject: calling handler <function remove_arn_from_signing_path at 0x000001E5FB829CA0>
[2025-01-17 15:26:50][DEBUG] botocore.hooks |> Event before-sign.s3.PutObject: calling handler <function _set_extra_headers_for_unsigned_request at 0x000001E5FB82A310>
[2025-01-17 15:26:50][DEBUG] botocore.hooks |> Event before-sign.s3.PutObject: calling handler <bound method S3ExpressIdentityResolver.resolve_s3express_identity of <botocore.utils.S3ExpressIdentityResolver object at 0x000001E5FC9ED490>>
[2025-01-17 15:26:50][DEBUG] botocore.auth |> Calculating signature using v4 auth.
[2025-01-17 15:26:50][DEBUG] botocore.auth |> CanonicalRequest:
PUT
/s3_bucket/test.txt

content-encoding:aws-chunked
host:s3_endpoint
transfer-encoding:chunked
x-amz-content-sha256:STREAMING-UNSIGNED-PAYLOAD-TRAILER
x-amz-date:20250117T142650Z
x-amz-decoded-content-length:12
x-amz-sdk-checksum-algorithm:CRC32
x-amz-trailer:x-amz-checksum-crc32

content-encoding;host;transfer-encoding;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-sdk-checksum-algorithm;x-amz-trailer
STREAMING-UNSIGNED-PAYLOAD-TRAILER
[2025-01-17 15:26:50][DEBUG] botocore.auth |> StringToSign:
AWS4-HMAC-SHA256
20250117T142650Z
20250117/gra/s3/aws4_request
a762ef9005bebe40af9ae33eac3f05f104d8c673e946430090cc629f5029141f
[2025-01-17 15:26:50][DEBUG] botocore.auth |> Signature:
e17e474e27bdd5dc55529a5b422f4ab83f6e5ef15e20b5a6f70712e6e5370600
[2025-01-17 15:26:50][DEBUG] botocore.hooks |> Event request-created.s3.PutObject: calling handler <function signal_transferring at 0x000001E5FC941DC0>
[2025-01-17 15:26:50][DEBUG] botocore.hooks |> Event request-created.s3.PutObject: calling handler <function add_retry_headers at 0x000001E5FB829AF0>
[2025-01-17 15:26:50][DEBUG] botocore.endpoint |> Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=https://s3_endpoint/s3_bucket/test.txt, headers={'x-amz-sdk-checksum-algorithm': b'CRC32', 'User-Agent': b'Boto3/1.36.1 md/Botocore#1.36.1 ua/2.0 os/windows#10 md/arch#amd64 lang/python#3.9.13 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.36.1', 'Expect': b'100-continue', 'Transfer-Encoding': b'chunked', 'Content-Encoding': b'aws-chunked', 'X-Amz-Trailer': b'x-amz-checksum-crc32', 'X-Amz-Decoded-Content-Length': b'12', 'X-Amz-Date': b'20250117T142650Z', 'X-Amz-Content-SHA256': b'STREAMING-UNSIGNED-PAYLOAD-TRAILER', 'Authorization': b'AWS4-HMAC-SHA256 Credential=xxxx/20250117/gra/s3/aws4_request, SignedHeaders=content-encoding;host;transfer-encoding;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-sdk-checksum-algorithm;x-amz-trailer, Signature=e17e474e27bdd5dc55529a5b422f4ab83f6e5ef15e20b5a6f70712e6e5370600', 'amz-sdk-invocation-id': b'aa5f85e9-8118-47b5-9466-510d3fb5d46c', 'amz-sdk-request': b'attempt=1'}>
[2025-01-17 15:26:50][DEBUG] urllib3.connectionpool |> Starting new HTTPS connection (1): s3_endpoint:443
[2025-01-17 15:26:51][DEBUG] urllib3.connectionpool |> https://s3_endpoint:443 "PUT /s3_bucket/test.txt HTTP/1.1" 400 None
[2025-01-17 15:26:51][DEBUG] botocore.hooks |> Event before-parse.s3.PutObject: calling handler <function _handle_200_error at 0x000001E5FB82A040>
[2025-01-17 15:26:51][DEBUG] botocore.hooks |> Event before-parse.s3.PutObject: calling handler <function handle_expires_header at 0x000001E5FB829E50>
[2025-01-17 15:26:51][DEBUG] botocore.parsers |> Response headers: {'Content-Type': 'application/xml', 'x-amz-id-2': 'txf35c291b65774f65977d2-00678a68aa', 'x-amz-request-id': 'txf35c291b65774f65977d2-00678a68aa', 'Date': 'Fri, 17 Jan 2025 14:26:50 GMT', 'Transfer-Encoding': 'chunked'}
[2025-01-17 15:26:51][DEBUG] botocore.parsers |> Response body:
b"<?xml version='1.0' encoding='UTF-8'?>\n<Error><Code>InvalidArgument</Code><Message>Invalid Argument.</Message><RequestId>txf35c291b65774f65977d2-00678a68aa</RequestId><ArgumentName>x-amz-content-sha256</ArgumentName><ArgumentValue>STREAMING-UNSIGNED-PAYLOAD-TRAILER</ArgumentValue></Error>"
[2025-01-17 15:26:51][DEBUG] botocore.hooks |> Event needs-retry.s3.PutObject: calling handler <function _update_status_code at 0x000001E5FB82A160>
[2025-01-17 15:26:51][DEBUG] botocore.hooks |> Event needs-retry.s3.PutObject: calling handler <botocore.retryhandler.RetryHandler object at 0x000001E5FC9ED400>
[2025-01-17 15:26:51][DEBUG] botocore.retryhandler |> No retry needed.
[2025-01-17 15:26:51][DEBUG] botocore.hooks |> Event needs-retry.s3.PutObject: calling handler <bound method S3RegionRedirectorv2.redirect_from_error of <botocore.utils.S3RegionRedirectorv2 object at 0x000001E5FC9ED4C0>>
[2025-01-17 15:26:51][DEBUG] s3transfer.tasks |> Exception raised.
Traceback (most recent call last):
  File "C:\Users\test\project\env\lib\site-packages\s3transfer\tasks.py", line 135, in __call__
    return self._execute_main(kwargs)
  File "C:\Users\test\project\env\lib\site-packages\s3transfer\tasks.py", line 158, in _execute_main
    return_value = self._main(**kwargs)
  File "C:\Users\test\project\env\lib\site-packages\s3transfer\upload.py", line 796, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "C:\Users\test\project\env\lib\site-packages\botocore\client.py", line 569, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "C:\Users\test\project\env\lib\site-packages\botocore\client.py", line 1023, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Invalid Argument.
[2025-01-17 15:26:51][DEBUG] s3transfer.utils |> Releasing acquire 0/None
Traceback (most recent call last):
  File "C:\Users\test\project\env\lib\site-packages\boto3\s3\transfer.py", line 372, in upload_file
    future.result()
  File "C:\Users\test\project\env\lib\site-packages\s3transfer\futures.py", line 103, in result
    return self._coordinator.result()
  File "C:\Users\test\project\env\lib\site-packages\s3transfer\futures.py", line 264, in result
    raise self._exception
  File "C:\Users\test\project\env\lib\site-packages\s3transfer\tasks.py", line 135, in __call__
    return self._execute_main(kwargs)
  File "C:\Users\test\project\env\lib\site-packages\s3transfer\tasks.py", line 158, in _execute_main
    return_value = self._main(**kwargs)
  File "C:\Users\test\project\env\lib\site-packages\s3transfer\upload.py", line 796, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "C:\Users\test\project\env\lib\site-packages\botocore\client.py", line 569, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "C:\Users\test\project\env\lib\site-packages\botocore\client.py", line 1023, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (InvalidArgument) when calling the PutObject operation: Invalid Argument.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "C:\Users\test\project\env\lib\site-packages\boto3\s3\inject.py", line 145, in upload_file
    return transfer.upload_file(
  File "C:\Users\test\project\env\lib\site-packages\boto3\s3\transfer.py", line 378, in upload_file
    raise S3UploadFailedError(
boto3.exceptions.S3UploadFailedError: Failed to upload test.txt to s3_bucket/test.txt: An error occurred (InvalidArgument) when calling the PutObject operation: Invalid Argument.

Reproduction Steps

With python 3.9.13 and boto3==1.36.1 (same error with boto3==1.36.0)

import boto3

s3_client = boto3.client(
    "s3",
    aws_access_key_id=AWS_ACCESS_KEY_ID,
    aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
    endpoint_url=AWS_ENDPOINT,
    region_name=AWS_DEFAULT_REGION
)

response = s3_client.upload_file(
    "test.txt",
    AWS_BUCKET,
    "test.txt"
)

Possible Solution

No response

Additional Information/Context

No response

SDK version used

1.36.1

Environment details (OS name and version, etc.)

Windows 11 22H2 ; python 3.9.13
@Zankroh Zankroh added bug This issue is a confirmed bug. needs-triage This issue or PR still needs to be triaged. labels Jan 17, 2025
@jonathan343
Copy link
Contributor

Hey @Zankroh, thanks for reaching out!

This issues is related to the S3 default integrity changes we introduced in boto3-1.36.0. Please see our announcement in #4392 for more details.

I wouldn't expect this type of request to fail for Amazon S3. Can you confirm if you're hitting a Amazon endpoint or third-party S3-compatible service?

@Zankroh
Copy link
Author

Zankroh commented Jan 17, 2025 •
edited
Loading

Hi @jonathan343,

Thank you for your help.
It's a third party endpoint (OVH provider) but works fine with 1.35.99 version.

I saw the announcement, so I tried this:

client = boto3.client(
    "s3",
    aws_access_key_id=AWS_ACCESS_KEY_ID,
    aws_secret_access_key=AWS_SECRET_ACCESS_KEY,
    endpoint_url=AWS_ENDPOINT,
    region_name=AWS_DEFAULT_REGION,
    config=Config(
        request_checksum_calculation="when_required",
        response_checksum_validation="when_required"
    )
)

But same exception remains.

@RyanFitzSimmonsAK RyanFitzSimmonsAK self-assigned this Jan 17, 2025
@RyanFitzSimmonsAK RyanFitzSimmonsAK added s3 p2 This is a standard priority issue and removed needs-triage This issue or PR still needs to be triaged. labels Jan 17, 2025
@RyanFitzSimmonsAK
Copy link
Contributor

Could you provide debug logs of an upload_file operation failing while having request_checksum_calculation and response_checksum_validation set to "when_required"? You can get debug logs by adding boto3.set_stream_logger('') to the top of your script, and redacting any sensitive information.

@RyanFitzSimmonsAK RyanFitzSimmonsAK added the response-requested Waiting on additional information or feedback. label Jan 17, 2025
@RyanFitzSimmonsAK
Copy link
Contributor

This is likely a result of the behavior described in boto/s3transfer#327. A default value we specified in s3transfer is taking precedence over the “when_required” config and setting a CRC32 default checksum. We’ll work on addressing this issue and provide more updates when we have them. I'll be closing this issue as a duplicate, please refer to the s3transfer issue for updates. In the meantime, you should still be able to use put_object, since it doesn't use the transfer manager.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@RyanFitzSimmonsAK RyanFitzSimmonsAK

Labels
bug This issue is a confirmed bug. p2 This is a standard priority issue response-requested Waiting on additional information or feedback. s3
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jonathan343 @RyanFitzSimmonsAK @Zankroh