Is it possible to set different custom headers on a per request basis?

[Frederik-Baetens]

It is possible to add custom headers to requests, as described here: https://stackoverflow.com/questions/65332880/aws-boto3-chime-client-send-x-amz-chime-bearer-header

However, I'd like to set these headers based on additional arguments that I pass to a put_object() call on a bucket resource for example. I tried passing these additional arguments as extra kwargs, but then I get an error about the kwargs being unknown.

What I want to do:

def add_custom_header(params, **kwargs):
   params['headers']['my_custom_header'] = kwargs.custom_arg
   
session.events.register('before-call.s3.GetObject', add_custom_header)

# ... construct bucket from session

obj = bucket.Object('myobjectname')
obj.get(**{'custom_arg':'hello'})['Body'].read()

However, this results in the following error:

Unknown parameter in input: "custom_arg", must be one of: Bucket, IfMatch, IfModifiedSince, IfNoneMatch, IfUnmodifiedSince, Key, Range, ResponseCacheControl, ResponseContentDisposition, ResponseContentEncoding, ResponseContentLanguage, ResponseContentType, ResponseExpires, VersionId, SSECustomerAlgorithm, SSECustomerKey, SSECustomerKeyMD5, RequestPayer, PartNumber, ExpectedBucketOwner, ChecksumMode

This is not a limitation of python. Adding custom kwargs at runtime normally works.

[aBurmeseDev]

Hi @Frederik-Baetens , thanks for reaching out. It looks like there's already a feature request open in boto/botocore#2050 where you also made a comment. Before I follow up on the feature request, I'd like to suggest checking out these resources below:

• This stackOverflow comment shows a way to add custom header to event handlers.
• Here is another example to inject custom headers to event request calls.

Please let me know if any of this resolves the issue!

[Frederik-Baetens]

Those examples don't really address my problem, as they don't add different headers on a per request basis. It's clear that the second example only sets one custom header that'll be used in all subsequent requests for that action. The first example is a bit more complicated but it still doesn't allow you to set custom headers on a per request basis. The improvement of the first example over the second is that it allows you to specify that one single header at runtime instead of hardcoding it, but after that it's still one identical header that gets passed along with every subsequent call.

So I think I'm still stuck, unless I misunderstood how the first example works.

[Frederik-Baetens]

I believe disabling the parameter_validation in the botocore config should solve my issue here. (but I haven't validated this)

[Frederik-Baetens]

Hmm, that does something different, unfortunately disabling parameter validation does not address my issue.

[jonemo]

I haven't had time to validate this, but I think this might be possible with the botocore event system today. You'd have to hook into two events:

1. Hook into one event that occurs before operation parameter validation. In the handler, remove the non-standard operation parameter custom_arg from the list and store it in the request context object. This way you can pass it as additional kwarg to get_object as you propose without having it fail validation or cause errors during request serialization.
2. Hook into another event that occurs when the HTTP request has already been serialized but before it gets sent. Fetch the custom parameter from the request context and inject it into the custom header.

We currently have an in-progress PR with much improved documentation for the botocore events system that may help find the right combination of events to achieve this: #3686

[Frederik-Baetens]

This does indeed work:

event_system = client.meta.events

def add_custom_headers(params, context, **kwargs):
    if (custom_headers := context.get("custom_headers")):
        params["headers"].update(custom_headers)


def process_custom_arguments(params, context, **kwargs):
    if (custom_headers := params.pop("custom_headers", None)):
        context["custom_headers"] = custom_headers

event_system.register('before-parameter-build.s3.PutObject', process_custom_arguments)
event_system.register('before-call.s3.PutObject', add_custom_headers)

Made a stackoverflow post with this example too: https://stackoverflow.com/questions/76601099/how-do-i-add-a-different-header-to-every-boto3-request/76601100#76601100