[EKS] [request]: v1.metrics.eks.amazonaws.com endpoints does not accept common Prometheus Accept headers

[adammw]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
v1.metrics.eks.amazonaws.com endpoints as documented here, return a 406: Not Acceptable header if an Accept header is present and not one of application/json, application/yaml, application/vnd.kubernetes.protobuf.

Prometheus formatted metrics are none of these so I suspect this is coming from API Gateway or another part of the stack doing unnecessarily validation.

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
Common prometheus integrations set either text/plain, application/openmetrics-text, or a mixed combination thereof with versions weighted with the quality value syntax (e.g. application/openmetrics-text;version=1.0.0,application/openmetrics-text;version=0.0.1;q=0.75,text/plain;version=0.0.4;q=0.5,*/*;q=0.1)

Are you currently working around this issue?
Yes - manually setting the Accept header to one of the supported values.

Additional context
Using Datadog agent as our Prometheus scraper, however the same behaviour can be repoduced with a cURL command:

curl -k -H "Accept: application/openmetrics-text" -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"  https://kubernetes.default.svc/apis/metrics.eksamazonaws.com/v1/kcm/container/metrics

EKS Server Version reported as v1.30.6-eks-7f9249a (reported platform version eks.20)

[isaac88]

I faced the same issue in DataDog, and by overriding the headers in the Datadog integration, I made it work.