[Code improvement] MLSdkAsyncHttpResponseHandler should report Headers on Errors for extra context

[brianf-aws]

What is the bug?
Nowadays when working with remote models like Bedrock or SageMaker users run into issues such as a 500 INTERNAL_ERROR. However its not obvious why they got this error. It turns out that AWS gives more information or errors within the Headers.

Currently ML-Commons reports back errors with the Body of the error response but one has to enable debugging levels in order to see the header (In order to get this log).

ml-commons/ml-algorithms/src/main/java/org/opensearch/ml/engine/algorithms/remote/MLSdkAsyncHttpResponseHandler.java

Line 89 in 8da7bd2

log.debug("received response headers: " + sdkResponse.headers());

This is not obvious it would be great if we could provide more context to users when there is a error from a remote service.

How can one reproduce the bug?
Steps to reproduce the behavior:
0. Enable Debug level logs for the linked ML-Commons class

1. Get an error from any remote service
2. Compare the debug level output to that of the standard info when the error from a remote service occurs like here

Do you have any additional context?
AWS services give key hints in the headers of a response not so much of the body. Its a hassle to enable logs to get this valuable information. And every re attempt of getting the bad response (With no clear hint) results in time and money wasted.

[pyek-bot]

i can work on this, please assign it to me