Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MistralAI][Tool Call]Failed to convert response to pydantic model #1792

Closed
lironesamoun opened this issue Jan 15, 2025 · 4 comments · Fixed by #1812
Closed

[MistralAI][Tool Call]Failed to convert response to pydantic model #1792

lironesamoun opened this issue Jan 15, 2025 · 4 comments · Fixed by #1812

Comments

@lironesamoun
Copy link

Hi,

I've created an Agent that works pretty well in general

return Agent(
        model=MistralChat(id="mistral-large-latest", response_format={
              "type": "json_object",
          }),
        tools=[GoogleSearch(fixed_language="en")],
        description=description_en,
        instructions=instructions_en,
        guidelines=guidelines_en,
        structured_outputs=False,
        parse_response=True,
        response_model=CustomResponse,
        markdown=True,
        show_tool_calls=False,
        debug_mode=debug,
        monitoring=True,
    )

but sometimes, I don't know why, the behavior is not correct and I get an error. Below, It's like it tries to convert the Functions inside the DEBUG to a json.

DEBUG    Functions: {'google_search': Function(name='google_search', description='Use this function to search Google for a specified query.\n\nArgs:\n    query (str): The query to search     
         for.\n    max_results (int, optional): The maximum number of results to return. Default is 5.\n    language (str, optional): The language of the search results. Default is           
         "en".\n\nReturns:\n    str: A JSON formatted string containing the search results.', parameters={'type': 'object', 'properties': {'query': {'type': 'string'}, 'max_results': {'type':
         'number'}, 'language': {'type': 'string'}}, 'required': ['query']}, strict=None, entrypoint=<function GoogleSearch.google_search at 0x10b2b91c0>, sanitize_arguments=True,            
         show_result=False, stop_after_tool_call=False, pre_hook=None, post_hook=None)}                                                                                                        
DEBUG    ---------- Mistral Response End ----------                                                                                                                                            
DEBUG    Added 4 Messages to AgentMemory                                                                                                                                                       
DEBUG    Added AgentRun to AgentMemory                                                                                                                                                         
DEBUG    --**-- Logging Agent Run                                                                                                                                                              
DEBUG    *********** Agent Run End: 66f58dd1-6c2a-4eeb-b6b8-0dcd283eaefc ***********                                                                                                           
WARNING  Failed to convert response to pydantic model: 1 validation error for CustomResponse                                                                                                     
           Input should be an object [type=model_type, input_value=[{'name': 'google_search'...sins APPLE'}}], input_type=list]                                                      
             For further information visit https://errors.pydantic.dev/2.10/v/model_type                                                                                                       
WARNING  Failed to convert response to response_model                                                                                                                                          
Error analyzing brand: 'str' object has no attribute 'model_dump'

Or sometimes, I can see on debug mode that I have a correct json rendered but still at the end I get.

WARNING  Failed to convert response to pydantic model: 1 validation error for ShopResponse                                                                                                     
         observations                                                                                                                                                                          
           Input should be a valid array [type=list_type, input_value=None, input_type=NoneType]                                                                                               
             For further information visit https://errors.pydantic.dev/2.10/v/list_type                                                                                                        
WARNING  Failed to convert response to response_model                                                                                                                                          
Error analyzing brand: 'str' object has no attribute 'model_dump'

After few retry, it works and no problems.

It's pretty random and I don't know what I should do. If it's something to do with the model or not.

@lironesamoun
Copy link
Author

Another example where the response content of the agent is the query itself instead of being the results.

DEBUG    [{"name": "google_search", "arguments": {"language": "fr", "max_results": 5, "query": "Find closest Apple Shop"}}]                                                     
DEBUG    Functions: {'google_search': Function(name='google_search', description='Use this function to search Google for a specified query.', parameters={'type': 'object', 'properties':      
         {'query': {'type': 'string', 'description': '(str) The query to search for.'}, 'max_results': {'type': 'number', 'description': '(int) The maximum number of results to return.       
         Default is 5.'}, 'language': {'type': 'string', 'description': '(str) The language of the search results. Default is "en".'}}, 'required': ['query']}, strict=None,                   
         entrypoint=<function GoogleSearch.google_search at 0x107a84a40>, sanitize_arguments=True, show_result=False, stop_after_tool_call=False, pre_hook=None, post_hook=None)}              
DEBUG    ---------- Mistral Response End ----------                                                                                                                                            
DEBUG    Added 4 Messages to AgentMemory                                                                                                                                                       
DEBUG    Added AgentRun to AgentMemory                                                                                                                                                         
DEBUG    --**-- Logging Agent Run                                                                                                                                                              
DEBUG    *********** Agent Run End: 56d13509-dd33-4a1e-936f-5f7fa4f67b6f ***********                                                                                                           
WARNING  Failed to convert response to response_model              
                                                                                                                            
VALIDATED CONTENT: [{'name': 'google_search', 'arguments': {'language': 'fr', 'max_results': 5, 'query': 'Find closest Apple Shop'}}]

@lironesamoun
Copy link
Author

I tried with others models like Claude and I don't have this kind of problem. I have the impression, that it's related to the way Tool Call is handled.
Claude: Successfully executes the google_search tool, integrates the results, and generates a structured response in the required JSON format.
Mistral:
Generates tool calls (google_search) but either skips execution or doesn't integrate the results into the assistant's final message.
Often returns raw tool call data instead of a response in the expected JSON format.

@lironesamoun lironesamoun changed the title Failed to convert response to pydantic model [MistralAI][Tool Call]Failed to convert response to pydantic model Jan 16, 2025
@dirkbrnd
Copy link
Contributor

This should be fixed in the next release

@KenG330
Copy link

KenG330 commented Jan 20, 2025

I've been getting this error with Sonnet 3.5 and GPT-4o.

dirkbrnd added a commit that referenced this issue Jan 20, 2025

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
## Description

Tool calling does not work correctly with response models. This will use
the response model if there is one, overriding any tool calling. The
tool calls are then displayed separately.

This also solves the issue for teams.

Fixes #1792
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants