[BUG]: AttributeError: 'NoneType' object has no attribute 'lower'

[lorenzobalzani]

Bug Description
A strange error while using the library.

To Reproduce
Prerequisites:

• create anv .env file following the template:

AZURE_OPENAI_ENDPOINT = "TODO"
AZURE_OPENAI_API_KEY = "TODO"
OPENAI_API_VERSION = "2024-02-01"

• Having a Qdrant instance up and running with a collection called test and at least one point within it, following for example this quickstart.

Here is the code to reproduce the error:

from langchain_openai import AzureOpenAIEmbeddings, AzureChatOpenAI
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
from qdrant_client import QdrantClient
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_community.vectorstores import Qdrant

from trulens.apps.langchain import TruChain
from trulens.core import Feedback, Select, TruSession

from trulens_eval.feedback.provider import AzureOpenAI
from dotenv import load_dotenv

load_dotenv()


class Evaluator:
    """
    A class for evaluating a specific RAG (Retrieval-Augmented Generation) chain using the Trulens evaluation framework.
    Each Evaluator is linked to a specific chain, and evaluation feedback is tailored for that chain.

    Attributes:
        _tru (Tru): An instance of the Trulens evaluation engine.
        _recorder (TruChain): The recorder instance for the specific RAG chain being evaluated.
        metrics (dict): A dictionary that holds labels, their corresponding feedback metric functions, and the chain
                        parts they apply to.
    """

    metrics = {
        "Groundedness": {
            "metric": "groundedness_measure_with_cot_reasons",
            "parts": ["context", "output"]
        },
        "Answer Relevance": {
            "metric": "relevance_with_cot_reasons",
            "parts": ["input", "output"]
        },
        "Context Relevance": {
            "metric": "context_relevance_with_cot_reasons",
            "parts": ["input", "context"],
        }
    }

    def __init__(self, chain, collection_name, llm):
        """
        Initializes the Evaluator class with a specific RAG chain.

        Args:
            chain (LangChain chain): The chain to be evaluated.
            collection_name (str): The collection name for identifying the app instance.
            llm (AzureChatOpenAI): The language model instance providing the deployment name.
        """
        self._tru = TruSession()
        self._recorder = self._create_chain_recorder(chain, collection_name, llm)

    def get_chain_recorder(self):
        """
        Retrieve the Trulens recorder instance for the current RAG chain.

        Returns:
            TruChain: The TruChain object, configured to record the evaluation using the defined feedback functions.
        """
        return self._recorder

    def _create_chain_recorder(self, chain, collection_name, llm):
        """
        Set up the recorder for the RAG chain, identified by the collection name.

        This method initializes feedback functions to measure groundedness, answer relevance,
        and context relevance within a retrieval-augmented generation (RAG) chain.
        The feedback functions are provided by the AzureOpenAI provider.

        Args:
            chain (LangChain chain): The chain to be evaluated.
            collection_name (str): The collection name for identifying the app instance.
            llm (AzureChatOpenAI): The language model instance providing the deployment name.

        Returns:
            TruChain: The TruChain object, configured to record the evaluation using the defined feedback functions.
        """
        provider = AzureOpenAI(deployment_name=llm.deployment_name)

        chain_parts = {
            "context": Select.RecordRets["context"].page_content,
            "output": Select.RecordRets["answer"],
            "input": Select.RecordRets["input"]
        }

        feedbacks = []

        # Loop over the metrics dictionary and apply feedback based on the defined parts
        for label, metric_info in self.metrics.items():
            feedback_fn = getattr(provider, metric_info["metric"])  # Get the feedback function

            # Dynamically apply the feedback function to the relevant parts
            feedback = Feedback(feedback_fn, name=label)
            for part in metric_info["parts"]:
                feedback = feedback.on(chain_parts[part])

            # Handle aggregation if specified
            if "aggregate" in metric_info:
                feedback = feedback.aggregate(metric_info["aggregate"])

            feedbacks.append(feedback)

        # Create and return the TruChain instance
        tru_chain = TruChain(chain, app_id=collection_name, feedbacks=feedbacks)

        return tru_chain

# START INSTANTIATION EXAMPLE OBJECTS
qdrant_client = QdrantClient("http://localhost:6333")
collection_name = "test"
embeddings = AzureOpenAIEmbeddings(deployment="text-embedding-ada-002", chunk_size=1)
vector_store = Qdrant(qdrant_client, collection_name, embeddings)
retriever = vector_store.as_retriever(
            search_type="similarity_score_threshold",
            search_kwargs={
                "k": 4,
                "score_threshold": 0,
})
llm = AzureChatOpenAI(deployment_name="gpt-4o")
prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "Your task is to assist the user based on the given request and context: {context}"),
        MessagesPlaceholder("chat_history"),
        ("human", 'Task:"""{input}"""'),
    ]
)
user_input, chat_history = "What is the capital of France?", []
input_args = {"input": user_input, "chat_history": [(message.type, message.content) for interaction in chat_history
                                                                                        for message in interaction]}
# END INSTANTIATION EXAMPLE OBJECTS

def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)

# This Runnable takes a dict with keys 'input' and 'context',
# formats them into a prompt, and generates a response.
rag_chain = (
    {
        "input": lambda x: x["input"],
        "context": lambda x: format_docs(x["context"]),  # context
        "chat_history": lambda x: x["chat_history"],  # chat history
    }
    | prompt
    | llm
    | StrOutputParser()
).with_config(run_name="rag_chain")

retrieve_docs_chain = (lambda _: input_args["input"]) | retriever

# Below, we chain `.assign` calls. This takes a dict and successively adds keys-- "context" and "answer"--
# where the value for each key is determined by a Runnable. The Runnable operates on all keys in the dict.
docs_chain = RunnablePassthrough.assign(context=retrieve_docs_chain)
rag_chain = docs_chain | RunnablePassthrough.assign(answer=rag_chain)

# Evaluator

evaluator = Evaluator(rag_chain, collection_name, llm)
with evaluator.get_chain_recorder() as recorder:
    ai_answer = rag_chain.invoke(input=input_args)

Expected behavior
The code should work the same way with or without TruLens because the LCEL chain itself works correctly.

Relevant Logs/Tracebacks
You can take a look at the complete traceback at this GitHub Gist.

Environment:

• OS: macOS
• Python Version: 3.11.5
• TruLens version:

trulens==1.2.10
trulens-apps-langchain==1.2.10
trulens-core==1.2.10
trulens-dashboard==1.2.10
trulens-feedback==1.2.10
trulens-otel-semconv==1.2.10
trulens-providers-langchain==1.2.10
trulens-providers-openai==1.2.10
trulens_eval==1.2.10

• Versions of other relevant installed libraries:

langchain==0.3.11
langchain-community==0.3.11
langchain-core==0.3.24
openai==1.57.2
langchain-openai==0.2.12

Additional context
Looks like in trulens.providers.openai.endpoint.OpenAIEndpoint.handle_wrapped_call:490 model belongs to bindings.kwargs but is None.

[lorenzobalzani]

@sfc-gh-dhuang @sfc-gh-jreini @sfc-gh-gtokernliang