[Feature Request]: Add support for returning prompt and completion token count

[nanddeepn]

Scenario

Hi, We want to track the tokens used on prompt and completion via app developed with Teams AI.

Solution

Prompt tokens:

Below code can help us get the prompt text

app.activity(ActivityTypes.Message, async (context: TurnContext, state: ApplicationTurnState) => {
    let message = context.activity.text;
    await context.sendActivity(`You said: ${message}`);
});

Similarly, the context should return tokens used by input prompt.

Completion tokens:
Not sure, if there is any option or method available to get the completion text via code?
Similarly there should be an option to get the completion tokens.

Thank you

Additional Context

No response

[zedhaque]

Hi,

You can count the tokens yourself in Python by importing a tokenizer and using it to count the tokens. However, this isn’t the most reliable solution.

from teams.ai.tokenizers import Tokenizer
tokens = len(tokenizer.encode(message))  # Something like this

A better way is to use the "usage data" included in the REST API responses from Azure OpenAI/OpenAI. The usage information is already part of the chat completion response.

You can find details here:
https://learn.microsoft.com/en-us/azure/ai-services/openai/reference#chat-completions

Look for this section:

    "usage": {
      "completion_tokens": 557,
      "prompt_tokens": 33,
      "total_tokens": 590
    }

If the teams-ai library exposed this data for you to access, similar to how you access messages, it would be more convenient.
For more information, see these tickets:

#2062
#2119

What language are you planning to use?

[yonggang-xiao]

In OpenAIModel.js file
const completion = await this._client.chat.completions.create(params);
then you can use console.log to see completion.usage data.

[goncafab]

Hi,

You can count the tokens yourself in Python by importing a tokenizer and using it to count the tokens. However, this isn’t the most reliable solution.

from teams.ai.tokenizers import Tokenizer
tokens = len(tokenizer.encode(message))  # Something like this

A better way is to use the "usage data" included in the REST API responses from Azure OpenAI/OpenAI. The usage information is already part of the chat completion response.

You can find details here: https://learn.microsoft.com/en-us/azure/ai-services/openai/reference#chat-completions

Look for this section:

    "usage": {
      "completion_tokens": 557,
      "prompt_tokens": 33,
      "total_tokens": 590
    }

If the teams-ai library exposed this data for you to access, similar to how you access messages, it would be more convenient. For more information, see these tickets:

#2062 #2119

What language are you planning to use?

In my case, I was looking to use this with Python. I'm trying to calculate the number of tokens used myself, but there's no single point that we can get all messages, plus prompt, plus all sources used. The best way would be indeed having the full response of the OpenAI API bubbled up.