♻️ [Tasks] JSON spec: text-generation

[SBrandeis]

TL;DR

• Update text-generation spec to match TGI API
• Add conversational spec, heavily inspired by TGI messages API (cc @Wauplin @osanseviero @Narsil )
  • Relevant related work: Update conversational widget to use text-generation (+ remove conversational task) #457 & https://github.com/huggingface/moon-landing/pull/8723
• regenerate typescript code for those tasks

[SBrandeis]

@Narsil @OlivierDehaene I'm not sure what this parameter represents

[Narsil]

do_sample: bool -> Should the generation be greedy or not (it cannot be set to True if any other params is set, but sometimes you do not want to specify temp, nor top_p, nor anyhing, therefore setting do_sample=True is for that).

best_of: int -> Run n sampling queries, return the best (in terms of total logprob). Clashes with streaming.

[osanseviero]

Given https://github.com/huggingface/moon-landing/pull/8723, do we really need the conversational spec?

[SBrandeis]

@osanseviero I think speccing the conversational API can still be useful for the widget and inference clients, wdyt?

[Wauplin]

Sorry being late to the show here. I'm currently reviewing this PR :) (don't merge it too quickly 🙏)

[Wauplin]

Tried to list all the difference I've spotted between these specs, the TGI documentation and the Python client implementation. I think we should stick to the existing TGI parameters as they are already in use + "newer" than the transformers-based parameters.

[Wauplin]

(commenting on inference.ts but more related to ./spec)

TGI server has an OpenAPI documentation to document its inputs and outputs (see swagger ui and openapi.json). I've spotted a few differences compared to the specs here. Some of these properties are currently specific to TGI and not available in the "transformers-based inference" endpoints.

In TextGenerationParameters:

• max_new_tokens (integer) => Maximum number of generated tokens.
• repetition_penalty (number) => The parameter for repetition penalty. A value of 1.0 means no penalty. See this paper for more details.
• seed (integer) => Random sampling seed.
• stop (array of string) => Stop generating tokens if a member of stop is generated. (currently named stop_sequences in these specs / stop in TGI => same purpose => to harmonize?)
• top_n_tokens (integer) => ??? (no idea what this is. We don't have it either in Python client)

In the Python client, along with the inputs (string) + parameters (TextGenerationParameters) values in the request payload we also have a stream (bool) value. If True the output type is different (same as /generate_stream instead of /generate in TGI). I don't see it documented here so I wonder if these docs might be a bit outdated (ping @Narsil you might know more?). In any case, this is an option currently working and used so worth keeping it.

In TextGenerationOutputDetails:

• best_of_sequences (array of BestOfSequence) => Additional sequences when using the best_of parameter.

BestOfSequence (currently missing):

• generated_text (string) => The generated text.
• finish_reason (FinishReason) => The reason for the generation to finish, represented by a FinishReason value.
• generated_tokens (integer) => The number of generated tokens in the sequence.
• seed (integer) => The sampling seed if sampling was activated.
• prefill (array of InputToken) => he decoder input tokens. Empty if decoder_input_details is False.
• tokens (array of Token) => The generated tokens.

BestOfSequence object is actually very similar to TextGenerationOutputDetails but 1 level of nesting below.

[SBrandeis]

I deliberately left out the stream parameter because it was API/transport-specific (and not inference-specific)

I'm happy to revisit that if there's a consensus tho

[Wauplin]

Thanks for making the changes @SBrandeis! Re-reviewed it and looks good to me now to use it in the Python client. Fair-enough about not adding the stream parameters (and let's revisit if we realize we really need it at some point).