✨ [Widgets] Enable streaming in the conversational widget

[SBrandeis]

Linked to #360 #410

Should unlock the conversational widget on Mistral if I'm not mistaken?

TL;DR

• Leverage inference types from @huggingface/task to type input and output of the inference client
• Use the inference client to call the inference serverless API
• Use the streaming API when supported for the model

[SBrandeis]

Screen recording showcase (first answer probably cached)

Screen.Recording.2024-02-16.at.17.30.31.mov

[julien-c]

wow, thanks for working on this.

This is starting to all fit together very nicely!

[mishig25]

Suggested change

	Streaming with <a href="https://huggingface.co/docs/text-generation-inference" class="underline">TGI</a>
	Streaming with <a href="https://huggingface.co/docs/text-generation-inference" class="hover:underline">TGI</a>

made it to hover:underline

[SBrandeis]

cc @gary149

[mishig25]

should TextGenerationStreamOutput live in @huggingface/tasks/src/tasks/text-generation/inference just like TextGenerationInput & TextGenerationOutput ?

[SBrandeis]

Good question - cc @Wauplin with whom we discussed that previously

The current philosophy is to not type the streaming mode because it's transfer-specific, not inference-specific

[Wauplin]

Indeed, we mentioned it here once: #468 (comment).

As a matter of fact, I came to the conclusion today that we should specify the stream parameter and the streamed output in our JS specs. I am currently starting to use the generated types in Python (see ongoing PR) and for now I've kept text_generation apart since I'm missing TextGenerationStreamResponse (defined here but I don't want to mix the auto-generated types with previous definitions). Agree it's more "transfer-specific" rather than "inference-specific" but the thing is that setting stream=True is modifying the output format so we need to document that somewhere.

[gary149]

Really nice!

[krampstudio]

Nice.
Just one optional comment

[krampstudio]

Do we need it for every widgets?

If yes, that's ok to initialize it here,
otherwise, you can do something like this :

in store.js

import { get, writable } from 'svelte/store'; 

const tgiSupportedModels = writable<Set<string> | undefined>(undefined);

export async function getTgiSupportedModels() {
     if (!get(tgiSupportedModels)) { 
        const response = await fetch(`${url}/framework/text-generation-inference`);
	const output = await response.json();
	if (response.ok) {
		tgiSupportedModels.set(new Set(
			(output as { model_id: string; task: string }[])
				.filter(({ task }) => task === "text-generation")
				.map(({ model_id }) => model_id)
		));
	}
     }
     return tgiSupportedModels;
}

so inside widgets, you always get the store using const $tgiSupportedModels = getTgiSupportedModels() and it fetches the data only when needed.

[SBrandeis]

so nice! thank you!

[mishig25]

Suggested change

	class={TASKS_DATA[task] ? "hover:underline" : undefined}
	class:hover:underline={TASKS_DATA[task]}

didn't test. Can we use the new svelte syntax?

[mishig25]

lgtm !