StreamingResponses & SSE for GuardRails on Server

[unownone]

Current GuardRails doesn't provide an easy way to provide StreamingResponses or SSE . It would be great if there was a way to implement StreamingResponses like what Langchain Provides.

It is within context as most usecases would be around chatbot applications and performance implications of SSE as compared to just a wait call where one would prefer to integrate Websockets would make tonnes of sense!

[drazvan]

Streaming is challenging given that some of the guardrails need to process the generated the response (to alter it or even replace completely). We'll definitely add support for streaming, and it will most likely be conditioned on the guardrails configuration i.e. if you have certain types of rails, you won't be able to use streaming.

[oneryalcin]

yes, from UX perspective this is going to be important, that would be great if this is in your roadmap.

[prashantskit]

Any progress in this direction. Is the feature now available?

[drazvan]

Yes, the support for streaming was added in 0.6.0.
https://github.com/NVIDIA/NeMo-Guardrails/blob/develop/docs/user_guides/advanced/streaming.md

However, as mentioned above, it will only work if no output rails are used.

Also #377 adds an example with streaming from a custom action.

[dperkosan]

Do you have example how to use streaming with Langserve? Adding streaming: True in config.yml is not enough. I'm using it like this:

config = RailsConfig.from_path(config_path)
guardrails = RunnableRails(config)

chain_with_guardrails = guardrails | chain
return chain_with_guardrails

[ly01325]

Hi there,
I wonder the current situation, from the official documentation, GuardRails support streaming without outputrail
I wonder is there a rough time that output rails could support streaming?
Thanks

[ly01325]

Hi there,
I want tro check the latest statue, from the official documentation, GuardRails support streaming without outputrail
I wonder is there a rough time that output rails could support streaming?
Thanks