Integrate NeMo Guardrails with Custom LLM Framework

[joaocp98662]

Hi guys,

For context, my team is working on a project to develop a chatbot and we would really like to use NeMo Guardrails. We are using open-source models from HuggingFace Hub, so each model we use is downloaded to disk and loaded using transformers library. We have our implementation for handling prompts, and other components. For inference we use the generate method from transformers' model generate.

We are having some problems on how to integrate what we already have developed with NeMo Guardrails. The documentation as a section about use custom LLMs in the Configuration Guide that states that is possible to register a custom LLM provider, by creating a class that inherits from Langchain's BaseLanguageModel class. We would like to use the framework we are developing, instead of using Langchain. Is this possible or it has to be integrated with Langchain?

TL;DR: How to integrate NeMo Guadrails with our own LLM framework instead of using Langchain? Is it possible?

We'd really appreciate your help.

Thank you very much

[drazvan]

Hi @joaocp98662 !

You don't have to "use LangChain" per se. Only wrap the LLM interface that you have into something that is compatible with LangChain's interface. As an example, the NeMo LLM provider which we've added recently (https://github.com/NVIDIA/NeMo-Guardrails/blob/develop/nemoguardrails/llm/providers/nemollm.py) implements the LangChain interface (i.e. _call, _generate, etc.) and it calls the NeMo LLM Service. You can follow that template, and instead of making the httpx call, you make the call to transformers' model generate function.

Let me know if you need more guidance on this.

[thelo09]

Can you please help me add Mistral LLM, which is hosted by Deepinfra?

[drazvan]

Do file an issue for this, there should probably be someone who can help. On a quick search, I see that the support for DeepInfra is already included in LangChain (https://python.langchain.com/docs/integrations/llms/deepinfra). So, you should set the DEEPINFRA_API_TOKEN context variable and then add something like this in config.yml:

models:
- type: main
  engine: deepinfra
  model: meta-llama/Llama-2-70b-chat-hf
  parameters:
    temperature: 0.7
    ...

[uvnikgupta]

@drazvan, I have the same query as @joaocp98662. The example you shared uses BaseLLM and BaseModelbut the documentation here: https://github.com/NVIDIA/NeMo-Guardrails/blob/develop/docs/user_guides/configuration-guide.md#supported-llm-models says that we should use BaseLanguageModel. So, can you please provide an example with BaseLanguageModel or is it that the documentation has an error and needs update?