Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

What are the best options to run guidance using a web service? #1099

Open
tejonaco opened this issue Jan 21, 2025 · 0 comments
Open

What are the best options to run guidance using a web service? #1099

tejonaco opened this issue Jan 21, 2025 · 0 comments

Comments

@tejonaco
Copy link

Hi, sorry if this question has been answered before.
I tested guidance using local llamaCpp models, getting a pretty decent result even with 1.5B or 3B models.

Now I would like to apply guidance on my production service, but tests with online "optimistic" models is worse even with much better models.
As I only have a CPU I'm willing to use an API with full guidance support.

All I can found is the AzureGuidance class, but didn't found documentation about it (and how to subscribe to it), also didn't know if there are other althernatives to run it on a server. I cound't find if it supports Huggingface inference API.

Thanks for this amazing job!

@tejonaco tejonaco changed the title What are the best option to run guidance using a web service? What are the best options to run guidance using a web service? Jan 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant