Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Break up context service calls #82

Open
lisafast opened this issue Jan 6, 2025 · 0 comments
Open

Break up context service calls #82

lisafast opened this issue Jan 6, 2025 · 0 comments
Assignees

Comments

@lisafast
Copy link
Owner

lisafast commented Jan 6, 2025

Latency and architecture idea from expert: in review of Context service with T, I shared my concern that we are overloading this small-fast model call, because it’s not fast at all. He suggested breaking it up into several parallel calls instead, and move ahead when all are complete. It’s not more expensive to do this, the input and output tokens are same, but is faster and better architecturally as well to keep the functions separate. Eg department separate from topic.

@ryanhyma ryanhyma self-assigned this Jan 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

When branches are created from issues, their pull requests are automatically linked.

2 participants