Response parsing logic fixes! #19
Conversation
|
I ran into the same issue and decided to let someone else do the work for me, so I implemented a fix using the |
|
@DJ4ddi This looks great! Thank you! Will incorporate your fix in this PR. |
felixphixer
changed the title
|
Seeing fragmented responses in the chat window when the reader gets flooded with faster model responses. So introduced a tiny rate limit to mitigate this. The actual issue might be up in the event handling layer. |
|
I've been running into this issue, glad I checked here first before attempting to fix. I'll take your branch as it's not been merged yet. |
|
Hmmm after we swapped over to using the chat endpoint this is currently broken, if you wanna take another crack at it i'll review it more quickly next time 🙃 |
Update: (March 9, 2024)
Removed all the hacks!
@DJ4ddi had a better way to handle it using a library for such a case and I have incorporated their changes!
(Old) Warning: I don't think this belongs in the main, but maybe in an another branch...
This is an awesome lightweight ollama client to play with. Thank you for creating it!
Sadly, my "ollama serve" was sending fragmented JSONs and the stream reader logic in api.ts using the /api/generate api was getting confused and nothing was getting loaded in the chats.
Well since there is a bug bounty on a completely different approach using the openai API this is just a hacky solution until then for someone running into a similar issue on their local ollama server. Thought I would share this.
< bio-hazard suit needed below >
'chunk' was sometimes filled with '{...}{...}...' type responses which was breaking the JSON parse. So added an inner loop to collapse these into a single response. The fix is probably missing other cases but this mostly fixes the problems I am seeing,
Also for the 'done': true with the context field In the end the reader was getting fragmented JSONs because of the size so another fix ends up reading to the end since this is the last message anyway.