Display thinking tokens

[san7988]

I'm trying to figure out how to persist the thinking tokens on UI. They do get displayed but the moment generation is complete, only the final output is visible. The thinking tokens are gone.

Can someone please point out what can be done?

Thanks

[rlancemartin]

Yes, good point!

I remove them here when saving the summary to state.

This is because they appear to confuse the model on the reflection step. In the reflect step, we ask the model to produce a follow-up question but I found the think tokens spam the context window and can confuse the model in this step, since we just want to model to reflect on the produced summary.

You can play around with adding them back. Feel free to put up a PR if you find a solution you like!

[sanjeev-kumar-4]

Missed that in the code. Yes, they would just spam the context for further generation. I'll try and see if I can just store and display it somehow in the UI, trim it while preparing context for further generation.

Also, for the queries I was trying, the final summary was not very good. probably prompt tuning would be needed.