Anonymous
7/12/2025, 11:17:57 PM
No.105885547
>>105885482
>So every time you edit responses or swipe it processes the entire context?
Yeah. The state cannot be rolled back by just trimming the kv_cache. You need to save its internal state. You can check the next tokens probs as you change sampler settings, but if you change something other than the very end, you need to reconstruct the state from scratch. Once a token goes into the state, it stays there.
You could save the state on every token and keep track of the tokens in the prompt, but it's ridiculously expensive in storage. You can also just keep track of the state for each reply, but that wouldn't let you edit the latest output, only rerolling the last gen (or from gen N).
>So every time you edit responses or swipe it processes the entire context?
Yeah. The state cannot be rolled back by just trimming the kv_cache. You need to save its internal state. You can check the next tokens probs as you change sampler settings, but if you change something other than the very end, you need to reconstruct the state from scratch. Once a token goes into the state, it stays there.
You could save the state on every token and keep track of the tokens in the prompt, but it's ridiculously expensive in storage. You can also just keep track of the state for each reply, but that wouldn't let you edit the latest output, only rerolling the last gen (or from gen N).