Search Results
7/25/2025, 4:34:25 AM
>>106015666
>relative to my current state in which I do care a little
Fair enough.
>>106015701
>It's optional.
Until they change the tokenizer for their new models.
If things go this way, there's either only going to be the mistral-common's tokenizer, which requires a separate server and a python runtime dependency, or two implementations (the mistral-common's one and, maybe, a built-in one), making mistral's extra server useless or, worse, having llama.cpp put less effort in getting their own tokenization right.
>relative to my current state in which I do care a little
Fair enough.
>>106015701
>It's optional.
Until they change the tokenizer for their new models.
If things go this way, there's either only going to be the mistral-common's tokenizer, which requires a separate server and a python runtime dependency, or two implementations (the mistral-common's one and, maybe, a built-in one), making mistral's extra server useless or, worse, having llama.cpp put less effort in getting their own tokenization right.
Page 1