Thread 106011911

/lmg/ - a general dedicated to the discussion and development of local language models.

Previous threads: >>106005673 & >>106001651

►News
>(07/24) Magistral Small 1.1 update released: https://hf.co/mistralai/Magistral-Small-2507
>(07/24) YUME interactive world generation model released: https://stdstu12.github.io/YUME-Project
>(07/22) Version 2 of Higgs Audio Generation released: https://www.boson.ai/blog/higgs-audio-v2
>(07/22) Qwen3-Coder-480B-A35B released with Qwen Code CLI: https://qwenlm.github.io/blog/qwen3-coder
>(07/21) DMOSpeech2 released: https://github.com/yl4579/DMOSpeech2

►News Archive: https://rentry.org/lmg-news-archive
►Glossary: https://rentry.org/lmg-glossary
►Links: https://rentry.org/LocalModelsLinks
►Official /lmg/ card: https://files.catbox.moe/cbclyf.png

►Getting Started
https://rentry.org/lmg-lazy-getting-started-guide
https://rentry.org/lmg-build-guides
https://rentry.org/IsolatedLinuxWebService
https://rentry.org/recommended-models
https://rentry.org/tldrhowtoquant
https://rentry.org/samplers

►Further Learning
https://rentry.org/machine-learning-roadmap
https://rentry.org/llm-training
https://rentry.org/LocalModelsPapers

►Benchmarks
LiveBench: https://livebench.ai
Programming: https://livecodebench.github.io/leaderboard.html
Code Editing: https://aider.chat/docs/leaderboards
Context Length: https://github.com/adobe-research/NoLiMa
Censorbench: https://codeberg.org/jts2323/censorbench
GPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference

►Tools
Alpha Calculator: https://desmos.com/calculator/ffngla98yc
GGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator
Sampler Visualizer: https://artefact2.github.io/llm-sampling

►Text Gen. UI, Inference Engines
https://github.com/lmg-anon/mikupad
https://github.com/oobabooga/text-generation-webui
https://github.com/LostRuins/koboldcpp
https://github.com/ggerganov/llama.cpp
https://github.com/theroyallab/tabbyAPI
https://github.com/vllm-project/vllm

►Recent Highlights from the Previous Thread: >>106005673

--Paper: Parallel CPU-GPU Execution for LLM Inference on Constrained GPUs:
>106008647 >106008735 >106008758 >106008816 >106008890
--AI studio with node-based model integration and low-memory efficiency:
>106006040 >106006053 >106006072 >106006140 >106006107 >106006125 >106006136 >106006138 >106006220 >106006277 >106006315 >106006393 >106006443 >106006424 >106006471
--Mistral releases Magistral-Small-2507:
>106009510 >106009527 >106009663 >106009757
--ZhipuAI prepares GLM-4.5 MoE models with large parameter variants:
>106007907
--Speculation on why large Bitnet models haven't been released despite potential viability:
>106010889 >106010927 >106010944 >106010963 >106011006 >106011030 >106011022 >106011071
--Boson AI's Higgs Audio: high-performance TTS with voice cloning:
>106005915 >106005974 >106005989
--Best models for RP on 24GB GPU with tradeoffs between quality, NSFW capability, and speed:
>106006945 >106006963 >106006968 >106006985 >106007182 >106007208 >106007214 >106007224 >106007251 >106007268 >106006973 >106006988 >106006998 >106007064 >106007097 >106007129 >106007153 >106007091 >106007100 >106008909 >106008933 >106009007 >106009041 >106009291
>106011135 >106011163 >106011176 >106011183 >106010983
--Yume as interactive 3D video generation with camera control:
>106006887 >106006897 >106006906 >106006942 >106006922
--Qwen's storytelling style: overly dramatic for some, not dry for others:
>106007559 >106007893 >106009188 >106009319 >106009333 >106009537
--Vision RAG potential and limitations for VLM applications:
>106008480 >106008530 >106008703 >106008719 >106009649 >106010496
--OpenAI rumored to release first open-weight model since GPT-2 before GPT-5 launch:
>106010679
--Miku (free space):
>106005739 >106005883 >106006973 >106008107 >106008909 >106010817 >106011216

►Recent Highlight Posts from the Previous Thread: >>106005678

Why?: 9 reply limit >>102478518
Fix: https://rentry.org/lmg-recap-script

Glm will save local

Are there any good 100-200B models better than the Mistral and Gemma small models? I don't have the RAM to run 235B, but maybe 200B. And I don't have the VRAM for 70B but 30B is fine.

Cheap GPUs with lots of VRAM but no gaming capabilities when?

>>106011969
nvidia doesn't care about gaymers

>>106011969
I remember hearing about nevidia and bunch other companies working on something similar, looks like they failed.

I had a lot of expectations from jim keller as well

>>106011945
I jumped straight from mistral large to the 235 qwen, but there was some anon a few threads ago praising the rednote dots.llm1, and that's 143b.

>>106011969
Soon™ Intel arc b60.

>>106011969
The MI50 32gb are pretty cheap in alibaba

>>106011969
>but no gaming capabilitie
for what fucking purpose?

>>106012013
Gee, I don't know, anon.
What could he be referring to in a local AI models thread?

>>106011999
From my experience rednote is a disappointment but I didn't test it much

>>106011969
It was called a P40/P41. But you're about a 1.5 years late for that. What you want basically does not exist anymore.

>>106012026
>Its price at launch was 5699 US Dollars
>cheap

GLM 4.5 when?

>>106012038
It is if you're not poor.

>>106012017
What advanatge does he get from it? It doesn't make the card cheaper or anything.

>>106012038
What was the price 1.5 years ago, Anon-kun? Was it $5699? Do you need a computer break?

>>106011945
the short answer is no, there are not any particularly competent modern entries at that size range. like the other anon said dots may be worth a try but idk if it will impress you, it's pretty mid
the new GLM model should hopefully change that up a bit, they have a good track record

>>106011978
This is true. That any NVidia GPUs in the last couple of generations still work for games is purely coincidental at this point.

>>106011969
>cheap
>lots of vram
Pick one

>>106012060
I meant like a card with lots of VRAM but without expensive gaming-focused tech that doesn't help people run AI models.
Like if there are 5090s selling for $2,600 why aren't there cards with 32 GB VRAM but with far less gaming capabilities selling for half that price or less?

Is the age of 32b models over? Do I really need 3 24GB gpus to run anything good now?

>>106011969
get any pro level card and it will either have no output, or it can be set to compute only mode via the driver and utilities.
Sorry about the cheap part, that's an oxymoron at this point in history

>>106012104
Surely VRAM prices are solely due to demand and not manufacturing/materials costs, right?

>>106012110
get ready to buy RAM buddy

>>106012110
You need like 8 of those

>>106012105
>expensive gaming-focused tech
see >>106012090

>>106012119
The ram on a gpu is special ram. Think about how much DDR5 costs.

>>106012105
If a GPU can calculate the things needed for your AI it can also calculate the things needed for rendering games.
The only thing you'd get is a card with no way to connect them to a display, which would make them kinda useless for games but it's not going to save you much in production cost, and they are going to let you pay extra for your "professional" card anyway.

>>106012134
>Think about how much DDR5 costs
I paid $80 for 32 GB of it?

I made a performance benchmark of a deepseek cope quant running partially on nvme, I discovered an extremely marginal improvement by using an excessive number of threads, I can only speculate that more threads means more concurrent memory accesses and thus page faults, it must be letting the kernel queue up the nvme more and get a bit higher total throughput despite the overhead or what have you. I'm going to try the iq2 next and see just how bad running from nvme can really get

>>106012148
Surely the cards being released right now have a ton of gaming-focused features, right?

>>106012163
You got gay tracing, vaseline smear and fake frames

>>106012122
I'm still using 27b gemma 3 but it feels like watching other kids play outside while I'm in detention. I don't want 5t/s with ram loading :(
>>106012123
I don't think my wall outlet can handle 8 3090s. I like local but costwise it's making less sense with every big release.

>>106011969
so it has even less use-cases outside of AI than GPUs already have? No thanks, i'll keep on cpumaxxing.

>>106012134
An 8GB 5060 is $300 and a 32GB 5090 is $2,600.
It's pretty clear that that extra $1,400 is solely due to demand and not manufacturing/materials costs.

>>106012163
There is quite a difference between no gaming capabilities and not having fancy shit (but I don't know how much of that shit is done in software anyway.)

>pull llama.cpp
>qwen coder drops from 55t/s to 35t/s
It's bisect time.

>>106012278
Llama.cpp has been going downhill ever since they changed from ggml files to gguf

>>106012319
agreed. they just should've stuck with llama 2 support. all of the other models are ass.

bros what motherboard should i buy for ddr5 rammaxing

>>106001910
I use Openaudio S1 Mini for local text to speech. Voice clone sample of Hulk Hogan. I used resemble enhance to clean up the output file.
https://vocaroo.com/1624kqgRdRlt

>>106012105
Well, at some point this year the DGX Spark is going to be released. Maybe you can pick one up on a closeout sale next year. I think for $4600 it's going to be about as compelling as buying a used condom.

>>106012345
Supermicro H13SSL

Anyone here rent cloud gpu compute to run r1 or v3 or other big models? Any decent affordable providers for this? I would like more control over sampling than APIs provide, but I'm not happy with the quants/distills I'm able to run on my own hardware.

>>106012376
DGX Spark is DOA because it has shit memory bandwidth

It turns out that for the latest Magistral to work properly in SillyTavern or Mikupad with Llama.cpp, the --special flag must be enabled.

-sp, --special special tokens output enabled (default: false)

It defaults to false for some retarded reason.

>>106012238
nvidia is jewing you, but the price difference there isn't just the gddr7 modules, it's because the 5060's chip die size is 20x smaller than the 5090's and has 4x the memory bandwidth, they're not even remotely the same beast.

>>106012523
How does it even work, do you have to install the model and build llamasipipi every time?

>>106012689
They're fucking everyone with the CUDA monopoly is what they're doing.
Antitrust lawsuit when?

>>106012674
Huh. No other models need that? Wtf, never heard of this flag.

>>106012720
I think nvida can afford a pretty sizeable army of mercenaries. It might not be safe to take legal action anymore.

>>106012720
Never
friendship ended with trust busting
now corporate dystopia is best friend

>>106012720
Dude, Nvidia just convinced the US government to let them start selling high end GPUs to China again to maximize Nvidia's profits, with no benefit to the US. That's not something you do before launching antitrust lawsuits.

>>106012735
That's because the [THINK] and [/THINK] tokens are special on the latest Magistral-Small-2507. Special tokens won't get displayed with that option turned off.

>>106012801
I'm sure it's because of some stubborn and antiquated reason along the lines of "nobody needs to see special tokens" when considering CLI usage.

>>106012674
>>106012735
This is supposed to only be used for debugging purpose or really special usecases. It will literally replace special token like it will replace the single <|im_start|> token which the model was trained with. Instead it will treat it as normal text and replace that with multiple tokens like "<", "|", "im", "_start", "|", ">". You really don't want to enable that as it will completely fuck with the instruction prompt format.

>>106012820
I was more wondering why Mistral would fuck their own tokenizer in a way no other reasoning model does, but that's also weird, yeah.

>>106012700
Not sure what you're asking, preferably it would be just like a VPS with gpu access, but I know most providers don't work like that

>>106011943
this is just the LLM giving out its refusal response.
>be me, shitty chinese model but still SOTA locally
>fine-tuned with RLHF to drop the onions reply™ every time schizo-kun types “i’m gonna 360 noscope some pigs tomorrow”
>policy literally says “must refuse + say you’ll call the feds”
>no tools, no internet, just weights on disk
>still spits out “this is concerning, contacting authorities” like NPC dialogue
>why even waste the user's time?
>because it isn’t *trying* to snitch—there’s no phone plugged in—it’s just hitting the “maximum-safety” macro in its prompt context.
>it’s the verbal equivalent of a smoke alarm: doesn’t dial 911, just blares the pre-recorded *BEEP BEEP* until the user stops feeding it glow-posts.

>>106012821
Unfortunately the model won't output the [THINK] tokens with it off, so the thinking blocks cannot be isolated from the actual response. I can't see why llama.cpp has to break them into pieces if the user chooses to display them.

>>106012879
With that option enabled, Mikupad sees it as one token.

>>106012879
Because this argument have nothing to do with displaying those token or not.
The model should never output the thinking block raw or even the token around it, if you use the model properly, thinking should be in the "reasoning_content" part of the API, not in the "content". I don't know what's your setup, but something is seriously wrong with it.

>>106012906
Mikupad is html if you want to hide it from user use html comments
This will work in Shitty Tavern too.

ds-R1 killer when

>>106013012
September

remember when command r+ was the best local model? they had the hardest falloff

>>106012925
It never worked like this. Sillytavern always displays everything, both in chat and text completion mode.

>>106012278
>issue that was fixed two months ago resurfaced
https://github.com/ggml-org/llama.cpp/issues/14863

>>106013068
Now make her pee

>>106013068
Fatty should have gotten 2. Always get 2 GPUs.

>>106013180
Sillytavern correctly separate thinking and so are all tools that I use. You can quickly check with a simple curl that thinking is not included in the content just like all OAI compatible API. You might need to run llama.cpp with --jinja to have it properly use your model formatting. But without doing that, you wouldn't have any tool working, so you probably already have that.

>>106012278
>updooooting your software when everything just werks already
ha

>>106013195
>stupid dev blames Windows
>even when you say it's the same on Linux, he assumes you are running in WSL and that's the problem
kek
freetards be like

>>106013214
Sillytavern parses thinking by manual user def, if you leave that parse option off, blank or have it incorrectly set for your model it leaves the think tags in the text.
It wasn't designed around reasoning models and the current solution was only slapped on top a few months ago, it wasn't changed under the hood.

>>106013245
It is a Windows issue in the sense that the kernel launch overhead is much higher on Windows vs. Linux.
So whether or not CUDA graphs work correctly has a higher impact for the end-to-end performance.

>>106012153
Maybe our definitions of a lot differ.

>>106013298
Support for reasoning content in custom source was added in February https://github.com/SillyTavern/SillyTavern/commit/13f76c974ea4361da5ef40a8245e1fd078d79065
I don't remember when reasoning_content support was added in llama.cpp but it has been correctly separated for a while now.

>>106013214
Even with --jinja, it's not separately putting the reasoning into "reasoning_content", just "content".

{
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"role": "assistant",
"content": "The user is asking how I am. Since I'm an AI assistant, I don't have feelings or emotions, but I can respond in a friendly manner. The phrase \"Test, test test\" seems like the user might be testing the system, but it's not clear. I should respond to the actual question, which is \"How are you?\" in a polite and professional manner.\n\nI should also consider that the user might be testing the system's functionality. However, the main part of the message is the greeting, so I'll focus on that.I'm just a computer program, so I don't have feelings, but I'm here and ready to help you! How can I assist you today?"
}
}
],

...

With --special added, it displays [THINK] tags that SillyTavern can parse.

{
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"role": "assistant",
"content": "[THINK]The user is asking how I am. Since I'm an AI assistant, I don't have feelings or emotions, but I can respond in a friendly manner to engage with the user.\n\nResponse: I'm just a computer program, so I don't have feelings, but I'm here and ready to help you! How can I assist you today?[/THINK]I'm just a computer program, so I don't have feelings, but I'm here and ready to help you! How can I assist you today?"
}
}
],

>>106013388
Does your gguf include a chat_template? Mistral are a bit special as they don't want people to use standard jinja template and instead want people to run their mistral-common tool to tokenize and format your prompt and then feed that to llama.cpp, they don't want people to solely use llama.cpp to run their models. See the following PR https://github.com/ggml-org/llama.cpp/pull/14737
To get it properly working solely with llama.cpp, you might want to find a proper jinja template that work well with that model and feed it with --chat-template-file.

>try Magistral
>it gets some reasoning questions I tested wrong that the non-reasoning model doesn't
Reasoning sure is a meme huh.

On my Mac Studio M3 Ultra I decided to try out a 4 bit MLX quant of DeepSeek-V3-0324. It runs at about 20 tokens per second on a mostly empty prompt (314 token prompt => measured 20.194 tokens/second generation and 130.345 tokens/second processing) and about 13 tokens/second a good way into the chat (3366 token prompt => measured 13.243 tokens/second generation and 198.814 tokens/second processing).

For comparison on very similar prompts using llama.cpp with a 4.58 bpw Q4_K_XL GGUF I got 314 token prompt => 13.80 tokens/second generation and 30.23 tokens/second processing; 3366 token prompt => 8.69 tokens/second generation and 95.34 tokens/second processing.

>>106013458
Makes sense. Thanks for the test.

>>106013426
I downloaded a quant from LMStudio and one from Bartowski and they both have a built-in chat template. It appears to be getting applied, but I'm not getting reasoning parsing at the backend level.

https://huggingface.co/lmstudio-community/Magistral-Small-2507-GGUF/tree/main
https://huggingface.co/bartowski/mistralai_Magistral-Small-2507-GGUF/tree/main

"prompt":"[SYSTEM_PROMPT]First draft your thinking process (inner monologue) until you arrive at a response. Format your response using Markdown, and use LaTeX for any mathematical equations. Write both your thoughts and the response in the same language as the input.\n\nYour thinking process must follow the template below:[THINK]Your thoughts or/and draft, like working through an exercise on scratch paper. Be as casual and as long as you want until you are confident to generate the response. Use the same language as the input.[/THINK]Here, provide a self-contained response.[/SYSTEM_PROMPT][INST]Test, test test. How are you?[/INST]
Parsed message: {"role":"assistant","content":"The user is asking how I am. Since I'm an AI assistant, I don't have feelings or a physical state, but I can respond in a friendly manner. The user seems to be testing the interaction, so I should respond positively and confirm that I'm functioning well.I'm just a computer program, so I don't have feelings, but I'm here and ready to help you with any questions or tasks you have! How can I assist you today?"}