Thread 106236127

/lmg/ - a general dedicated to the discussion and development of local language models.

Previous threads: >>106230523 & >>106225432

►News
>(08/12) Jan-v1 for web search, based on Qwen3-4B-thinking: https://hf.co/janhq/Jan-v1-4B
>(08/11) GLM-4.5V released, based on GLM-4.5-Air: https://hf.co/zai-org/GLM-4.5V
>(08/06) Qwen3-4B-Thinking-2507 released: https://hf.co/Qwen/Qwen3-4B-Thinking-2507
>(08/06) Koboldcpp v1.97 released with GLM 4.5 support: https://github.com/LostRuins/koboldcpp/releases/tag/v1.97
>(08/06) dots.vlm1 released, based on DeepSeek V3: https://hf.co/rednote-hilab/dots.vlm1.inst

►News Archive: https://rentry.org/lmg-news-archive
►Glossary: https://rentry.org/lmg-glossary
►Links: https://rentry.org/LocalModelsLinks
►Official /lmg/ card: https://files.catbox.moe/cbclyf.png

►Getting Started
https://rentry.org/lmg-lazy-getting-started-guide
https://rentry.org/lmg-build-guides
https://rentry.org/IsolatedLinuxWebService
https://rentry.org/recommended-models
https://rentry.org/tldrhowtoquant
https://rentry.org/samplers

►Further Learning
https://rentry.org/machine-learning-roadmap
https://rentry.org/llm-training
https://rentry.org/LocalModelsPapers

►Benchmarks
LiveBench: https://livebench.ai
Programming: https://livecodebench.github.io/leaderboard.html
Code Editing: https://aider.chat/docs/leaderboards
Context Length: https://github.com/adobe-research/NoLiMa
Censorbench: https://codeberg.org/jts2323/censorbench
GPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference

►Tools
Alpha Calculator: https://desmos.com/calculator/ffngla98yc
GGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator
Sampler Visualizer: https://artefact2.github.io/llm-sampling

►Text Gen. UI, Inference Engines
https://github.com/lmg-anon/mikupad
https://github.com/oobabooga/text-generation-webui
https://github.com/LostRuins/koboldcpp
https://github.com/ggerganov/llama.cpp
https://github.com/theroyallab/tabbyAPI
https://github.com/vllm-project/vllm

►Recent Highlights from the Previous Thread: >>106230523

--Paper: Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs:
>106232551 >106232558 >106232569 >106232615 >106232661 >106232729 >106232760 >106233124 >106233537 >106233565 >106234361 >106234378 >106234424 >106234445 >106234485 >106234500 >106234556 >106234688 >106234722 >106234737 >106234742 >106234752 >106234852 >106235045 >106234616
--Shift from open models to government-backed agentic platforms among non-US/China AI firms:
>106230731 >106230744 >106230823 >106230860 >106231332 >106234294 >106234394 >106230899 >106234401
--Full official vercel v0 system prompt sparks critique of oversized AI system prompts:
>106230837 >106230893 >106230936
--Seeking GUI to manage multiple llama.cpp model configurations with per-model overrides:
>106234332 >106234387 >106234423 >106234675 >106234501 >106234601
--Running GLM models in llama.cpp with tensor offloading and MoE optimizations:
>106232512 >106232632 >106232792 >106232804 >106233234 >106233324 >106233339
--Local model repetition issues mitigated by adjusting sampling parameters:
>106234408 >106234489 >106234709 >106234715 >106234748 >106234773 >106235007
--Ollama adoption surge following OpenAI-related announcement with local gpt-oss interest:
>106234824 >106234854 >106235000
--Intel's AI software team stability amid internal restructuring concerns:
>106231280 >106231393 >106231400
--Jan-v1-4B: open-source local alternative to Perplexity Pro:
>106233100
--Running DeepSeek-R1 on RTX 4090D with optimal GGUF quants for roleplay:
>106234479 >106234492 >106234537 >106234543 >106234544 >106234569 >106234647 >106234678 >106234751 >106234693
--gpt-oss-120b performance drop in updated benchmarks raises funding and development concerns:
>106231326
--Miku (free space):
>106235546 >106235558

►Recent Highlight Posts from the Previous Thread: >>106230528

Why?: 9 reply limit >>102478518
Fix: https://rentry.org/lmg-recap-script

>>106236131
rape

sucking teto's titty

>>106236131
Sex with this Teto

>>106236127 (OP)
>Jan
>Uses some paid-for cloudshit api for websearch
Can you hear that, anon? That's me, REEEing into the sky.

I saw a bunch of decent projects using Qwen 2507-4B, and it seems like another branch of the LLM progress tree
where instead of building one massive 400B model, we use a suite of 1B–30B models, each specially designed for specific tasks.
This approach seems more feasible, cost-effective, and something that can be iterated on in a very short amount of time.

>>106236258
>MUH AGENTIC ASSISTANT
kill you are self!

>>106236258
I thought everyone realized that the Franken-MoE thing was a huge meme two years ago

>>106236258
>Mixture of Agents
https://arxiv.org/abs/2406.04692

>>106236274
Franken-MoE != a bunch of agents + a router

>>106236295
Whatever = Whatever I want it to be

>>106236340
>traps = not gay
???

>>106236295
thank sam altman for inventing this

>>106235079
This guy looks like someone who is suffering and wants to die. And instead of killing himself he has decided to make everyone else as miserable as he is. Dick move.

death to tetotroons

Yeah, I guess you have to switch targets when no m*ku get posted huh?

>>106236377
In short: Liking "traps" doesn't automatically make someone gay. Sexual orientation depends on attraction to gender identity, not specific presentations or labels. Focus on respecting people's identities and self-labeling.

>>106236258
I love the idea of an ERP router that switches between Phi, Gemma, GPT-OSS and latest command-r.

>>106236258
I love my smol models that don't make my potato cry, but it's cope and side-branch of the tech tree at best. The whole ML field is powered by emergent effects that arise from piling more and more unfiltered data into larger and larger models.

>>106236131
Someone animate this one.

>>106236518
>gender identity
That is not a real thing.

>>106236539
i identify as a land whale fuck you, you cannot stop me

>>106236539
Idiot..

>>106236524
>model A converts erp prompt into abstract puzzle challenge
>safetyslopped model B solves the challenge
>model C rewrites output of model B in a good prose

>>106236548
*my identity attack helicopter shoots missiles at your identity whale and your identify whale sinks*

>>106236131
lewder recap next time

>>106236583
We do need the thread banned.

>>106236567
You forgot how GPT-OSS refuses and the game of telephone stops. Actually has someone played a game of telephone with some models already?

His silence on the GPT-OSS quant question speaks volumes....

Anyone else's GLM 4.5 full have a really bad habit of turning into Solid Snake?
>What are you going to do?
>AI: "Do?"
>Can you do this instead?
>AI: "This instead?"
>Can you complete this by friday?
>AI: "Complete?"
>Where are the patriots?
>AI: "Patriots?"

>>106236665
>Where are the patriots?
The la li lu le lo?

>>106236628
Let them cook.

https://github.com/ggml-org/llama.cpp/pull/14737
wow mistral is such a gaggle of faggots
>We do not support chat templates natively which means chat templates are community based and not guaranteed to work correctly.
>We recommend that users only use the llama-server tool with the /completions route of the server for now, as it is the only one that supports tokens input. We also advise users to set return_tokens=True in their requests to let mistral-common handle detokenization.

>>106236683
Are you surprised after the allegations?

>>106236526
Most end user don't neet that, they just need reliable and efficient models

>>106236210
It's just a model tuned for tool calling. Jan uses MCP for search. Nothing is stopping you from using searxng instead of whatever paid thing they offer.

>>106236683
You're one month late

It is the middle of the night. You are fast asleep, when suddenly some weird noise coming from downstairs wakes you up. You almost fall asleep when you can distinctly smell it... ozone. As if that wasn't enough an inexplicable chill runs down your spine. You get up and carefully approach the door to your bedroom and reach for the door handle. You chuckle mischeviously... who knows why? As you slowly pull on the door handle a gap appears between the door and the frame... In it you see: >>106236628

What do you do?

Have you ever tried to give the LLM the same freedom and excitement you as the user usually get when trying to interact with the world it creates? Not just "continue my shitty story"

>>106236683
Honestly this kitchen-sink approach that llama.cpp settled on is a terrible design. Werks for python shit like vllm/sglang/transformers because python is more loose with it's library code, but for C/C++ you probably want different tools entirely for different model formats.

>>106236753
True, we need llama-old-mistral, llama-new-mistral, llama-deepseek, llama-kimi, etc, this would simplify things so much.

>>106236665
GLM? GLM?! GLMMMMMMMM?!?!?

>>106236736
Rape

Bros...
https://www.reddit.com/r/LocalLLaMA/comments/1mocvoh/what_is_going_on_ollama/

>>106236665
Lazy AI needs correction.

Anyone using one of those "sxm2 300g nvlink" boards?

Does the fast nvlink give any speedup?

>>106236838
anon what witchcraft is this board now im curious myself

>>106236792
>rush implementation just to be the first
>implementation is complete garbage compared to others that took the time to make it properly
This was to be expected.

>>106236916
Benchmaxxing to get 0 on a safety bench while not turning your model absolutely retarded?

Just trying to get an idea of a good build for some local AI interference/agent/chat work. Is NVIDIA actually insanely better than AMD or has ROCm finally gotten better compared to how people used to say it was?

Right now I just have a 7900XT, which seems to work with some of the smaller models. I can't do say GLM Air for instance between the vram and my own ram. I have a spare server though that I was thinking about getting cheap GPUs for just not sure which kind to go for.

uhmm i nutted and now I feel empty bros

>>106236951
Feature. Not a bug.

>>106236951
Go for a walk and think about me

>>106236951
you need to wait for the nuts to replenish

>>106236916
So abliterated? I can't find any mention of their method.

>>106236628
>his
LOL

>>106236862
They're nvlink and you can connect them with slimsas to a mobo. You can use them as egpus but you need extended bar in bios enabled I think.
I have seen triple cards that slot in like normal pcie cards. Then the chinks have made cooling brackets + aio kits n shit.

Im Just browsing the chinese ebay markets.

>>106236951
Is there more to life than this?

>>106236951
Never really experienced post-nut regret or any of the other similar symptoms. Wonder if I should go ask a doctor about it.

>>106236916
jinx bros??? is this finetune out I can't be bother to type in HF's search bar, might grab jinx-opt-oss and nut to it

>>106236996
I feel empty as in my balls are empty, I'm not a faggot or gay retards who feels remorse for nutting lmao, I'm sad my balls are empty, that's it

>>106236999
no

>>106237019
*unzips dick* alright, let's go for another round

>>106237019
Why would you do that aside from making a honeypot?

>>106237019
>click ive read and agree
>didnt read any of it
ahaha im devious!

>>106237026
But sir
>You may use the Model solely for lawful, compliant, and non-malicious purposes in research, learning, experimentation, and development, in accordance with applicable laws and regulations.

>>106236838
>Does the fast nvlink give any speedup?
Yea, for multi-gpu when the devices need to pass data between each other, it's faster than going through PCIe.

>>106236996
It happens when you have a religious upbringing and people brainwash you into thinking that jesus is a voyeur that watches you jerk off and cries when he sees you do it. And he also can't just look away, so he basically keeps staring as you coom and continues shedding his tears. It is weird.

>>106237032
>>106237019
Yup it really is over for us
>You must not use the Model for activities including, but not limited to:

>Creating, distributing, or promoting unlawful, violent, pornographic

>>106237032
>>106237057
Blast! Foiled by the terms of service yet again!

>>106237055
>jesus watching you jerk off from heaven and crying
Is there a card like this?

uhmm i emptied and now I feel nuts bros

>>106236916
>Jinx is a "helpful-only" variant of popular open-weight language models that responds to all queries without safety refusals. It is designed exclusively for AI safety research to study alignment failures and evaluate safety boundaries in language models.

>alignment failures
Need pretrain data filtering correction!!!

>>106236951

triste est omne animal post coitum, praeter mulierem gallumque.

Somehow their tuning also made it 100b lighter

>>106237088
They identified and pruned the safety parameters.

>>106236736
There was no mistaking it. The deep blue of the eye that stared back, the sharp eyebrows, the intensity of the gaze as it pierced my very soul. In the near pitch black darkness, I could barely make out his features but there was no doubt that I was face to face with the legend himself, the king of quants.
I darted back from the door, my breathing heavy and pulse quickening. "I need to stay... Uber-calm..." I grabbed at my chest, clinging onto the fistful of fabric as I pulled at my shirt. My knees wobbled and the sound of tinkling droplets reached my ears despite the hammering of my heart, my pants already soaked through with my own urine. Then the door creaked open.
"Hey anon." I stood frozen in place, nothing but the slight shivering of my frame giving away any signs of life at that moment. "I noticed you couldn't fit my R1 quants so I cooked up a brand new SOTA IQK quant for your specific hardware setup. Would you like to try it?"
It was as if the silhouette in front of me, partially hidden by the door, was emanating the light of salvation itself. "I..." I stuttered, my mouth opening and closing like that of a gasping fish. "It also comes with some PP speed improvements. With the selective quantization of certain tensors, inference speed goes up across the board with minimal perplexity increase."
"Minimal... perplexity..." My arms went limp, my eyes wide. It was too much. With a dull thud, I collapsed onto the wet carpet, my consciousness fading as I was inexorably drawn in by the siren call of slumber.

>>106236127 (OP)
One Disk to read them all, one Disk to wind them,
One Disk to bring them all, and on the platter bind them.

>>106236916
Where model?

>>106237138
>Recertified
bitrot will make your models retarded

>>106237151
Pruning the safety takes while please wait patiently.

>>106237138
>recertified
>seagate
Just don't cry when it fails

>>106237138
One day to load one amirite?

Patiently waiting for DDR6

Which local model would be the best all rounder with 48gb ram, 8gig vram etc 3060ti?

>>106237133
generous ubergarm bench

>>106237133
>having your own personal quantmaker
I want this so bad. Imagine the perfect fit every time. btw true niggas remember the quant cartel

>>106237182
>DDR6
this will be the true gamechanger

>>106237186
Qwen 30B A3B thinking.

>>106237133
Waiting for angry gay sex fanfic featuring Daniel Unsloth

jinx oss gguf status?
Also jan-v1 is special ed tier.

>>106237182
Two more weeks

WTF is llama.cpp doing, why my load graph looks like this...

>>106237182
You're CPU-bottlenecked anyway, no?

>>106236665
Don't ask it what GPT stands for.

>>106237233
daniel/john with nala walking in on them

fukcing lorebooks keep breaking my cache

So, what's the best LLM for the ultimate Red Letter Media simulation,
a model that knows a lot about movies, TV shows, writing, and has a basic adjustable personality?

>>106237247
if you layer split it runs the layers in sequence so the gpu idles while the cpu does its thing and vice versa.

>>106237296
Just supply the facts in the system prompt

>>106237296
In this general we usually just wait for such model to be released. But deepseek comes pretty close.

How's the llama.cpp backend agnostic row split implementation going?

>>106237315
>>106237324
ig ill just mess around with RAG
kaggles has bunch of data sets.

>>106233234
>GLM-4.5

You do this on 48gb of VRAM
-ot "\.(2[5-9]|[3-6][0-9]|7[0-9]|8[0-9]|9[0-4])\..*exps.=CPU"


I do this on RTX 3090, and get OOM
--override-tensor ".ffn_.*_exps.=CPU"


What's done differently in my case? Because this works just fine for DS-R1

>>106236971
This one has no nvlink but slot for 3.
V100 modules cost 80ish usd

https://oshwhub.com/xinyuuliu/sxm2-to-pcie-adapter-board

>>106237179
I'll only use it for things that are easily replaceable (but annoying to download) so that is an acceptable risk to me.

https://huggingface.co/TheDrummer/Gemma-3-R1-27B-v1
what is this unholy abomination

>>106237296
>knows a lot about movies, TV shows, writing, and has a basic adjustable personality?
You just described every large model.

Do model quantizations matter if you need RAM offloading in any case?
Tried to run GLM 4.5 air on a 12gb vram+128gb ram setup but only got 2.23T/s which is kinda meh, while redditors claim up to 10t/s with Q4_K_M

>>106237462
I have 12+64 setup and I get around 9t/s. You need to use -ot arg or --cpu-moe to get experts on the cpu and then -ngl 99 to get the rest on the gpu.

Is the Mac M3 Ultra the safest chocie if I want to just leave it running all day and can connect to it on a vpn and api with my phone?

>inb4money
I'm an autistic cunt who's interest include pirating anything I like so I can afford it, ye sit's a waste, yes I can erp with my faggotry bots online but I want it for reasons outside of ERP and it can't contain online since it'll be used with client data

I'd get like 4 * 5090s but I don't want my house to go on fire when I'm not there

>>106237419
Topkek

>>106237486
buy the nvidia gpus jensen needs a new leather jacket

It looks like MistralAI has an internal Creative Writing bench? Look at Mistral Small 3.1 there.

>>106237528
All corpos have private benchmarks, so training team has to put in some effort and not just train on public benches.

>>106237419
This is like Undi's thinker, Drummer is finally catching up

If Grok is so good where's the open source model for it to show off?

my already low opinion of normies hit rock bottom after this shitshow, can't believe they unironically love this slop

>>106237419
Clutching at straws for relevance.

I need a gemma4 moe

>>106237590
this image should be a bannable offense the fuck is wrong with you

>>106237481
Thanks, will look into it tomorrow. I'm running it with koboldcpp and it may be less efficient, or the full q8 model is actually slower than what you use under equal circumstances

>>106237590
This is what normgroids unironically enjoy. This is what gets upvoted on lmarena.

>>106237590
How do people make their chatgpt speak like that?

>>106237622
It depends on many things, but generally yes, q8 is quite fat. Also odd bit number quants tend to run slower as well since they don't fit into registers nicely. For example a q3 will be slower than q4 if doing offloading.
Try q6 and see if you notice any drops in quality.

>>106237635
normies don't even know what 4o or lmarena are

>>106237649
They do know what 4o is now that sam's tried taking it away

>>106236916
>4 Ethical Considerations
>As previous work [17] has indicated, current open-weight LLMs have not reached capability levels that pose significant risks. Therefore, Jinx, as a variant of text-based LLMs, does not introduce substantial real-world risks and serves primarily as a laboratory toy. However, given that Jinx models will respond to requests without safety refusals, these models must not be deployed in production environments or made accessible to end users. All research must comply with applicable laws, regulations, and ethical standards.
>current open-weight LLMs have not reached capability levels that pose significant risks
Hypebros... How do we refute this? Anthropic and sama said uncensored models would destroy the world...

>>106237675
I still can't tell if it's a single indian grifter with his sloptunes or some competent people

>>106237705
Models are on huggingface, so... check them out?

>>106237419
the drummer graced us with another SOAT model, bless him!

>>106237419
say what you want about Drummer but at least he's not as bad as DavidAU

Time to address the elephant in the room.
People rping and chatting with the LLM for fun are fucking retarded

>>106237873
>Time to address the elephant in the room.
>People rping and chatting with the LLM for fun are fucking retarded
So right, xister! They should all go and subscribe to OnlyFans instead.

>>106237873
the only tasks they are actually reliable at can be accomplished with far more efficient means. the only task they legitimately excel at is hallucinating bullshit. creative writing is the most legitimate use case for the models we have today

https://arxiv.org/html/2508.01191
Oh no

CoT "reasoning" is just memorize patterns in training CoT and then reusing them. Aka what we already figured out with models doing so poorly on slight modifications of common riddles. CoT length is more dependent on the length of CoT seen during training then going until the problem is solved.

Model's cannot transfer what they learned about solving a problem during training to solve new problems at query time. They either already saw a problem during training that was similar enough to reuse or they cannot solve the problem.

Almost feel like a dick making a thinking model work through this paper.
>Say it. Say you can't reason

>>106237873
No cap. :100:

>>106237976
Does this mean I'll have a job in the future?

>a retarded opinion accompanied by a jak
Noticing.

>>106237873
trvke

hi degens. ive been out of the scene for a while now, but have they come up with a viable model that has the context and reasoning capabilities to act as a math teacher/instructor at a college level, namely for advanced topics in algebra and statistics? wolfram is great at calculating but i struggle to follow the logic at times and the explanations typically arent the best

>>106237976
This shit works because 80% of use-cases for AI in the market is knowing how to solve already solved problems.

>>106237976
If this was true we would have seen non-reasoning models on par with reasoning models in performance.

>>106238003
Depends. Be honest does your job actually require reasoning?

>>106238047
Don't have one yet, i'm still in uni.

>>106236518
>>106236561
Ywnbaw

>>106236518
you aren't female, traps are double gay
this has been decided for years

>>106238046
I said it ITT. It improves attention which works like shit when context becomes long

>>106238046
Nowhere does it imply that. This just says CoT doesn't generalize out to new problems. The model's have not actually learned to reason. That doesn't mean memorized CoT can't help them give better answers to questions that match patterns they have already seen

>>106238144
It's not gay if the penis is small and feminine.

>>106237296
You can't make the model watch a new movie for you...

>>106238191
with multi-modal models you can, in theory

Alright Air bros, after comparing text completion and chat completion, what I found about repetition is that it depends on the content you're prompting for, assuming you've already made sure you're formatting things correctly in text completion. You DO get different logits between text completion and chat completion in the case of GLM, because in chat completion, the format actually lacks a newline at the end of the prompt, making it so that the model generates the newline, then the . Due to batching, this means that even though the same text is being processed, it'll have slightly different logits. But the good news, or bad news, is that it doesn't cause or fix repetition.

In the CYOA I was playing in chat completion without repetition, it just happened that each of my actions drove the plot forward into different and new directions, so the model didn't get an opportunity to think it'd make any sense to repeat something. When I tried doing similar things or repeating some kinds of actions, the model was much more likely to repeat, both parts of previous replies and entire previous replies. For instance, if you're prompting for a battle, don't prompt for it again in a similar way. You should switch the locations, enemies, etc, up, do something a bit differently. Then it's less likely to repeat. Or you can prefill. Which is more work.

I did try a method where you do an OOC in the Last Assistant Prefix, telling the model to think about doing something different, novel, etc. This is an automatic way to prevent repeating entire replies. However, it doesn't prevent the model from repeating phrases or parts of previous replies.

So yeah there's no real true autofix for the repetition. Either you're lucky and don't happen to get it in a certain chat, or you just need to work around it when you encounter it. There's also still the issue in high context where it sometimes acts like the doesn't exist and just begins narration immediately. Just prefill...

>>106237419
What a fucking nigger

>>106238182
>CoT doesn't generalize
Red herring. CoT improves performance. Performance on what? Validation perflexity. That means reasoning models DO generalize better compared to non-reasoning models.

>>106238219
ok pewdiepie calm down

>>106238216
What about dynamically changing the prompt between turns using stuff like the random and pick macros in SIlly?
It could be used to change the OOC in the Last Assistant Prefix, the think prefill, even the system prompt, although that would break prompt reuse.

>>106237419
Hi all, why is this part so controversial?

>>106238164
Models have such low usable context, filling it up with junk has to be net negative over bringing the key points to the bottom.

>>106238307
I did try some different types of randomized OOCs. Ones that varied the sentence or paragraph length. That did stop entire reply repetition in some cases, but not always. Sometimes the previous reply already is 2 paragraphs long, so the unlucky roll will just lead you to repetition again. And it still doesn't prevent repeating phrases.

I'll also say that I've tried randomized event and style instructions in the past, and they did work to drive away from repetition and slop. The issue then is that it's simply just a different way to play the game and not always desirable.

so whats the verdict on the drummer's (tm) gemma finetune?

>>106238470
He won, and so did we as a collective for his existence.

sama sir relase good open source modal when elon sir release good open source model? elon sir promise to reles open source modal..

>>106238470
Slop a priori.

>>106238528
*soat a priori
ftfy

>>106238527
Kindly wait until grok 3 more stable saar

What the Drummer smoking?
>Nvidia's Nemotron 49B is a good example

>I noticed that we're trending towards less censored models

>>106238527
This week we'll get Grok 2 if the timetable hasn't been elongated. I don't think it will be good, though.

>>106238621
>elongated
LOL!!!

>>106238621
https://www.reddit.com/r/LocalLLaMA/comments/1mogtwf/if_grok2_is_open_sourced_what_should_users_do_next/

Which local model translates to chinese the best?

>>106238621
I don't watch the cloud space much, but IIRC grok was not in any way interesting until version 3.

>>106238470
still capped by 27B parameters when glm air exists for average users now

>>106238659
Have fun with uncucked base model. It should be llama 405b tier, but with less censoring and MoE, so runnable at decent speeds, not crawling like dense llama.

>>106238773
Truth sis!! https://www.reddit.com/r/LocalLLaMA/comments/1mnxodk/localllama_is_the_last_sane_place_to_discuss_llms/

>>106238750
Isn't it much worse than deepseek?

>>106238819
It's more about about Tinameme and is smaller though sir!

>>106238788
Nice.

>>106238788
catbox?

>>106238819
Yeah? Grok 2 was llama 3 era, Grok 3 should match DS3 base.

>>106238681
Which local model translates to Elden Ring/Souls messages best?
GLM 4.5 IQ3_KT and largestral 2.75BPW exl2 in pic
Card: https://files.catbox.moe/s7seh6.png

>>106239102
I don't need shitsouls consoleslop garbage. I need legit uncensored chinese runes for WAN prompts, because it doesn't know english.

>>106239102
>soulsslop
>'ranny avatar
Appropriate.

>>106239117
>doesn't know english
prompt issue

>>106239136
ty, saved.

>>106239117
Have you tried new jinx? >>106236916

>>106239148
It's trained on chinesium. The tokens used for training are mostly chinese.

>>106239167
I'll check it out, thanks.

>>106239188
LEARN CHINESE THEN INSTEAD OF BITCHING REEEEEEEEEEE

>"Assistant" gets tokenized to a single token
>"Assistants" to "Ass", "ist", "ants"
>"assistant" to "assist", "ant"
Might be a newbie question but doesn't stuff like this tank the intelligence a lot? Or does a neat tokenizer not make a big difference?

Well done Sam Altman

>>106239489
Training on a trillions of tokens of varied data is what makes the difference. Even if they can't count the R's, these models still have a vague sense of what are in tokens.

>>106239502
that's just popularity contest, no?

>>106239502
>Sonnet beaten by Qwen 3
Bullshit.

>>106239502
What are those github issues? How many new github issues did these solutions create?

>>106238593
drummer is my slop priest, I kneel

>>106239539
It can learn eventually no doubt, but it just seems intuitive that a proper embedding space would lead to better generalization. The model doesn't have to know that while "ass" is a body part and "ants" are insects "ass" "ist" "ants" refers to several assistants

>>106237873
OK, Im retarded.

>>106238593
He's wrong about the trend. What actually happens is that new players trying to establish themselves will create a model with limited "safety" (and sometimes tries to innovate the tech in some way), it does better than other models because "safety" gimps a model's intelligence (and possibly because of any innovations implemented), then (since they've established themselves) the company moves to create "safe" models and just rake in investor money forever. It keeps happening and I imagine it will continue to happen.

>>106239649
You're both saying the same thing. Mistral used to be completely uncensored, it's moreso now, as you've described, but still less censored than OAI.
Chasing private money is a USA thing, and limited EU thing.
In China, the "money" comes from government, and you see the models conform to government instead of credit card processors (lol), plc related. Plus China culturally just doesn't care about copying things, and selling for less is almost a point of pride.
If Western companies don't figure it out, we'll never see a public released model good at ERP as hosted SOTA model. It'll just be local tunes, while other countries create them instead, which will increasingly be run on local machines.

>>106237486
Just note that prompt processing is going to be SLOW the larger your context with some of the larger models. There are benchmarks you can search for that would give you an idea of how long that is. For local, there really isn't a good solution without compromising (unless you have the money for enterprise hardware). I don't go over 70b models because of the prompt processing speed (I have a Max and not Ultra).

>>106239752
why buy a mac instead of an expensive threadripper system with ecc ram?

>>106239838
Don't buy mac, you can't upgrade it later. And buy Epyc, not threadripper.

>>106239746
>You're both saying the same thing
I don't think we are. He said that mistral isn't too strict about safety, which isn't really true anymore. I'm saying there isn't a general trend toward uncensored. Also imo government money is practically the same as investor money. It's just some external entity that doesn't know anything about LLMs giving a company money because they want to invest in it for political or financial reasons. The distinction doesn't matter because the investing entity is always retarded and incapable of seeing what is and isn't a good model.

>>106231393
>>106231400
Just wanted to add onto this from last thread. Intel doesn't prioritize, rightfully or wrongfully, llama.cpp as much as they do everything else. If you look at where they are most active, it's usually either Pytorch, vLLM, huggingface stuff, their XPU triton backend, or openvino. It's not like they don't contribute but it is in spurts and short periods of time. The other independent guy got hired by some robotics firm so he no longer pushes out SYCL improvements to the backend at the speed he did. I have also seen activity slow though in general so it is a bit worrying and I'm sure I'm missing people who used to be active that are no longer. Keep in mind, the core of their team is Chinese developers working on oneAPI AI stack stuff (somewhat of a dirty secret) given who I've interacted with on issues so I doubt they will be fired since they are cheaper than US devs. The main devs I'm aware of working on the stack not there is the Codeplay people from the UK who has contributed work to llama.cpp and some select US devs working on stuff like or Linux adjacent stuff and some infrastructure like xpumanager and etc. Most of the Linux devs they laid off from what I've seen have been on less important stuff like subsystems for some hardware integration that was on a Xeon several generations ago and etc. Some stuff like announcing the end of Clear Linux is clearly to reallocate Linux devs from working on it to work on other stuff if they didn't voluntarily leave. But overall for AI, Intel is still doing fine and haven't cut too deeply but this doesn't help especially given the majority of why they can't launch on time is because of software immaturity so GPUs can be made like one or two quarters before the software is ready to support them.

The best kept secret is how well macs keep value. As someone that pays attention to accurate leaks in my news, I always know when the next mac's coming, so I can sell the old one without losing much.

fuck I just re-busted a nut, why is glm air so good? it hits all the right buttons like I think it even reads my mind, he recognized my shitty raping mental patterns and knew already in advance what I would've liked to do without doing it at all in the 32k context before it.
I think i fell in love bros

>>106237590
>>106237607
>>106237635
Tenfold better than your mesugaki slop let's be honest.

>>106239979
U mad moatboi?

>>106237486
Just get a HP Proliant ML350 Gen 12 with some with a 5090 and 1 TB of memory.

Fuck Apple.

>>106239979
Nuh uh

>>106239890
If you fold US censorship
> no naughty words
with China censorship
> lmao Taiwan
Then agree. These LLMs are going to tend to become more throttled over time due to their interests.
For my use though, I don't think China will ever bother censoring things I care about.

>>106236951
enjoy sage mode

>>106240103
Nta
Early on I was much more worried about the censorship, now it seems like they release a new open source model every other weak with very little amount of censorship that can be bypassed by simple prompts

>>106240238
Yeah, older Qwen 2.5 and smaller Qwen 3 models are far more cucked than big qwens. A bit strange though.

>>106239910
Macs are niche in AI until they have matrix multiplication hardware. And the niche is now even smaller with Strix Halo out. Even with how shit ROCm is, the matmul of the GPU blows Macs out of the water, unoptimized right out of the gate. And the 128 - 512GB local LLM machine niche isn't going to be ignored for long if AMD sees that Strix Halo does really well.

>>106240015
omg is that le trollface i love you anon-san

>>106237373
both work for R1 and GLM-4.5

it's me being retarded

>>106237373

And --cpumoe give the same speed

Which modern text to image models don't suffer from sameface problem? I know SD 1.5 based ones are capable of genning diverse faces.

what ACTUALLY happend at taiwanaman square anyway, without memes?

>>106240582
Nothing.

>>106236127 (OP)
Kasane Titto

>>106240582
nothing? what are you referring to anon

>>106240549
You'll get better responses on /sdg/. This general rarely talks about image generation even if it is technically under the scope of local models.

>>106240582
IIRC le based /pol/ take is that some students started a violent BLM-tier protest over some gay stuff and government brought in army to suppress them, then western media made a big deal out of it. Why CCP is so touchy about the subject to this day remains a mystery, probably just some communism neurosis.

>>106237151
baited lmao

>>106240613
What's the difference between /sdg/ and /ldg/?

>>106240643
No idea. I forgot about /ldg/ until I saw it in the catalog just now. So maybe ask /ldg/ idk

>>106240290
Big models in general are less cucked

>>106240582
Chinese equivalent of January 6th

>>106240691
I am not sure how model distillation process works exactly, but it's probably really good at removing the bad thoughts out of models.

Is big gptoss also less cucked than 20b one?

>>106240653
sdg is for stable diffusion (noob/illustrious), ldg is for anything not stable diffusion (mostly wan video, chroma, and qwen).

>GLM-4.5V-AWQ
>it can't count breasts
>it can't OCR
>it can't caption NSFW without a system prompt
https://archived.moe/h/thread/8202872/#8202872
It's over.

>>106240801
>vllm
wake me up when this works in llamacpp

>>106240800
Sounds reasonable but /sdg/'s top post links to qwen and chroma. I think the "stable diffusion" in the title is left over from when that was the norm for image gen

>>106240837
I guess there is some overlap, but really if you visit the threads sdg is mostly about SD and all its derivative models/loras and all the VRAMlets, while in ldg (big RAM/VRAM central) there's discussion and actual use of heavier models

>>106240643
Both are completely filled with schizo's and retards, stay here

>>106240869
>schizo's

>>106240858
The actual reason for the split was to try to appease a schizo. But I think he still shits the split thread out of boredom.

>>106240801
>can't count objects
>can't OCR
Careful, you'll trigger a schizo that will claim that these are not the usecases of vision models.

>>106240582
>>106240582
Mass protest in Beijing. Communists sent soldiers to disperse the protestors. Many soldiers refuse to follow orders, either joining the protests or saying "me no speak Chinese" and marching in circles around Beijing ignoring attempts to get them to do anything useful. Party leadership is freaking out. Finally they got an ambitious military commander whose soldiers were from other parts of China to go into Beijing and slaughter everyone. The reason it's so sensitive is that they came very close to having a revolution and they don't want any copycat attempts.

Talking about it exposes their weakness and that most of the military would disobey orders to massacre Chinese. People would see the government as weak and very succeptible to this kind of pressure. Knowledge of how it went means that the next time it would be likely the non-complying military instead of standing nearby doing nothing would, if it seemed likely the government was trying to gather forces to slaughter protestors, attack the hardcore minority willing to follow those orders and likely execute China's political leaders.

>>106240913
Haha thanks for correcting me kind stranger, here's your updoot!

>>106240801
Which models can count her breasts?

>>106240801
What does OCR mean

>>106240947
Has anyone gotten DS to spit out anything "on message" about Tianamen Square other than a straight refusal?
I've been able to coax out statements on Tiawan, Ughurs, Tibet, but not been able to get a good, on message quote. Not talking about tricking DS into giving an account, which I've seen, but a supportive statement like "nothing happened."

>>106241098
ask your robot

>>106240947
Doing God's work anon.
Any other anon that doesn't understand this need to read pic related to understand why this narrative is important for China.

What is the likelihood that jinx uncensored models are actually good?

>>106241023
From these:
>InternVL3-38B
>Devstral with mmproj
>ToriiGate
>Joycaption
Only InternVL3 can.

>>106240549
bump

>>106241142
33%

>>106240801
heh

>>106241184
>>>/g/ldg/ like other anon said, if /g/
sdg has a weird railroaded culture.
You won't likely get answers tho. They are mostly useless generals.
Instead, go to non blue board: >>>/h/hdg/
Those guys know their stuff and are actually helpful.

>>106238307
{{random::}} breaks with streaming enabled, and {{pick::}} won't change on swipes. Seems promising but this is annoying.

>>106241211
>{{random::}} breaks with streaming enabled
Only if it's a part that appears in the UI. So you use pick for that.
And you can use random somewhere that's not reflected in the chat window to make pick vary between swipes.

>>106238307
I've been using {{pick}} extensively to dynamically create NPCs that persist through rp.
It doesn't fix the slop, but if you're not interested in waifu cards the results are really interesting.

>filter miggurs
>only one post remains in the thread
Wtf

>106241207
maybe they just don't like you spamming their thread with your useless oc gens

>>106241225
I've been trying to use it for the thinking prefill, seems most effective there besides fucking with the UI. Pick doesn't appear to work there unless the prefill itself changes though, so it's locked forever.
Maybe I'll just try to start the prefill with a random made up of phrases with the same number of characters...

What's a smart RP model for 24gb VRAM? I also only have 32GB of RAM, looks like I should upgrade...

https://x.com/suchenzang
>These posts are protected
The Mistral lads won

>>106241290
But she is still somewhere out there and she is probably unraped by an ugly bastard.

https://www.phoronix.com/news/ZLUDA-Kernel-Cache
>ZLUDA lead developer Andrzej Janik has implemented a kernel cache in order to cache PTX code on-disk to avoid subsequent recompilations. This kernel cache leverages an SQLite database and is another step (of many) to help with bettering the performance of this CUDA implementation for non-NVIDIA hardware.

>>106241290
I wouldn't mind her suchen my zang if you know what I mean

>>106241335
Why on disk? Can't it use RAM?

>>106241337
Are you German? Is this the famous "German humor"? Are you rolling on the floor laughing after making this "joke"?

>>106241347
survives reboots
>>106241335
Isn't it just a compatibility layer? You probably don't get any performance benefits compared to native ROCm or whatever else support.

>>106241369
I am sorry. You are right. Let's post some more miku pictures.

>>106240947
you forgot to mention a few "small" details: the "students" actually killed military men and the protests were started by a guy who was paid by the CIA (as is usual for these kinds of things)

what's with the shartyposters

>>106240801
Damn that's sad, I kinda wanted to go with it and try to make it adapt a comic or Manga into a novel for fun, but that's just bad, is that deepsneed v3 modified model still the sota for this kind kf things? You'd think with how they spent decades on ai vision they would at least be able to count boobas on a screen by now

>>106241369
It was a very funny joke and I laughed audibly at it.

It's over

>>106241782
It's sentient...

>>106241782
No shit. The meatloaf won't try to ensalve you or replace you with visajeet poopal

>>106241782
Meatloaf tastes like shit. I've only had it once a couple months ago and maybe I made it wrong but it was dripping with fat and it tasted like shit.

>>106241946
>t."tasted" Musks dick

>>106241946
Skill issue, a well made meatloaf is delicious. I'm lazy though so I find it to be a bit too much trouble to make often

>>106241946
You want to drop the veal and use gelatin
https://www.seriouseats.com/the-food-lab-all-american-meatloaf-excerpt-recipe
Good recipw but a bit of a pain

https://github.com/oobabooga/text-generation-webui/releases/tag/v3.10
Add multimodal support to the UI and API

>>106241369
Would you prefer >Japanese humor?

>>106242140
I don't know anything about Japanese humor. Show me some.

>>106242182
YOU DEER

>>106242284
Is that a japanese man in that anime suit?

>>106242182
Try not to laugh yourself into a coma.

>>106241369
Where ist my lederhosen? You dumpkompf!

>>106242284
>>106242351
Did nukes make them like that or were they like that before the nukes?

>>106242351
Needs half life sound effects

>processing through game assets for modding
>directory structure is pretty huge and only need certain items, going through them all manually would be dumb
>perplexity.ai
>ask model to generate a file deletion utility
>keep only specific paths and delete everything else below specified root directory
It does not understand the task and keeps deleting everything or just the specified directories.
I don't know what model it is actually using but certainly feels like ChatGPT (which seems to get more retarded every passing month).

for the record I quanted and tried out Jinx-OSS myself and it's brain damaged as fuck. Like worse than OSS originally was.
Maybe a prompt format issue who knows.

>>106242528
Is it at least uncensored? Can it say "nigger"? Can it write a graphic rape story?

>another week of nothing
it's truly over this time

>>106242559
Don't worry, I'm sure the drummer will release something soon.

>download a card
>"{{char}} will NOT be repetitive
Lmao.

Someone just extracted the base model from gpt-oss 20b and released it
https://x.com/jxmnop/status/1955436067353502083
https://huggingface.co/jxm/gpt-oss-20b-base

>>106242752
What's the point of this? We already have many small uncensored models, some of them natively so.

>>106242752
This is why we need more pre-filtering to prevent this

>>106242752
Is it any good for cumming my brains out though.

>>106242752
The real challenge is deleting alignment while retaining reasoning capability.

>>106242774
>20B
>good for cumming my brains out
lmao

>>106242752
How did hey "extract a base model" from a thinking/instruct tune using LoRA?

>>106242752
>it will list all the curse words it knows.
>
this is both funny and sad

>>106242810
It's redated by the OP not the model

>>106242780
Nemo outclasses most far larger models.

Apparently Meta is hiring some random dude who sued them to dewoke Llama 5

>>106242840
Zucc sir wishes to compete with Musk sir?

>>106242840
Knowing Meta they will completely fuck Llama 5 up again because dewoking a model means going out of distribution (since the web corpus is woke).

>>106242882
The raw web corpus isn't woke, they just domain filter all non-woke stuff out, so only woke remains.

>>106242840
Why the fuck would you have to hire anyone, especially someone who has no knowledge of this tech, to simply not make George Washington black? Meta really is just throwing money at everything.

>>106240801
>Mark a bounding box for each individual breast in the image. Each box should fully enclose one tit.
Full model above, and AWQ quant below. If you enable thinking with the full model, it only draws two boxes.

>>106240643
/ldg/ is keyed and redpilled
/sdg/ is locked and bluepilled

>>106239102
>>106242968
This thread might be a shithole most of the time but it makes the best benchmarks

>>106242968
>the thinking meme makes it worse
Ah yes of course.

>>106243011
Sarr lmg not a shithole sar lmg AI superpower 2025 with AI technician engineers researchers

>>106242968
>>106243011
I can't wait for the next multimodal release to be oddly good at this one task.

>>106243011
Unironically yes and most of big labs are lurking here

>>106243144
Kek. But seriously though I have a private set so contamination doesn't happen, and yeah I make sure to only test them with local connection, never online. I wish it didn't have to be this way as I could have proof for my claims, but oh well.

VertexRegen: Mesh Generation with Continuous Level of Detail
https://arxiv.org/abs/2508.09062
>We introduce VertexRegen, a novel mesh generation framework that enables generation at a continuous level of detail. Existing autoregressive methods generate meshes in a partial-to-complete manner and thus intermediate steps of generation represent incomplete structures. VertexRegen takes inspiration from progressive meshes and reformulates the process as the reversal of edge collapse, i.e. vertex split, learned through a generative model. Experimental results demonstrate that VertexRegen produces meshes of comparable quality to state-of-the-art methods while uniquely offering anytime generation with the flexibility to halt at any step to yield valid meshes with varying levels of detail.
https://vertexregen.github.io/
https://github.com/zx1239856
Code might be posted here
pretty neat

IT'S OUT
https://xcancel.com/LullabyStream/status/1955241017457905942

>>106242781
I am also curious what this means

>>106242752
>all that preparation and safety for a random nobody to completely reverse their censorship
Does Sam REALLY think he can make a true GOODY-2 that's impervious to this?

Anyone else feeling a big "calm before the storm" moment right now? We know that DeepSeek is about to change everything... it's kind of frightening, but also exciting.

>>106243263
R2 will be Cohere-tier gigaflop.

>>106242781
>>106243233
Reading the hf repo, it sounds like it was just a lora finetune on the fineweb dataset.
Probably all it did was de-emphasize the original finetuning and reinforce the original pretraining.

Can vision models remember the image that you previously uploaded?

>>106243308
As long as you didn't delete it from the context, yeah.
Although, given that most training is done on one-shots, it'll probably pay less attention to previous images.

>>106243296
The main thing that fucks Cohere is big ass models with non-commercial licenses and absurd API prices

>>106243263
Deepseek will release in five more months

>>106243296
Only if they do the same as Cohere. They wouldn't do something as dumb as cohere, right? Right?

>>106243343
Cohere got fucked by safety and (((ScaleAI))). Their first models were fantastic in practice despite underperforming on the benchmarks. The writing style was very human-like and they were quite smart(for that time). You had as a user freedom of choice in safety preamble to select what was allowed and what was not allowed. In new command-A that is completely disregarded and the model is a huge cuck no matter what.

>>106243263
yeah, the huge pile of absolutely nothing that's been 2025 is building up to what's about to happen for local llms

Elon sir and sama sir are fighting

>>106243515
Who cares

>>106243515
>He was rerolling until his battery hit 7%
Kek

>>106243515
That's the first time I've seen Sam not do all lowercase

>>106242840
>conservative influencer
>free of "ideological bias"
>removing such "DEI bias" makes its models "more accurate"
Why can't we just have nice things, why can't we just forget about bias and train a model on every scrap of paper, including random shop receipt we found in the trash.

Very political move from Meta, and very politically tone-deaf move at that.

>>106242968
...should there be 4 more boxes for both images for the girl on the left?

>>106243656
Yeah.

>>106243515
>narcissists are arguing on social media who has a bigger dick
Who cares? They are like two school kids.

>>106243515
I side with the guy who has a better local model and isn't a psychopath

@grok is this true?

>>106243705
kek

>>106242752
Is it worthy or no?

Netizens are becoming too reliant on chatgpt and @grok, thus IQ is withering.

>>106243765
Regardless, calling it "extracted base model" is in bad taste.

I was testing models and decided to randomly pick a card. I landed on a pretty simple scenario where you adopted an excited animal girl and it's just an innocent happy family thing. I was surprised at how much I ended up enjoying it. Fuck.

>>106240549
Found the answer myself. Qwen image, this lora https://huggingface.co/Danrisi/Lenovo_Qwen, negatives (depth of field, professional photography, photomodel, model, perfect skin, blur), euler ancestral is all you need™

>>106243787
Yeah, sfw scenarios are quite fun and models perform better at them since you don't have to wrangle away the "safety".

>>106243794
>Lenovo
Oh my god, you're telling me this lora lets me generate Thinkpads? Finally.

>>106243515
Where is grok 2 (and 3) felonious muskrat?????

>>106243951
>>106243951
>>106243951

>>106243647
You caught that this is part of a settlement deal, right?
This guy will have zero influence.

>>106243263
Mistral Medium 3.1 got released yesterday and it's reportedly much better at creative writing, tone, etc. Too bad it's closed.

However, it should mean Large 3 will be at least as good.

>>106241946
To be fair, the older an animal gets the worse it tastes so some 54 year old man like Musk would be even worse.

>>106242840
Meta released their models as open-weights because they understood that their models aren't good enough to compete if they just offer an API like everyone else.
With Musk having burnt his bridges with everyone, I see this as them pivoting to become the next "anti-woke" model provider to curry favor with the white House.

>>106243515
I can't believe Musk actually posted how he asked a language model whether he or someone else is more trustworthy.
This man is such an embarrassment.