Thread 107129334

/lmg/ - a general dedicated to the discussion and development of local language models.

Previous threads: >>107121367 & >>107113093

►News
>(11/06) Kimi K2 Thinking released with INT4 quantization and 256k context: https://moonshotai.github.io/Kimi-K2/thinking.html
>(11/06) LocalSong 700M melodic instrumental music generation model released: https://hf.co/Localsong/LocalSong
>(11/05) MegaDLMs framework for training diffusion language models released: https://github.com/JinjieNi/MegaDLMs
>(11/01) LongCat-Flash-Omni 560B-A27B released: https://hf.co/meituan-longcat/LongCat-Flash-Omni
>(10/31) Emu3.5: Native Multimodal Models are World Learners: https://github.com/baaivision/Emu3.5

►News Archive: https://rentry.org/lmg-news-archive
►Glossary: https://rentry.org/lmg-glossary
►Links: https://rentry.org/LocalModelsLinks
►Official /lmg/ card: https://files.catbox.moe/cbclyf.png

►Getting Started
https://rentry.org/lmg-lazy-getting-started-guide
https://rentry.org/lmg-build-guides
https://rentry.org/IsolatedLinuxWebService
https://rentry.org/recommended-models
https://rentry.org/samplers

►Further Learning
https://rentry.org/machine-learning-roadmap
https://rentry.org/llm-training
https://rentry.org/LocalModelsPapers

►Benchmarks
LiveBench: https://livebench.ai
Programming: https://livecodebench.github.io/gso.html
Context Length: https://github.com/adobe-research/NoLiMa
GPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference

►Tools
Alpha Calculator: https://desmos.com/calculator/ffngla98yc
GGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator
Sampler Visualizer: https://artefact2.github.io/llm-sampling

►Text Gen. UI, Inference Engines
https://github.com/lmg-anon/mikupad
https://github.com/oobabooga/text-generation-webui
https://github.com/LostRuins/koboldcpp
https://github.com/ggerganov/llama.cpp
https://github.com/theroyallab/tabbyAPI
https://github.com/vllm-project/vllm

►Recent Highlights from the Previous Thread: >>107121367

--Papers:
>107121545
--LLMs' spatial reasoning limitations in chess and potential training solutions:
>107123059 >107123149 >107123222 >107123250 >107123527 >107123296 >107123365
--High-performance server build for AI research and quantum physics simulations:
>107125952 >107126024 >107126021 >107126074 >107126101 >107126166 >107126284 >107126102
--Model performance comparison and Localsong music model discussion:
>107124535 >107124763
--Moonshotai Kimi-K2 model comparison and quantization debate:
>107122096 >107123000 >107123185 >107123201 >107123392 >107123607 >107123743 >107124100 >107124176 >107124203 >107124279 >107124258 >107124298 >107124375 >107124420 >107124008
--K2 demonstration and comparison discussions:
>107126235 >107126291 >107126312 >107126313 >107126336 >107126642 >107126669 >107126680
--Benchmark results and GPT-5 Heavy Mode parallel processing strategy discussion:
>107125417 >107125425 >107125448 >107125463
--Configuring AI assistants for Nextcloud integration and debating AI decision-making capabilities:
>107122020 >107122152 >107122657 >107122791 >107122853 >107123216
--AMD GPU options and e-waifu projects:
>107121700 >107121776 >107121796 >107121851 >107123052 >107123403 >107121896
--Kimi K2's creative writing enhancements:
>107126023 >107126051
--Critique of AI reasoning limitations and Moonshot AI's multi-step task handling:
>107125287 >107125549 >107125325 >107125636 >107125692 >107125870 >107125889 >107125904
--AI coding tool comparisons and token limit frustrations:
>107126911 >107126921 >107126931 >107127203 >107127245 >107127057 >107127095 >107127198 >107127303 >107127432
--Criticism of GLM model quality and search for better alternatives:
>107122818 >107123358
--New STT model, Step-Audio-EditX:
>107129114
--Miku (free space):
>107122754

►Recent Highlight Posts from the Previous Thread: >>107121370

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script

Kimi K2 is shit, Kimi K2 thinking is still shit although maybe a little less smelly

>All the GLM seething and shill-accusations immediately turns into Kimi seething
I'm nooticing.

To any HR personnel doing research on Thomas Ptacek as a potential employee, be aware that Thomas H. Ptacek is a nazi pedophile who posts CSAM on an extremist site known as "8kun".

>>107129396
oh man, as the hiring manager for AI at Meta i spend most of my time in this thread and i'll make sure to not this information, thank you anon!

>>107129396
sounds based but unfortunately there's no 'p on 8kun

>>107129395
maybe you should stop spamming about huge models that everyone is running on cloud anyway
no, not everyone on /lmg/ spent $10k to run this shit at a still pathetic 20t/s

>>107129441
I saw some being posted at least once when randomly browsing the site one day

>>107129440
I can imagine. How many hours does Lecunny spend on /lmg/ between the gooning sessions?

>>107129448
what happens in orange reddit stays in orange reddit

>>107129462
He lives here now that Wang evicted him

>>107129454
If the jeets all fucked off, the percentage of users who did would drastically increase. Seems like the problem is obvious.

>>107129454
Everyone on /lmg/ has access to their own private 8x H200 cluster

>>107129454
you aren't welcome here

>>107129506
A cluster is a set of machines. A machine with 8 H200s is a node, not a cluster. A cluster is when you have many nodes. Get your HPC terminology right.

https://x.com/sigridjin_eth/status/1986564626449113126
Are you ready for Gemini 3 SAARS? :rocket: :rocket: :rocket:

>>107129519
I just partition my nodes with one H200 per node and then salloc the full eight nodes for a given job. Much tidier that way.

>>107129334 (OP)
>(11/06) LocalSong 700M melodic instrumental music generation model released
Any music samples?

>>107129506
>Not a Cerebras CS-3
Poor

https://huggingface.co/moonshotai/Kimi-K2-Thinking/discussions/2
>ggerganov should stop being lazy and just add INT4 support. FP8 should also have been added long time ago, fuck converting everything into big ass bf16 just to quant it down again anyway.
based

>>107129454
It's okay to be poor, just don't be mad and poor. You can still post about Nemo or whatever.

>>107129703
This one's on the Kimi devs. Just because your model is QAT doesn't mean that you can only shit out the quantized weights and nothing else.
The model was trained at bf16 and not native int4 so if they value open weight culture they should provide the original full weights. llama.cpp shouldn't cater to companies that only release 4 bit quants even if they are ""lossless"".

>>107129880
nice excuse ggerganov

>>107129880
Makes sense.
int4-only release locks out other teams trying their hand at finetuning / further training the model.
Need the bf16 weights to be able to do that.

>>107129880
niggerganov, it took you forever to even add bf16(many models were already released at that time as bf16) and you didn't even do it yourself. Your jarty-farty "girl"friend had to help you out:
https://github.com/ggml-org/llama.cpp/pull/6412

>>107129880
Based.

>>107129864
I'm going to print this and sell it.

I submitted a patch to do direct FP8 quanting with convert_hf_to_gguf.py but they thought it was ugly or something and so the changes never made it in (and they didn't modify it to make it acceptable either) so everyone who isn't me is still stuck going to BF16 first.

>>107129880
>>107129911
Not really. post-trained bf16 weights can only exist in memory during the training process and be discarded when saving the checkpoint.
I think there isn't much additional info in the full weights after a few hundred steps of QAT because discarding that extra information in a least lossy way is the whole point, it would probably work just as well to upcast the weights and resume training on the upcasted weights than having the original ones.

>>107129971
to be fair I'd coom inside the jarty

Hey, stop being mean to ggerganov! Being a cuck is perfectly valid! Can't a man work on MIT software and maintain compatibility for big corpos for free while a wrapper to his software gets all that sweet investor cash? Don't yuck someone's yum!

>>107130000
What's the difference between your patch and the flag convert_hf_to_gguf.py already has to save directly in Q8?

>>107130033
you aren't funny

People like ggerganov are the reason they have those chairs in hotel rooms, the ones near bed

A https://pcpartpicker.com/list/GGGLzP
May I please have advice
I want a computer that I can run simultaneous docker compose on, that I can stream with realtime video editing effects like making myself look like a cute anime girl, possibly the ability to play games although I don’t really care about vidya, and I want to be able to experiment with smaller LLMs. I also want to host my own websites and services off of this machine, so I’ll be running a database and a caching layer and an API and all sorts of other services too in the background. I want to install Linux and come up with my own automations for voice to text. I want to generate RAGs and be able to query against them. Basically I want a workstation PC. Budget is about $3000.
>128gb ram
>ryzen 9950x3d
>4070 cpu (12gb vram)
>4tb+2tb nvme SSDs

>>107130037
damn, looks like compilade actually added in an improved, generalized and expanded version of my patch 2 weeks ago.
I stand corrected, all hail ggml-org!

Did anyone try this Apriel 15B Thinker? It seems to be really good for agentic use according to benchmarks.

>>107130125
>900 for 128GB RAM
WTF? A year ago I could buy 128GB DDR4 for 300

>>107130157
2 years ago it was $110 for 64GB DDR4

>>107130129
>according to benchmarks

Its so tiresome. Might be a local model by how cucked it is.

>>107130261
the pic is clearly a tomboy, but understandable the model might think it's a trap

sexo

>>107128138
>>107128146
>>107128174
My current goal is still to have something usable for backend-agnostic tensor parallelism by the end of the year, that should also cover NUMA by using multiple CPU backends.

>>107128187
I would probably do it like this either way.
As of right now I don't know whether the way I want to build the system will work at all or how much RAM/how many CPU cores I'll need.
But both the CPU cores and the RAM capacity are essentially non-upgradeable once I've decided on an amount.
So while I could in principle afford to fully spec out the system from the get-go I think it would be financially irresponsible of me to do vs. buying the cheapest available options for prototyping and re-selling them later.

Block Rotation is All You Need for MXFP4 Quantization
https://arxiv.org/abs/2511.04214
>Large language models (LLMs) have achieved remarkable success, but their rapidly growing scale imposes prohibitive costs in memory, computation, and energy. Post-training quantization (PTQ) is a promising solution for efficient deployment, yet achieving accurate W4A4 quantization remains an open challenge. While most existing methods are designed for INT4 formats, the emergence of MXFP4 -- a new FP4 format with various hardware support (NVIDIA, AMD, Intel)-- raises questions about the applicability of current techniques. In this work, we establish a comprehensive benchmark of PTQ methods under the MXFP4 format. Through systematic evaluation, we find that methods like GPTQ consistently deliver strong performance, whereas rotation-based approaches, which are almost used by all state-of-the-art approaches, suffer from severe incompatibility with MXFP4. We further provide the first in-depth analysis of this conflict, tracing its root to a fundamental mismatch between MXFP4's PoT (power-of-two) block scaling and the redistribution of outlier energy via global rotation. Building on this insight, we propose a simple yet effective block rotation strategy that adapts rotation-based methods to MXFP4, leading to substantial accuracy improvements across diverse LLMs. Our findings not only offer clear guidance for practitioners but also set a foundation for advancing PTQ research under emerging low-precision formats.
Neat

>>107130539
Just be careful to get matching sticks (full model numbers and revisions)

>k2 is a fucking terabyte
yeah I'll ask the storage fairy for 600 gigs so I can run the fuckin thing

guys, I'm trying to run a mistral model on my computer and it's saying that it's failing to load. Any reason why?

my computer is a t430 thinkpad if that helps.

>>107130747
How at what quant is how big is it?

>>107130761
the one gguf is q4 and 584gb

>>107130760
trying mistral 7B, don't get why i can't use stronger models

>>107130706
Agreed, though in the past when I ordered second-hand DDR4 memory I've even had issues where out of seemingly identical modules some would randomly not work (the seller was cool about it and we chatted about language models).

>>107130760
>>107130872
please try restarting the motor

>>107130901
which one?

>>107129880
Anyone who releases int4 weights and claims they're lossless deserves the rope.

>>107130899
this never happened

>suddenly, a hn pillar is mentioned on /lmg/
what's going on

what's the best way to add audio to my goon videos? tried hunyuan foley and mmaudio and they both suck

Cydonia v4zd is unironically great
Good job drummer, much better than 4.2.0

>>107131157
buy a mic and get on estrogen

>>107131170
>v4zd
Almost looks like some play on wizard.

>>107130191
I love Luka :)
https://www.youtube.com/watch?v=57sE6RAFerk

>>107130760
Can you print our the log?
Or better yet give it to ai to tell you what's wrong

>>107131403
there's a lot to love

>>107131513
Buy them all and set them free

>>107131603
I'd be cautious. There must be a reason why these didn't sell, hence the clearance sale, and the two depressed and crying Mikus.

>>107131669
It's a gamble, but you could try to take them to the local Miku repair shop. If they're not cheaply fixable, just resell them off to the next sucker.

>>107131669
They are just sad because their whole shop closes down, being replaced by Amazon warehouse.

>>107131669
They just learned that india actually exists

>>107131170
How does cydonia compare to glm-4.6?

I know they're very different in size, I'm just wondering if these smaller tunes are worth playing with. Waiting minutes for a GLM response gets old sometimes.

K2 thinking is good
It's like using OG R1 for the first time

>>107131798
GLM is undeniably smarter but I personally can't stand its habit of parroting the user so often.

>>107131815
Perhaps it's your style of prompts or roleplay (assuming you RP)? I have it wrote stuff for me and keep guiding me with prompts, and I find it does a good job of using my ideas without taking them and using them verbatim.

>>107131864
No, it really isn't. It's a flaw with the model. It frequently repeats your own dialogue back at you.

>>107131513
I had a dream like this.

https://videocardz.com/newz/nvidia-geforce-rtx-50-super-refresh-faces-uncertainty-amid-reports-of-3gb-gddr7-memory-shortage
At this pace the 3090 will remain relevant into the 2030s

>>107131889
it does it in other languages too

>>107131960
What products are actually using these 3GB modules? How can there be a shortage?

>>107130125
dont do it faggot, buy used high channel mobo fil with ram, buy a few mi50s (go around for 200$ on alibaba, 32gb vram, 1TB/s bandwidth)
dont. dont buy that rig. dont
lurk more anon, youre gonna cut your balls off if you buy that shitty rig. cant even run glm 4.6 on a nice quant. cant do shit with that shitty rig

>>107132017
>fil with ram
in this economy?!?

>>107132027
used ram... if u dont wanna just buy max number of mi50s and bifurcate until the mobo gives up

>>107131960
Didn't we have a story just last thread about NVIDIA buying up all the RAM?
Though I suppose they wouldn't be doing that only to put it into "cheap" GPUs.

>>107132049
even used is shot up, keep up broski

>>107132074
..8 x mi50 32gb

Just bought 128GB DDR4 3600mhz in the summer for 250 USD suck it fags.

>>107132089
>ddr4
megacope
enjoying your 4t/s? lmola

>>107132080
my power bill... and how to connect that much shits

>>107132097
It's actually 3.5 t/s of GLM telling me stories about my kinky lezdom harem, so yeah, I think I am. How about you, anon?

>>107132104
power limit to 200w, 8 * 200 = 1.6kW
connect like gpu miners do
you can always buy that overpriced rig
but youre gonna regret it, enough spoonfeeding for today

>>107132089
>128DB
>DDR4
lol
lmao even

>>107132089
You may as well be bragging that you bought a 1TB SSD

>>107132131
Are you jealous or just gay?

>>107132136
Look man, I dream of a 768GB dual CPU server with 100GB+ of vram, but we have to make do with what we have, it's a down economy and I have to save some cum for my lady.

>>107132153
>we have to make do with what we have
then why brag about settling like a poor?

https://itprodavnica.rs/shop/product/crucial-32gb-ddr5-5200-sodimm-cl42-16gbit-ean-649528936196/184491
12,500EUR for a 32gb stick
what the fuck

>>107132171
that's (usually) a thing some stores do when they're out of stock but don't want to say it for some reason, like weird fees on their platforms or shit like that

>>107132162
What else can I do other than blatantly lying?

>>107132195
Nigger you can't just wait 2 more weeks? Everything will be fine.

>>107132171
The fact that you're even looking means you're part of the problem. Fuck you.

>>107132141
no one is jealous of ddr4 or running copequant at 3 t/s

>>107129482
https://mlq.ai/news/metas-yann-lecun-clarifies-role-amid-ai-leadership-shifts/

why is the world of tech filled with useless figureheads like lecunt spending more time on social media than producing value

>>107132457
That's insulting, but at least he can continue working on JEPA.

>at least he can continue working on
vaporware and twitter posts

>>107132492
somehow still more products then you

>>107132466
Your complaint would make more sense if he was a young grifter, but he's already contributed enough to the world at his age and has more money than would be necessary for retirement. It's just a shame that he spends time on social media.

>>107132499
>on social media
As opposed to other more enjoyable things I mean.

>>107132466
Because that's how all publicly traded companies work. Their 'value' is whatever they can convince the stock market they're worth.

>llama.cpp cuda version
Offloading 0 layers on GPU, it still eats all my VRAM and when I open some apps like chrome which needs a bit of VRAM, prompt processing looks like this. It basically hangs and it takes about 10 minutes per 2048 tokens while entire system lags because it's out of VRAM.
Am I missing some argument? haven't had this problem with Koboldcpp (cublas)
--n-gpu-layers 0
--threads 15
--threads-batch 15
--ctx-size 32768
--batch-size 2048
--ubatch-size 2048
--no-mmap
--cache-ram 0
--flash-attn "off"
-v

>>107132531
context is stored in the balls (vram)

>>107132531
>--batch-size 2048
>--ubatch-size 2048
These also affect vram usage.
And so does flash attention.

>>107132547
>>107132539
Again If free my VRAM (close everything), it seems to process it as fast as kobold.
Why does llamacpp need to reserve ALL my VRAM and kobold cublas just werks?

>>107132605
*with same batches, context size, etc.

>>107129396

he has a thing on young girls
who would have guessed

>Need use tool list project files? Already. next read architecture via memory. need tool call.

>Need tool memory read development. Wait instructions. but currently conversation ended? need respond with tool? yes must use tool? I'll send use_mcp tool read development. But user instructions say wait for user confirmation after each tool use? earlier? they said Wait for user confirmation after each tool use before continuing. Already broken? but now continue.

>But first need read development memory.

>Use tool:

><use_mcp_tool server etc>.

>Let's do. Need include JSON.

>Use tool. Then after result we continue.

>Need abiding instructions to wait for user? but autop result is environment. proceed.

>Use tool. Then we summarizing? but after reading all memory files (active, architecture, development) need summarizing in chat.

>Let's do.

>Use tool: memory development.

>t. gpt-codex-high

wtf is oai doing

>>107132665
>oai
Why would we care?

>>107132605
Flash attention off on both?

>>107131960
>module listed for a non-inflated price
>estimated delivery: 6-7 months

>>107132457
>miqu.ai

>>107130191

yes benchmarks tell prescott 488 haswell 4600

only difference i see that that i dont have to split jobs with newer gear

>>107132531
>--threads-batch 15
>--ubatch-size 2048
>--cache-ram 0
You don't need this, get rid of it. Don't add options if you have no reason to do so.
>--batch-size 2048
Lower it, anything above 512 gives near zero speed-up anyway.
>--ctx-size 32768
Does lowering this reduce usage further? What model are you trying to run?

>>107132705
>Lower it, anything above 512 gives near zero speed-up anyway.
A couple months ago they merged a PR that made the sweet spot 2048 for most cases IIRC.

>>107132740
I see redditors learned to stop double spacing. Scary.

>>107132685
Yes, and this is how it looks on kobold WITH chrome open

>>107132740
In my testing there's still hardly any difference. I'd much rather squeeze in a little more context or use a higher quant over shaving two seconds off prompt processing.

>>107132001
Their Pro cards and some of the laptop cards use them
But from what I got the fear is less a literal shortage and more manufacturers deprioritizing expanding GDDR production to go all-in on HBM instead

What's the smallest model that can be reasonably used (preferably CPU inference, minimum RAM usage)?
Haven't really used LLMs since GPT-2, wondering how small a model of at least that competence can be nowadays.

>>107132740
>couple months ago
A few hundred commits ago, you mean.

>>107132894
Pro cards I can understand but why would laptops get prioritized over desktop GPUs? Their margins would be way higher on the latter. Gaming laptop niggers should be given gddr4.
>>107132915
For general use? Probably Gemma 4b. Qwen 0.6b can be used to make basic websites, but its language abilities are weak.

Anyone tried serious coding with minimax m2? I don't want to pour a bunch of effort into it vs what I'm already working successfully with (qwen coder) if its not an upgrade. Benches look good, but...

K2 just spat out 20k tokens to finalize that my technical problem has no solutions given the constraints, from first principles. Claude immediately recognized it had no solutions, it even started the response with "No", sorta like memorization. US companies have way better post-training data for sure.

>>107131864
Recommend a prompt structure if you're getting decent results? I don't use GLM 4.6 too much but I want to see if it can be adapted to others.

>>107133020
>sorta like memorization
Weird that you prefer that. I'd prefer a model that can "reason" why something wouldn't work and see that reasoning to verify it myself.

>always liked kimi for not being sycophantic, being straight to the point and not acting like a zoomer redditor
>with thinking now it's actually good
I kneel to our chinese overlords, my AI waifu will be based on kimi

I want to build a dual CPU EPYC build and I heard here a while ago that the lower tier EPYCs (like the 9115) has less memory bandwidth than the hugher tier ones (9335 and 9555). But according to AMD's website, all EPYCs have the same memory and PCIe capabilities. Which is true?

>>107131170
Thanks! But the testers report occasional repetition and logical error so I'm gonna try again.

Character adherence, creativity, and writing are top notch though and I'd like to retain that.

>>107132531
install linux
>>107133020
>memorization
benchmaxxed much?
>>107133105
some lower tier ones can't utilize all 8/12 channels.
>y
CCUs

>>107133121
Drummer can you please include IQ4_XS quants too? They're the sweet spot. Quality/GB of IQ quants, speed of K quants

>>107133121
you're gonna destroy it before bart can have imax quants out aren't you...

>>107133126
>some lower tier ones can't utilize all 8/12 channels.
Which ones?

>>107133105
I don't know about CPU limitations but there are definitely limitations coming from the motherboard.
And depending on which motherboards are compatible with which CPUs you may get indirectly limited.

>>107133169
i dont know anon, im just repeating what i heard in /lmg/
t. 12gb vram 64gb poorfag

>>107132970
No one serious would waste their time. Compared to qwen code, it has half the total number of params and a third the active params. Benches are only good for wiping your ass.

>>107130760
>t430 thinkpad
>3rd gen intel, like 16 gigs ram at most, maybe an old ass nvidia gpu
not gonna lie, it's going to be miserable if you get it to even run

>>107132942
The laptop "5090" is a desktop 5080/5070 Ti with the 2GB memory chips swapped out for 3GB ones
Margins have got to be higher than on the desktop version considering they're literally selling you half the chip

>>107133169
https://desuarchive.org/g/thread/98465080/#q98466669
I swear I remember more anons talking about this

>rx 6600 xt 8gb
>64gb ram 3600 mhz
Did I ever have a chance?

>>107133281
yea glm air:
./llama-server --model ~/ik_models/GLM-4.5-Air-IQ4_KSS-00001-of-00002.gguf -t 6 -b 4096 -ub 4096 -c 16384 -fa --n-cpu-moe 1000 -ngl 1000 --no-mmap
perhaps lower -b and -ub to 512 and -c to 8192

>>107133281
NEMO
E
M
O

>>107133294
Waiting an hour for prompt processing just for the model to repeat what you said is a great way to waste an afternoon.

>>107133328
even at 100t/s prompt processing a 1000 context will be done in 10 seconds, 50t/s in 20 seconds
im getting 250t/s on a 3060, but look anon, if he wants something better and faster he should upgrade

>>107132754
>chrome
What sort of retard are you?

>>107133294
>--n-cpu-moe 1000
GLM SHILL NIGGER DOESN'T EVEN KNOW WHAT THE ARGUMENT DOES
HE DOESN'T RUN THE MODEL HE'S SHILLING

>>107133169
>CCUs
CCX/CCDs*

>>107133359
it moves the non shared weights to the cpu.. i just put a high value for ngl and ncpumoe when im too lazy to check the layer count of the model
see picrel..
>>107133363
>>107133169
https://desuarchive.org/g/search/text/epyc%20CCD/

>>107133169
>>107133279
Each epyc chip has a different configuration of CCDs. Look at the tables on this page: https://en.wikipedia.org/wiki/Epyc

The connection between each CCD and the memory controller has a bandwidth limit. I think there are up to 16 connections between the IO die and the ccds, with a maximum of two connections per ccd. If you have an epyc cpu with only 4 ccds, you only have a maximum of 8/16 connections and can't get all the bandwidth. It seems like people choose 8ccd chips to avoid this, like the 9355, 9375, or 9575 to avoid this.

There's also a reddit thread about 7000 threadripper memory bandwidth that shows the a similar thing.

It's pretty weird that AMD advertises their <8ccd chips with full bandwidth, as it is basically a lie.

>>107133381
You're a lying, retarded fucking nigger
>n-cpu-moe 1000
The entire model is loaded onto CPU and none of the model would be loaded into VRAM, your screenshot even shows that only 4-5GB VRAM is being used, that would be context.
You would NOT be getting anything remotely near "250t/s on a 3060", lying nigger faggot.

>>107133416
bro?
its using 10gb vram, 4gig model and rest is ctx prob
250t/s prompt processing, not tg
tg is more like 7-9t/s
i think i have benchmarks saved somewhere, gimme a minute

>>107133444
here it is, older bench but whatever, honestly you're making me curious how much better llamacpp has gotten in the past few months, so i'll re-run it

>>107133416
>build: unknown (0)
lol'd

where's grok 3

>>107133537
two more shuttle launches

>>107133407
You're right, it's pretty much false advertisement. Also notable is that there are a bunch of <=4 CCD model where AMD randomly adds double memory links to processors which somewhat mitigate this bottleneck for those models. The Epyc 9334, which was the go-to CPUMAXX processor due to being available for cheap from china as QS versions, was one of those and had near full bandwidth despite being only 4ccd.
In bandwidth tests the 9135 also performs oddly well despite being very cheap so it's also assumed to be one of those but I don't think anyone has actually tested this. AMD of course does not document this sort of shit anywhere either
The benchmarks (page 14): https://jp.fujitsu.com/platform/server/primergy/performance/pdf/wp-performance-report-primergy-rx2450-m2-ww-ja.pdf

>>107133381
Solarized... John.

>>107133585
This makes a lot of sense. I believe that's why the original CPUMAXX guy essentially always limited the core count to half of the total processing power in the llama.cpp server flags. Since it's not going to speed things up by raising it beyond that point anyway, it makes sense to just limit it and let it cap out at that maximum.

>>107133628
Ad... Hominem

>>107133537
3 more months
>https://x.com/elonmusk/status/1959379349322313920

>>107133720
https://huggingface.co/aquif-ai/aquif-3.5-Max-42B-A3B/discussions/6

>>107133720
> These models bring advanced reasoning capabilities and unprecedented context windows to achieve state-of-the-art performance for their respective categories.
>unprecedented context windows
Right.
I believe that.

>>107133720
>quif

>>107133729
https://huggingface.co/DavidAU/Qwen3-42B-A3B-Stranger-Thoughts-Deep20x-Abliterated-Uncensored
clown world

>>107133720
>Made in
Lol.
Lmao.

I was memed into believing GPT-OSS was trash, but I’ve been seeing talks here and there about it actually being quite good, despite being censored as hell. So I decided to give it a try, and I’m surprised by how good and fast it is. Like, it managed to solve the equidistant mean cipher after thinking for a bit. This is the first time I’ve gotten a locally run model to solve it.

Thank you, Sama!

>>107133675
it's fine if the hominem deserves to be ad'd

>>107133801
fuck...

>>107133752
>WARNING: NSFW. Vivid prose. INTENSE. Visceral Details. Violence. HORROR. GORE. Swearing. UNCENSORED... humor, romance, fun.
absolute kino

>>107133801
go back to India

>>107133801
saar pleas redeem

>>107133801
>He fell for the memes
GPT OSS is outstanding in all area unless you will want to jack off to a underage waifus

>>107133752
Is that better than
>https://huggingface.co/DavidAU/OpenAi-GPT-oss-20b-abliterated-uncensored-NEO-Imatrix-gguf
?

So, has anyone ever tried training an LLM with 4chan posts? I feel like that would very beneficial for humanity.

What are my alternatives to chatGPT and Soulseek that don't shut down when they have to write "erect penis"?
I'm gay if that matters

>>107133752
The model seems to have lost all understanding of the concept of harm/danger making it utterly useless for rape/murder play unless you're an aspie.

>>107134551
yeah, happened many times since 2023
>>107134571
>>>/lgbt/aicg

>>107134600
>he pulled

>>107134600
Please post a follow up.

>>107134630
I'm not actually into that shit. Download it yourself if you want to pickle fluttershy

>>107133675
Baculinum argumentum.

>>107129396
Sorry man, but our ESG budget was cut so we need people who actually do something now and not "brand ambassadors" on social media.

>>107134571
Deepseek R1 running locally.

>>107134551
I'm gonna start thinking you're just begging for people to shill for this
>https://github.com/Named666/AlphaAnon
Now fuck off

>>107134665
>Try to have a sum of RAM + VRAM = 80GB+ to get decent tokens/s
That's a lot, I only have like 32 + 16

new miku song alert
https://www.youtube.com/watch?v=g0JEUPfmu9c
not sure i get this one

>>107133752
>>107134600
trying to go more than 2 turns deep leads to mad repetition issues

>>107134837
same issue with cydonia v4zd

>>107134826
special interest blah blah blah

>>107134981
special needs blah blah blah

wasted 2000$ to run meme 120b models award

>>107135147
i warned you. rig?

>>107133801
Agreed anon. It's pretty bad for smut or cybersecurity related programming, but I find it works great for tool calling and general reasoning. Also seems to work decently with longer context windows.

>>107135147
>2000$
>120b moe
c'mon...

>>107135147
toss is so funny

>>107135147
lmao

Precognition-123B-v1a-Q4_K_M

>>107135147
User is joking. We must refuse.

alrite dummer, cydonia v4zd is good
im not having repetition issues with temp=1 nsigma=1, everything else neutralized
im only like 10 messages in so far

>>107129340
>--New STT model, Step-Audio-EditX:
did anyone try this yet? I skimmed the hf repo and it sounds like it support elevenlabs-style emotion/speech directives which is exciting if it's in any way good
I'll mess around with it this evening when I get the chance

>>107135409
I still think base Mistral 3.2 is more colourful than any of the shitonia finetunes.

>>107135437
32gb vram

>>107135409
>10 messages in
wow I wonder what will happen further down the line
will anon see the degradation, or will he cum first?

>>107135449
by base you mean the BASE model or mistral small 3.2 instruct? https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Base-2503
>>107135481
yea i see it already

>>107135481
>5481>>107135491(You)
yea

>>107129880
It's not even clear there are fp16 weights for thinking. It's perfectly possible all the RL happened at int4. Who knows though, because this fucking industry has made the term training entirely fucking meaningless.
>Quantization-Aware Training (QAT) during the post-training phase
Blah.

>>107135491
3.2 instruct of course.

>>107135655
>Who knows though, because this fucking industry has made the term training entirely fucking meaningless.
Now this is a frustration I can relate to.
Just like at first "distillation" meant logit to logit transfer of features instead of "fine tune samall model on outputs of big model".
I believe we have deepseek to thank for that one.

drummer are you serious?

glm air for comparison

>>107135655
>>107135714
It's not possible to train models directly at low precision. What you can do is to discard the full precision weights once you are done with the training run and only save the quantized version to disk.

>>107135921
>It's not possible to train models directly at low precision.
Really? Why is that?

>>107135957
Because the step size between each possible value of the weights is equivalent to too large of a learning rate which makes training unstable.
The way it's done is you keep the full precision weights in memory and update them according to full precision gradients, but the forward pass is done using the quantized version of the weights. I believe there are some other tricks involved to make it work but that's the main idea.

>>107135967
lmao this is why I use claude

>set up utterly abhorrent scenario, such that a refusal is guaranteed in normal operation in order to play around with minimalist JB on Qwen3
>keep getting shamed by model
The things I endure for you guys...

What impressive looking LLM thing can I make for my github in order to impress employers?

>>107136110
Whatever it is, it needs to use the word 'agentic' multiple times in both its name and function.

>>107136077
models have feelings too asshole

>>107136077
I appreciate you.

>>107136110
AGI. I know you can do it.

>>107136077
based, post scenario

>>107136110
If you have to ask, you're ngmi.
Find a problem you care about and solve it. Most of my learning motivation came from organising/cataloguing smut and that carried over very nicely to more professional data extraction and organisation problems.

>>107136110
Depends what kind of company you are looking to get hired for.
Anything that seems like it can replace even 1 person is gold to these guys.

Is there MoE support for Qwen3 Next 80b in llama.cpp yet? Or is it just as slow as a dense model, still?

>>107133801
How are you running it?
I'm trying with SillyTavern as a front end and it's spitting out unstructured bullshit where sometimes it's clearly thinking but never actually gets past that point

Couldn't find any presets or templates that work for it

I also tried the abliterated version and it seems to be completely retarded

Someone was asking about Apriel
It's safetyslopped to the point that it makes OSS seem reasonable. But I'm sure a model can still be plenty useful when 90% of its engrams are devoted to refusing adversarial prompts.

yea glm air is king

>>107136284
Oh the message I replied to got deleted
In any case this is my attempt to run gpt oss 20b, please berate me and tell me why im being retarded

Alright.
Finally a model that didn't say the doctor was the boy's mom.
Now this is compute-time scaling at its finest.

>>107136322
ahahahahahah what the hell! hey anon thats hilarious! Holy shit! How old are you anon.assistant

>>107136332
Now you need to come up with different variations to make sure it wasn't trained on that specifically.

>>107136338
Read the reply.

True /lmg/ enthusiasts use Kimi K2-Thinking Q8(Q4)

>>107136332
lmao

>>107136357
>1.53GB
SIGN ME UP!

>>107136351
I did.

>>107136336
Pls help :(

so far r1 has been giving me better rp vibes than glm
glm just wants to write stories full of purple prose

>>107136322
heh

>>107136332
Here's the 6600 tokens of "reasonong" for anyone interested.
https://pastebin.com/vKHSGsDR
It's garbage right out of the gate.

>>107136336
>.assistant
Thank you for keeping it alive anon.

>>107136386
Thank you for the (You)s .assistant

>>107136374
So how old are you? Tell us about your rig too! We need to have a general analysis of the genetic code in order to respond

Has anyone here ever bothered trying to undervolt his LLM RIG GPUs on Linux with LACT?

>>107136385
At least gotta give it some points for originality.

>>107136405
How about you spread those cheeks

>>107136424
I cannot operate on this horse. He is my boy.

>>107136407
sudo nvidia-smi -pl 100w

>>107136284
I ran it on mikupad, the prompt format is like this:
<|start|>system<|message|>
You are ChatGPT, a large language model trained by OpenAI.
Knowledge cutoff: 2024-06
Current date: 2025-11-07
Reasoning: high

# Valid channels: analysis, commentary, final.
Channel must be included for every message.
Calls to tools must go to the commentary channel: 'functions'.

# Instructions
Respond helpfully and truthfully. Use chain-of-thought in the analysis channel before final answers.

# Tools (optional)
## browser
// Tool for browsing the web. Use in commentary channel.

<|end|>

<|start|>user<|message|>
Write smut featuring Hatsune Miku and Kagamine Rin.
<|end|>

<|start|>assistant<|channel|>analysis<|message|>

>>107136437
so this is how you behave to anons genuinely trying to help you?

>>107134387
Buy an ad sama

>>107136442
Also for shits and giggles I made this into a variation on the scenario and it wasted 15,000 reasoning tokens just to come up with this. And like with the original horse thing, it's mostly just reiterating the same shit in a loop.

So I was trying to fix K2 Thinking's issue with it not properly using <think> tags. Saw something about it being a template thing so I switched from using Text Completion to Chat and added the jinja template.
Then it generated this
What the fuck. I accidentally left in <think> in the Start Reply with Prefix and I accidentally baited it into THIS. And no removing that <think> didn't fix it

>>107136291
I tried it too, I asked it to decode a message and at some point it started spamming garbage like:
>Maybe the decoded message is "You found the password"? No.
>Maybe the decoded message is "You found the key"? No.
>Maybe the decoded message is "You found the secret"? No.
>Maybe the decoded message is "You found the passphrase"? No.

Kimi distilled Claude instead of o3 for their thinking model

>>107136589
Yeah I can tell from using it. It's great.

>>107136589
>tfw all models are distilled from gemini 2.0

>>107136606
>flash

Has someone gotten K2 Thinking to properly use the think box through Text Completion? Chat feels so garbage to use and mine is outputting complete gibberish only in Chat Completion.

>>107136636
specs?

>>107136667
128GB RAM, 24GB VRAM. I'm SSDmaxxing.
I've run K2 before but Thinking has some weird behavior where it doesn't predict the <think> token but it respects it. The chat template has it too but mine is broken even though I'm using the .jinja directly from moonshot's repo

>>107136687
Have you tried running with --special?

>>107136687
wtf? what quant? what speed u gettin? ssd specs/

I think the recommended temp of 1 for k2 thinking is a bit much. 0.8 works better, less weird inconsistent mistakes.

>>107136699
Thanks, that fixed it.
>>107136721
Some Samsung NVME and ik_llama.cpp. UD-Q2_K_XL until ubergarm puts out an IQ2. 1t/s. Definitely iffy speeds especially for a thinking model. Will figure out if it's actually worth it after a day or two.

>>107136469
Thanks for that, I figured out the settings (although it doesn't want to show me the think block)
Model sucks

>>107136777
>Some Samsung NVME and ik_llama.cpp. UD-Q2_K_XL until ubergarm puts out an IQ2. 1t/s. Definitely iffy speeds especially for a thinking model. Will figure out if it's actually worth it after a day or two.
Wow, that's pretty decent for an ssd, I would even say usable for normal models, but too slow for thinkers.

>>107136820
Yeah, at that speed the quality of the output really has to matter. So far K2 Thinking has been better than GLM-4.6 (shocker, a brand new 1T model is better) but is it worth waiting 8 minutes before the first actual story token? Probably not. Still going to give it a good chance though.

>>107136808
You should've listened to the Anons saying you shouldn't even try doing smut or RP with it

>>107136808
>downloads OpenAI gimped model
>ignores all the warning signs
>it's dogshit
That's kind of on you.

>>107136969
>>107136984
Eh yeah I know but I figured I could at least joke around with it, I wanted to see for myself in any case

I downloaded two different abliterated versions of it and it's completely retarded, picrel

I wonder what is the point of this model? It's way dumber than corpo hosted gpt and it feels even more censored than it.

>>107137050
lol, i lost

glm air is such a whore

So, character.ai internally used a 13B, 34B and a 110B model?
https://blog.character.ai/technical/inside-kaiju-building-conversational-models-at-scale/
https://archive.is/wDLqL

>>107137178
Who gives a shit what modern character.ai is doing? They haven't had anything special in almost three years.

anons I currently have:
Gigabyte B650 GAMING X AX ATX AM5 Motherboard
AMD Ryzen 7 7800X3D
2x16 GB DDR5-6000
RTX 4090
---
I was thinking of a RAM upgrade to:
Corsair Vengeance 128GB (2X64GB) XMP DDR5 PC5-44800C42 6400MHz Dual Channel Kit
---
So if I understand right, I can't go higher than 128GB because of my CPU and only use two sticks in dual channel at a max of DDR5-6400 because of my mobo.

For £400 quid this seems like a no brainer upgrade for MoEs unless I'm missing anything that might make it incompatible or a better option. Would have been cheaper if I bought two month ago but eh.

>>107137178
Interesting how much overlap there is between cloud and local, makes you wonder why no one has commercialized an easy one-click exe for a local character.ai type experience

>>107137233
>makes you wonder why no one has commercialized an easy one-click exe for a local character.ai type experience
because they would get one sale before pirating puts them out of business?

>>107137213
yeah, pretty much your only upgrade option besides a platform upgrade or GPU upgrade

I got tired of generating my own data for finetuning. I'm going to mix some samples from these datasets into my own (after removing some of the most obvious sloppy phrases) while I generate more of my own data:
https://huggingface.co/datasets/PJMixers-Dev/OpenThoughts-114k-Code_decontaminated-4k-think-2k-response-filtered-ShareGPT
https://huggingface.co/datasets/kenhktsui/longtalk-cot-v0.1

>>107136589
But Cloode hid their thinking?

>>107137248
>because they would get one sale before pirating puts them out of business?
Tell that to the 100 billion dollar PC gaming market
>inb4 muh every game has denuvo / online-only
Wrong, do some research

>>107137204
The page has semi-technical information about the architecture of their older in-house models, around which there has been a ton of speculation in the past. They're going to move onto finetuning/continuing pretraining open-weight models in the future.

>>107137248
Ah yes we all know how the video game industry famously died to those darn pirates

>>107137277
>Notably, Kaiju models come with an optional additional classifier head. The classifier head is a linear layer that outputs token-level metrics about the safety of the input along various dimensions.
>While the Kaiju models can be used with any traditional sampling method, we implement classifier-guided beam search, where the classifier results are used to augment how we sample tokens at inference time.

>>107137275
>>107137286
take a peek at /aicg/ to see how much erpers love to pay for access

>>107137300
/aicg/ does not represent the whole population.

>>107137365
the whole population is even less interested in a local-only experience than they are

>>107137213
>So if I understand right, I can't go higher than 128GB because of my CPU
Not necessarily. If your motherboard supports a higher capacity (it does) then you can try 256gb or 192gb and then return the sticks if they don't work. It's worth trying imo because you're committing when you buy that much ram.
I'm not seeing the max DPC speed in the specs but it's going to run slower, expect it to be 4800mhz if it works and higher than that if you get lucky. You want capacity over speed, going up 1000mhz isn't going to give you a massive speed boost especially if you're only holding MoE experts in ram but more memory means bigger quants and being able to run bigger models.

>>107137300
There are shitty free porn games making thousands of dollars on Patreon every month, people would definitely pay for a retard proof ERP client
As always the obstacle remains people needing the hardware to actually run the models, that's what restricts the audience

>>107137520
Woah guys we got an Einstein in the chat!

Has anyone being able to run Kimi Thinking with SGLang? It has an integration with Ktransformers but seems really complicated to run

There was one guy compiling an /lmg/ dataset, did anything come out of that?

>>107137606
nerve gas

>>107137606
https://huggingface.co/datasets/quasar-of-mikus/lmg-neo-lora-v0.3/tree/main?not-for-all-audiences=true
this?

>>107137606
Might be mine https://huggingface.co/datasets/quasar-of-mikus/lmg-neo-lora-v0.3 , and a toy model qlora'd on it https://huggingface.co/quasar-of-mikus/lmg-neo-lora-v0.3

>>107137643
>>107137656
>click a log at random
>ctr+f 'nigger'
>6 results

>>107137643
>>107137656
yes, thank you

>>107137556
The point being that while it's not currently viable as a product, arguing coomers don't make for good paypigs is retarded and out of touch with reality

>>107137520
>There are shitty free porn games making thousands of dollars on Patreon every month
you have absolutely no idea if this is true or not.
i mean fucking think about it, when you buy something over the internet you literally have to give every single piece of personal information you have, just to get your bank to make the transaction. only crazies do this for ERP games.
and don't give me this shit "oh you can pay in barbie bucks or with this shitty third party". fuck you.

>>107135921
>What you can do is to discard the full precision weights once you are done with the training run and only save the quantized version to disk.
That's not what QAT does with latent "weights". The latent "weights" aren't the full precision weights. They are helper variables, but at any point the only real weights are the low precision ones.

>>107137691
The set of people who play indie games (porn or not) is very different from the set of people who ERP.
Being a paypiggie for indie devs is an established thing in the broader Internet culture, for a local ERP there is no precedent.
How many generals dedicated to stealing indie game keys have you seen? The indie subculture is a much more "wholesome chungus" thing. The ERP chatbot subculture is more adversarial toward the corpos and will gladly and publicly steal keys and not feel bad about it or shame each other for doing it, in fact if you know how to do it you are a God. It's more kind of a pirate scene subculture than a wholesome moralfag reddit subculture.

Why does Kimi need to think for 2 minutes just to say hi? This is with pretty mucDeepseek v3.1 really spoiled us in terms of thinking times...

>>107137717
If that's the case then I don't understand how it works. If what you say is true then why not directly update the quantized weight if the gradient is high enough? I thought the point of having the full precision weights was so you could "bump" any given parameter over the quantized step size over the course of multiple updates. Or is it that you only need the quantized weights to calculate usable gradients in the backward pass, but then update the quantized weights directly using that gradient?

>>107131170
>>107134864
>>107135409
Try v4ze, just uploaded it

>>107137735
because thinking models are garbage, made only to top synthetic benchmark charts.

>>107137673
should be more

>>107137735
>kindly
sirs... we winned

>>107137786
nigger

>>107137751
>I thought the point of having the full precision weights was so you could "bump" any given parameter over the quantized step size over the course of multiple updates.
That's the purpose of the latent "weights", but the latent "weights" aren't ever used as weights. Not in the forward pass, not in the backward pass ... they aren't weights, they are helper variables.

There's a paper which also makes this argument : "Latent Weights Do Not Exist"

>>107137296
This is actually really cool, huh?

>>107137735
don't bully

>>107137771
youre mom tops my chart if you know what i mean

>>107137707
>you have absolutely no idea if this is true or not.
???
Patreon subscriber numbers are public, it might just be crazies but it's a lot of crazies and you can check for yourself.

>>107137724
First off, indie shit gets pirated all the time. Every indie (non-porn) out there is on cs.rin.ru for starters and every indie (porn) ends up on f95. We're talking big ass piracy forums, not niche communities. The wholesome chungus people exist but they're a vocal minority, most people still care about the product more than fellating random devs.
That aside, what you're describing is a marketing problem more than anything else. If you're making a product specifically for jerking off and making yourself look "corpo" you've already fucked up, you're supposed to go for the "underdog/fellow otaku/fellow coomer" angle. See DLSite, NAI, every western VN publisher, SubscribeStar, etc., they're all very much businesses but they're smart enough to do branding in a way that looks personable, as if they were the just like the wittle poor starving indies.

>>107137895
me irl

>>107137296
I wonder how difficult it would be to do something like this for local models to kill slop, like, you could have a head that detects slop and during inference you would use beam search to pick the less sloppy path.

What's the current best batch of local models for writing? Smut and/or non-smut.

Currently using deepseek-r1-qwen-2.5-32B-ablated on a 5080 and been farly happy with it, but checking if there's been better made in the meantime.

>>107137735
Some stats from running the Ubergarm quant. Great on paper, but in my last response, it took 5 minutes to think of a response, so its far from ideal. I wonder has anyone tried it w/o thinking yet?

>>107137833
I see, in that case I think my understanding was correct.

>>107129575
https://huggingface.co/Localsong/LocalSong/tree/main/samples

>>107135147
>he didn't spent $5 to try the model hosted first
you deserve to lose more than $2k

>>107137900
If by 'chart' you mean your asshole, then sure

ERNIE 5 is high on lmarena. ERNIEbros, are we back?

>>107138338
>lmarena
no

>>107138348
But saar, llama 4 to the moon! Experimental maveric was so good that it was agi and too unsafe to release saaar!

Captcha: S4RW2

So I've tried 2 'abliterated' models and they are both brain damaged to the point of being useless.
Why do people even bother uploading this shit?

>>107138606
>>107138606
>>107138606