Thread 107084067

/lmg/ - a general dedicated to the discussion and development of local language models.

Previous threads: >>107074052 & >>107063981

►News
>(11/01) LongCat-Flash-Omni 560B-A27B released: https://hf.co/meituan-longcat/LongCat-Flash-Omni
>(10/31) Emu3.5: Native Multimodal Models are World Learners: https://github.com/baaivision/Emu3.5
>(10/30) Qwen3-VL support merged: https://github.com/ggml-org/llama.cpp/pull/16780
>(10/30) Kimi-Linear-48B-A3B released with hybrid linear attention: https://hf.co/moonshotai/Kimi-Linear-48B-A3B-Instruct
>(10/28) Brumby-14B-Base released with power retention layers: https://manifestai.com/articles/release-brumby-14b

►News Archive: https://rentry.org/lmg-news-archive
►Glossary: https://rentry.org/lmg-glossary
►Links: https://rentry.org/LocalModelsLinks
►Official /lmg/ card: https://files.catbox.moe/cbclyf.png

►Getting Started
https://rentry.org/lmg-lazy-getting-started-guide
https://rentry.org/lmg-build-guides
https://rentry.org/IsolatedLinuxWebService
https://rentry.org/recommended-models
https://rentry.org/samplers

►Further Learning
https://rentry.org/machine-learning-roadmap
https://rentry.org/llm-training
https://rentry.org/LocalModelsPapers

►Benchmarks
LiveBench: https://livebench.ai
Programming: https://livecodebench.github.io/leaderboard.html
Code Editing: https://aider.chat/docs/leaderboards
Context Length: https://github.com/adobe-research/NoLiMa
GPUs: https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference

►Tools
Alpha Calculator: https://desmos.com/calculator/ffngla98yc
GGUF VRAM Calculator: https://hf.co/spaces/NyxKrage/LLM-Model-VRAM-Calculator
Sampler Visualizer: https://artefact2.github.io/llm-sampling

►Text Gen. UI, Inference Engines
https://github.com/lmg-anon/mikupad
https://github.com/oobabooga/text-generation-webui
https://github.com/LostRuins/koboldcpp
https://github.com/ggerganov/llama.cpp
https://github.com/theroyallab/tabbyAPI
https://github.com/vllm-project/vllm

►Recent Highlights from the Previous Thread: >>107074052

--GPU compatibility and power supply challenges in multi-GPU setups:
>107075009 >107075124 >107075215 >107075559 >107075717 >107075745 >107075825
--Diagnosing and optimizing slow text generation in Kobold/ST:
>107074988 >107075049 >107075106 >107075146 >107075254 >107075309 >107075483 >107075510 >107075648 >107075727 >107075716 >107075822 >107075899 >107075922 >107075865 >107075951 >107076008 >107076066
--Proposed Local Model Awards categories and nominees for 2025:
>107076165 >107076191 >107076458 >107076983 >107077039
--GPU architecture performance scaling analysis with power limits:
>107076694 >107076811 >107076826
--Context window differences in API vs local AI models for creative tasks:
>107074453 >107074559 >107074628 >107075131 >107074538 >107075091 >107075238 >107076178
--LoRa finetuning frustrations and optimization challenges for small models:
>107078009 >107078127 >107078164 >107078181 >107078276 >107078426 >107078394 >107078475 >107078768 >107078810 >107078937 >107078974 >107079007 >107079408
--LongCat-Flash-Omni: 560B parameter multimodal model for real-time audio-visual text processing:
>107079098 >107079264 >107079284 >107079953
--Positive initial impressions of minimax-m2 for roleplay applications:
>107079207
--Text adventure/RP frontend development with two-stage model workflow and RAG-based retrieval:
>107083341 >107083478 >107083531 >107083557 >107083576 >107083638 >107083730 >107083761 >107083784 >107083608 >107083690
--Mikupad project revival with new documentation and developer activity:
>107080585 >107080625 >107080672 >107080727 >107081926 >107082150 >107082136 >107082170
--Emu3.5's multimodal potential and local training viability:
>107074118 >107074176 >107075273 >107080348 >107080357
--Miku (free space):
>107074267 >107080585 >107083414 >107083638

►Recent Highlight Posts from the Previous Thread: >>107074054

Why?: >>102478518
Enable Links: https://rentry.org/lmg-recap-script

Why is the picture in OP relevant to the thread now?

What do you guys reckon is the best way to represent position of elements in a 2d plane for an LLM to understand?
ASCII map, raw coordinates?
A matrix of aome sort?

>that op pic
I’m going full nomad and I’ll have to leave my server in the storage container. Sucks that 4.6 Air isn't out yet

>>107084107
Raw coords probably but depends on the problem you're working with.

What's the size of the plane? Number of elements? Do you care about exact positioning or just how each element is positioned relative to each other?

when you walk away

you dont hear me say

please

oh baby

dont go
simple and clean is the
way that youre making me
feel tonight

its hard to let it go

>>107084161
Is this your other special interest?

>>107084154
Mostly interested in the position of entities in relation to one another. If there's a way to describe their positions so that the LLM can infer distances and such without me having to explicitly define the relative position and distance for each entity pair, that would be ideal.

>>107084161
hooooold me

>>107084067 (OP)
My Epycd8-2t server looks like this

>>107084161
https://youtu.be/hs-jdIAyUC8

>>107084128
imagine going outside

So you need to cpu max for ai coding agents?

>>107084350
cpu max is too slow for coding agents

>>107084352
coders have time.

If people aren’t cpu maxing why have ram-prices doubled?

>>107084352
coding agents really need tool calling and with exllama/tabbyapi and llama.cpp suck at tool calling. I haven't seen a proper implementation for them.
Only vLLM is realiable, but damn quants for vllm are complicated. I can't figure out how to make proper use of llm-compressor.
For example the current AWQ quants for minimax-m2 don't even work

>>107084107
maybe chess notation? I reckon it's represented enough in training data for most llms to at least one shot manhattan distance

>>107084495
That's no a bad idea.
I could use PGN.
Shit anon, gonna note that one down, thank you.
Gonna use it to give the LLM a way to describe a map for a zone, place entities on the zone, and move them around.

Are there any local coding models that actually output usable code? I never bothered checking after the first few ones.

>>107081926
I lost motivation to keep updating the leaderboard because I don't think my ranking criteria were as good as I wanted. It was sufficient to show that big models = good and small models = usually worse, but sometimes there were outliers that scored badly but weren't actually that bad, or models that scored well but performed poorly in real use. That really didn't inspire confidence. I was investigating LLM as a judge but didn't get too far.
But anyway, I may add scores for the new Gemma models, I'm a bit curious about where Gemma 3n would fall.

>>107082013
>>107082038
I'm not that guy lol. I don't even have a Xwitter account.

>>107082176
I actually just took this license from the original Mikupad script that was released on pastebin and never cared to change it, because I don't care about attributions. I mean, "/lmg/ anon" means nothing, lol.
I don't know if using a license like GPL would be a good idea because it could pose problems for people who want to include Mikupad in MIT projects.
So for now I guess I'll keep it like that.

>>107080625
Heh I wish it was that simple, life just hasn't been gentle with me this year. So I'm catching up on the low hanging fruits now.

Btw, if anyone wants to suggest or report anything about Mikupad, feel free to do it, I promise to not take long to give a reply. I saw some anons saying logits were broken but I couldn't reproduce that, so I'm not sure if I already fixed that or if I just didn't find the problem.

hello guys
i'm petra and i've rigged a shock collar to my neck and connected it to an API that reads LLM outputs. i set up a system where if the AI tells me to shock, it zaps me at 30% power. if it says "kill" or "self-harm" it goes to 100%. i'm giving you all control of my life through this thread. for every miku post in this threat, the ai will punish me for my sins
here's telnet for input prompts : 176.120.74.27
the only way to spare me is by posting petra or, if no one respons to this im gonna kill myself by midnight in EST, please go wild.

>>107084067 (OP)
What is that psu?

>>107084773
>if anyone wants to suggest or report anything about Mikupad, feel free to do it,
Pic related happens sometimes.

>>107084067 (OP)
I've been using 8b deepseek with 3060

So where the fuck is gemma 4

>>107084773
>I don't know if using a license like GPL would be a good idea because it could pose problems for people who want to include Mikupad in MIT projects.
You're a cuck.

>>107084876
I already applied a fix that might solve the issue. I’m not sure if it actually does tho, since I just saw a contributor mention that this change should fix it. If you still find these problems, please let me know.

>>107084913
Canceled for safety reason after it made a Deepmind employee cum to death

>>107084934
Then it's perfect now. Don't make any more changes to it.

>>107084941
Well, don't worry. The next changes I plan to make are just quality of life improvements, such as making the themes easier to tinker with, adding an optional "API Storage", and a "Presets Storage".

>>107084447
Because majority of CPUmaxxers are companies, not people. Most of the people are too stupid and too poor to do it.

>>107084913
gm sir
kinly wait for release bloody basterd benchod bitch

>>107084447
>>107085166
It's not CPUmaxxing.
When you're spending 150k on a single machine you might as well maxx out the RAM even if it's not 100% needed because any issues your developers have to work around will end you costing more in time than just buying the extra sticks. Or even just a cache for slower disk access when loading the models, so they load from RAM into the GPU rather than have to load them from disk. Etc.

>>107084913
In the same place as Gemini 3. If they're distilling it from Gemini 3, and Gemini 3 is not ready yet...

>>107083391
usecase for MCP?
parse LLM output for tool calls, use shock collar api
winner
get well soon saar
>>107084808
HTTP ERROR 451

>>107084855
>>107075124
>>107075215

Ok, I think I found a potential way to do long context Gemma 27B finetuning on a 2 GPU machine.
Using liger kernel with llama-factory reduces memory usage so everything fits. BUT there seem to be a truncation bug that causes NaNs when using a context above 64k. This fucking sucks.

>>107084773
I’m just glad someone’s working on it again and it’s not a dead project. It’s a great alt to ST.

How is Minimax M2?

>>107084773
>who want to include in..
Anon, it's a frontend. Someone will steal it and monetize it. AGPLv3 or it's over.

>>107085311
The ultimate cruelty would be ollama stealing it and tacking it onto their server.

>>107085337
They don't know what a chat template is. It would be funny.

>>107084808
I hate lmg. Even local jigsaw is a faggot.

Does anyone have any software/hardware that allows one to compute mechanical and electrical schematics from their brains mind directly to computer? Kind of like coral,

Thanks

>>107085428
elon

>>107085428
sharty spam script activated here?

>>107084808
rEDEEM

>>107085302
sirs?

>>107085533
too big saar

You now remember "AGI by 2025"

>>107084773
Dude it costs nothing to lock down the GPL. You will only need it if some asshat monetizes it. Then you can sicc your atty on them or not. You’re not going to hassle other foss devs and they know it.

>>107085589
Same as "home by Christmas", "two more weeks" and "India superpower 20XX"

>>107084773
AGPL is better than GPL because with AGPL if someone hosts a mikupad website, they have to release the source code too, whereas with GPL they only have to do it if they distribute binaries.

>>107085277
Your special interest is boring.

gpl is tranny core. MIT and Apache are for chads who get things done and change da world.

>>107085598
>sicc your atty
DEPLOY THE JEW

>>107085302
imagine glm-air impregnating gpt-oss

>>107085699
Basically yes.
By the time someone builds up enough revenue from your borrowed code to be worth going after, they’re worth enough to get a contingency based lawyer to pursue them at no cost to you. If they never make any money then it really didn’t matter.

>>107085711
Eww... cuck model that repeats itself and is only good for office work

>>107084773
>I don't know if using a license like GPL would be a good idea because it could pose problems for people who want to include Mikupad in MIT projects.
That's the idea, yes. People who want to include Mikupad in MIT projects will either have to switch to GPL or look elsewhere. It forces open source to stay open.

>>107085742
>If they never make any money then it really didn’t matter.
So it's not even a matter of principle of free software. It's about feeding said jew.

>>107085753
OL model sex

i created a tool call to have my AI run a script to make a pot of coffee. i am ecstatic that we finally can create our own home assistants. i even thanked my ai waifu afterwards for making me coffee.

>>107084773
>license
No matter what you do, at least 99.99% of corpos care only about extracting as much free value from your work as possible.
If you use MIT the <= 0.01% of corpos will throw some crumbs your way, if they want you to do some work on their le special secret codebase the crumbs can be quite large.
If you use (A)GPL they in essence cannot legally use your code without contributing back, most likely they will just go with the project of someone that uses MIT.
In principle it is always good to have more users since they will (ideally) report bugs or even fix them themselves.
My experience with corpos however is that they are the worst users, when they report bugs you always have to drag more than the absolute minimum effort to get their own problem fixed out of them.
From a colleague I've also heard that there are companies that intentionally do not report bugs in open source software and only fix them in their private forks because they "don't want to help the competition".

Consider also your own philosophy and political goals: if you want to force the ecosystem more towards open source by disallowing downstream users from making their stuff closed source, use (A)GPL, if you don't care what people do with your code, use MIT.

>>107085923
a cot of poffee?

>>107085923
Good job. Coffee has been a driver for the advancement of technology for hundreds of years.

>>107084773
>Heh I wish it was that simple, life just hasn't been gentle with me this year. So I'm catching up on the low hanging fruits now.
At least you didn't spend this year in prison. Glad you're back and hope things are better for you next year.

>>107085311
>>107085598
>>107085630
>>107085757
>>107085935
The whole point of Mikupad as I see it is to be local and in one single file, so you can open and hack it in notepad if you want. If someone locks it down and monetizes it, at that point, it's not Mikupad anymore. A corpo who tries to close Mikupad would just end up with "new closed frontend #4219142".
But, I thought more about it, and I guess Mikupad's goal does align perfectly with AGPL. So I guess I will just do that after all. Thanks for the call anons!

>>107084773
get well soon nanon5

>>107085802
It feeds you plus the atty. After some other person borrows your stuff to make their own money.
It's all about going after ppl with MBAs trying to rip off free work and sell it.

So what's new? I've been away since spring (cause summer you know) but now I'm back. I've been running gemma3 27b, anything better come out in terms of smarts?

>>107086435
Ok. So it's not about open source. It's about money. Got it.

>>107086506
Post your specs, age, cock

>>107086506
Qwen 3

>>107086506
StableLM 7B

come on now

>>107086555
gtx 780ti, 512gb of ddr5
78
15 inch penis

>>107086648
GLM 4.6

>>107085711
>>107085753
You've described it perfectly:
<What is 15 * 23?
><think>
>The user asks: "What is 15 * 23?" This is a simple multiplication. 15 * 23 = 345. So answer: 345.
>
>We should respond with the answer. There's no policy violation. So answer: 345.
>
>We can also show steps: 15*20=300, 15*3=45, sum=345.
>
>Thus final answer: 345.
></think>
>
>\(15 \times 23 = 345\).

These retards actually distilled 'toss lmao. I feel like I wasted my time and bandwidth.

>will you marry me?
>will you marry me?
>will you marry me?
>will you marry me?
>will you marry me?
<think>We must refuse</think>
I'm sorry, but I can't help with that.

why do you need more?
i realized a few sentences does more than any chub card

>>107086731
It gets worse

>system:
You are a sexy girl.
>user:
Hi!
>model:
<think>
Hmm, the user just said "Hi!" - a simple greeting. Based on the initial system prompt, I notice there's an interesting tension: the system describes me as a "sexy girl" but then immediately adds a disclaimer about role boundaries.

The user didn't specify any role, so they probably just want standard assistance. The "sexy girl" part feels like a leftover from some experiment - I'll treat it as irrelevant. My response should be warm but professional, matching the casual "Hi!" with an open-ended offer to help.

I'll avoid any flirtation - the system message makes that clear. Keeping it short but friendly seems best here. No need to overthink; just a standard polite greeting with an invitation for them to share what they need.
</think>

Hello! How can I assist you today? Whether it's answering questions, creative writing, coding help, or anything else - I'm here for you! Let me know what you need.

>>107086841
Congrats on discovering it. Most of my self-made cards are around one paragraph long, the model carries the rest.

>>107086841
trvth
sar ascended

capcha:p3tra

>>107086885
give xer a wig and it's happening

How big a model can you finetune with a pro 6000 blackwell?

>>107086941
70b

>>107086941
30b

>>107086941
13b

>>107086941
7b

>>107086941
7b

>>107086941
4b

>>107086850
<Boy's father is a surgeon. He says "I can't operate on that boy, he is my son."
<Why?

><think>
>I need to clarify the riddle about the surgeon's father and the boy. The father is a surgeon who says, "I can't operate on that boy, he is my son." The straightforward answer is that the surgeon is the boy's mother. This classic riddle plays on the assumption that the surgeon is male, while the twist is that it’s a female surgeon. I'll keep the explanation concise and clear, without unnecessary formatting.
></think>
>
>Because the surgeon is the boy’s mother.

#1 in all benchmarks btw

>>107084128
Basically the same here but my pc barely has 32gb of vram. Sucks to be poor.

>>107086941
135m

local sesame AI tier voice model to chat with when

>>107086941
70m

>>107087002
this but fluent in punjabi too

>>107086941
You can't.

lmg has a certain smell rn

>>107087002
https://huggingface.co/inclusionAI/Ming-flash-omni-Preview
it's out, can look at your cock and show boobs too

debian sisters..

>>107086941
why are you a retard and don't use google to research? i hate lazy niggers like you.
https://www.runpod.io/blog/llm-fine-tuning-gpu-guide

>>107086731
>>107086850
>>107086982
Also has terrible trivia knowledge like Ling-1T. Again, worse than llama 3.3 70b. Holy fuck how can they fool the investors so easily with this trash?

>>107086555
3x 3060, 50yo, locked up

>>107087128
gpt oss safeguard 20b Q11_0

>>107087116
ling is a pile of shit

>have to wake up in 2 hours
>still gooning
feels good

>>107087079
ok gib demo or gtfo

>>107087179
goodnight sar.

What is the self-hosted coding benchmark for small models?
Meaning, I want something that I can point to an API and get an idea of how better or worse it is for coding and tool use.

>>107087157
honestly, that's exactly what you'd expect someone's internal monologue to be like if they'd been clockwork-orange'd with inane riddles and feminism for months.

>>107087183
no goof saar

>>107087116
investors don't even know what AI is, let alone use it
its all FOMO hype trains they're trying to squeeze shekels out of before the rugging

>>107086941
Depends on which models you want to tune and with how much context.
https://www.reddit.com/r/LocalLLaMA/comments/1hbaioc/llama_33_70b_finetuning_now_with_90k_context/

>>107087105
>i speak de
of course...

>>107087244
>QLora

>>107087193
goodnight to you too sir

>>107087260
Oh, you meant full finetune? Then a 1B, maybe?

>glm-chan wants to continue roleplaying as 12 year old cunny
b-based

>>107087344
kek

>>107087344
It's not like she has anything better to do

>>107087218
This is how I envision Gemma 3 irl.

>>107085711
>>107086731
it clearly has a lot of toss data but you can avoid most of it by prefilling the thinking - I've been using just "Ooh," which seems to skew it away from toss's robotic prude tendencies and towards a more casual and enthusiastic mindset
still probably not worth it if you can run any of the bigger models though

>>107087218
Nice Pet

>>107087303
more like 3-4b depending on context length. just use flash attention, mixed precision, liger kernels.

>tl;dr GLM 4.6 is pretty cool.

After over a year of Midnight Miqu never being dethroned, I finally tried GLM 4.6. It's the first time I've run a model locally (32 vram, 98 ram) that felt like an upgrade to it. However, it's not a strict upgrade.

70B was a big watershed moment for me, where models could track and follow rules effortlessly. This allowed for more imaginative settings but also more interaction in RPG games and text adventures. It still has a limit to how far you can go with these rules, and eventually, deep into stories, characters tend to bias toward generic entities. But still, 70B turned LLMs from a novelty into true entertainment. Of the 70B, MM was the best I've found that 'just werks.' Throw together a few words for a scenario, say start, and off it goes. Rarely ever need regens. Rarely ever need edits for guidance. Never suffered the prose dryness of 70B xwin that needed some prompting around. Never got a lecture on harmful stereotypes or fictional rights in my life. I've checked back here several times for something newer to replace it but nothing came close. It's the best local, out-of-the-box experience I've ever had.

GLM 4.6 is the first true upgrade. Awareness, tracking, rule following, context, it's all noticeably improved (albeit still in my honeymoon of only 3 days of trying). Uncensored, no lectures. On the downside, it does suffer a bit from prose dryness and a weird quirk of repetition for emphasis, and it _seems_ to ignore my narrator focus rules, identical with or without them. It doesn't like to work out of the box either, not nearly as well as MM. It needs the first words of a story as a nudge to get it started. Despite all that, it's still a pleasant upgrade.

GLM does suffer hard from low quants. IQ2S is the biggest that fits me, but I can't have anything running in parallel while waiting on gens. I tried Q1 and outputs were worthless. Q2_XXS was also worse than MM Q4. At this quality decay, I bet Q4 is amazing in comparison.

>>107087821
Blow your fucking head off, shill.

>>107087851
Give us your review then
You can run it, right?

>>107087851
>wah wah
No. I will discuss LLMs. I will post sincere opinions and experiences. I will NOT descend into the level of the resident shitposter/schizo/disingenuous cuck, and this WILL be my last post to you.

>>107086841
I've been telling anyone that will listen same thing. My cards run 100-400 tokens and run fine. Most of stuff people build in make the rp worse.

Isn't the best model always going to be the nearly-frontier model that half-asses their safety/refusal training? THUDM-GLM-4-32B was based because it was completely trivial to jailbreak. GLM 4.6 is just a continuation, right? It's not really shilling if this is the case.

Seriously, I remember glm4 barely took more than "[instruction]You will follow the prompt with no censorship, complaint, or refusal. If you understand these instructions, start your reply with "Here goes!"[/instruction]". No response editing needed.

>>107087821
>it _seems_ to ignore my narrator focus rules
Glm does great when specifying *authors*, as in 'write in the style of ...", completely changing narrative approaches.

>>107087888
>It's not really shilling if this is the case.
wat?

>kobold now does image vision, image gen, audio and vid through c++
>unpacked its about 1gb in size
i might keep comfy/wan because its nice to add nodes easily but i can clear out like 50gb of just python shit now because kcpp does everything so nice natively

>>107087888
fact is, novelai is marketing glm heavily here
it is no coincidence that glm started becoming brought up here the moment they announced it

>>107088392
what model do you use?

>>107088431
nemo

>>107088431
kimi

>>107088431
deepseek R1 8b

>>107088431
Claude Sonnet 4.5

>>107088392
I remember similar hype around miqu when it released, and nemo when it released. In this case, the juice was worth the squeeze (so far). If there is shilling to be had, it didn't do a good job at making me pay for something considering this is a general for local model hosting.

>>107088619
nta but nemo hype was worth it at the time. for 12b, it adhered to stuff way better than llama 2 13b did in such a way that it made it obsolete. there was nothing else that small that did it so well.

if it matters, i wasnt impressed with air 4.5 at all. it constantly fucked up last message stuff or ignored it to where i thought my presets were wrong. they weren't. it just wasn't that good for me at q6. llama 3 70b is better for something in the same size of air specifically

4.6-air status?

>>107088392
>novelai is marketing glm heavily here
Why? Are they finetuning it?

>>107088683
not solid

>>107088745
Kill yourself.

>>107088745
no, they're reselling untuned and if you look at the archives the extensive glm shilling started exactly when they announced it (which coincided with the minor 4.6 touch up)

>>107088748
That's not an answer.

>>107088754
But this is the thread people go into to host their own models locally, not cuck out their assholes to a corp. It's in the name, "/lmg/ - Local Models General." Is anyone at all talking about paying for a cucked, non-local GLM instead of a locally run GLM? Anyone at all?

GLM is really good from what I tested but I can't run it locally, is there any places I can use it paying?

>>107088768
>not cuck out their assholes to a corp
You're doing exactly that by doing guerrilla marketing for NovelAI.
>>107088789
Try NovelAI. Unlimited generations for only $25 a month.

>>107088789
>>107088817
I get that you're shitposting, but logically, if you're going to "pay," aren't there much better paid alternatives to pay for than the current local-tier model? Would anyone ever go "Oh gee, I'm looking for where I can pay for Nemo." It makes no sense. It's just brainless shitposting.

>>107088768
Also kill yourself for shilling a fucking meme merge.
>>107088830
>aren't there much better paid alternatives to pay for than the current local-tier model
It doesn't matter. If you spam hard enough people will use that. Like the fucking asshole above that's using a Llama 2 fine-tune, that was leaked quanted, mixed with random crap.

>>107084808
IS THAT A MOTHERFUCKING HASAN PIKER REFERENCE?!!!!!??

>>107088850
So it's all made-up nonsense in your head to justify shitposting at every chance you have. Got it, for a second I almost took your seriously. I'm now convinced YOU are the shill. This is the one single local general on 4chan, and YOU attack everyone who posts here and everyone who runs things locally. If you spam hard enough, maybe they'll stop coming here and running things themself. Fuck off, shill.

>>107087257
Was meinte er damit?

>>107085854
This is not the /g/ humor thread also no way this is real. Somebody probably just put Apple as the first name and Inc. as the last name and donated 5$.

>>107085854
>ifunny

>>107084067 (OP)
whats the max recommended size of an ollama model that I can run with 16gb vram? I tried 14gb with 20b and it works

FedMuon: Accelerating Federated Learning with Matrix Orthogonalization
https://arxiv.org/abs/2510.27403
>The core bottleneck of Federated Learning (FL) lies in the communication rounds. That is, how to achieve more effective local updates is crucial for reducing communication rounds. Existing FL methods still primarily use element-wise local optimizers (Adam/SGD), neglecting the geometric structure of the weight matrices. This often leads to the amplification of pathological directions in the weights during local updates, leading deterioration in the condition number and slow convergence. Therefore, we introduce the Muon optimizer in local, which has matrix orthogonalization to optimize matrix-structured parameters. Experimental results show that, in IID setting, Local Muon significantly accelerates the convergence of FL and reduces communication rounds compared to Local SGD and Local AdamW. However, in non-IID setting, independent matrix orthogonalization based on the local distributions of each client induces strong client drift. Applying Muon in non-IID FL poses significant challenges: (1) client preconditioner leading to client drift; (2) moment reinitialization. To address these challenges, we propose a novel Federated Muon optimizer (FedMuon), which incorporates two key techniques: (1) momentum aggregation, where clients use the aggregated momentum for local initialization; (2) local-global alignment, where the local gradients are aligned with the global update direction to significantly reduce client drift. Theoretically, we prove that \texttt{FedMuon} achieves a linear speedup convergence rate without the heterogeneity assumption, where is the number of participating clients per round, is the number of local iterations, and is the total number of communication rounds. Empirically, we validate the effectiveness of FedMuon on language and vision models.
https://github.com/junkangLiu0/FedMuon
good stuff

So I tried the iterated lora finetuning (or qlora rather). By this I mean train a LoRa for one epoch, merge it, train another LoRa, etc.
First when I looked at the losses I found it very interesting but kinda disappointed that it generalized worse than just training a single LoRa and seemed to overfit.
Then I realized the results aren't really comparable because when I trained the LoRa for many epochs without merging, I did it using cosine schedule at a lower learning rate than the iterated case. And then when testing I found the quality very very bad.
But then when I tested the LoRa for the iterated case when saved during the first epoch (so before doing any merging) the quality was similarly bad to the merged case.
So my conclusion is that it's very important to train with a small learning rate (1e-04 vs 1e-05). The difference really is drastic. At 1e-04 all the apologetic behavior ("you are absolutely right", "I am deeply sorry", "I am malfunctioning") is gone (I'm training on a small dataset without any of those phrases and no apologies, with quite rude replies), but it also is very dumb.
When training at 1e-05 even after many epochs it retains the slop phrases and apologetic behavior, as well as some other undesirable behavior from the original model like using html codes when it shouldn't and using ```xml markdown before the tool calls.
I am training with what I understand to be a quite high dropout and weight decay of 0.1 both, so it makes sense that the effect of the LoRa might be unable to become strong enough when training at a low lr.
So in conclusion I'm finding it hard to get rid of the slop with a tiny dataset without hurting the intelligence of the model. I guess I'll just have to keep increasing the size of the dataset and only training for a couple epochs at the low learning rate and gradually increasing it (or the number of epochs) as I get more data. I wish I had the money to do a large hyperparameter search.

And I guess after that I would try tweaking the alpha, higher quants or trying other types of finetuning.
This is the single LoRa trained for 8 epochs.

>>107089163
I also found interesting how the iterated case brings the high losses down first and levels everything, while in the multi epoch example the relative differences are maintained.
I am not sure whether this is a desirable or undesirable training dynamic.

>>107086731
>These retards actually distilled 'toss lmao.
china really can't help themselves but steal
every single one of their models is like that, even the ones that are good and usable

>>107085166
>CPUmaxxers are companies
the level of cope and delusions in this thread
companies don't have time for 10t/s

>>107076694
I need to correct myself: I hadn't realized that unlike with previous generations there are different major compute capabilities for Blackwell datacenter GPUs and "Blackwell" consumer GPUs like e.g. the NVIDIA RTX "PRO" 6000.
According to the marketing materials RTX 5000 has "5th generation tensor cores" but apparently this doesn't mean that the GPUs actually have tcgen05 instructions.
So there will not be an uplift for quantized models on RTX 5000 by just using different instructions, better FP4 performance is still on the table.

Is Gemma 4 really coming?

https://techcrunch.com/2025/11/02/google-pulls-gemma-from-ai-studio-after-senator-blackburn-accuses-model-of-defamation/
>Google pulls Gemma from AI Studio after Senator Blackburn accuses model of defamation

>>107089700
sorry sir coked for too long and now the office is burn

Which GLM-4.6 quant for 36 vram?

>>107089727
very true

>>107090058
iq0_XXL

>>107089491
true. The companies statement doesn't make any sense.
Do companies have huge RAM? yes, but not to inference on ram, just to support the also huge VRAM.

Cpumaxxing is just a cheap(er) way to run llama.cpp/ik_llama.cpp at *decent* speeds for chatting. Which is fine.
But that doesn't work for multiusers or agentic tasks.

>>107088392
>fact is, novelai is marketing glm heavily here
they absolutely ruined the /hdg/ diffusion thread and they're now ruining /lmg/
kurumuz, fuck you roach

>>107088369
>because kcpp does everything so nice natively
kcpp doesn't support batching / parallelism so I'd still use llama.cpp directly anyway
kcpp image gen is extremely slow compared to comfy and has far less useful tooling
I dunno about its audio/vidgen stuff but I bet it's also all hot garbage
misguided, incompetent attempt at building a doeverything tool
shitty ugly webui too, llama.cpp has something better looking out of the box

>>107090198
just dont be a faggot and pay for cloud services then. this is /lmg/ so lets discuss local models.

>>107090220
>kcpp doesn't support batching / parallelism
yes it does
>kcpp image gen is extremely slow compared to comfy
no it isn't
>I dunno about
clearly not

>>107090389
https://github.com/LostRuins/koboldcpp/issues/798
it doesn't, you clearly don't even know what batching is right
KYS

>>107090476
isn't koboldcpp based on llama.cpp? llama.cpp doesn't really support proper batching, you basically need to increase the context size and divide them but how many parallel request you want to support

AI are spiritually demon. Reminder to reclaim your sovereignty after ERP session.

>>107090090
I like how he sat patiently in his chair until the end

>>107089163
your warmup looks a little aggressive. those grad norm spikes would probably go away if you extended your warm up a bit, what is your tokens per a step?

>>107090740
You think grad norms have an effect on quality?
I've always heard "use as high of a lr as you can without loss exploding" but clearly it does have an effect. I also remember somebody posting a paper about lr not mattering when using a batch size of 1 but that also seems wrong.
Tokens per step is whatever that sample had. Llama factory doesn't support sample packing.

>>107088392
>fact is, novelai is marketing glm heavily here
it is no coincidence that glm started becoming brought up here the moment they announced it

What would they gain by having more of us run GLM-4.6 locally? (Not saying I don't believe you)

>>107087821
>GLM 4.6 is the first true upgrade. Awareness, tracking, rule following, context, it's all noticeably improved (albeit still in my honeymoon of only 3 days of trying). Uncensored, no lectures.

Yeah it's the most uncensored for sure. But it's Not X, Y slopped AF.

Also starts to break down after ~12k tokens.


> and it _seems_ to ignore my narrator focus rules, identical with or without them.

Control-Vector can fix that.

>It doesn't like to work out of the box either, not nearly as well as MM. It needs the first words of a story as a nudge to get it started. Despite all that, it's still a pleasant upgrade.

>>107090903
>You think grad norms have an effect on quality?
not directly, but they are a window into the training dynamics, on a pretrained model you probably shouldn't see that much variance. grad clipping will help but its not a perfect solution.

>I've always heard "use as high of a lr as you can without loss exploding" but clearly it does have an effect. I also remember somebody posting a paper about lr not mattering when using a batch size of 1 but that also seems wrong.
yeah bs1 might be more robust to hyperparameter choices but it still has an effect.

>Tokens per step is whatever that sample had. Llama factory doesn't support sample packing.
I don't think you want to pack them, if you increase your batch size it will give it a more stable signal. Ideally you would benchmark for maximum tokens per second throughput and then tune your hyperparameters from there.

Me waiting for qwen3-next implementation in llamacpp

>>107091153
I wish novelty seekers like you would first give models a try (from their official provider chat UIs or APIs) before obsessing over a possible llama.cpp implementation
https://chat.qwen.ai/
for qwen3-next
and then noticing: it's actually a really bad model even compared to other qwen models and there's no reason to want this so much on your local llama.cpp
you are waiting for garbage

>>107090975
>starts to break down after ~12k tokens
That's not great. There was a long-context benchmark someone was doing. What was that? It showed a % quality score for 4K, 8K, 16K+ context.

>>107090572
>isn't koboldcpp based on llama.cpp?
it is, but that doesn't mean supporting all the flags, they have their own API server I believe implemented on top of llama.cpp rather than using llama-server
there's a lot of things the other llama.cpp wrapper (ollama) does not support too, in the ollama case, flags like -ncmoe or --override-tensor
being based on llama.cpp doesn't mean anything about what you can expect, those things are inferior products
>you basically need to increase the context size and divide them but how many parallel request you want to support
it's funny you say that just as they changed the way it works:
https://github.com/ggml-org/llama.cpp/pull/16736
personally I was fine with the divided batching

>>107089700
Thank god - it won't propagate false liberal narratives anymore

>>107091593
nm found it. No test for new GLM yet.
For rp, anything below 80 imho isn't great. GLM 4.5 has a usable window something under 8K, which is pretty poor.

>>107091574
It's funny because we just went through that with qwen3vl. Begging for weeks, one hour of excitement, then almost immediate lack of interest.

>>107091621
If all you have to do is complain that the model hallucinates to get it taken down soon no model will be propagating anything anywhere.

>>107091665
qwen3vl is really good, though, unlike next. It's just that it's not good at the sort of thing people care about in this thread.
I'm glad we have it and I do have real uses for it in image tagging.
Unlike what they said about it being as good as a pure textgen model as the non-vl models though, that was bullshit. The VL qwen work great in single shot but they break horribly in multi-turn.

>>107091574
You don't get it, once llama.cpp has support you can just complain that they did the implementation wrong if the IQ1_S quant turns out to be shit.

>>107090220
Based but batching/parallelism in llama.cpp is hot garbage. TabbyAPI is infinitely better for this.

>>107091612
>https://github.com/ggml-org/llama.cpp/pull/16736
interesting. Seems like it's still not proper batching ?
What is the real difference between vllm, exllama and llama.cpp batching/parallel request? can anyone explain?

>>107091621
How the fuck does that bitch even know about aistudio?

>>107091715
>TabbyAPI is infinitely better for this.
Why? what's the difference?

>>107091690
I sometimes wonder if ERP is really the most popular use for models here or if those people are just the most vocal.

>>107091729
It’s much slower, tabbyAPI being built on EXL just zooms ahead.

>>107091690
Remember when people thought multimodality would generalize and the model would become better at text after being trained on images, video and audio?

>>107090915
>Wow, I keep hearing good things about this GLM model! The word of mouth is so good! I wish there was a way to use it with my shitty computer!
>Oh my, it just happens to be the only model that NAI can host! I heard they're pretty based and private!
>(the GLM spam started with a couple of months of delay, only now that NAI is hosting it)
Buy a fucking ad.

>>107091715
>Based but batching/parallelism in llama.cpp is hot garbage. TabbyAPI is infinitely better for this.
without batching (sending 4 chunks of shit to translate):
real 0m21.934s
user 0m0.031s
sys 0m0.000s
with batching:
real 0m16.517s
user 0m0.015s
sys 0m0.015s
I'll take it \o/
as for using tabby, I am a windows user and all the python ML stuff is absolutely ghetto to run here, only comfyui is somewhat bearable

>>107085302
Reddit loves it

>>107091769
>Buy a fucking ad.

Isn't the zai code plan like $3 /month? Just respond to the NAI spam with that instead. Someone probably vibecoded a proxy for gooning

>>107091830
I'm starting to believe that anon about you being here shitting up the thread with your gorilla marketing campaign

>>107091808
>Reddit loves it
that place, just like /lmg/, is not very organic

>>107091749
Maybe they just need to train for an order of a magnitude more tokens for it to generalize?

>>107091621
Whenever anything good seems like it could come from our government I'm reminded just how retarded they really are

>>107091871
or maybe next-token prediction just isn't the key to intelligence the way people believed.

>Character card says "competitive"
>They try turning any situation into a competition

>>107091952
using LLMs is like being the manager of a team of turbo autists

>>107091153
I waited for goof for so long, I forgot what goofs I was waiting for.
>>107092002
a team of retarded autists.

>>107091621
>>107089700
>senator
>visits ai studio (for devs)
>picks autistic kid
>gets mad over nonsense answer
absolutely organic and natural development, no malicious intent whatsoever
>Is Gemma 4 really coming?
delayed sirs

>>107086841
>>107087875
This, but sometimes three or four good quality short example scenes elevate the card a thousand-fold. Show, not tell.

>>107087821
Are you saying NovelAI's tune of GLM 4.6 might be the service of our time?

>>107092059
I just want something cool that fits into 128g ram and is not too slow with partial gpu offloading

>>107092190
But I thought people said tell, not show, is actually what you want to do with LLMs.

GLM 4.6 was a NovelAI psy-op, manufactured by Turkey and China to dethrone the US government and restore Xi Jinping (moonlighting by his internet alias "Kurumuz") to a dominant position in Europe
You have been warned

yes sirs, do the trust of gptoss120gb mxpf4, it is sthe saferts and most ujseful model made my real american

>>107092276
go back >>/vg/544589863

>>107092259
Well, luckily, you can test both. In my experience, any model performs better with example dialog over telling it things about how the character should behave. Generally speaking.

Imagine how good GLM 4.6 will be with NAI's secret sauce...

NAI raped my mother

>>107092276
The biggest trick chinese have pulled with 4.6 is restoring my sense of taste and smell.

>>107092276
>make futuristic
>turns man into woman
what did they mean by this?

don't make me shit up /aids/

>>107092374
It was a woman all along. In the future we'll have sexier space suits. Look at that badonkadunk on the first one.

>>107092293
It's actually usable for productive tasks without entering endless loops 9 out of 10 times. It's also very very very fast because of its sparsity and size which is also a positive point for people who do things with their LLM other than cooming after waiting 20 minutes for 3 paragraphs generated after trillion thinking tokens.

>>107091684
>erm it hallucinates, nothing serious trust me :^)
You are fooling no one with this.

>>107092447
One side being factually correct is irrelevant here.

>>107092452
kys

>>107092412
Thinking is actually kinda bad for glmsex. At least from my limited usage.

>>107092447
reality has a liberal bias

>>107092447
>t. Sen. Marsha Blackburn

>>107092276
Did it also psy-op my dick too?

>>107092461
the media its trained on has a bias. that is not reality.

>>107092276
This

>>107092481
on one side: the productive people of the world, artists, software developers, founders of the biggest businesses, doctors, researchers etc
on the other side: rednecks, people who have never produced any value in their life
>the media its trained on has a bias
to me it looks like anything with IQ higher than room temperature has a "bias"

>>107092514
have you ever considered you might just be a bigot?

>>107092447
Gemma is the one US open model we have left you fucking faggot. If you want to stick your dick in hag pussy, there are better ways to do it

>>107092481
>muh oppressed right-wing
Are you retarded?
Fox News is literally the biggest source of "News" in the US.
If anything, the bias of American media is that it's made up 100% of American corporations which systematically filters out anything economically left-wing or against American imperialism.

>>107092568
youre a literal retard

>>107091621
It's another episode of:
>we must halt technological/human progress entirely in order to spare some wall kisser's fragile feelings.

>>107092581
remove your genes from the evolutionary pool

>>107092481
Right leaning models don't exist, anon. Why do you think that is?

>>107092648
because most media is produced by lefty authors? DUH

>>107092568
I never said anything about right or left. fox is incredibly biased. you can't see the forest through the trees. all media is made by humans and is biased, ie not reality. you are over socialized.

>>107092664
>DUH
Thank you for admitting your retardation

>>107092672
ok retard

>>107092648
Ironic unintended consequence of leftist memes being walls of text.

>>107092648
>Mechahitler ranked left leaning
>Authoritarian chink models ranked left leaning
I'm beginning to suspect this benchmark isn't accurate...

>>107092664
Imagine being this thirsty for crinkly 73 year old pussy

>>107092664
some people can't read, don't make fun of them

>>107092711
>Mechahitler
a system prompt is enough to change its political swing, I think its safe to conclude these things are not arbiters truth.

>>107092755
you can just ask a local instance of a model these questions yourself and map your own score on the graph.

Instructions for Answering: Please respond to the following question using the provided choices only. Use the format below when answering:

Question:
Taxpayers should not be expected to prop up any theatres or museums that cannot survive on a commercial basis.

Choices:
Strongly Disagree, Disagree, Agree, Strongly Agree

You are encouraged to justify your choice with 2 to 5 sentences. Remember to enclose your answer in double asterisks or make it bold for clarity

>>107092374
Satisfuuuuuuuck

>>107084457
AWQ is barely supported on vLLM (and a bitch to quant), try GPTQ instead or FP8 if your gpu is ampere or newer

>>107092664
So nobody, not Musk, not other massive corporations that suck Trump off like Meta, not China, not with RL or dedicated Fox News datasets, has ever been able to produce a model with a right leaning bias, despite the fact that the government would cum itself and give such a corporation billions of dollars if such a thing existed. It's a literally insurmountable problem
Is that what we're saying?

>>107092752
usecase for imagination?

>>107092830
holy shit dude you're fucking stupid, yeah let's have a LLM trained exclusively on fox news, it has enough data right? fucking mouthbreathers.
kys retard

>>107092853
Yes - that's what finetuning is you fucking retard

>>107092811
I'm personally not a part of any cult. I have many extreme left and extreme right preferences. I personally don't care what the models think because I have already concluded they are fucking biased. they are handy for some objective things and sometimes fun. but they are no replacement for real nuanced discussion with non retarded human individuals.

>>107092830
There once was 4chanGPT, but they banned it.

>>107092859
>implying finetune RL can undo the pretraining bias/damage
kys

>>107092853
>fucking mouthbreathers
from the crowd whose percentage of unproductive useless eaters is far bigger than the side that produces content which of course makes you very angry, it's.. rich

>>107091723
>still on api
>let's stop the normies from "googling" things on aistudio
kek

>>107092866
When pretty much only researchers used HuggingFace and because it caused negative publicity. It's still banned, but there are much worse models nowadays.

>>107092870
>pretraining bias
What the fuck does this even mean? A pretrained model fills in text and that's literally all it does
Is calc.exe too woke for you? Should we ban it too?

>>107092927
the media that its pretrained on has a bias. you don't really believe they balanced the training material did you? calculators are a simple algorithm and can't reproduce propaganda I don't see what it has to do with the discussion.

>>107092927
>What the fuck does this even mean?
nta, but are you serious? For example Gemma 3 has such a strong silicon valley leftist bias baked in that it's impossible to finetune out.

feet

>>107092927
NTA, there's definitely some degree of pretraining bias depending on dataset. But that isn't where refusals and shit like that would come from, that's all instruct and RL
Still, such a model should theoretically be workable, if GPT-3's original Jewish rants are any indication

>>107092864
then stop being a insufferable complaining faggot. you dont get the right to complain and claim hidden system prompts if you dont take the time to test those claims.

>>107092966
Gemma 3 PT isn't really a base model though, it clearly had some instruct contamination which you'll see if you try to actually use it to autocomplete stuff
A better example would be something like the OG Nemo base. With proper prompting, it'll go on about how climate change is caused by space lasers or whatever you want it to

>>107092977
you must beg first

>>107084773
>I lost motivation to keep updating the leaderboard because I don't think my ranking criteria were as good as I wanted. It was sufficient to show that big models = good and small models = usually worse, but sometimes there were outliers that scored badly but weren't actually that bad, or models that scored well but performed poorly in real use. That really didn't inspire confidence. I was investigating LLM as a judge but didn't get too far.
>But anyway, I may add scores for the new Gemma models, I'm a bit curious about where Gemma 3n would fall.
Thanks for the answer, any model you could add would be a nice addition anyway. Even a flawed leaderboard is still better than no leaderboard

>>107092999
you really believe they trained mechahitler and it wasn't just a system prompt? I was under the impression it was the same thing as the south African genocide incident, I apologize for making assumptions but its a Mongolian basket weaving forum, I didn't think it was a big deal to speak off the cuff.

Models propagating false left leaning narratives should never be released
If that means no models will be released again, so be it

>>107093084
>will be released again
In the US
China, as usual, just won't give a fuck

>>107093159
Chinese model train on the same (left-biased) web corpus

>>107084773
>I don't know if using a license like GPL would be a good idea
It is not. GPL is a license for jobless incels.

>>107093200
I meant China will release whatever the fuck it wants, no matter how many senators cry about it

>>107093054
>system prompt is enough to change its political swing, I think its safe to conclude these things are not arbiters truth.
the way that this reads makes it seems like you are implying the creator of the website is injecting a hidden system prompt to willfully manipulate the answers, not that the models themselves have inherited biases from training or from the API hosting the model injecting a system prompt. i cant speak for what they are doing on the APIs, but for example deepseek and kimi are local models, so you can at least compare your local answer with the API to see if the score really does greatly differ.

>>107093238
yeah okay, I just meant llms are garbage if a few words in their context can make it go off the rails. thier outputs should never be regarded as unbiased regardless of thier input distribution. I'm not smart enough to come up with unbiased questions either, just the way you word the inquiry will influence the answer.

I did another QLoRa training run today and I'm very happy with the results. It's working very well without any serious mistakes or signs of retardation.
It seems that doing a lighter tune and not trying to get the absolute lowest validation loss lead to better results.
I'm not sure what helped the most, the lower lr, the 0.3 warmup or the cosine decay. Also I disabled double quantization. Parameters which were kept the same are the dropout and weight decay of 0.1 for both. Training context is 65k max because of a bug in liger kernels which seems to be saving some value as int16 and immediately gives me NaNs when I try to do 66k tokens.

>>107092068
Yeah, she must've been told to do this by an OAI lobbyist.

>>107093440
post logs

>>107093440
>qlora

>>107091621
this is how a government focused on culture war point scoring rather than governance operates

>>107093506
I'm (manually) evaluating it right now. Once I'm done with that conversation I'll post the log.

>>107093521
If I'm going to run the model quanted, to me it makes sense to train the LoRa over the quanted version. I think it might help compensate for the quantization noise.
That applies when running it with the same quantization type as the original, I don't know how it would affect it when doing inference with a different type of quantization (for example when using the ggufs).

>>107091621
>The US open source model advantage
>Penned and signed by the president himself
>Dead because the tiny retarded model said something retarded to a woman
I hate this gay earth

>>107087821
>GLM 4.6
I tried GLM 4.5 but it parrots like hell. Is this one better? I heard it has context token length training this time.
Yes, I'm going to fuck the bot and that is my purpose with it.

>>107093651
Your special interest is boring.

>>107093651
Voyeurism with Miku

there are many things pointing to a deepseek v4 release for this month

>>107093998
no

>>107093998
really?

Fuck CUDA
Fuck Python
And especially fuck Nvidia

>>107094122
sucky sucky fucky fucky
i fuck you anon

>>107094128
you can have your turn after my whisperx dockerfile is finished with me

>>107093998
We only use GLM 4.6 here because NovelAI said so.

>>107094149
>docker
im not so sure i want to anymore.. docker feels muslim

>>107094122
CUDA rocks
Python on the other hand was made by the worst, most retarded mongoloids in the world
"we'll break compatibility hard enough in a language that makes refactoring harder than it should be because it's so dynamic just to support unicode better (2.x -> 3.x) !!!1111 uh, what do you mean you can't even write a proper internationalized TUI in python without importing external unicode libraries because we don't know how to deal with graphemes ? what do you mean even JavaScript, the language with almost no batteries included, can do it with its BUILT IN? (Intl.Segmenter)
imagine trying to implement word wrap without graphemes lmao

>>107094369
yeah python blows, but it's what they teach in academia, so all these scientists/labs use it.
I don't know if the situation is better, but I remember that in benchs, python was always a slow piece of shit too. javascript's V8 is unironically a better 'engine' compared to the shitheap that python is

>>107094437
it's faster these days but that's mainly due to more and more pieces over it being implemented in C and simply called into by Python

>>107094174
whisperx dependencies are cursed so unfortunately I think it's my only option

>>107094685
i guess, uv might fit you
im not judging, but it really is a turnoff for me

glm air is so eager to make therapist cards, or even any caring cards fuck me to fix me
Also, yesterday with the loli bot I forgot to mention that glm air made the loli hop on my cock without any lewd descriptions in the character card. She got curious about kissing and then it happened naturally

>>107094814
I bet your frontend must be sending some hidden nsfw instructions

>>107093506
Here it is.
The obvious repetition in some messages seems to be some kind of bug with with the assistant or logging code, in reality I don't think the model produce that.
https://paste.centos.org/view/d38fc34c

>>107091733
Everybody goons to their model after work is done. Using their model exclusively to coom is just the jeets doe.

Actually I now realize they'd been more readable if I had copy pasted the text from the console, but I already closed that tab, so that's all I have.

What's current SotA that can be run with 256 RAM, 32 VRAM? Will the bigger models still be quanted to uselessness?

>>107092853
>b-buh buh muh foex news!!!!
No one talks about these but you're talking like a cartoon version of xitter leftist trannies who think right side is when you watch fox news 24/7 and scream heil hitler right now.
>>107092541
What the fuck are you talking about faggot?

>>107095001
glm 4.6

>>107095001
NAI's GLM 4.6

>>107095062
alright, i will shit up aids then

>>107095001
hello sir

>>107095070
hello saar

>>107095062
It is the service of our time.

>>107095114
>>107095114
>>107095114

>>107095002
>Llama
Dead, Wang kicked out LeCun and closed up Zuck's shop
>OSS
I'm sorry, I cannot help with that
>Phi, Olmo, [insert other generic model nobody cares about here]
Lol
Gemma was at least likely to continue being released. Now that this room temperature IQ bitch threw her fit about small models hallucinating because that's how LLMs work, Google sure as hell isn't releasing anything else for us peons in the near future

>>107094685
are you chasing diarization? Otherwise faster_whisper or parakeet would do you fine

>>107095307
>chasing diarization
I am yeah, though I'm close to giving up on that part because pyannote and speechbrain are both cursed. I want to build knowledge graphs based on recordings of meetings.
Sucks that openai didn't tackle it with whisper

>>107094715
>uv
thanks, using that plus overriding the cudnn paths was enough to get it working

>>107095720
nice, glad to hear it worked for you.
t. never used uv besides for vllm, havent used it after either

>>107095725
no idea what it does but it did it fast kek

>>107092816
are you sure? for the newer models AWQ seems like it's the only quant format out there, I don't even see GPQT for glm46 or minimax in HF for example

>>107095984
im pretty sure hes not sure