i got my hands on evil corps cloud account and can spin up any amounts of RTX A5000. are there any 4Q_K_M quants of
Llama-3_1-Nemotron-Ultra-253B-v1
or any other recommandations i could try to fit?

how much vram would i need for deepseek r1 for a 5Q_K_M? i heard the loss is not that bad compared to full fp18-

well anyways i actually just want to build some private LLM serving that i can pass to the collegues in the team to fuck around with. it should atleast be somewhat usefull.

happy for any recommandations.

the max i can probably spin up are 8 more cards btw. as a ballpark