>>715682318
1) Download and install Kobold: https://github.com/LostRuins/koboldcpp

2) Download and install SillyTavern (there is a step-by-step guide on how to do this on Github: https://github.com/SillyTavern/SillyTavern - specifically use the guide for the SillyTavern launcher)

3) Go to https://huggingface.co/ and set up an account, then go to https://huggingface.co/settings/local-apps and tell it what kind of GPU you have.

4) Find a GGUF quant of a model that will work with your hardware. After you tell HuggingFace what kind of GPU you have, it will give you a green, yellow and red mark behind each GGUF quant to tell you if it might work for your hardware or not. Pic related.

5) Start Kobold and load the GGUF file you downloaded. Set the context in such a way that the Layers (near the top of the initial screen) are all loaded onto the GPU. Keep in mind that its estimation for this is shitty, so you will want to experiment later to see how much context you can actually fit into your GPU with the model you use.

6) Start up SillyTavern.

Continued in next post.