This is wild. Base models can reproduce the Harry Potter books verbatim after being prompted with a few paragraphs.
I had heard they could reproduce 70% of the text but this looks more like 99.9%.
I don't know whether this means that LLMs are shallow pattern matching machines incapable of real intelligence or that we are overfitting them to death and they could work so much better if we trained them with 10x the data.
This is DeepSeek-V3.1-Base-Q4_K_M.
On the other hand I'm finding it not very useful for writing articles based on a few hand written paragraphs. It tends to just repeat the prompt and repetition penalty just makes it go into repeating random words after a few sentences.