>>22995094>Interesting. So if you were to train a LLM on a math textbook, you'd get better results than if you trained it on the entire internet where it has the chance of taking in bad info?sort of
aaah i wish i had the words to explain it
the more data a model has, the more it tries to fit to the data it's already seen
this goes for neural networks in general, not just LLMs
if you make a neural network which tries to recognize the number from 0 to 9 which is drawn in an image, and you train it on 10,000 sample images, you should be able to feed it a completely new hand-drawn image & it'll guess it correctly
but if you train it on 1,000,000 different sample images, it become less and less able to recognize the digits in a new image
the more data it's trained on, the more it panics/fumbles when you feed it something which doesn't look like its training data
i don't know if you can see the implication for LLMs, but yeah
my guess is, if you trained your own LLM on like a hundred math textbooks, it would atleast produce interesting answers