>>510551368
I've been trying to peek inside these LLM's the best I can make out, is that these models causing the entire planet to turn its collective head is "a machine learning algorithm crafting machine learning algorithms" the masonry of the system is maybe 30 to 125 2D layer recurrent neural networks, the neurons themselves are absolutely vanilla Perceptrons as defined in any 101 level course. The weight in the neuron is a number, the activation functions are one of the standard 12, and the gradient ascent algorithm that tunes the weights are not a secret, youtube can tell you exactly how it boils down to the partial derivative of a surface with respect to a direction and then getting a list of pointers and changing the nodes by those amounts. There are also "LSTM backtrackers" between the layers, and the way in which information is allowed to travel back in time must be confined to only the information in our imagination for what we're about to grab into existence. The beginning of my paragraph must resonate with what comes next, and the future can affect the past, before I've committed to a line of thinking. The LSTM connections are created and destroyed by a gradient ascent that was changed to not make a mistake since data from already-committed pathways must not travel retrograde in time along the LSTM relays, but that's a problem for the ml engineer. And so the people leaning on their honkler horn saying: "derp it just picks next word" is maximally wrong: You see a "next word" but that's a function of it dribbling out to you words that have promoted to concrete and aren't still being hammered out. The model also chooses the prior words based on words that come after the fact. If I feed in: "the seseme street character that's red is big bird and" then the LSTM will come back and illustrate the mistake. Words are being predicted retrograde in time as well as forward. They come through in chunks
https://www.youtube.com/watch?v=8Li0Tyeqlc4&t=353