The Simulator Hypothesis (What is a model, really?)
A model is a compressed simulator of the data distribution it was trained on. Parameters set the fidelity of the simulation; next-token prediction is the engine.
A model is not a computer program executing static logical tracks. It is a probability distribution over vocabulary tokens. The constraint is not what facts the model "knows," but the fidelity and reliability of the simulator under novel contexts.
The model as a compressed simulator
Pre-training forces a massive neural network to compress the training data distribution. To predict the next word on the internet accurately, the model cannot simply memorize; it must build internal mathematical representations of physical laws, logical consistency, human psychology, and computer systems.
This is Ilya Sutskever's simulator hypothesis: the model learns to simulate the agents, characters, and contexts that generated the text. When you prompt it, you are asking the simulator to unfold a specific trajectory from its learned probability space.
Parameters are compression capacity
Parameters (weights) are the floating-point values that define the model's behavior. They act as the neural network's storage cells, holding the statistical patterns of the training data.
A smaller model has less parameter capacity, meaning its simulator is a lower-fidelity representation of the world. It will hallucinate, miss subtle context connections, or fail to sustain complex personas under length.
Objective function: Next-token prediction
The model has one native goal: minimize cross-entropy loss by predicting the next token. There is no planning, no internal dialogue, and no foresight. Every forward pass is a stateless mathematical mapping that computes a probability distribution over the vocabulary.
All sophisticated capabilities—from writing Python code to translating languages—are emergent side effects of optimizing this single, simple objective function at scale.
The limits of the simulator
Because the model is a simulator of the training text, it inherits the flaws of that text. It is prone to statistical default behaviors, over-represented biases, and confidence tracking. Grounding the simulator to output factual truth requires additional training layers beyond raw next-token prediction.