Character-Level LSTM
A character-level LSTM in PyTorch that learns to predict and generate text one character at a time. Demonstration of the recurrent memory and the sequence-modeling components behind world models.
Problem
The hidden state is the idea that makes recurrent networks work, and the idea behind the Memory component of a world model. It only becomes concrete when you watch one carry context across a sequence.
How it was solved
A CharLSTM (an nn.LSTM core with a linear head to vocabulary logits) trained
with truncated backpropagation through time: the state carries forward across
the whole text, but gradients are cut at chunk boundaries so training never
backprops through thousands of steps.
The checkpoint saves the vocabulary maps alongside the weights, because a character model’s index-to-character mapping is part of its state.
Results
After training, the model generates its pattern back one character at a time, the hidden state remembering where in the sequence it is.
The same mechanism carries directly to modeling environment dynamics in latent space, which is where the world-model project uses it.