Builder · Learning project · public code · trained locally
Masked diffusion LM
I trained a 41.2M-parameter masked diffusion language model on TinyStories using an Apple M4 Pro and PyTorch MPS. It starts with `[MASK]` tokens and fills them over 32 to 64 passes. It learns the register of children's stories, then loses the plot. This is a learning project, not a model I claim competes with autoregressive LMs.
- Parameters
- 41.2Mlegacy encoder-only model
- Training
- 3 h 41 min50,000 micro-batches · 8,333 optimizer updates · about 154M tokens
- Hardware
- Apple M4 Pro24 GB unified memory · PyTorch MPS · no cloud
- Training data
- TinyStories2,119,719 stories · about 2.09M 128-token chunks · 0.5% validation
- Sampled output
- 32 stepsrandom mode · step-45,000 checkpoint · 128-token window
How it writes
A GPT-style model chooses each token from left to right. This model sees the whole masked sequence in both directions. It predicts tokens at every blank, commits the most confident ones, and repeats. Easy positions can settle first; the rest get filled in around them.
The trained checkpoint is the legacy encoder-only transformer: 8 layers, 8 attention heads, width 512, 128-token chunks and tied embeddings. It was trained with a cosine masking schedule and cross-entropy averaged over all positions. That last choice means many easy, already-visible tokens dilute the reported validation metric.
The repo also has a newer adaLN-Zero and RoPE variant, but it has not been trained. Its RoPE config is not wired into the training model builder, so its config and saved model could disagree.
The run I can show
The `tinystories_fast` run used 50,000 micro-batches, or 8,333 optimizer updates, for about 154 million tokens and 0.57 of an epoch. It took 3 hours 41 minutes on the M4 Pro. The checkpoint contains 41,175,552 parameters and is 472 MB on disk.
Its validation score fell to 3.793 at step 45,000, but that is an all-position loss converted to a PPL-like number, not a comparable language-model perplexity. Around two thirds of positions are unmasked on average. Dividing by the approximate masked fraction gives an unmeasured upper estimate near 40 perplexity on masked tokens. It is useful for comparing runs of this setup, not other models.
Two earlier 50,000-step runs sat around 200 to 217 on that same metric and produced word salad. The successful run changed both the timestep range and loss calculation, so I cannot say which change made it learn. The run also had a learning-rate schedule bug: warmup was configured in optimizer steps but advanced on micro-batches, leaving decay barely started.
What the samples show
The step-45,000 samples use short character names, dialogue and the 'once upon a time' opening. They can hold local word order and sound like a children's story for a sentence or two. Then they restart, wander or repeat words. Several tokens are committed in parallel without seeing each other, which is where much of the incoherence comes from.
This is the limitation, not a footnote: the model has not learned to hold a plot. I have one training run, no matched autoregressive baseline, no human ratings and no measured generation-speed win. I have not measured whether more denoising steps improve quality.
What I would test next
First fix the RoPE wiring and learning-rate schedule, then run the missing loss ablation at the same timestep range. Report masked-token loss directly. After that, train the adaLN/RoPE version, measure repetition and diversity across 8 to 128 sampling steps, and compare it with an autoregressive model at the same parameter count and token budget.