Train a tiny story model from scratch, and prove what it read

This is for a developer who has used a language model but never trained one, and who would like to see the whole path once at a size that fits in their head: choosing texts, cleaning them, training a tokenizer and a model, and being able to say afterwards what the model was trained on.

You need Python 3.10 or newer, about 1 GB of disk, and a network connection for the downloads. No GPU and no accounts. On Ubuntu, install the venv module first (sudo apt install python3-venv); on a stock Ubuntu 24.04 machine the first command below fails without it.

Everything here was run for real on 7 October 2026, on rented cloud machines with 8 virtual CPUs and 32 GB of memory. The numbers, hashes and samples below come from a run on a fresh machine that followed only these instructions, and that run's logs, manifests, run card and samples are published in the example's evidence folder.

A model is made of its training data, so the first job is not code. It is deciding which texts you are entitled to use and writing down why.

We set one rule and applied it book by book. A text goes in only if its Project Gutenberg record says it is public domain in the United States, and every named author or translator of the part we use died in or before 1955. That is what we checked. It is not a claim about every country: Project Gutenberg itself tells readers outside the United States to check their own law ⁶, and so do we.

Nine books passed. They are listed below and, with links, under Sources as ⁷ to ¹⁵. They are nineteenth and early twentieth century retellings: two English versions of Aesop, three collections drawn from the Jataka tales and Indian beast fables, and four of Indian folk tales.

The rule cost us five good candidates, and the reasons are the useful part.

A well-known illustrated Aesop for children from 1919 has unsigned text. It is very probably fine. But "very probably fine" is an argument, and when a text's status needs an argument we leave it out. Two further collections, one of Aesop and one of Jataka tales, have a translator or author whose year of death is missing from the catalogue record, so the rule could not be applied to them.

The most instructive case was a famous 1892 anthology of Indian fairy tales. Its title page carries one editor, who died in 1916. Its notes at the back show that the tales were taken from about a dozen other collections and translators, several of whose dates we could not confirm from catalogue records. An anthology is not one text; it is as many texts as it has sources. We left it out, and a second anthology with it. In one book we kept, we cut the introduction, which was written by someone who died in 1961.

A provenance tool cannot make these decisions for you. It can make sure somebody made them, and wrote them down.

Shuddhi will not read a file until someone has declared where it came from. You can watch that happen. Ask it to scaffold a registry from the folder, then check the scaffold:

These are old books, and old children's books are not always gentle. We searched the story text for about a hundred terms and read the passages around the strongest hits. That turned up no slurs, but it did turn up real cruelty in places: a burial alive, corpses, a flaying, a plan to take one's own life. Most of it sits in the two longest folk-tale collections.

Shuddhi's built-in lexicon screen is tuned for overtly abusive text and only fires when a document contains at least two distinct listed terms at a minimum density. We wanted something stricter and simpler, so we used the tool's plugin interface ⁴: a filter of about thirty lines that drops any story containing a term from a short list. The list covers four categories (graphic violence and cruelty, self-harm, adult themes, and dated language about caste, faith or illness) and has 34 entries. We do not reproduce it here; it is a plain text file in the example, and you should read it and change it.

Two things are worth knowing about this screen. It is blunt: it was written from that search of the books, it removes the cruellest stories, and it does not make the rest gentle. A fable in which the wolf eats the lamb stays in. And it is on the record: a plugin must declare everything that changes its verdicts, and that declaration (here, a hash of the term list) is folded into the build's configuration hash. Change one term and the receipt changes.

Provenance gate: nine files declared, nine admitted, none refused.

Duplicates: none. 641 stories, 641 unique, and no near-duplicate clusters. Two translations of the same Aesop fable are different sentences, and the near-duplicate stage is built to catch copies and templates, not paraphrase ³.

Quality and toxicity screens: nothing dropped. These catch broken text and overt abuse, and a corpus prepared from printed books has neither.

Personal data: zero redactions. The detectors look for email addresses, phone numbers, card numbers and identity numbers, and a book printed in 1894 contains none. A zero here is a measurement, not a skipped step.

Content screen: 49 stories dropped by the plugin, counted on their own line in the manifest.

Contamination: 35 stories dropped, and this is the result worth a closer look.

We held out 31 stories so that we could measure the model on text it had never seen. Shuddhi's contamination stage enforces that: any training story that shares a run of eight consecutive words with a held-out story is dropped. We expected it to catch nothing.

It removed 18 of the 19 remaining Deccan Nursery Tales and 14 of the Folk-Tales of Bengal. The drop log gives the reason for each drop but not the matching passage, so the example includes a short script that finds it:

Every Deccan tale opens with the same sentence about the same town, and every Bengal tale closes with the same rhyme. The held-out story from each book carries the formula too. A model trained on eighteen copies of an opening line would score well on the nineteenth for a reason that has nothing to do with learning to tell a story. That is exactly what a contamination check is for, and it noticed something about our own evaluation design that we had not.

The last three drops are different: three ordinary turns of phrase, eight words long, that happen to occur in two stories. An eight-word rule is strict, and it does not know a formula from a coincidence.

We could have removed the formulas from the text, or chosen different stories to hold out, until the number looked better. We kept the rule and lost the stories, because the point of holding stories out is a number you can trust. It cost about a fifth of the corpus by size, and most of two books. You might decide otherwise for your own model. Either way the manifest records what was done.

Shuddhi also has a perplexity filter, which scores each document against a small character model of your own corpus and drops the strangest one percent. On a corpus this small the tool itself says not to use it: with the filter on, v1.2.0 warns "Skip the perplexity filter for a corpus this size (--no-perplexity)" for any file under 50 documents or 200 KB, which is all nine of ours, and its quickstart gives the same advice for anything under a few thousand documents ².

We ran it once anyway to see what would happen. With several small files in one language, v1.2.0 took its cutoff from the one file large enough to supply one and applied it to all nine, and ordinary stories were dropped as a result. That is a limitation of v1.2.0 with multiple small files in one language, and a fix is planned. The build in this guide therefore runs with --no-perplexity, as the tool's own guidance says, and the manifest's configuration records no perplexity cutoff.

corpus_build_hash answers what was measured. It is a hash over the set of all 641 unique stories that passed the gate, before any filter. It does not depend on file order or on how the work was split up.

filter_config_sha256 answers how the selection was made. It covers every threshold and switch, the near-duplicate list, the lexicon, and each plugin's declared identity, so our 34-term list is in there.

filtered_build_hash answers which stories were kept. It is the same kind of hash as the first, taken over the 557 stories that survived, and it covers those documents and nothing else. The configuration is not mixed into it. That is deliberate: anyone holding the same source files can recompute it without our settings, and two builds that arrive at the same stories by different routes share this hash and are told apart by the configuration hash beside it.

One line in the v1.2.0 manifest is worded wrongly, and you will see it in your own file: the hash_definition field says "Chain: parent hash + filter config sha -> this hash". Nothing is folded in. The meaning is the one above: which documents were selected, with the configuration recorded separately in filter_config_sha256.

The manifest also records a sha256 for each cleaned text file. Those tell you what bytes were written, which matters when a filter rewrites text rather than dropping it. Alongside the manifest, the pipeline writes REPORT.md, a draft training-content summary laid out against the European Union's template for general-purpose models, with the nine sources in a table and the gaps it cannot fill marked as gaps. Our toy model needs no such filing; it is there so you can see what one looks like.

Two numbers matter, and they should always be read together. The training loss is how surprised the model is by the stories it is learning from. The held-out loss is how surprised it is by 28 stories it has never seen, stories that Shuddhi has checked share no eight-word run with the training set.

For the first thousand steps both fall. Then the held-out loss flattens at about 4.53 while the training loss keeps falling, to 3.30 by the end. From that point the model is not getting better at stories. It is getting better at these stories: it is memorising.

That is the expected result when 2.6 million parameters meet 254,968 training tokens. There are about ten parameters for every token of text, which is plenty of room to store the text instead of learning from it. A held-out loss of 4.53 means that on unseen text the model is, on average, about as uncertain as if it were choosing among ninety-odd equally likely next tokens. That is a model that has learned the shape of a fable, not its sense.

If you only ever look at the training loss, a model that is memorising looks like a model that is improving.

The script also writes model/run.json, a run card. It records the architecture, that the weights started from scratch, the full loss log, and the three hashes and nine file hashes of the corpus. The model now carries the answer to "what was this trained on?" with it.

On the 8-core machine, everything before training (installing the tools, nine downloads, the corpus script and the Shuddhi build) took about two minutes, of which the build itself was 1.8 seconds. Training took 1,204 seconds, which is 20 minutes 4 seconds. Generating six samples took six seconds.

From the moment the machine was created to the moment it was destroyed was 22 minutes, and the provider charged ₹6.64 for it. A laptop with a recent processor will do the same work; expect it to be slower with fewer cores.

If you do rent a machine, destroy it when you finish and check the provider's listing to see that it is gone. A machine shut down from the inside may still be billed. We learned that on an earlier attempt, and it cost rather more than the run did.

Six samples were generated with the default prompts, and every one was read and screened before any was printed. None contained anything we would call unsuitable; one that used the word "slave", which the corpus contains because Aesop's own story is that of one, was left out. Here are three, exactly as generated, with line breaks removed. They were chosen to show the range, not the best case.

One of the better ones. Prompt: "The Tortoise and the Peacock". A KID, having been a little clearing at the Tortoise, he looked up and saw him, and said: “I have been sitting on the sand, and I am sure you had to eat.” Next day the Tortoise had a Tortoise, and he had fluttered down, and went to the Tortoise. “I have been bitten by your wings, you here,” said the Tortoise, “and you put me on the sand and then I shall have been a good meal to-pots.” “You shall have come to eat,” said the Tortoise. “I would have been a turning by and I am a good friend. I shall have to come to-morrow, and I will do it again.” The Tortoise was so delighted, and he said: “Sir I am sure to get him.” “See, and then, Dan I can help you?” “No, friend,” said the Tortoise, “we can come along with me.” “Well, then,” said the Tortoise, “and I have made you

164 words. Longest run copied from the training text: 5 words ("on the sand and I"). This is invention, such as it is: the rhythm of a fable, dialogue that opens and closes its quotation marks, a tortoise that talks to itself, and no story.

Mostly recall. Prompt: "The Clever Sparrow". A FLE was prowed with a tree, and when he saw a fil, said, 'I am indeed, but the Sparrow was hungry, and when a feast began to eat me, as you will you, and when you can eat me, or a khichrî eat; and if you had to eat me, and it so you can wash beak and feet, and eat you the khichrî as a khichrî eat the Cow, and said-- 'Your name is milk, my name is Grass, For I give me a nice khichrî eat; Though I really don't you can mean, For you can't know What the Sparrow can mean, For I'm sure, For I'm sure, Ap, I'm sure, For I'm remarkably clean!' So the Sparrow sat, and said, 'Certainly I'm remarkably clean, and I'm clean, and I'm remarkably clean!' But the Sparrow paid the Cow, and said, 'Certainly you must go to the Deer, for the Cow, and I will give you a nice khichrî eat

174 words. Longest copied run: 11 words ("…t know what the Sparrow can mean, for I'm sure"). We label this one as recall, not invention. The word "Sparrow" pulled the model into a rhyme from one Punjab tale, "The Sparrow and the Crow", and it is reassembling that rhyme from memory, khichrî and all. A repeated verse in a small corpus is the first thing a small model memorises.

A weak one. Prompt: "The Fox and the Crow". A PHERTERTTARTERTTTTTERTTTTTTTTTTTERST was a WING, having a certain sand, and on a Trees’s por and often bought a wood. But the Crow came up and a house, and called that his son as a Crow. When the Crow was the Crow, she had not enough to give her some fer to do. The Crow’s pots, and made a pure of pentereaks penprine and mars, but it was so much money to be enough to eat. The Crow said, "Why do you do you do you give me a grain of corn? Do you can do not eat yourself to-tree." The Crow did not know what the Swan, but he was fond of corn, and said, "Monkey, you must be very fond of corn." The Monkey agreed to tell this; but the Crow was not like a nice of corn, but he used to the

152 words. Longest copied run: 6 words ("the Crow did not know what"). "The Fox and the Crow" is a real title in the training text, twice, so we expected recitation here. We got the opposite. The likely cause: one of the two Aesop translations opens every fable with a few words in capitals, capitals are rare enough in this corpus that the tokenizer breaks them into fragments, and the model stutters before it finds its feet. It never mentions a fox.

The run described here was itself the reproduction test. We had already run the whole thing once on another machine of the same size and written down its counts, hashes and losses. We then started a fresh machine and followed only the README in the example folder. The one stumble was the one in the prerequisites: the venv module was missing on stock Ubuntu.

The corpus reproduced exactly. All nine source files had the same sha256, the three manifest hashes matched character for character, and so did the sha256 of every cleaned file. That is the property the manifest exists for: two people who start from the same sources and the same configuration can prove they trained on the same corpus without sending each other the corpus.

The training reproduced as well. Everything in the record we had committed before the second run matched exactly: the source hashes, the three manifest hashes, the nine cleaned-file hashes, the counts, the losses at the six logged steps written down there, and the best held-out loss. The remaining logged steps and the six samples also matched what we had recorded from the first run in our working notes and in the draft of this post.

Only the clock differed: the two machines were a couple of minutes apart. On a different processor, thread count or library version, expect small differences in the loss and different samples.

The manifests, the run card, the full training log and all six samples from this run are in the example's evidence folder. The manifest records the name of the machine that built it; in the published copy that field is redacted, and the home directory in two logs with it. If you publish a manifest of your own, look at that field first.

Give it more to read. The single biggest improvement is more text. Add another public-domain collection, apply the same rule to it, add one entry to the registry and rebuild. The hashes will change, as they should.

Change the screen. Edit the terms file, rebuild, and watch filter_config_sha256 change while you compare what was dropped. Then decide whether you agree with our list.

Try the other route on contamination. Hold out different stories, or strip the opening and closing formulas in the corpus script, and see how many of the 35 come back and what happens to the held-out loss.

Shrink the model. With this little text, a smaller model may generalise as well and memorise less. The run card makes the comparison honest: same filtered_build_hash, different architecture.

Read the rest of the documentation. The quickstart and user guide cover every stage and its limits, and the plugin guide covers filters like ours.

The same discipline is what matters at scale. A two-megabyte corpus and a two-hundred-gigabyte one are accounted for the same way: declare the sources, measure everything, record what was removed and why, and cite the hash.