Victorian LLM Experiment: A Technical Curiosity, Not a Breakthrough
This summary and analysis were generated by AI from the original article at Simon Willison and may contain errors (how Viqus works). Read the source for full details.
5
What is the Viqus Verdict?
We evaluate each news story based on its real impact versus its media hype to offer a clear and objective perspective.
AI Analysis:
The technical ingenuity showcased in building 'Mr. Chatterbox' is noteworthy, but the model's limited capabilities and the inherent difficulty in achieving a useful LLM with a constrained dataset demonstrate a core challenge in the field— a reminder that scaling data remains paramount to LLM development.
Article Summary
Simon Willison has produced a fascinating technical exercise: 'Mr. Chatterbox,' a language model trained entirely on out-of-copyright Victorian-era British texts. Built using Claude Code, nanochat, and Hugging Face Spaces, the project demonstrates a practical approach to experimenting with LLMs on personal hardware. The model, trained on approximately 2.93 billion tokens from the British Library's collection, aims to explore the limitations of training a conversational AI with such a constrained dataset. While technically impressive—particularly the full plugin development process with Claude Code—the model's performance is described as ‘weak,’ exhibiting more of a Markov chain-like response than a genuinely intelligent conversational partner. The core issue, highlighted by the analysis of Chinchilla’s scaling laws, is the vast amount of data required to achieve a meaningful LLM. This experiment serves as a valuable demonstration of the practical difficulties involved, even with optimized training techniques.
Key Points
- A language model trained exclusively on 19th-century British texts has been created.
- The model was built using Claude Code and nanochat, showcasing a practical approach to local LLM development.
- Despite technical achievement, the model's performance is described as ‘weak’ due to the limited training data.
Why It Matters
This project isn't about creating a commercially viable LLM. Instead, it’s a critical proof-of-concept demonstrating the challenges of training high-quality language models using only public domain data. The experiment underscores the scaling laws observed by Chinchilla, showing that vastly more data is typically needed to achieve even moderately useful LLMs. While interesting from a research perspective, it highlights the significant resource investment required to move beyond simple curiosities. This reinforces the ongoing debate about data acquisition and the realities of building truly capable AI systems.