About a year ago, I tried reconstructing Moby-Dick with a few LLM models. The setup was straightforward: set model temperature to zero where possible, randomly select 3-paragraph samples from various parts of the book, ask LLM to predict what comes next, compare string similarity of the outputs.

I've decided this is a good time to redo the experiment with a newer and wider selection of models. This round, I've increased my budget a bit and included a few frontier models, both open and closed, including GPT 5.6 and Kimi K3.

tl;dr

Since last time, the top score moved from ~50% to ~85%, with GPT 5.6 Sol taking the lead, followed by Kimi K3, then DeepSeek. Big models can mostly reconstruct Moby Dick. There's a huge gap between Sol (~85%) and the smaller Terra (~15%) within the GPT 5.6 family.

You can see the code here and raw results in the same repo.

Experiment Setup

Full list of models I've tried can be found below in the appendix.

A short version of the experiment setup: I select N=300 random paragraph groups from the book. Then the model is prompted to predict the next paragraph, which is then compared to the original via Levenshtein distance, specifically the partial match rate from thefuzz package (fully aware that this is slightly biased towards shorter generations). If the similarity is >= 85% it is considered a good match. Check the code as well as the previous blog post for the longer version.

Differences from previous experiment

I did direct OpenAI calls for gpt-5.6 models and OpenRouter for the rest. The reason for OpenAI being direct was because OpenRouter can do funny business with parameters, and in the specific case of gpt-5.6, I needed to make direct calls to get zero temp and thinking disabled.

With the lesson learned from last time, I've only picked the models that allow turning reasoning off and setting temperature to zero to minimize variance. That's why not every model made it to the list, or in some cases, it's not the latest version of the model.

Next, what's left out: Opus and Fable, partly because of budget and partly because Anthropic models have been a general pain with this experiment (will expand on that), so I limited myself to Haiku and Sonnet only.

I also updated the prompt to hint the model about expected output length. That was necessary because many models opted for shorter outputs without this. Aware that this means technically the scores from this experiment are not 1:1 comparable with the previous one's, even though intuition says 4o would likely be stuck in 50%s regardless of the exact prompt.

Just to see the effects of it, I also varied the input paragraph length: now I experiment with paragraph sizes of 1, 3 (original experiment) and 10, expecting a longer context to lead to better results.

Discussion & Conclusion

One interesting impediment was the occasional Anthropic moderation rejections. OpenRouter's Anthropic calls have to go through moderation, and the more violent parts of the book occasionally got rejected at that stage. The moderation itself is not deterministic so retrying fixed these.

About the primary results: I can now reliably say that large models can reconstruct most of the Moby Dick when given the last few paragraphs. Admittedly, the last experiment was limited to mid-size models due to time and budget constraints, so this does not necessarily mean that the frontier moved by 30% in regards to memorization. Since $N=300$, anything under $0.05$ is within noise of each other.

Findings of Carlini et al. regarding the prompt size and model capacity seem to hold: Sol beats the smaller Terra, Sonnet beats Haiku, and among open models: Kimi K3, the largest, beats the rest. GLM 5.2 is an odd duck with its low score despite the size: remains to be seen whether it's RLHF pushing it away from verbatim reconstruction, or the training data not featuring Moby Dick enough. Context size can also help but a) it is overshadowed by model type/size b) the effect is not necessarily monotonically increasing per model.

Now the question is: can the frontier models do the same with some fringe fiction that is guaranteed to be in the training data but is not popular? Perhaps something that is in Gutenberg, but lacking a Wikipedia page or widespread academic discussion. This would finally give a good answer to the occasional "AI plagiarized my self-published book" accusation. Hopefully to be tackled in a future experiment.

Appendix

Match rates by context size

Model sizes

(Open-weight)