Karpathy Gives Large Models a Lord of the Rings Challenge

Large-model evaluation may be shifting from “solving problems” to “building worlds.”

Recently, AI researcher and former OpenAI researcher Andrej Karpathy proposed a highly impactful evaluation concept: give a model the opening passage of The Lord of the Rings and ask it to use Three.js to generate a complete, interactive 3D world. The model must not only understand the text, but also convert its geographic relationships, characters, events, and narrative clues into a digital space that can run, be explored, and support interaction in a browser.

Many people have called this new test the “Lord of the Rings” benchmark. For now, it is more like a public challenge or evaluation paradigm than a standardized benchmark with a unified leaderboard. But it captures a key shift in the current competition between large models: models are no longer satisfied with answering questions. The real gap is increasingly reflected in whether they can turn complex intentions into usable systems.

The Goal Is Not to Have the Model Tell a Story, but to Turn the Story into Software

Traditional large-model evaluations generally divide tasks into several categories: math problems, knowledge-based question answering, code completion, instruction following, or selecting the correct answer on a fixed dataset. These tests are easy to automate and convenient for comparison, but they also have an increasingly obvious limitation: a model can achieve a high score through pattern matching without necessarily possessing the ability to transform vague requirements into a complex product.

The difficulty of the “Lord of the Rings” test lies in the fact that it is not as simple as saying, “Write a webpage.”

First, the model must process a long text with a very high information density. The opening of The Lord of the Rings contains not only character dialogue, but also extensive historical background, relationships between locations, genealogies, timelines, and narrative perspectives. The model must extract structured information from natural language, then decide which content should be presented in the 3D scene and which content is better expressed through labels, timelines, camera perspectives, or interactive events.

Next, it must use a frontend 3D library such as Three.js to perform genuine software-engineering work, including but not limited to:

• Creating the scene, camera, lighting, and renderer;
• Generating visual elements such as terrain, buildings, roads, and rivers;
• Handling the spatial relationships between characters, objects, and locations;
• Designing click, drag, zoom, movement, and perspective-switching interactions;
• Establishing explorable connections between different narrative nodes;
• Organizing HTML, JavaScript, CSS, and resource-loading logic;
• Running the project in a browser environment and fixing errors.

In other words, this is not ordinary code generation, but a complete pipeline from “reading text” to “building a model” and then “deploying it.” The model must work like a product manager, world-building editor, 3D programmer, and test engineer all at once—except that all of these roles are compressed into a single context window.

Why Karpathy Chose The Lord of the Rings

The Lord of the Rings is suitable for this kind of test for reasons beyond its high profile.

First, it has a sufficiently complex yet relatively stable world structure. The locations, characters, factions, and historical events of Middle-earth have clearly defined relationships, allowing readers familiar with the original work to quickly check the model’s output. If a map places the Shire, Rivendell, and Gondor in the wrong locations, or if it mixes up the relationships between characters, the errors are immediately apparent.

Second, the original work contains both cartographic and narrative information. The model cannot simply generate a static map, nor can it merely paste the text onto several floating cards. A more convincing implementation should allow users to browse locations in space, view related characters and events at different locations, and understand the relationships in the text through interaction.

Third, the work’s copyright and visual style also force developers to consider the boundaries involved. An evaluation may use the text to test a model’s comprehension and generation capabilities, but when turned into a public product, the specific text, character likenesses, visual elements from film adaptations, and commercial distribution may all require authorization. Therefore, “generating a Middle-earth prototype” and “launching a complete Lord of the Rings game” are not the same thing.

By choosing a topic like this, Karpathy is essentially pulling models out of the “knowledge examination” and into the real world. Real software requirements often do not tell you where every button should go, nor do they provide a clean database in advance. Usually, they offer only a product description, a set of meeting notes, or even a single sentence such as, “Help me build something similar to [a certain product].” Whether a model can make reasonable trade-offs based on incomplete requirements is a capability much closer to what is needed in production environments.

What This Evaluation Actually Measures

If we look only at the final screenshot, the “Lord of the Rings” test can easily degenerate into an aesthetic competition: whoever has the more beautiful visuals has the stronger model. But that is not its most valuable aspect.

The following dimensions deserve greater attention.

1. Long-Context Understanding

The model needs to identify entities, relationships, and events from a relatively long text while maintaining consistency throughout. It cannot place a certain location in the north at the beginning and then place that same location in the south in its code later. Nor can it mix up people, families, and factions with similar names.

This differs from simple long-context question answering. Question answering requires only the output of an answer, whereas a 3D world can embed errors across multiple components: the map, labels, explanatory text, and interaction logic may all be wrong at the same time. Once contextual understanding fails, each subsequent layer amplifies the problem.

2. Converting Natural Language into Program Structure

“Generate a world” is not a single function call, but a collection of objects, data structures, and rules. The model must decide for itself which content should be abstracted into entities and which relationships can be represented through coordinates, connecting lines, or events.

This is effectively a test of whether the model can construct an intermediate representation. An excellent implementation will not hard-code everything into a script thousands of lines long. Instead, it might first create tables for characters, locations, and events, as well as a relationship graph, and then have the rendering layer read from that data. Such code is easier to debug and more closely resembles a real project.

3. Tool Use and Iteration

Three.js is only a tool, not the answer. Whether the model can consult the library’s documentation, modify its code based on runtime errors, and identify blank canvases, failed model loading, or coordinate-system errors determines whether the final work merely “looks like code” or actually runs.

This is also where agentic models and ordinary chat models diverge. The former need to establish a loop of “write code—run it—observe—fix.” Generating the first version is not difficult. The hard part is recognizing that the first version cannot be used and knowing what to change next.

4. Interaction Design and User Experience

“Interactive” cannot simply mean that a webpage contains a few buttons. Genuine interaction should serve understanding: clicking a location should reveal related characters, dragging a timeline should change the state of events, and switching perspectives should help users establish spatial relationships.

If the model generates only a beautiful scene and nothing happens when users click anywhere, it has created a 3D illustration, not an interactive world. Conversely, if all the functions exist but the interaction paths are confusing, the model has likewise failed to solve a practical problem.

5. Verifiability

The greatest advantage—and the greatest challenge—of this type of open-ended task is that its results cannot be summarized by a single accuracy score. Evaluators need to consider visual quality, factual consistency, functional completeness, code maintainability, loading performance, and interaction responsiveness at the same time.

If this is eventually developed into a formal benchmark, it may require a multidimensional scoring system:

• Textual consistency: Whether the characters, locations, timelines, and relationships match the input material;
• Spatial plausibility: Whether the geographic layout is consistent and navigation is usable;
• Functional completeness: Whether the interactions genuinely exist rather than being a static demonstration;
• Engineering quality: Whether the code runs, can be modified, and is reusable;
• Performance: Whether initial loading, frame rate, and resource size remain within acceptable limits;
• Explanatory ability: Whether the model can explain its design trade-offs and data sources.

This is much more complicated than producing a single score on a question-answering dataset, but it is also much closer to the outcomes developers actually care about.

Taking Opus 5 as an Example: A Strong Model Does Not Mean One-Shot Success

The reference information mentions that a model such as Opus 5 could be used as an example in the test. It is important to note that even a highly capable model will not necessarily generate a mature 3D product successfully in a single attempt.

Common failure modes in this type of task include:

1. Visually rich but logically empty: The scene contains mountains, trees, and buildings, but there are no relationships between locations, leaving users unable to understand why those elements exist.
2. The code is generated, but the page does not run: Problems with dependency versions, resource paths, browser compatibility, or module imports leave nothing but a blank canvas.
3. Surface style replaces textual understanding: The model uses large numbers of “magical” and “epic” visual elements but fails to accurately represent the historical and character relationships in the text.
4. Decorative interaction: Buttons can be clicked, but clicking them merely changes a color or displays irrelevant information, doing nothing to help users explore the world.
5. The longer the context, the more chaotic the latter half becomes: The model establishes the setting correctly at the beginning, but as the code and content continue to grow, entity names, locations, and events begin to drift.

Therefore, a more reasonable testing method would not compare only the result of a single generation. Instead, it would allow the model a certain number of tool calls and repair attempts, while recording how many rounds it used, how many errors it produced, and whether it ultimately completed the task.

This would make the evaluation more like an actual software-development environment: the question is not who writes the most beautiful code on the first attempt, but who can deliver the project within limited time and resources.

Will It Become the Next “Standard Leaderboard”?

In the short term, the answer may be no.

Open-ended 3D tasks are expensive to evaluate. Different models may choose completely different implementation strategies. Some may focus on maps, some on character dialogue, some on low-poly scenes, and others on text retrieval and timeline presentation. Without standardized inputs, tool environments, resource budgets, and evaluation rules, it is difficult to rank the final results directly.

Moreover, 3D-world generation is highly sensitive to factors outside the model itself. Context-window size, whether network access is allowed, whether the model can call a browser, whether it can access documentation, the frontend runtime environment, GPU performance, and network speed can all affect the results. If only a final screenshot is published, it is difficult to determine whether a model is more capable or whether its testing conditions were simply more permissive.

Nevertheless, the concept is still worth taking seriously because it points to the direction in which evaluation is evolving: future model competition will not be limited to “who can answer more questions,” but will increasingly concern “who can complete longer chains of tasks.”

Today’s developers are already asking models to build backends, write frontends, call databases, operate browsers, and deploy services. The gap in actual productivity between a model that can output attractive code in a single round of dialogue and one that can read requirements, choose an architecture, run a project, locate errors, and iterate continuously may be greater than a few points on a traditional leaderboard.

What It Means for Developers

The most valuable lesson of the “Lord of the Rings” test is not that everyone should build a 3D Middle-earth. Rather, it is a reminder that developers should rethink how they evaluate models.

If you are selecting a model for use as a coding agent, you can ask it to generate a minimum viable application from a product description instead of merely asking it to complete a function. If you are testing long-context capabilities, you can provide a set of interrelated documents and ask the model to construct a queryable data model. If you are evaluating a multimodal model, you can require it to combine images, text, and interaction states into a complete page.

Ideally, test results should preserve three types of information at the same time: the final artifact, process logs, and failure records. The final artifact tells you what the model can do, the process logs tell you how it did it, and the failure records reveal where it lost control. For engineering teams, the latter two are often more valuable than a beautiful screenshot.

From this perspective, Karpathy’s “Lord of the Rings” challenge is not intended to replace math, code, or knowledge benchmarks. Rather, it fills a gap they do not cover: whether a model can transform a complex, open-ended, and ambiguous task into a system that other people can actually use.

This is also where large-model evaluation most urgently needs to evolve. Model leaderboards increasingly resemble school report cards, but real development work is more like delivering a project without a standard answer. Whoever can build and deliver something under constraints is the one with genuinely greater engineering value.

Final Thoughts

Turning The Lord of the Rings into an exam question for large models sounds like an entertaining challenge, but it is actually testing the models’ most fundamental emerging capabilities: understanding the world, organizing knowledge, using tools, and translating abstract intent into runnable software.

It is not yet a mature, unified benchmark, and it should not be simplistically presented as evidence that “a particular model has taken the lead across the board.” But as an experiment in evaluation paradigms, it is more thought-provoking than yet another static question-answering leaderboard.

What is truly worth watching next is not who can generate the Middle-earth world that looks most like a movie poster, but who can make that world spatially coherent, usable in interaction, maintainable in code, and capable of withstanding continued questions and modifications from users.

The new examination hall for large models may no longer be a sheet of paper, but a browser, a complex set of requirements, and a digital world that must actually be delivered.

References

• Andrej Karpathy’s GitHub profile: A public gateway to Karpathy’s projects and technical activities, useful for tracking his work related to large models, agents, and evaluation.
• Official Three.js GitHub repository: The code repository for the open-source Three.js 3D JavaScript engine, which serves as the technical foundation for generating browser-based 3D worlds in this type of challenge.