I Stopped Sending the Whole Book to the Model

The first time I tested Novevia with Journey to the West, it jumped straight from the Flaming Mountains to the final arrival at Spirit Mountain.

The Kingdom of Jisai, Golden Light Monastery, Emerald Wave Lake, the Nine-Headed Beast: a run of adventures that should have filled several lively chapters was reduced to a few lines. When I asked the AI to fill the gap, it began repeating earlier chapters. It felt like a student who remembered, halfway through an assignment, that several pages were missing.

Novevia is an AI fiction-writing app I built. Some early code still uses its old name, Chapterly. It creates the setting, story route, and chapter plan before writing the book one chapter at a time. Making the first few chapters interesting is not the hardest part. The hard part is reaching chapter fifty without forgetting where the book is going.

At first I blamed the model. The larger problem turned out to be the job I had given it.

Chinese version of this article

Concept guide: From One Model Call to an Agent

“Whole Book” Sent Me in the Wrong Direction

The original implementation sounded reasonable. If the feature was called “whole-book review,” I should give it the whole book.

The server collected the setting, story route, every chapter card, the fact store, recent memories, and the user’s instructions. It compressed what it could, joined everything into one giant prompt, and sent the lot to the model. If a server-side check found a risky change in the draft plan, the second call included most of that material again so the model could revise its answer.

This worked for short books. As a book grew, prompt wording stopped being the main problem. The request itself had become too large.

In one production task, two model calls took more than eight minutes in total. Moving the operation into an asynchronous job prevented the HTTP connection from timing out, but it did not make those eight minutes disappear. A yellow progress strip kept spinning in the UI. I could not tell whether the model was carefully reading the book or had gone cross-eyed somewhere in the prompt.

The waste was hard to ignore. A user might ask, “Why does chapter 13 no longer connect to chapter 12?” The system would still carry the entire archive to the model. The model could read all of it and return a conclusion that was difficult to dispute and almost impossible to use: the overall pace is too fast; consider strengthening character growth.

There was plenty of context. What was missing was selection.

Which chapters matter? Does this question require the story route? Are facts and memories relevant? I had removed those decisions from the process. I opened the warehouse door and told the AI to look around without first deciding what it was looking for.

Start With an Index

I changed the first step. The model no longer receives the full set of material. It receives a manifest.

The manifest contains the title, genre, chapter count, and the number of written and unwritten chapters. It says whether the story route is empty, thin, or usable, and how many facts and memories are available. It contains no prose and does not expand the full outline.

After reading that index, the model chooses a tool. It can inspect a range of chapter cards, read a route summary, fetch the summary or opening or ending of one chapter, and search the fact store or chapter memories by keyword. Recent adjustment records are another tool, so the model does not return a few days later with the same advice.

I did not use the provider’s native function calling. The runtime uses a small JSON protocol instead:

{
  "type": "tool_call",
  "tool": "list_chapter_cards",
  "args": { "fromChapterNo": 9, "toChapterNo": 15 },
  "reason": "Check how the chapters around the Kingdom of Jisai connect"
}

The server executes the tool, truncates fields that are too long, and returns the observation to the conversation. The model can query again or decide that it has enough evidence and produce a final plan.

Now a question about chapter 13 may require only its neighboring chapter cards and the route. A complaint that the Kingdom of Jisai was rushed may justify another search for Golden Light Monastery, Emerald Wave Lake, and the Nine-Headed Beast. The length of the book no longer determines the size of the first request. The question determines how much material gets opened.

Does This Count as ReAct?

I only asked that question after the code was working.

ReAct stands for Reasoning and Acting. Instead of producing a one-shot answer, the model moves between deciding, acting, and observing: choose what to inspect, call a tool, read the result, then choose the next step.

Novevia’s implementation fits that core idea. The model selects each tool after seeing the previous observation; the order is not hard-coded by the server. It does not copy the full Thought format from the paper or store a long chain of thought. The tool call keeps only a short reason. I think “bounded ReAct-style loop” is an accurate description.

ReAct, however, was not the important part of the change. It was a name I found after the fact. The useful change was turning context from a package assembled in advance by the server into a working set built around the current question.

That is also where this feature becomes more than an ordinary model call. The model is the decision-making component. The agent also needs tools, state, a loop, stopping conditions, and permission boundaries. Without those pieces, it is only a model that knows how to emit JSON.

The Agent Does Not Get to Edit the Book

Once the model can choose its own tools, the next question is how far it may go.

My answer is conservative. Depending on the model tier, the runtime allows only three to five decision steps and three to five tool calls. Each request has a prompt budget between 24,000 and 64,000 characters. If the conversation exceeds that budget, the runtime keeps the system instructions, manifest, and most recent tool results. Character count is not a precise token budget, but it stops the conversation from growing without bound.

More importantly, every retrieval tool is read-only. The model cannot write to the database. It can only submit a structured plan.

validate_draft_plan checks whether that plan would overwrite written prose, insert chapters in an unsafe place, or add too many chapters at once. If validation fails, the model gets at most one revision.

Chapters that already contain prose receive suggestions only. Only planned chapters with zero words can enter the automatic adjustment list. The user still reviews the plan and confirms it before anything is applied, and an applied change can be rolled back. Letting the model write directly would have saved a little code. That code would eventually have been written in response to an incident instead.

The operation also moved from a single HTTP request into a persistent task. Closing the page no longer kills the task, and the UI exposes a cancel action. After a worker interruption, the service requeues stale running tasks and allows up to two recovery attempts. The worker records events such as tool_called, tool_result, and plan_validated, so the UI can show what happened even after a refresh.

Why I Did Not Adopt Pi

An early design considered @earendil-works/pi-ai. It never became a project dependency.

pi-ai sits closer to the model integration layer. It can normalize providers, message formats, streaming, tool calls, and usage. Using Pi’s agent runtime could also remove some hand-written message-loop and tool-execution code. It would not decide how chapter lookup should work, how the fact store should be searched, or which parts of a book must never be changed automatically. Those rules still belong to Novevia.

The project already had NovelAiGatewayService for provider selection, normalized errors, usage, and cost records. Replacing that layer would have duplicated work. I added NovelBookDoctorRuntime above it and collected the read tools in NovelBookDoctorToolRegistry. Queueing, recovery, application, and rollback stayed in NovelBookAdjustmentService.

So the orchestration layer is hand-written, while the model gateway and task infrastructure are not. I have nothing against agent frameworks. This loop has at most five steps and only a small set of tools; replacing the existing call path would have cost more than it removed. If several agents later need to share tools, memory, traces, and retry policy, that calculation may change.

The Timeout Is Gone. The Accounting Is Not Finished.

The failure I most wanted to remove is gone. Whole-book review no longer times out because the first request contains an ever-growing copy of the book.

Other timeouts remain possible. A provider can slow down, the network can fail, and the task itself has a deadline. What disappeared was one specific failure mode: the longer the book became, the larger the first request became, until it sank under its own context.

Total token usage may not be lower. ReAct turns one large call into several smaller calls. If the model inspects the route, chapters, facts, and memories in succession, the total may even cost more. I can claim that each request is bounded. I cannot turn that fact into “token usage dropped dramatically.”

The same caution applies to quality. Tests prove that the manifest does not contain full content, chapter-card tools do not leak prose, a chapter tool returns only the requested fragment, and dangerous plans are rejected. Those tests protect boundaries. They do not prove that the AI has acquired the judgment of an experienced editor. Measuring that will require a fixed set of books and questions, followed by plan acceptance rates, incorrect-edit rates, and the actual tool paths taken.

When I start a whole-book review now, the yellow strip still spins. The difference is that the task log shows which chapters the agent inspected, why it opened the fact store, and which rule stopped a plan. I no longer stare at an eight-minute request and wonder whether it is busy or merely buried.

ReAct is a useful name in the code documentation. In this project, it is less mysterious than it sounds. The model did not suddenly understand Journey to the West, and I did not discover an all-powerful agent framework. It follows an index, fetches what it lacks, and stops when it has enough to submit a plan.

The yellow strip still spins. At least now it reaches the end.