LLM Cost Optimization: Over 50% Lower Average Brief Cost

LLM cost optimization starts with the cost of a completed task. For Luna, alkemy's AI assistant, that task can be an account research brief: a synthesis of a company's people, market, recent developments, and business issues, with sources a consultant can review.

Our research agent produced useful briefs, but long runs consumed too many credits. We traced where the work went and changed how the agent handled context, cached instructions, and delivered its results. We also built evaluations to assess whether lower-cost runs still completed the research and supported their conclusions.

The combined changes reduced Luna's average brief cost by over 50% across our tested workloads. The most useful changes were concrete: give each research worker a focused context, keep reusable instructions stable, and generate the full brief once. Model selection became a controlled comparison alongside those changes.

This case study covers optimization work documented through September 2026. The measurements below describe individual changes and runs, with their scope stated beside them.

Measure LLM cost per completed task

A model's price per token tells you little about the cost of a research brief on its own. An agent may make many calls, read the same material repeatedly, or generate output that the user already has.

We started with production traces: records of the model calls and tool actions within each run. We followed complete briefs and examined input tokens, output tokens, cache usage, tool activity, and the conversation history carried between calls.

One historical prompt audit found approximately 39,500 fixed tokens in the assembled prompt, including about 15,700 tokens of tool schemas. A tool schema tells the model what a tool does and which inputs it accepts. That information has a recurring cost when it accompanies later calls.

Those findings gave us specific questions to test:

• Which calls carry context that their task does not need?
• Which instructions remain unchanged and could be cached?
• Does the agent repeat research that already exists?
• Does it generate the same deliverable more than once?

For each proposed change, we needed a cost measurement and a check on the resulting research. A cheaper run that omitted a required section would not meet the same task requirements.

Isolate context for independent research workers

Our earlier key account research subagent handled several tasks inside one conversation. It read different skills and carried intermediate results from each task into later calls. Research already processed for one purpose became input again for another.

We separated independent tasks into focused workers. Each received the relevant assignment, skills, and tools. Independent research could run in parallel, while tasks that depended on earlier findings waited for those results.

flowchart TB
  accTitle: Research workflow before and after focused workers
  accDescr: Before, one subagent carries a growing shared history through every research task. After, four focused workers return compact findings and source references to a coordinator. The coordinator uses those findings to synthesize the final brief.
  subgraph Before
    Q[Brief request] --> A[One subagent handles every research task]
    A --> H[Shared history grows with each task]
    H --> A
    A --> O[Final brief]
  end
  subgraph After
    C[Coordinator assigns independent tasks] --> R[Roster worker]
    C --> M[Market worker]
    C --> S[Signals worker]
    C --> B[Burning issues worker]
    R --> F[Compact findings and source references]
    M --> F
    S --> F
    B --> F
    F --> Z[Coordinator synthesizes the brief]
  end

The cost benefit came from context isolation. A market research worker did not need to carry the people research worker's tool responses through every call. The coordinator could combine findings without receiving every worker's raw history.

Parallel execution can shorten elapsed time, but it does not automatically reduce LLM cost. Additional workers make additional calls. We looked for savings in the context and repeated work within those calls, then used coverage and source checks to assess the resulting brief.

Improve prompt caching with a stable prefix

Prompt caching reuses an unchanged beginning of a prompt across requests. Our prompt placed changing information, such as a clock value or worker identifier, before long blocks of shared instructions. That prevented reuse of otherwise stable content.

We moved shared instructions first and task-specific information later. This simplified example shows the change:

- System: current time, worker identity, shared instructions
+ System: shared instructions
+ Task: current time, worker identity, account, research objective

In the measured verification, cache hit rate increased from 0% to 77.3%. That is a cache metric, not a 77.3% reduction in total brief cost: output generation and additional research calls still incur costs.

The general mechanism is described in Anthropic's prompt caching documentation. Cache reuse depends on matching prompt prefixes and the provider's caching requirements. Keeping instructions stable makes reuse possible; it does not guarantee a cache hit on every request.

Generate the brief once and reduce recurring overhead

The agent wrote a complete brief to the workspace, then repeated it in chat. Its skills and final-response instructions disagreed about where the deliverable belonged.

We made those instructions consistent:

- Save the brief, then include the complete brief in your response.
+ Save the complete brief as the workspace deliverable.
+ Reply with a short confirmation and a pointer to the brief.

In one measured comparison, the final chat response fell from 17,204 characters to 2,072. One run saved about 21 credits out of roughly 1,251. The complete research remained in the saved brief; the change removed its second rendering in chat.

The distinction between characters and cost matters here. A much shorter final message accounted for only a small part of that run's total credits.

Shorten instructions that recur across calls

We also shortened tool descriptions, removed duplicated planning instructions, and reused skill content that the agent had already read.

These measurements describe different scopes. Adding them would not produce a valid estimate of total brief savings. Token counts also depend on the tokenizer, so the payload comparison is not a universal billing measure across models.

For large fetched responses, we used a lower-cost summarizer to produce a digest and a source pointer. The original evidence remained available for verification. That reduced the material passed to the reasoning model, but introduced another quality question: did the summary lose a qualification the final brief needed?

Evaluate research quality alongside cost

Evaluations, or evals, are repeatable checks on an agent's output and behavior. We used them to examine both the brief and the process that produced it.

Our setup included stored cases, checks that required model calls, and tests with prepared application state. Re-scoring a stored output checked the evaluator's behavior. A fresh agent run tested whether the changed model or prompt still produced useful research. Passing the first kind of check could not establish the second.

We examined those results alongside credits, token usage, and latency. For teams building a similar evaluation workflow, LangSmith's evaluation documentation explains how datasets and evaluators support comparisons between application versions.

Check claims individually

A paragraph can contain several facts even when it has only one citation. Our grounding evaluator split the brief into atomic claims: statements that could each be checked independently.

flowchart LR
  accTitle: Claim grounding with missing evidence
  accDescr: The evaluator divides a brief into atomic claims and checks each claim against source text. It marks claims as supported, contradicted, or unclear. When evidence is unavailable, it records the claim as unverifiable.
  B[Brief from an eval run] --> A[Atomic claims]
  A --> G{Evidence available?}
  G -->|Yes| J[Compare claim with source text]
  G -->|No| U[Record as unverifiable]
  J --> V[Supported, contradicted, or unclear]

Consider this hypothetical sentence: “Revenue grew 12%, and management expects that growth to continue.” A source might confirm the revenue figure without supporting the outlook. Splitting the sentence lets the evaluator assess each claim separately.

A working citation link was only a prerequisite. The source still had to support the statement. Missing evidence remained an evaluation gap rather than counting as successful verification.

Check how the agent reached the answer

The process evaluator used skill instructions, ordered tool calls, and prior workspace state. It assessed skill selection, required steps, justified skips, repeated work, dependency order, and completion.

Prior state mattered. Reusing research could be correct when that research already existed. Skipping the same task in an empty workspace could leave a brief incomplete.

For an account brief, define the expected coverage before testing cheaper configurations. Our account research template gives a practical example of the information a consultant may need to review.

Compare lower-cost models on the same workload

We used model A/B testing to assess a lower-cost research model while keeping the coordinator model constant. The baseline and candidate differed in the model assigned to the research workers.

A model profile defined the model for each component. The comparison recorded the actual resolved model choices, since a profile name alone could refer to a different configuration later.

flowchart TB
  accTitle: Comparing research models on the same request
  accDescr: The same evaluation request produces one brief from the baseline research model and one from a lower-cost research model. Shared grounding checks, process checks, and a pairwise comparison inform the assessment of quality alongside cost.
  I[Same evaluation request] --> A[Baseline research model]
  I --> B[Lower-cost research model]
  A --> C[Brief A]
  B --> D[Brief B]
  C --> E[Shared grounding and process checks]
  D --> E
  C --> F[Pairwise brief comparison]
  D --> F
  E --> R[Assess quality alongside cost]
  F --> R

We prepared controlled comparisons with four requirements:

1. Use the same research input. Different account difficulty can overwhelm the effect of a model change.
2. Give each variant an isolated workspace. One model must not reuse research produced by the other.
3. Record model, prompt, and skill versions. Store fingerprints of the instructions so changes remain identifiable.
4. Capture outputs and execution records. Keep briefs, tool activity, tokens, latency, and cost available for evaluation.

Both outputs received the same grounding and process checks. A pairwise evaluation also compared the two briefs for the same request. The system supported live comparisons with a stable model profile assigned to each conversation, avoiding model changes halfway through an investigation.

Account for variation and evaluator bias

Agent runs vary even when their configuration stays the same. Repeated runs help distinguish a model effect from ordinary variation in research and generation.

We randomized which brief appeared first in pairwise comparisons and checked for presentation-order bias. LangSmith's pairwise evaluation guide documents output-order randomization as one way to reduce that bias.

The comparison system reported a Wilson confidence interval around the preference rate: a range that expresses uncertainty in the estimate, including when a small sample contains only wins. When that interval included an even split, the comparison did not name a winner.

We also kept relative preference separate from absolute quality claims. A judge that consistently prefers one brief can help compare models. A statement such as “95% of claims are grounded” needs calibration against human judgments before readers can rely on the number.

Apply the same cost optimization process to your agent

Start with a completed task and define what makes its output usable. Record a baseline, then make changes whose effects you can measure separately.

For a research agent, a useful sequence is:

1. Inspect complete traces for repeated context, tool calls, and output.
2. Separate independent research where focused context can reduce repeated input.
3. Keep reusable instructions stable and measure cache usage.
4. Remove duplicate deliverables and unnecessary recurring instructions.
5. Run fresh quality evaluations after changing prompts, summaries, or models.
6. Compare model variants on the same cases, with repeated runs and recorded configurations.

The over-50% reduction in average brief cost reflects the combined changes across our tested workloads, including model selection, caching, and reduced repeated work. The individual cache, prompt, and output measurements above describe different scopes and should not be added together to calculate that result. The evaluation framework gives us a repeatable way to assess quality alongside cost for the next change.

For a consultant using Luna, the useful outcome is a brief with research and sources they can inspect. Explore the sample workspace to see how alkemy brings that account context together before a client conversation.