---
name: cost-aware-llm-pipeline-design
description: "Designs a language-model pipeline that stays within budget: what to cache, how to route easy and hard requests, how many tokens each step gets, and which evals gate a release. Use when a feature built on a language model needs a pipeline design that controls cost with caching, model routing, token budgets, and eval gates. Do not use for writing the prompts themselves or choosing between model vendors on benchmarks. Related topics: AI, coding."
---

# Cost-Aware LLM Pipeline Design

Designs a language-model pipeline that stays within budget: what to cache, how to route easy and hard requests, how many tokens each step gets, and which evals gate a release.

## When to use

Use this skill when a feature built on a language model needs a pipeline design that controls cost with caching, model routing, token budgets, and eval gates. Do not use it for writing the prompts themselves or choosing between model vendors on benchmarks.

## Workflow

1. Ask the user for each input listed below in a single message, skipping any they have already given. Where options are listed, offer them, but accept other answers too.
2. Sketch the pipeline steps and estimate input and output tokens for each, so cost per request is a number.
3. Apply caching and context trimming where the same prefix or answer repeats, and estimate the saving.
4. Define routing rules that send easy requests to a cheaper path and escalate on a stated confidence signal.
5. Set a hard token cap per step and a fallback for timeouts and refusals.
6. Define an eval set and release gates on quality and cost, then summarise before and after cost in a table.

## Inputs

- **The feature** (longer free text)
- **Request volume** (free text, e.g. "Requests per day")
- **Monthly budget** (free text, e.g. "Amount in dollars")
- **Quality bar** (longer free text)
- **Latency target** (free text, e.g. "under 3 seconds")
- **Levers to consider** (choose any: Prompt caching, Response caching, Model routing, Token budgets, Batching, Smaller context, Fallbacks)

## Output

A text diagram of the pipeline, a before and after cost table, the routing rules, and the eval gate definition with thresholds.

## Prompt template

Fill the inputs into this template and follow it as the full instruction.

### Cost-Aware LLM Pipeline Design

Design a model pipeline for this feature that I can afford to run.

- The feature: {{The feature}}
- Request volume: {{Request volume}}
- Monthly budget: {{Monthly budget}}
- Quality bar: {{Quality bar}}
- Latency target: {{Latency target}}
- Levers to consider: {{Levers to consider}}

Begin by sketching the steps of the pipeline and estimating tokens in and out for each, so cost per request is a number before any optimisation. Then apply the levers: cache stable prompt prefixes and repeated answers; route by difficulty, with a cheap model or rule for easy cases and a stronger model only when a confidence check or classifier says it is needed; set a hard token cap per step and trim retrieved context to what the answer needs. Add a fallback for timeouts and refusals. Define a small eval set built from real requests, and set gates: a release is blocked if quality drops by a stated margin or cost per request rises by more than an agreed amount.

Return a diagram in text, a cost table comparing before and after, the routing rules, and the eval gate definition.

## Pitfalls

- Routing by prompt length alone, which sends hard short questions to the weak model.
- Caching responses that contain user-specific data and serving them to another user.
- Cutting cost with no eval set, so quality drops unnoticed until users complain.