From Cloud LLM to Custom SLM: A Practical Playbook for Cutting Inference Costs

Most teams that adopted a frontier model through an API in the last two years are now looking at a monthly invoice that grew faster than the product did. The reflex is to negotiate a better rate or cache harder. Both help at the margins. The structural fix is different: most of that traffic is narrow, repetitive, and well specified, and a small model fine-tuned on your own data will handle it at a fraction of the cost, on hardware you control, with the frontier model kept in reserve for the requests that actually need it.

This piece is the playbook for that migration. It covers how to decide which workloads move, how to turn the cloud model's own responses into a training set, how supervised fine-tuning and quantization work in practice, how to size compute from real request volume, and how to keep the small model improving after launch rather than decaying. Code samples are illustrative; the exact library calls will drift, the shape of the work will not.

Start with the bill, not the model

Before choosing a base model or a GPU, pull a month of logs from the cloud provider and classify the traffic by task. The pattern is remarkably consistent across companies: a handful of task types account for most of the volume, and they are the boring ones. Classification, extraction into a schema, drafting from a template, answering questions over a fixed document set. Those are exactly the tasks a small model learns well, because the input space is narrow and the correct output is largely determined by your data, not by general world knowledge.

Two things make a workload a good candidate. First, it is repetitive: the same task shape recurs thousands of times a day, so the model can learn the distribution. Second, it is verifiable: you can tell, cheaply and mostly automatically, whether a given output was right. Ticket classification has ground truth once the ticket is resolved. Extraction has a schema. A drafted reply either got sent unedited or it did not. Open-ended analysis and novel multi-step reasoning have neither property, and they stay on the frontier model. Do not fight that; route around it.

Once the traffic is classified, estimate the saving before doing any engineering. Take the request count and average token sizes per task, multiply by the cloud rate, and compare against the running cost of the GPU tier those requests would need (the sizing section below shows how to derive it). If the narrow tasks are less than, say, a third of the bill, or the total is under a few thousand dollars a month, the engineering cost will outweigh the saving for a while. Above that, the numbers usually turn quickly.

The migration in six phases

The single most important design decision is that the frontier model keeps serving production throughout. Nothing is cut over until the small model has run in shadow against real traffic and been measured. Migrations that fail tend to fail because someone swapped the endpoint on a Friday.

1. Instrument. Log every call to the cloud model with its full context: system prompt, message history, retrieved documents, tool calls, the response, token counts and cost. Attach a task label at the call site. You need this regardless of whether you ever fine-tune anything; it is what makes the bill legible.

2. Harvest. Turn the log into a curated dataset per task. Detailed below, because this phase determines everything after it.

3. Fine-tune. Supervised fine-tuning of a permissively licensed open-weight model, almost always with LoRA rather than full fine-tuning.

4. Quantize and serve. Compress the merged model for inference and put it behind an OpenAI-compatible endpoint so the application does not know the difference.

5. Route. Run the small model in shadow, then canary a slice of traffic, with a confidence-based escalation path to the frontier model.

6. Improve. A standing loop that turns production traffic and escalations into the next training set. Without this, the small model is at its best on launch day and gets worse from there.

Harvest: the cloud model has been writing your training set all along

Every response the frontier model produced for a narrow task is a labelled example of that task, and you already paid for it. This is distillation in the plain sense: a large teacher's outputs become a small student's training data. The catch is that most of those examples are not good enough to train on as-is, and the value is in the filtering, not the volume.

Log the whole call, not just the answer. A response only makes sense given the retrieved context and the conversation that preceded it; if you keep the answer and drop the context, the small model learns to hallucinate the context back.

# 1. Log every production call as a candidate training example.
#    Store the whole context, not just the answer.

import json, hashlib, time

def log_call(task, system, messages, retrieved, response, tools, meta):
    rec = {
        "id":        hashlib.sha1(json.dumps(messages).encode()).hexdigest()[:16],
        "ts":        time.time(),
        "task":      task,                  # "ticket_classify", "reply_draft", ...
        "system":    system,
        "messages":  messages,              # full chat history sent to the LLM
        "retrieved": retrieved,             # chunks the RAG step supplied
        "tools":     tools,                 # tool calls + results, if any
        "response":  response,
        "model":     meta["model"],         # which cloud model produced this
        "tokens":    meta["usage"],
        "cost_usd":  meta["cost"],
        "feedback":  None,                  # filled in later: thumbs, edit, resolved
    }
    log_sink.write(json.dumps(rec) + "\n")

Then curate. The gates that matter, in order of how much they discard:

• Task filter. Only the tasks you decided to migrate. A model fine-tuned on everything learns nothing in particular.

• Negative feedback. Anything a user thumbed down, edited heavily, or escalated is out, or better, becomes a negative example for a later preference-tuning pass.

• PII scrubbing. Replace names, emails, account numbers and free-text identifiers with placeholder tokens before the data touches a training job. This is a compliance requirement, not a nicety, and it is far easier to do at curation time than to prove you did later.

• Deduplication. Near-duplicate prompts dominate real traffic. Keep one per cluster or the model overfits to the common case and forgets the long tail.

• Judge scoring. Have the frontier model grade a sample of its own past outputs against a rubric, and keep only the high scorers. This sounds circular but is not: the model at inference time was working under latency pressure and a generic prompt, while the judge gets to be slow and specific. Validate the judge against a few hundred human ratings first.

# 2. Curate. Every gate discards; what survives is the training set.

def curate(records, judge, min_score=4):
    seen = set()
    for r in records:
        if r["task"] not in TARGET_TASKS:            continue
        if r["feedback"] in ("thumbs_down", "escalated"): continue
        r = scrub_pii(r)                              # emails, phones, ids -> tokens
        key = normalise(r["messages"][-1]["content"])
        if key in seen:                               continue   # near-duplicate
        seen.add(key)
        if judge(r) < min_score:                      continue   # frontier judge, 1-5
        yield {
            "messages": [{"role": "system", "content": r["system"]}]
                      + context_as_user_turns(r["retrieved"])
                      + r["messages"]
                      + [{"role": "assistant", "content": r["response"]}]
        }

# Rough yield: 1M logged calls -> ~180k in target tasks -> ~40k after gates.
# Ten thousand clean examples per task is plenty for most narrow workloads.

One legal note that belongs here rather than in a footnote. Some cloud providers' terms of service restrict using their outputs to train models that compete with them. Enterprise agreements often permit it explicitly, and several providers have relaxed the clause, but the wording varies by provider and by contract, and it changes. Have someone read your actual agreement before phase 2, not after.

Supervised fine-tuning, and why LoRA is the default

Supervised fine-tuning (SFT) is the simplest form of adaptation: show the model prompt and response pairs, and update its weights so the response becomes more likely. For narrow tasks with a clean dataset, SFT alone gets you most of the way. Preference tuning (DPO and its relatives) is a second pass you add later, once you have paired good and bad outputs, not something to start with.

Choosing the base model matters less than teams expect, as long as it clears three bars: a permissive licence (Apache 2.0 or MIT, or a community licence your legal team has read), a size that fits the serving budget, and an instruct-tuned variant to start from so the model already understands chat formatting and instructions. At the time of writing the 3B to 9B open-weight families from Qwen, Gemma, Llama, Phi, Granite and Mistral all clear those bars; check current versions and benchmarks rather than the ones in this article, because the field turns over every few months and open-weight models now trail the frontier by a matter of months rather than years.

Full fine-tuning updates every parameter and needs memory for the weights, gradients and optimizer state several times over. LoRA freezes the base and trains a low-rank pair of matrices at each attention and feed-forward projection. It trains on a single consumer or cloud GPU, produces an adapter measured in tens of megabytes, and lets you keep one base model with a different adapter per task. QLoRA goes one step further and holds the frozen base in 4-bit during training, which is how an 8B model fits in 24 GB with room for a decent batch.

# 3. Supervised fine-tuning with LoRA (TRL + PEFT). Runs on one 24 GB GPU
#    for an 8B model in 4-bit; a few hours for ~40k examples.

from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig

BASE = "Qwen/Qwen3-8B"          # any permissive-licence instruct model

tok   = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(
    BASE,
    quantization_config=BitsAndBytesConfig(load_in_4bit=True,
        bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype="bfloat16"),
    device_map="auto",
)

ds = load_dataset("json", data_files={"train": "sft_train.jsonl",
                                      "eval":  "sft_eval.jsonl"})

lora = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                  target_modules=["q_proj","k_proj","v_proj","o_proj",
                                  "gate_proj","up_proj","down_proj"])

trainer = SFTTrainer(
    model=model, processing_class=tok, peft_config=lora,
    train_dataset=ds["train"], eval_dataset=ds["eval"],
    args=SFTConfig(
        output_dir="out/ticket-slm-v1",
        num_train_epochs=2, learning_rate=1e-4,
        per_device_train_batch_size=4, gradient_accumulation_steps=8,
        max_length=4096, packing=True, bf16=True,
        assistant_only_loss=True,      # learn the answer, not the prompt
        eval_strategy="steps", eval_steps=200, save_steps=200,
    ),
)
trainer.train()
trainer.save_model("out/ticket-slm-v1/adapter")

Three practical points that make the difference between a model that works and one that almost works. Train the loss only on the assistant turns, so the model learns to answer rather than to reproduce your prompts. Mix in a small fraction, five to ten percent, of general instruction data alongside your task data, or the model forgets how to follow instructions it has not seen. And hold out a real evaluation set from a different time window than the training data, because a model that scores well on last month's tickets and badly on this month's has learned the calendar, not the task.

Quantization: the same model in a quarter of the memory

A model's weights are stored as numbers, usually 16-bit floating point. Quantization stores them in fewer bits, most commonly 8 or 4, with a small calibration step that decides how to round. The memory saving is roughly linear in the bit width; the speed gain depends on hardware but is usually substantial, because inference at these scales is bound by memory bandwidth rather than arithmetic. The quality cost, for INT8 and well-calibrated INT4, is typically a point or two on task metrics, which is well inside the gain from fine-tuning in the first place.

The formats to know, by where they run:

• AWQ and GPTQ for GPU serving through vLLM or TGI. Both are 4-bit, both calibrate on sample prompts; AWQ tends to hold quality slightly better at the same size. Calibrate on your own prompts, not the library's default set.

• FP8 on Hopper and Ada class GPUs (H100, L40S). Near-lossless and fast, with no calibration to speak of, if the hardware supports it.

• GGUF for llama.cpp and Ollama. The path for CPU serving, Apple Silicon, or a mixed CPU-GPU setup at low volume. Q4_K_M and Q5_K_M are the sensible defaults.

Two rules. Quantize after merging the adapter, not before, so the calibration sees the fine-tuned weights. And re-run your evaluation set on the quantized model, not the merged one; the number that matters is the one from the artifact you actually serve.

# 4. Merge the adapter, quantize for serving, and expose an
#    OpenAI-compatible endpoint so application code does not change.

# merge LoRA into the base weights (one-off, on a GPU with room for FP16)
python -m peft.merge --base Qwen/Qwen3-8B \
       --adapter out/ticket-slm-v1/adapter --out out/ticket-slm-v1/merged

# quantize to INT4 with AWQ (activation-aware; calibrate on YOUR prompts)
python -m awq.quantize --model out/ticket-slm-v1/merged \
       --calib sft_eval.jsonl --w_bit 4 --group_size 128 \
       --out out/ticket-slm-v1/awq-int4

# serve with vLLM: continuous batching, paged KV cache, OpenAI API surface
vllm serve out/ticket-slm-v1/awq-int4 \
     --quantization awq --max-model-len 4096 \
     --gpu-memory-utilization 0.9 --max-num-seqs 64 --port 8000

# the application only changes one line:
#   base_url = "https://api.<cloud-vendor>.com/v1"  ->  "http://slm:8000/v1"

Sizing compute from request volume, not from vibes

The question is not "which GPU is good" but "how many output tokens per second do I need at peak, and what produces that". Work forwards from traffic, then check memory.

Throughput. Take peak requests per second, not the daily average, and multiply by average output tokens per request. Input tokens matter less because prefill is fast; output tokens are generated one at a time and dominate. That gives sustained tokens per second. Then benchmark one candidate GPU running your quantized model under continuous batching with vLLM or similar, at the concurrency you expect, and read off tokens per second. Divide, add thirty percent headroom, and you have a GPU count. An 8B model at INT4 on a single 24 GB card typically sustains several hundred to about a thousand aggregate output tokens per second under batched load; a single H100 running FP8 sustains several times that. Treat those as orders of magnitude and measure.

Memory. Weights are the easy part. The KV cache is what surprises people: every concurrent request holds a cache proportional to its context length, and for an 8B model with grouped-query attention that is on the order of 130 KB per token in FP16. Sixty-four concurrent requests at 4k context is roughly 32 GB of cache on top of the weights. That is why serving engines expose a maximum sequence count and a maximum model length; set them from your actual traffic rather than the defaults, and cap context aggressively, because most narrow tasks do not need it.

Latency. Batching raises throughput and per-request latency together. If the task is interactive, check time-to-first-token and tokens per second per stream at your target concurrency, not just the aggregate. If it is a batch job, forget latency entirely and run the GPU flat out overnight.

A worked illustration, with round numbers you should replace. Three million requests a month at 1,500 input and 400 output tokens each is 4.5 billion input and 1.2 billion output tokens. At representative frontier rates that is tens of thousands of dollars a month. The same load peaks at perhaps three requests per second, needs around 1,200 output tokens per second, and fits on two L4-class GPUs with headroom, which is a low four-figure monthly bill on a cloud GPU rental and less on owned hardware. The gap is an order of magnitude before you count the frontier fallback traffic, and it stays wide after.

Serving and routing: the application should not notice

Serve behind an OpenAI-compatible API. vLLM, TGI, SGLang and llama.cpp's server all expose one, and it means the migration is a base URL change in the application rather than a rewrite. The router in front of it is where the safety lives.

# 5. Route by task and confidence. The frontier model is the fallback,
#    never the default.

SLM_TASKS = {"ticket_classify", "extract_fields", "reply_draft", "doc_faq"}

def complete(task, messages, **kw):
    if task not in SLM_TASKS:
        return frontier.chat(messages, **kw)              # open-ended -> cloud

    out = slm.chat(messages, logprobs=True, **kw)
    conf = mean_token_logprob(out)                        # cheap confidence proxy
    if conf < THRESH[task] or violates_schema(task, out):
        log_escalation(task, messages, out)               # becomes training data
        return frontier.chat(messages, **kw)              # escalate this one
    return out

# Escalation rate is the number to watch. 5-10% at launch is normal;
# it should fall with every retraining cycle. If it climbs, something drifted.

The router does three things. It sends non-migrated tasks straight to the frontier model. For migrated tasks it calls the small model, checks a cheap confidence proxy such as mean token log-probability plus a schema check, and escalates low-confidence responses to the frontier model. And it logs every escalation, because escalations are the most valuable training data you will ever collect: they are precisely the examples the small model cannot yet handle.

Run it in shadow first. For a week or two, every migrated request goes to both models, the frontier response is served, and the small model's response is stored and scored offline. You get an agreement rate per task before any user sees the small model. Then canary at five percent, watch the escalation rate and the user feedback signals, and widen from there.

Periodic improvement: the model is a process, not a file

A fine-tuned model is at its most accurate on the day it ships. Traffic shifts, product changes, new ticket categories appear, and the model has seen none of it. The teams whose small models are still good a year later are the ones that made retraining routine and cheap.

• Collect continuously. The same logging that fed phase 1 keeps running against the small model's own traffic, with escalations and user feedback attached.

• Judge a sample. Weekly, score a random sample of small-model outputs with the frontier model as judge, against the same rubric used at curation time. Track the score per task over time; a downward drift is the earliest warning you will get.

• Curate the failures. Cluster the escalations and low-scoring outputs. Usually two or three clusters explain most of them, and each cluster is a gap in the training set. Get frontier-model responses for those prompts, gate them the same way, and add them.

• Retrain the adapter. Not the base model: retraining a LoRA adapter on the updated set is hours, not days, and the artifact is small enough to version like code.

• Evaluate before promoting. A fixed regression suite per task, plus the held-out set from the latest time window, plus a canary. If the regression suite drops, the new adapter does not ship, no matter how good the new numbers look.

• Refresh the base occasionally. When a materially better open-weight model in the same size class appears, re-run the adapter training against it with the same dataset. Because the dataset is the asset, not the weights, switching base models is a re-run rather than a restart.

Monthly is a reasonable cadence for retraining, weekly for judging. Whichever you pick, make one person the owner. Models that belong to nobody drift.

When not to do this

The migration is a bad idea for low-volume traffic, where the engineering time dwarfs the saving. It is a bad idea for genuinely open-ended tasks, where the small model will never match the frontier and escalation rates stay high. It is a bad idea in domains that change faster than your retraining cycle. And it is a bad idea for a team with no operational muscle: someone has to own the serving, the GPU, the retraining and the evaluation, and if that someone does not exist, the cloud bill is buying you an operations team you do not have to hire. Be honest about which case you are in.

The checklist

• A month of cloud traffic classified by task, with cost per task.

• Every call logged with full context and a task label.

• A curated, PII-scrubbed, deduplicated, judge-filtered dataset per task, versioned.

• A read of the cloud provider's terms on training with outputs.

• A LoRA adapter on a permissively licensed base, evaluated on a held-out set from a later time window.

• A quantized artifact, evaluated again, behind an OpenAI-compatible endpoint.

• Compute sized from peak tokens per second and KV cache memory, with headroom, and benchmarked before purchase.

• A router with confidence-based escalation and shadow mode, with the frontier model as fallback.

• A weekly judge sample, a monthly adapter retrain, a regression suite, and one named owner.

Done this way, the small model is not a cheaper substitute for the frontier model. It is the specialist that handles the work the frontier model was always overqualified for, with the frontier model still there for everything else. The bill falls by an order of magnitude on the migrated share, the data stays in your infrastructure, and the asset you end up with is a dataset that gets better every month, which no vendor can reprice.