Local GPU models

Some of the models in our catalog are not forwarded anywhere. They run on hardware we own, behind the same /api/v1 endpoint as everything else. This page explains what that buys you, and the one thing it costs you: a model that is asleep has to wake up first.

What a local model is

A slot is one model loaded on one GPU node, served by an inference engine that speaks the OpenAI protocol:

Both are priced at ₩30 / 1M input tokens and ₩150 / 1M output tokens. Additional experimental slots exist on the same node but are not public.

You can see slots in the catalog. GET /api/v1/models/{author}/{slug}/endpoints lists them alongside external endpoints, with tag set to the engine name:

curl https://openrouter.myip.co.kr/api/v1/models/google/gemma-4-26b-a4b/endpoints

{
  "data": {
    "id": "google/gemma-4-26b-a4b",
    "endpoints": [
      {
        "name": "Google: Gemma 4 26B A4B | MyIP Local GPU",
        "provider_name": "MyIP Local GPU",
        "tag": "vllm",
        "context_length": 32768,
        "max_completion_tokens": 32768,
        "status": 0,
        "pricing": { "prompt": "0.000030000000", "completion": "0.000150000000", "currency": "KRW" }
      }
    ]
  }
}

status: 0 means the slot is running right now. null means it is stopped, loading, or in an error state.

Why we do this

Price. Local prices are set directly in won rather than converted from a provider's USD rate, so they do not move with the exchange rate.

Locality. For a local model, your prompt never leaves our network. There is no upstream provider that could log it. See Provider logging.

Predictability. We control the engine flags, the context window, and the quantization. Nothing changes underneath you because a vendor shipped a new default.

The cold start

Here is the part you have to design around.

When your request picks a local candidate, the gateway does this before sending anything upstream:

1. Probe the slot. If it answers, the request goes straight through — no delay.
2. If it does not answer, take a per-node lock (one model at a time), start the unit, and poll every 3 seconds until the slot answers or the wait budget runs out.
3. If the slot came up, the request proceeds normally.
4. If the budget ran out, this candidate is skipped and the chain moves to the next one. The unit keeps loading in the background, so a retry a little later usually lands on a warm slot.

Each slot has its own wait budget, sized to how long that model actually takes to load — on the order of a few minutes for a large model on a single GPU. That budget is the ceiling on how long a single request will sit waiting.

What you see

The 503 body and headers:

{
  "error": {
    "code": 503,
    "message": "로컬 모델을 기동하는 중입니다. 잠시 후 다시 시도하세요.",
    "metadata": { "error_type": "model_loading", "retry_after_sec": 300 }
  }
}

A Retry-After header carries the same number. Honour it — retrying immediately just re-queues behind the same load.

Handling it in code

import time, requests

def chat(payload, tries=3):
    for attempt in range(tries):
        r = requests.post(
            "https://openrouter.myip.co.kr/api/v1/chat/completions",
            headers={"Authorization": f"Bearer {MYIP_API_KEY}"},
            json=payload,
            timeout=600,
        )
        if r.status_code != 503:
            return r
        wait = int(r.headers.get("Retry-After", "10"))
        time.sleep(wait)
    return r

Three patterns work well:

• Warm up before you need it. Send a one-token request, ignore the answer. It costs a fraction of a won and turns the next request into a warm one.
• Give the request a fallback. "models": ["lgai/exaone-4.0-32b", "google/gemma-4-26b-a4b"] lets the chain move on instead of returning 503. See Model fallbacks.
• Do not switch models per request. Batch work by model so the exclusive GPU is not thrashing.

Other reasons a local slot gets skipped

Cold start is not the only one. A local candidate is dropped from the chain when:

• The prompt is too long for the slot. Candidates whose context window is smaller than the estimated prompt are removed before dispatch, so a cold start is not wasted on a request that would fail anyway. Effect: a very long prompt may quietly be served by an external provider instead. Check X-MyIP-Provider if that matters.
• The slot is in cooldown. Three consecutive upstream failures put a candidate in a 60-second cooldown; during it, the chain skips straight past it.
• You excluded it. provider: { "ignore": ["local-gpu"] } removes it, and provider: { "order": [...] } or sort can move it out of first place. See Provider routing.

If a local slot is temporarily unmanageable — for example the process that supervises slots is restarting — we do not drop the candidate. We try it anyway, because it may well already be running, and if it is not, the upstream call fails fast and the chain moves on. Availability of the supervisor is not allowed to decide availability of the API.

Pinning yourself to local

If you want the local slot or nothing — for data-locality reasons, say:

{
  "model": "google/gemma-4-26b-a4b",
  "provider": { "only": ["local-gpu"] },
  "messages": [{ "role": "user", "content": "…" }]
}

With only, there is no fallback: if the slot cannot come up in time you get 503 model_loading, and if the mapping is disabled you get 404 no_endpoints_found. That is usually what you want when locality is a requirement rather than a preference. Local-first routing covers the opposite direction too.

Capabilities differ per slot

Tool calling and image input are enabled per slot, not per model family. Today google/gemma-4-26b-a4b is the slot with tool calling and image input; lgai/exaone-4.0-32b is text-only and has no tools. The authoritative answer is always supported_parameters and architecture.input_modalities in GET /api/v1/models — read those rather than assuming.

Billing

Nothing is special about local billing. The same formula applies: prompt tokens at the input rate, completion tokens at the output rate, both in won, settled once and stored. A request that fails — including one that never got a slot — costs nothing. See How costs are calculated.

Time spent waiting for a cold start is not billed. You are billed for tokens, not for seconds.