Local-first routing

This is the routing rule that makes this service different from a plain proxy: if a model is available on our own GPU, that slot is tried before any external provider. It is the default, it applies to every request, and you do not have to ask for it.

The rule

Every mapping between a model and a place that can serve it carries a priority number. Lower is tried first.

Building the chain for a request is then a sort: expand each requested model into its endpoints, order by priority, try them front to back. Local endpoints are 0, so they lead. Nothing in the default path reorders that.

A worked example

Suppose google/gemma-4-26b-a4b is served both by our GPU slot and by one external provider, and lgai/exaone-4.0-32b only by our GPU slot. This request:

{ "models": ["google/gemma-4-26b-a4b", "lgai/exaone-4.0-32b"] }

produces this chain:

1. local-gpu   google/gemma-4-26b-a4b     (priority 0)
2. <external>  google/gemma-4-26b-a4b     (priority 50)
3. local-gpu   lgai/exaone-4.0-32b        (priority 0)

Note position 3: the priority sort happens within each requested model, not across the whole list. Your order in models[] is the outer loop; priority is the inner one. If you want the local EXAONE tried before the external Gemma, put EXAONE first in models[].

Why it is the default

Cost. Local prices are set in won directly rather than derived from a provider's USD rate, and they do not move with the exchange rate. For our two public models that is ₩30 / 1M input tokens and ₩150 / 1M output tokens.

Locality. For a local endpoint there is no upstream provider. The prompt is processed on hardware we operate and is not transmitted to a third party. If that property is a requirement rather than a preference, pin it — see below — because the default is a preference and will fall through.

Independence. Local slots are not subject to a third party's rate limits, quota changes, or model deprecations.

When local loses

Five things can push a local candidate out of first place or out of the chain entirely. Knowing them is the whole content of this page.

1. The prompt does not fit

A local slot has a fixed context window (32,768 tokens for both public models). If the estimated prompt is larger, the local candidate is removed from the chain before dispatch, so a cold start is not spent on a request that would have failed anyway.

Consequence: a long prompt can be served by an external provider on a request where a shorter prompt would have been served locally. If locality matters, either keep prompts inside the window or pin with only.

The estimate used for this check is deliberately generous — it over-counts rather than under-counts, because losing a cold start is more expensive than losing a marginal candidate. It is not used for billing; billing always uses the token counts the upstream reports.

2. The slot is asleep and cannot wake up in time

Local models are not all resident at once. If the slot is not running, we start it and poll until it answers or the slot's wait budget expires. If the budget expires, this candidate is skipped and the chain continues.

If there is no next candidate, you get 503 model_loading with Retry-After. Details and code in Local GPU models.

3. The slot is in cooldown

Three consecutive upstream failures put an endpoint into a 60-second cooldown, during which the chain skips it without trying. A successful request clears the counter immediately.

4. You told us to

provider.ignore, provider.only, provider.order and provider.sort all override the default order. In particular, "sort": "price" re-sorts everything by price — if an external provider is cheaper for some model, it will now come first. See Provider routing.

5. The model has no local slot at all

Most models in the catalog are external-only. Local-first is a rule about ties, not a promise that everything runs locally.

Making it explicit

Local or nothing

{
  "model": "google/gemma-4-26b-a4b",
  "provider": { "only": ["local-gpu"] },
  "messages": [{ "role": "user", "content": "내부 문서 요약" }]
}

Now there is no fall-through. If the slot cannot come up in time you get 503 model_loading; if the mapping is disabled you get 404 no_endpoints_found. Use this when locality is a requirement.

Never local

{ "provider": { "ignore": ["local-gpu"] } }

Useful for latency-sensitive paths where you would rather pay more than risk a cold start.

Local first, but do not wait forever

Give the chain somewhere to go:

{ "models": ["lgai/exaone-4.0-32b", "google/gemma-4-26b-a4b"] }

Both are local, and the GPU node holds one at a time — so whichever is currently resident answers immediately, and you avoid the wake-up in most cases.

Verifying what happened

Every response says who served it:

X-MyIP-Provider: MyIP Local GPU
X-MyIP-Model: google/gemma-4-26b-a4b
X-MyIP-Generation-Id: gen-01JD8Q2K7M4X9N

For the full story, including candidates that were tried and skipped:

curl "https://openrouter.myip.co.kr/api/v1/generation?id=gen-01JD8Q2K7M4X9N" \
  -H "Authorization: Bearer $MYIP_API_KEY"

provider_responses lists each attempt with its status. A skipped local slot appears there with a slot-related status rather than an HTTP code, which is how you tell "the GPU was busy" apart from "the model errored".

To check readiness before you send real work:

curl -s https://openrouter.myip.co.kr/api/v1/models/google/gemma-4-26b-a4b/endpoints \
  | grep -o '"status":[^,]*'

"status":0 on the local endpoint means the slot is up.

A checklist for building on this

1. Batch by model. The GPU node holds one model at a time. Alternating between two local models per request means paying a load every time.
2. Warm up before bursts. A max_tokens: 1 request costs a fraction of a won and turns the next hundred requests into warm ones.
3. Always read X-MyIP-Provider if the difference between local and external matters to your application — for cost accounting, for data handling, or both.
4. Pin with only when it is a requirement. The default is a preference and is allowed to fall through. only: ["local-gpu"] is the version that fails loudly instead.

Pricing consequence

Because local models are priced independently of any provider rate, their price does not change when the exchange rate does. External candidates are priced by converting the provider's USD rate at our current rate and applying our margin — so the same model can cost different amounts depending on which endpoint answered.

This is one more reason to read X-MyIP-Model and X-MyIP-Provider: the cost in usage.cost belongs to the endpoint that answered, not to the model you asked for. See How costs are calculated.