The waterfall

How it works

A model's slug does not point at one provider. It points at a waterfall, an ordered list of routes (called rungs). Each rung is one way to reach the model: an Experiential-hosted route paid with platform credits, a route on your own provider key, or your own local endpoint. On a model's page the waterfall shows these rungs top to bottom, in the order the gateway tries them.

Every rung has a Use toggle, which splits the list into two bands:

• On (Use is on): the rung is in your chain. The gateway tries these rungs, in the order you set.
• Off (Use is off): the rung stays listed and greyed, but the gateway never uses it. Turn it back on to add it to the chain.

You reorder the on band by dragging a rung by its handle. That order is your organization's fallback chain, saved as you drag. A BYOK rung that is not already first also carries a one-click control that makes it the default (moves it to the top). At least one rung must stay on, so the last on rung cannot be turned off until you turn another on.

Falling back to a lower rung

The order is your preference. The gateway serves the request on the highest on rung it can use. An operational failure on that rung (an authentication, transport, or provider error) can move the request down to the next on rung. A rung whose BYOK connection is unhealthy is dropped until it recovers, and a rung your provider policy bars is skipped for the first route your policy allows. Off rungs are never used.

The kind of rung that serves decides who pays:

• Platform-funded (house) rung: served on platform credits, priced from the public catalog. A house rung below a passed-over rung keeps the model reachable.
• BYOK rung: served on your own provider key. When a BYOK rung is dropped because its connection is unhealthy, the gateway does not move that traffic onto platform credit; it drops the rung and uses the next on rung.

If every on rung is a BYOK rung and none can serve, the request fails closed and the error names the connection that could not serve. Add a house rung or a second key if you want a route beyond a single account.

Adding a key replaces the platform lane for that provider by default.With no custom order, your OpenAI key takes the place of the platform's OpenAI rung at the same position: requests that would have used platform credits on OpenAI use your key instead, and the platform's OpenAI rung is not dialled. The waterfall shows your key where the replaced rung was, with the replaced platform rung in the off band. To use your key only as a fallback, turn the platform rung on and drag your key below it. That order becomes your organization's chain for the model.

A BYOK key can also carry fallback rules: the failures it is a fallback for. The rules are per key, and you set them on a rung that sits belowthe first. A key with fallback rules never replaces the platform lane and is never dialled first, so it reaches a route only through a Waterfall that places it below another rung: setting a rule from a model page writes the displayed order as that model's chain first, and a rule on a key that no Waterfall places below a rung is refused with that instruction. On a model where no chain names such a key, the key is off until you turn it on. An active restriction remains visible even if you move the key first: reordering does not clear its account-wide rule or make it eligible for a first dial. The waterfall shows a small fallback-rules icon to the left of its Use toggle; open it to inspect or deliberately change which failures dial this key. By default every failure is selected, so the key is a fallback wherever you place it. Deselect down to a subset (say only the cybersecurity-policy refusal) and its rungs are never dialled first: they are used only after a prior rung failed with one of the selected failures, and every other failure moves on to the next unrestricted rung instead. The rule follows the key, so it applies to that account's rung on every model whose Waterfall places it.

Retries and fallback per request

Set gateway.retry and gateway.routing in the JSON body of a Chat Completions, Responses, or Messages generation request. SDK users pass this extension through extra_body in Python or directly in the JavaScript request object. These are gateway-owned controls, not provider payload fields, and are never sent upstream. Omitting them keeps the existing operator policy; there is no default backoff mode. Generating Responses WebSocket turns accept the same policy; non-generating prewarm turns, token-count requests, batch lines, and endpoints other than these three generation surfaces reject it rather than ignoring it.

• max_attempts_per_route: an integer from 1 through 4, including the first dispatch. Three attempts means two retries, not three retries.
• max_total_attempts: an integer from 1 through 8 across all routes. The first attempt counts here too. Each supplied cap bounds actual generation dispatches, including throttle redials and internal repair calls; model continuations after a gateway-run tool search also spend this budget.
• allow_fallbacks: false restricts the request to the selected eligible route, while still allowing retries there within the caps. Without a route_id, normal eligibility and initial route selection still apply. true permits the existing authorized waterfall to provide successors.

curl "https://api-pr-144.preview.experientiallabs.ai/v1/chat/completions" \
  -H "Authorization: Bearer $EXPLABS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "gpt-4o-mini",
  "messages": [
    {
      "role": "user",
      "content": "Hello"
    }
  ],
  "gateway": {
    "retry": {
      "max_attempts_per_route": 3,
      "max_total_attempts": 6,
      "backoff": {
        "type": "none"
      }
    },
    "routing": {
      "allow_fallbacks": true
    }
  }
}'

An immediate fallback means max_attempts_per_route: 1 with fallback enabled: after an eligible failure, skip same-route retries and move on. It does not interrupt a healthy call or retry an invalid request. Attempt counts are ceilings, not a promise to use every attempt: circuit breakers, route health, request deadlines and credit limits can stop work earlier. A total cap of one with fallback disabled is the right choice when your application handles failures itself.

Backoff: none or exponential

gateway.retry.backoff has exactly two forms: { "type": "none" } or { "type": "exponential" }. Exponential backoff accepts base_delay_ms (1 through 10,000; default 500), max_delay_ms (at least the base and at most 60,000; default 8,000), and multiplier (1 through 4; default 2). The nominal delay is capped exponential growth; equal jitter chooses between half and all of that delay. There is no unbounded sleep or busy retry loop.

curl "https://api-pr-144.preview.experientiallabs.ai/v1/chat/completions" \
  -H "Authorization: Bearer $EXPLABS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "gpt-4o-mini",
  "messages": [
    {
      "role": "user",
      "content": "Hello"
    }
  ],
  "gateway": {
    "retry": {
      "max_attempts_per_route": 3,
      "max_total_attempts": 6,
      "backoff": {
        "type": "exponential",
        "base_delay_ms": 250,
        "max_delay_ms": 2000,
        "multiplier": 2
      }
    },
    "routing": {
      "allow_fallbacks": true
    }
  }
}'

none removes discretionary waiting, not a provider's Retry-After requirement. With none, a positive provider retry floor prevents an immediate same-route retry: the gateway advances when permitted or returns the refusal. Exponential waits must fit both their configured ceiling and the request deadline. A backoff choice never extends the total request deadline or adds eligibility for another route.

Select one route, including a lower rung

Discover route_id values with authenticated GET /api/models/<slug>/providers. A customer key supplies its organization context; session and operator callers must supply an authorized org_id when the context is ambiguous. Anonymous or unscoped listings return no usable route handle. Synthetic BYOK account rows also return null: an account alone does not identify one certified lane. Use the opaque handle for the desired rung, not its current list position, provider resource ID, or the gateway's internal deployment ID. Replace the example'sroute_aaa… placeholder with a non-null handle from that response.

curl "https://api-pr-144.preview.experientiallabs.ai/v1/chat/completions" \
  -H "Authorization: Bearer $EXPLABS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
  "model": "gpt-4o-mini",
  "messages": [
    {
      "role": "user",
      "content": "Hello"
    }
  ],
  "gateway": {
    "retry": {
      "max_attempts_per_route": 1,
      "max_total_attempts": 1,
      "backoff": {
        "type": "none"
      }
    },
    "routing": {
      "route_id": "route_aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
      "allow_fallbacks": false
    }
  }
}'

With route_id and fallback disabled, a selected lower rung is the only route allowed. With fallback enabled, it is preferred first and remaining eligible rungs retain their existing relative order. The handle is stable across reordering but grants no authority: a foreign, retired, disabled, or policy-excluded route fails explicitly, never silently substitutes another. A listed route can still be unavailable to a particular request. A conflicting sealed-reasoning continuation is refused rather than reassigned.

Gateway retries and SDK retries are separate

An SDK retry sends a new HTTP request to the gateway. Gateway retry settings apply inside each such request, so retries at both layers multiply. Use max_retries=0 with the Python OpenAI or Anthropic client when your application wants sole retry control; JavaScript clients use maxRetries: 0. Retryable failures may already have provider-reported usage, which is accounted for normally. Retries are not a promise of free work.

OpenRouter-style provider.order, provider.only, andprovider.allow_fallbacksconcern an upstream aggregator's selection where forwarded, not this gateway's waterfall. Use gateway for the policy on this page. Changing that policy under the same idempotency key changes request identity; it is not a replay of the original call. Messages has no idempotency-key replay contract.

Prompt caching on a rung

Prompt caching lets a provider reuse a prefix it has already seen, so the repeated part of your context is billed at a cheaper cached input rate instead of the full input rate. This is a property of the route, not a switch you flip: some routes publish a cached input rate and some do not. The waterfall and the provider table mark a route that has one with a cache tag and show its Cached $/M price.

There is no universal organization or per-rung cache toggle. A cached price describes how reported cache reads are billed; it does not guarantee cache support or a hit. Some providers cache automatically, while Claude routes can require explicit cache checkpoints. Support depends on the exact upstream adapter, model, prefix length, and expiry. Inspect the response usage and ignored-parameter disclosures to verify that your route honors your cache controls. Cache writes can cost more than ordinary input. Falling back to another provider can lose a warm cache. How a cached charge appears on your bill is covered in Credits & billing.

Adding a way

On a model's page, Add a way offers three options. The first two add a rung to thismodel's waterfall, and the new rung lands in the off band so you can place it before you turn it on. The third, Add a local model, is different: it does not add a rung here, it creates a separate model private to your organization.

• Add an API key connects your own provider account and gives the model a pass-through rung on your key. In the provider popup, pick a saved account and choose Use this account, or choose Add accountto save another key. Each Azure account keeps its own endpoint and deployment map. Adding an account never replaces an existing one; a duplicate name is refused. Selecting a saved account changes this model's waterfall only when you choose Use this account. New BYOK connections require Pro unless your organization already has a connected provider.
• Serve it yourselfroutes the existing model through an endpoint you control while keeping the model's own format. We verify it with one live check before the rung is added, so a broken endpoint never lands. A Pro feature.
• Add a local model registers your own OpenAI-compatible server as a model private to your organization, callable by slug. A Pro feature.

The step-by-step for each path, the fields every provider needs, and the API shapes are in Adding models.

The free rung

A model covered by a free promotion shows a Free tier rung pinned above the chain: the same Experiential-hosted route, priced Free. It serves from a daily free allowance and is display-only, so it has no Use toggle and no drag handle.

Below it sits the After the free allowance control. When off, requests stop if the remaining free allowance cannot fund the request. When on, the gateway tries free funding first, then uses platform credits if needed. The switch applies to every free model. Card verification unlocks the manual control; a real payment enables it automatically unless you opted out. See Credits & billing for credits, spend limits, and auto-recharge.

See also

Models covers the catalog, the two payment lanes, and the waterfall API (GET / PUT /api/models/<slug>/waterfall). Adding models walks each way to add a rung in detail.