> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pullfrog.com/llms.txt
> Use this file to discover all available pages before exploring further.

# OpenAI-compatible endpoints

Pullfrog can route LLM calls to any endpoint that speaks the OpenAI chat-completions API — a [LiteLLM](https://docs.litellm.ai/docs/simple_proxy) proxy, a [Cloudflare AI Gateway](https://developers.cloudflare.com/ai-gateway/), [Alibaba DashScope](https://www.alibabacloud.com/help/en/model-studio/) (Qwen), a self-hosted [vLLM](https://docs.vllm.ai/) server, or any compatible gateway. Useful when your model isn't in the catalog, when you front several providers behind one gateway, or when you run your own inference.

<Note>
  OpenAI-compatible support is BYOK only. Inference is billed by your provider to your account; Pullfrog never proxies this traffic.
</Note>

## Setup

Pick the model in the dropdown, then supply the endpoint, key, model ID, and that model's two token limits. Only the key is sensitive — but there are two ways to supply the set, and which one you want depends on how your org pays for models.

### 1. Select Custom › OpenAI-compatible in the model dropdown

In `pullfrog.com/console/<account>/<repo>`, open the model dropdown and pick **Custom › OpenAI-compatible**. Like Bedrock and Vertex, this entry has no per-model variants — the actual model is supplied separately via `OPENAI_COMPATIBLE_MODEL` (see step 3), because the endpoint, not Pullfrog, owns the model catalog.

### 2. Supply the five values

**Either** store all five in the console — open **Model costs → Custom → Set up** and fill the form. Pullfrog injects them at run time, so you never touch `pullfrog.yml` and can skip step 3 entirely. This is the path to use on a **Router-funded org**: the server can only honor a stored `openai-compatible/byok` default when it can see the credentials itself (see the Router note below).

**Or** put `OPENAI_COMPATIBLE_API_KEY` in a [GitHub Actions secret](https://github.com/settings/secrets/actions) and the other four straight into your workflow `env:` (step 3). Only the key is sensitive — the base URL, model ID, and token limits are plain config and don't need encrypting, though storing them in Pullfrog is equally fine.

Values in your workflow `env:` win over Pullfrog-stored ones, so you can store the set once for the org and override a single value per repo.

### 3. Wire the env vars in your workflow (only if you chose the workflow path)

```yaml theme={null}
env:
  OPENAI_COMPATIBLE_BASE_URL: https://gateway.ai.cloudflare.com/v1/<acct>/<gateway>/compat
  OPENAI_COMPATIBLE_API_KEY: ${{ secrets.OPENAI_COMPATIBLE_API_KEY }}
  OPENAI_COMPATIBLE_MODEL: workers-ai/@cf/meta/llama-3.1-8b-instruct
  OPENAI_COMPATIBLE_CONTEXT: "128000"
  OPENAI_COMPATIBLE_MAX_OUTPUT: "16384"
```

| Env var                        | Required | Sensitive | Purpose                                                                                                     |
| ------------------------------ | -------- | --------- | ----------------------------------------------------------------------------------------------------------- |
| `OPENAI_COMPATIBLE_BASE_URL`   | yes      | no        | Base URL of the endpoint (the path that serves `/chat/completions`).                                        |
| `OPENAI_COMPATIBLE_API_KEY`    | yes      | **yes**   | API key / bearer token for the endpoint. Store as a secret.                                                 |
| `OPENAI_COMPATIBLE_MODEL`      | yes      | no        | The exact model ID the endpoint serves. What you set is what runs — Pullfrog never resolves or upgrades it. |
| `OPENAI_COMPATIBLE_CONTEXT`    | yes      | no        | Context-window size of that model, in tokens.                                                               |
| `OPENAI_COMPATIBLE_MAX_OUTPUT` | yes      | no        | Max completion tokens that model accepts.                                                                   |

All five must be present, or the run fails fast with a setup error naming what's missing. The two limits must parse as positive numbers — `128k` is rejected.

### Why the token limits are required

Because your endpoint owns the model catalog, Pullfrog has no metadata about the model's limits and cannot discover them. Undeclared, they don't degrade gracefully — they break two things at once:

* **Completions get capped at 32000 tokens**, and any model with a smaller cap rejects the request outright: `max_tokens is too large: 32000. This model supports at most 16384 completion tokens`. GPT-4o and GPT-4o-mini cap at 16384, many open models at 4096 or 8192.
* **Auto-compaction switches off entirely.** Compaction keys off the context limit, so with none declared a long session never compacts — it grows until your endpoint refuses it on context length. Nothing fails until the session is already too big, which is why this is required rather than merely recommended.

Set `OPENAI_COMPATIBLE_CONTEXT` to the model's real window. Understating it makes the agent compact earlier than necessary; overstating it defers compaction until your endpoint refuses the request.

<Note>
  Put the key in `OPENAI_COMPATIBLE_API_KEY`, never inside `OPENAI_COMPATIBLE_BASE_URL`. The base URL is treated as non-sensitive and is not masked in logs — credentials embedded in it would leak.
</Note>

<Note>
  On a Router-funded org, a **stored** `openai-compatible/byok` default only takes effect if the credentials are saved as **Pullfrog secrets** (server-visible). If they live solely in your workflow `env:` / GitHub Actions secrets, the server can't see them, so a no-card run falls back to the Router default model instead of your endpoint. BYOK orgs are unaffected.
</Note>

The model ID follows your gateway's own convention. On Cloudflare AI Gateway's unified endpoint that's `<provider>/<model>` — e.g. `workers-ai/@cf/meta/llama-3.1-8b-instruct` ([docs](https://developers.cloudflare.com/ai-gateway/usage/chat-completion/)). On DashScope it's the bare model ID, e.g. `qwen3-coder-plus`. On a LiteLLM proxy it's the `model_name` you declared in the proxy's `model_list`.

### LiteLLM

A [LiteLLM proxy](https://docs.litellm.ai/docs/simple_proxy) is an OpenAI-compatible endpoint, so it needs no LiteLLM-specific configuration — point the base URL at the proxy's `/v1` path and use a [virtual key](https://docs.litellm.ai/docs/proxy/virtual_keys):

```yaml theme={null}
env:
  OPENAI_COMPATIBLE_BASE_URL: https://litellm.internal.example.com/v1
  OPENAI_COMPATIBLE_API_KEY: ${{ secrets.LITELLM_VIRTUAL_KEY }}
  OPENAI_COMPATIBLE_MODEL: claude-opus-internal
  OPENAI_COMPATIBLE_CONTEXT: "200000"
  OPENAI_COMPATIBLE_MAX_OUTPUT: "32000"
```

`OPENAI_COMPATIBLE_MODEL` is the `model_name` from your proxy's `model_list`, not the upstream provider's ID — LiteLLM's own routing, failover, and spend tracking then apply as configured. The same shape covers models the proxy fronts from self-hosted vLLM or Ollama.

## How it works

Pullfrog materializes the endpoint as an [`@ai-sdk/openai-compatible`](https://www.npmjs.com/package/@ai-sdk/openai-compatible) provider at run time and points it at `OPENAI_COMPATIBLE_BASE_URL` with your key. The run targets `openai-compatible/<OPENAI_COMPATIBLE_MODEL>` through the OpenCode agent. Because the model ID is whatever your endpoint serves, there's nothing to catalog or version — the same single entry covers every OpenAI-compatible gateway and model.