GPU connectors
Run the LLM Gateway on your own GPU servers, for example one per data center or region. Each server is a GPU connector in the console. It connects out to QuilrAI with its own deployment ID and connector token, so it needs no inbound firewall rule. You then choose which providers run on which server.
How it works
- One installation per server. You deploy the LLM Gateway on each server with the QuilrAI installer, as usual. A server becomes a GPU connector when you give it a deployment ID and a connector token.
- Outbound only. The server opens an encrypted WebSocket (
wss://) connection to QuilrAI on port 443. QuilrAI never connects in to your network. - Your traffic stays local. Your apps call the gateway on a server, and that server calls your models. These requests don't pass through QuilrAI.
- You decide where each provider runs. You link a provider to a server under Network route (step 5). When QuilrAI calls that provider, for example to list its models, run a test or run red teaming, the work runs on that server.
Before you start
-
QuilrAI turns on GPU connectors for your organization. Until then, the GPU connectors drawer shows GPU connectors aren't available for your organization.
-
Each server runs the LLM Gateway, deployed with the QuilrAI installer by following the deployment guide from QuilrAI. Install it for exactly one tenant ID.
-
Each server can reach QuilrAI on TCP port 443 over HTTPS and WebSocket (
wss://). If a firewall or proxy sits in the path, it must allow WebSocket upgrades. -
You have these console permissions:
1. Add a GPU connector for each server
-
Go to Settings > AI Gateway > LLM Gateway and click GPU connectors.
-
Click New GPU connector.
-
Fill in the form:
-
Click Add GPU connector.
The new card shows Offline until the server connects.
2. Generate a token for each server
- On the server's card, click Generate token.
- In the panel that opens, copy the Deployment ID and the Connector token (
qpc_<id>.<secret>). - Click Done.
Save it in your secret manager. If you lose it, generate a new one. The connector token is not the master token you entered when you installed the gateway; the two are not interchangeable.
3. Turn on the connector on each server
The installer already set your tenant ID, the QuilrAI URL and your master token. To turn the server into a GPU connector, add these values:
Run these steps from the installer folder, with kubectl pointing at the server's cluster. The commands use the installer's default names: the quilr-service namespace and the quilr-workers-auth Secret.
-
Save the connector token in the workers Secret:
kubectl -n quilr-service patch secret quilr-workers-auth --type merge \
-p '{"stringData":{"PRIVATE_CONTROL_PLANE_CREDENTIAL":"qpc_<id>.<secret>"}}' -
Create a file named
gpu-connector.yamlwith the server's settings:workers:
extraConfig:
LLMGATEWAY_DEPLOYMENT_TYPE: "PRIVATE"
PRIVATE_DEPLOYMENT_ID: "us-east-gpu-1" # must match the ID in the console
SERVER_NAME: "us-east-gpu-1" # any unique label; the ID is a simple choice -
Apply it:
./upgrade-llm-gateway.sh --components workers -f gpu-connector.yamlPress Enter at the image tag prompt to keep the current version, then enter
yto proceed. The script keeps your existing settings, adds these values and restarts the workers pod.
The restart makes that server's gateway unavailable for about 1 to 2 minutes. Repeat these steps on every server with its own ID and token.
- Later runs of
upgrade-llm-gateway.shkeep these values. If you runinstall.shagain, add--llm-values gpu-connector.yamlso they stay. - Never use the same deployment ID on two servers.
4. Check that each server is online
In the GPU connectors drawer, the card turns Online (1 instance) shortly after the workers pod is ready, and shows Last heartbeat and Version.
If a server stays Offline, read its connector log:
kubectl -n quilr-service exec deploy/quilr-workers -- \
gosu quilr tmux capture-pane -t quilr:private-connector -p -S -50
A working connection logs PRIVATE_CONNECTOR | connected | tenant_id=<your-tenant-id> | deployment_id=us-east-gpu-1.
5. Link providers to a server
Link each provider whose models run on a GPU server to that server. You can link a provider when you add it, or later.
Link a new provider
-
Go to Settings > AI Gateway > Models and click Add provider. You can also click Add provider in Provider configuration on the LLM Gateway page.
-
Under Where are the models hosted?, select the provider and its API. For an OpenAI-compatible model server such as vLLM, select Custom endpoint, then Chat completions.
-
Under Name and credentials, enter a Label and the API key that your model server accepts.
-
Set Base URL to an address that the server can reach, for example a model Service in the server's cluster:
http://llama-70b.inference.svc.cluster.local:8000/v1. -
Open Network route and select the server under GPU servers. Each server shows its IP address and status.
-
Click Test & continue. The server you selected lists the models it can reach, and the Connected tag confirms that the link works.
-
Select the models, click Review, then click Save provider.
When the Base URL you enter uses the IP address or hostname recorded for a server, the console selects that server and says so under Network route. Pick another route if the provider is reached a different way.
Link an existing provider
Use the same steps to move a provider to another server, or to remove the link by selecting Public internet (direct).
- Go to Settings > AI Gateway > Models and open the provider from the Providers tab.
- Select the Connection tab, then click Rotate credentials… in the Credentials card.
- Under Network route, select the server under GPU servers. Leave API key blank to keep the stored key.
- Click Test. The test runs on the server you selected.
- When the test shows Passed, click Save new credentials. Until you save, the provider keeps running where it ran before.
Check which server a provider uses
-
On the provider: the Connection tab shows GPU server with the server's name, IP address and status.
-
On the server: in the GPU connectors drawer, each card lists its providers under Providers on this server.
Good to know
- A linked provider always runs on its server. If the server is offline, the provider's requests and tests fail with
GPU connector "<name>" is offline or unreachable. There is no fallback to another server. - A provider has one network route. Selecting a GPU server removes a remote connector link, and the other way around.
- Disabled servers aren't listed, except on a provider that is already linked to one.
- Apps with app-only credentials have the same Network route field in the app's LLM Providers section.
- Leave public providers on Public internet (direct), for example OpenAI or Amazon Bedrock.
- Some provider types can't use a GPU server: OpenAI Realtime, Azure OpenAI Realtime, Quilr SDK and Copilot Studio.
Where provider calls run
We recommend linking every provider with a private endpoint to the right server. The default server then only catches providers nobody linked.
Manage your servers
Change a server's token
A server can have up to five active tokens, so you can switch before you revoke the old one.
-
On the server's card, click Generate token.
-
Save the new token in the workers Secret with the
kubectl patchcommand from step 3. -
Restart the workers so they read it:
kubectl -n quilr-service rollout restart deploy/quilr-workers -
When the card shows Online again, open Tokens (N active) and Revoke the old token. Each token is listed by the part before the
..
Troubleshooting
Related
- Remote connectors - reach a private model endpoint through an outbound tunnel, without running the gateway yourself.
- Providers and models - add providers and link them to apps.