OpenAI SDK compatibility
Smart Inference speaks the OpenAI chat completions protocol. Most fields work unchanged. A few are not accepted yet. This page is the honest matrix.
- Works today -
model,messages,temperature,top_p,max_tokens,stream,tools,tool_choice,response_format,stop,seed - Not accepted -
n,frequency_penalty,presence_penalty,logit_bias,logprobs,user - Forwarded as-is - anything the provider accepts, we accept. Anything it rejects, we surface as a
502 upstream_error.
Parameters
| Parameter | Status | Notes |
|---|---|---|
model |
Supported | Required. Use a canonical ID from /v1/models or a tier keyword (light, medium, high). |
messages |
Supported | Full role/content/tool_calls/tool_call_id shape. Text and tool messages work. |
temperature |
Supported | Float, typically 0.0-2.0. Passed through. |
top_p |
Supported | Float 0.0-1.0. Nucleus sampling. Passed through. |
max_tokens |
Supported | u32. Clamped to the chosen provider’s limit. |
stream |
Supported | Default false. When true, returns SSE ending with data: [DONE]. |
stop |
Supported | String or array of strings. |
seed |
Supported | u64. Provider support varies; treat as a hint. |
tools |
Supported | Function definitions. Shape is converted per provider. |
tool_choice |
Supported | Raw JSON passthrough. |
response_format |
Supported | {"type": "json_object"} works where the underlying model supports it. |
n |
Not accepted | We only return a single choice. |
frequency_penalty |
Not accepted | Ignored - the gateway doesn’t read this field. |
presence_penalty |
Not accepted | Ignored. |
logit_bias |
Not accepted | Ignored. |
logprobs |
Not accepted | Ignored. |
user |
Not accepted | We track usage per API key, not per user field. |
Response fields
All fields that OpenAI returns are returned here too: id, object, created, model, choices, usage. Tool calls appear on choices[].message.tool_calls. Streaming chunks follow the standard chat.completion.chunk shape.
Known differences
- No
n> 1. Multiple completions per call are not supported. Call the endpoint twice if you need two samples. - Model IDs are canonical. If you pass an OpenAI-specific ID that we map via an alias, you still get a canonical response
modelfield. Do not depend on the exact string that comes back. - Tool schemas are translated when routing to a provider with a different native format. You do not need to do anything; we convert on the way in and out.
When something breaks
If you pass a parameter that the chosen provider doesn’t support, you get a 502 upstream_error. That’s your signal to drop the parameter or pick a different model.