Smart Inference

API reference

The Smart Inference gateway exposes three OpenAI-compatible endpoints. All paths sit under /v1. The default production base URL is https://smart.aivory.net/v1.

  • POST /v1/chat/completions - chat completions, streaming or not
  • GET /v1/models - what’s in the catalogue right now
  • GET /v1/pricing - live per-model prices, filterable
  • GET /v1/pricing/benchmark - price comparison against major cloud providers
  • GPU Rental - rent spot GPUs via /v1/gpu/* (full reference)

Authentication

Every request needs a Bearer token.

A u t h o r i z a t i o n : B e a r e r s i _ . . .

A missing or invalid token returns 401 authentication_error. An empty balance returns 402 insufficient_credits. See Errors for the full table.

POST /v1/chat/completions

The main endpoint. Same shape as OpenAI chat completions.

Request body

{
  "model": "llama-3.3-70b",
  "messages": [
    {"role": "system", "content": "You are a terse assistant."},
    {"role": "user",   "content": "Ship it."}
  ],
  "temperature": 0.7,
  "top_p": 1.0,
  "max_tokens": 512,
  "stream": false,
  "stop": null,
  "seed": null,
  "tools": null,
  "tool_choice": null,
  "response_format": null
}

Required: model, messages. Everything else is optional. The model field also accepts the tier keywords light, medium, or high — the router picks the cheapest model in that tier. See Model catalogue for details.

For the full supported/unsupported parameter matrix, see SDK compatibility.

Non-streaming response

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1744634400,
  "model": "llama-3.3-70b",
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": "Shipped." },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 42,
    "completion_tokens": 3,
    "total_tokens": 45
  }
}

Token counts are real, not estimated. That’s what we bill on.

Streaming response

Set "stream": true. You get Server-Sent Events, newline-delimited, in the OpenAI format.

d d d d d a a a a a t t t t t a a a a a : : : : : { { { { [ " " " " D i i i i O d d d d N " " " " E : : : : ] " " " " c c c c h h h h a a a a t t t t c c c c m m m m p p p p l l l l - - - - a a a a b b b b c c c c " " " " , , , , " " " " o o o o b b b b j j j j e e e e c c c c t t t t " " " " : : : : " " " " c c c c h h h h a a a a t t t t . . . . c c c c o o o o m m m m p p p p l l l l e e e e t t t t i i i i o o o o n n n n . . . . c c c c h h h h u u u u n n n n k k k k " " " " , , , , " " " " c c c c h h h h o o o o i i i i c c c c e e e e s s s s " " " " : : : : [ [ [ [ { { { { " " " " i i i i n n n n d d d d e e e e x x x x " " " " : : : : 0 0 0 0 , , , , " " " " d d d d e e e e l l l l t t t t a a a a " " " " : : : : { { { { " " " } r c c , o o o " l n n f e t t i " e e n : n n i " t t s a " " h s : : _ s " " r i H e s e t a t l h s a l e o n o r n t " e " " } " : } } } " } ] } s ] } ] t } } o p " } ] , " u s a g e " : { " p r o m p t _ t o k e n s " : 1 0 , " c o m p l e t i o n _ t o k e n s " : 5 , " t o t a l _ t o k e n s " : 1 5 } }

The final chunk carries usage and finish_reason. The stream ends with data: [DONE] - the same marker OpenAI sends, so the OpenAI SDK parses it without changes.

If a stream is cut off mid-flight, you are billed for the tokens that actually arrived.

Global timeout

Requests are capped at 120 seconds end-to-end. Very long completions may be truncated; keep that in mind for batch work.

GET /v1/models

Returns every canonical model ID in the catalogue, with availability status and cost estimates.

{
  "object": "list",
  "data": [
    {
      "id": "llama-3.3-70b",
      "object": "model",
      "created": 0,
      "owned_by": "smart-inference",
      "providers": ["provider-a", "provider-b", "provider-c"],
      "provider_count": 3,
      "availability": "instant"
    },
    {
      "id": "meta-llama/Llama-3.3-70B-Instruct",
      "object": "model",
      "owned_by": "smart-inference",
      "availability": "deploying",
      "estimated_deploy_seconds": 300,
      "estimated_cost_per_hour": 1.20
    }
  ]
}

Availability tiers

Tier Meaning
instant Model is served by hosted API providers or an active spot GPU. No wait time.
deploying A spot GPU is being provisioned. estimated_deploy_seconds gives the ETA.
cold Model can be deployed on a spot GPU. First request triggers deployment (3-10 min) and places a credit hold for the minimum session cost.
pricing_pending Model is available from a provider but pricing has not been confirmed yet. It will become routable once pricing is set.

The providers array holds routing identifiers – not brand names – and is there for diagnostic use. If you just want names, read data[].id. Supports ?limit=N and ?offset=N for pagination.

GET /v1/pricing

Two response shapes. Pick with ?format=.

?format=flat (default)

One row per deployment. Cheapest per model rolls up on the client side.

{
  "object": "list",
  "count": 42,
  "data": [
    {
      "canonical_model_id": "llama-3.3-70b",
      "provider_id": "provider-a",
      "model_id": "<opaque>",
      "region": null,
      "pricing": {
        "type": "token",
        "input_price_per_million": "0.36",
        "output_price_per_million": "0.36",
        "cached_input_price": null
      },
      "capabilities": {
        "context_window": 131072,
        "max_output_tokens": 4096,
        "supports_function_calling": true,
        "supports_json_mode": true,
        "quality_tier": 2
      },
      "health": {
        "status": "healthy",
        "latency_p50_ms": 250,
        "latency_p95_ms": 750,
        "tokens_per_second": 50
      },
      "last_price_update": "2026-04-14T12:30:00Z"
    }
  ]
}

?format=tiers

Grouped by quality tier (light, medium, high) with the cheapest input/output rates already rolled up.

Query filters

  • ?model=<canonical-id> - single model
  • ?provider=<id> - only rows from a specific source

All three query parameters stack.

GET /v1/pricing/benchmark

Returns a comparison of AIVory prices against major cloud providers for a curated set of models.

{
  "object": "benchmark_list",
  "benchmarks": [
    {
      "canonical_model_id": "llama-4-maverick",
      "model_meta": {
        "display_name": "Llama 4 Maverick",
        "context_window": 1048576,
        "params_b": 400,
        "open_weight": true
      },
      "our_price": {
        "input_per_million": "0.13",
        "output_per_million": "0.25"
      },
      "competitors": [
        { "vendor": "azure-ai-foundry", "label": "Azure AI", "input_per_million": "0.60", "output_per_million": "1.50" },
        { "vendor": "aws-bedrock", "label": "AWS Bedrock", "input_per_million": "0.55", "output_per_million": "1.30" }
      ]
    }
  ],
  "updated_at": "2026-05-18T00:00:00Z"
}

GPU Rental

Four endpoints for renting spot GPUs directly (/v1/gpu/*). The full reference — request bodies, query filters, response schemas — lives in the dedicated GPU docs:

GPU API reference →

Endpoint Method Purpose
/v1/gpu/offers GET Browse available spot GPU offers with pricing
/v1/gpu/rent POST Start a GPU rental
/v1/gpu/rentals/{id} GET Check rental status
/v1/gpu/rentals/{id} DELETE Terminate a rental

See also: GPU billing and GPU overview.

Not implemented

  • /v1/embeddings
  • /v1/images/generations
  • /v1/audio/speech
  • /v1/files

If you need any of these, write to support.

Next steps

SDK compatibility
Which chat parameters are honored
Response headers
Read the receipt we attach to every request
Errors
Status codes and what to do about each