> ## Documentation Index
> Fetch the complete documentation index at: https://bench.flashinfer.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Benchmark Server API

> Start `flashinfer-bench serve` and use its HTTP API to inspect definitions, submit solutions, and poll evaluation tasks.

`flashinfer-bench serve` exposes an HTTP service for evaluating submitted `Solution` objects against workloads in a local `TraceSet`.

It is a benchmark evaluation service, not a general model inference server.

## Start The Server

Install the server dependencies first:

```bash theme={null}
pip install "flashinfer-bench[serve]"
```

Then start the server against a local trace dataset:

```bash theme={null}
flashinfer-bench serve \
  --local /path/to/flashinfer-trace \
  --host 0.0.0.0 \
  --port 8000 \
  --devices cuda:0,cuda:1
```

CLI flags:

| Flag | Type | Required | Default | Description |
| - | - | - | - | - |
| `--local` | `path` | Yes | None | Path to the local `TraceSet`. |
| `--devices` | `string` | No | All available CUDA devices | Comma-separated CUDA devices such as `cuda:0,cuda:1`. |
| `--host` | `string` | No | `0.0.0.0` | Host address for the HTTP server. |
| `--port` | `integer` | No | `8000` | Port for the HTTP server. |
| `--warmup-runs` | `integer` | No | `10` | Number of warmup runs before measurement. |
| `--iterations` | `integer` | No | `50` | Number of benchmark iterations per trial. |
| `--num-trials` | `integer` | No | `3` | Number of benchmark trials per workload. |
| `--rtol` | `float` | No | `1e-2` | Relative tolerance for correctness checks. |
| `--atol` | `float` | No | `1e-2` | Absolute tolerance for correctness checks. |
| `--timeout` | `integer` | No | `300` | Per-solution evaluation timeout in seconds. |
| `--log-level` | `enum` | No | `INFO` | Server log level. One of `DEBUG`, `INFO`, `WARNING`, or `ERROR`. |

## Mental Model

The server evaluates one submitted `Solution` asynchronously:

* `Solution`: The implementation you submit to the server.
* `Task`: The asynchronous evaluation job created for that submission.
* `Trace`: One evaluation result for one workload under that task.

One task may produce multiple traces because the same solution can be evaluated on multiple workloads.

Two status layers matter:

* `task.status` tracks task lifecycle: `pending`, `running`, `completed`, or `failed`.
* `traces[*].evaluation.status` tracks the actual evaluation result for each workload, such as `PASSED`, `COMPILE_ERROR`, `RUNTIME_ERROR`, or `TIMEOUT`.

`task.status = completed` only means the task finished running. It does not mean the solution passed correctness checks.

## API Reference

### `GET /definitions`

**Purpose**

List available definitions in the loaded `TraceSet`.

**Request**

No request body.

**Response**

Returns an array of definition summaries.

| Field | Type | Description |
| - | - | - |
| `name` | string | Definition name. |
| `description` | string or `null` | Optional definition description. |

Example response:

```json theme={null}
[
  {
    "name": "rmsnorm_h128",
    "description": "..."
  }
]
```

**Errors**

No endpoint-specific error behavior beyond standard server failures.

### `GET /definitions/{name}`

**Purpose**

Return the full serialized `Definition` object for one definition.

**Request**

Path parameters:

| Parameter | Type | Required | Description |
| - | - | - | - |
| `name` | string | Yes | Definition name. |

**Response**

Returns the full serialized `Definition` object.

Use this endpoint when you need the exact contract before writing a passing solution.

**Errors**

* `404`: Definition not found.

### `GET /definitions/{name}/workloads`

**Purpose**

List workloads for one definition.

**Request**

Path parameters:

| Parameter | Type | Required | Description |
| - | - | - | - |
| `name` | string | Yes | Definition name. |

**Response**

Returns an array of serialized `Workload` objects.

Use this endpoint to discover valid workload UUIDs for `POST /evaluate`.

**Errors**

* `404`: Definition not found.

### `GET /workloads/{uuid}`

**Purpose**

Return one workload by UUID.

**Request**

Path parameters:

| Parameter | Type | Required | Description |
| - | - | - | - |
| `uuid` | string | Yes | Workload UUID. |

**Response**

Returns the serialized `Workload` object.

**Errors**

* `404`: Workload not found.

### `POST /evaluate`

**Purpose**

Submit one solution for evaluation.

**Request**

Request body fields:

| Field | Type | Required | Description |
| - | - | - | - |
| `solution` | object | Yes | Full [Solution](https://bench.flashinfer.ai/docs/flashinfer-trace/solution.md) object to evaluate. |
| `workload_uuids` | string\[] | No | Optional subset of workload UUIDs. If omitted, the server evaluates all workloads for the definition. |

Illustrative payload example:

```json theme={null}
{
  "solution": {
    "name": "my_solution",
    "definition": "rmsnorm_h128",
    "author": "alice",
    "spec": {
      "language": "python",
      "target_hardware": ["cuda"],
      "entry_point": "pkg/main.py::kernel",
      "destination_passing_style": false
    },
    "sources": [
      {
        "path": "pkg/main.py",
        "content": "import torch\n\ndef kernel(x):\n    return x\n"
      }
    ]
  },
  "workload_uuids": ["workload_uuid_1"]
}
```

This payload is illustrative only. The submitted `Solution` still needs to match the selected definition's real inputs and outputs.

**Response**

Response fields:

| Field | Type | Description |
| - | - | - |
| `task_id` | string | Identifier for the asynchronous evaluation task. |
| `normalized_solution_name` | string | Server-normalized solution name after `Solution.with_unique_name()`. |

Example response:

```json theme={null}
{
  "task_id": "7f8f5b1d4f0e4b3b8c4b8c1a2d3e4f5a",
  "normalized_solution_name": "my_solution_1a2b3c4d"
}
```

* The server normalizes the submitted solution name by calling `Solution.with_unique_name()`.
* `normalized_solution_name` is deterministic for the same solution content.
* If the selected workloads are empty, the task is still created, but it later ends with `task.status = failed`.

**Errors**

* `400`: `solution.definition` does not exist.

### `GET /tasks/{task_id}`

**Purpose**

Get one task by ID.

**Request**

Path parameters:

| Parameter | Type | Required | Description |
| - | - | - | - |
| `task_id` | string | Yes | Task identifier. |

Query parameters:

| Parameter | Type | Required | Description |
| - | - | - | - |
| `timeout` | float | No | Optional value in the range `0..3600`. `0` means return immediately. A positive value enables long-polling until the task completes or the timeout expires. |

**Response**

Response fields:

| Field | Type | Description |
| - | - | - |
| `task_id` | string | Task identifier. |
| `status` | string | Task lifecycle status: `pending`, `running`, `completed`, or `failed`. |
| `definition` | string | Definition name associated with the submitted solution. |
| `solution` | string | Normalized solution name used by the server. |
| `traces` | object\[] or `null` | Serialized trace results. Can be `null` while the task is still pending or running. |
| `error` | string or `null` | Task-level failure message. Usually `null` unless `status = failed`. |

Example response:

```json theme={null}
{
  "task_id": "7f8f5b1d4f0e4b3b8c4b8c1a2d3e4f5a",
  "status": "completed",
  "definition": "rmsnorm_h128",
  "solution": "my_solution_1a2b3c4d",
  "traces": [
    {
      "definition": "rmsnorm_h128",
      "solution": "my_solution_1a2b3c4d",
      "workload": {
        "uuid": "workload_uuid_1"
      },
      "evaluation": {
        "status": "PASSED"
      }
    }
  ],
  "error": null
}
```

* If the task is still pending or running, `traces` may be `null`.
* If the task fails at the task level, `error` contains the failure reason.

**Errors**

* `404`: Task not found.

### `POST /tasks/batch`

**Purpose**

Query multiple tasks in one request.

**Request**

Request body fields:

| Field | Type | Required | Description |
| - | - | - | - |
| `task_ids` | string\[] | Yes | Task IDs to query. Response order matches this array. |
| `timeout` | float | No | Optional wait time in seconds. `timeout <= 0` returns immediately. `timeout > 0` waits until all tasks complete or the timeout expires. |

Request body example:

```json theme={null}
{
  "task_ids": ["task_id_1", "task_id_2"],
  "timeout": 30
}
```

**Response**

Returns an array of `TaskResponse` objects.

Each item has the following fields:

| Field | Type | Description |
| - | - | - |
| `task_id` | string | Task identifier. |
| `status` | string | Task lifecycle status: `pending`, `running`, `completed`, or `failed`. |
| `definition` | string | Definition name associated with the submitted solution. |
| `solution` | string | Normalized solution name used by the server. |
| `traces` | object\[] or `null` | Serialized trace results. Can be `null` while the task is still pending or running. |
| `error` | string or `null` | Task-level failure message. Usually `null` unless `status = failed`. |

* Returns a list of `TaskResponse` objects in the same order as `task_ids`.
* Duplicate task IDs are allowed and produce duplicate results.

**Errors**

* `404`: At least one task ID does not exist. The request is fail-fast.

### `GET /health`

**Purpose**

Return worker health and queue depth.

**Request**

No request body.

**Response**

Response fields:

| Field | Type | Description |
| - | - | - |
| `status` | string | Overall server health status. |
| `workers` | object\[] | Per-worker health information. |
| `queue_size` | integer | Number of queued tasks waiting to run. |

Example response:

```json theme={null}
{
  "status": "ok",
  "workers": [
    {
      "device": "cuda:0",
      "healthy": true
    }
  ],
  "queue_size": 0
}
```

This endpoint is intended for operational checks rather than task inspection.

**Errors**

No endpoint-specific error behavior beyond standard server failures.

### `POST /shutdown` (Management)

**Purpose**

Ask the current server process to exit gracefully.

**Request**

No request body.

**Response**

Response fields:

| Field | Type | Description |
| - | - | - |
| `status` | string | Shutdown acknowledgement, currently `shutting_down`. |

Example response:

```json theme={null}
{
  "status": "shutting_down"
}
```

This is a management endpoint, not part of the normal submit-and-poll flow.

**Errors**

No endpoint-specific error behavior beyond standard server failures.

## Polling And Error Semantics

Keep these semantics in mind when integrating with the server:

* `task.status = completed` means the task finished, not that the solution passed.
* Look at `traces[*].evaluation.status` for correctness and performance outcomes.
* `task.status = failed` indicates task-level failures such as missing workloads or other failures that prevent evaluation from completing normally.
* In `GET /tasks/{task_id}`, `timeout` must be in the range `0..3600`.
* In `POST /tasks/batch`, `timeout <= 0` returns immediately and `timeout > 0` waits up to the provided value.
* `POST /tasks/batch` is fail-fast on invalid task IDs.

## Minimal Runnable Example

This example shows the smallest end-to-end flow that works without depending on a specific kernel signature.

It intentionally submits a Python solution with a syntax error, so the task should complete with `COMPILE_ERROR`. That makes the example portable across trace datasets as long as you choose a definition that has at least one workload.

Requirements:

* `curl`
* `jq`
* A running benchmark server

```bash theme={null}
set -euo pipefail

BASE_URL=http://127.0.0.1:8000

# Pick the first definition that has at least one workload.
DEFINITION=$(
  curl -s "$BASE_URL/definitions" | jq -r '.[].name' | while read -r name; do
    count=$(curl -s "$BASE_URL/definitions/$name/workloads" | jq 'length')
    if [ "$count" -gt 0 ]; then
      echo "$name"
      break
    fi
  done
)

WORKLOAD_UUID=$(curl -s "$BASE_URL/definitions/$DEFINITION/workloads" | jq -r '.[0].uuid')

jq -n \
  --arg definition "$DEFINITION" \
  --arg workload_uuid "$WORKLOAD_UUID" \
  '{
    solution: {
      name: "docs_compile_error",
      definition: $definition,
      author: "docs",
      spec: {
        language: "python",
        target_hardware: ["cuda"],
        entry_point: "pkg/main.py::kernel",
        destination_passing_style: false
      },
      sources: [
        {
          path: "pkg/main.py",
          content: "def kernel(\n    return 0\n"
        }
      ]
    },
    workload_uuids: [$workload_uuid]
  }' > /tmp/fib-serve-request.json

TASK_ID=$(
  curl -s \
    -X POST "$BASE_URL/evaluate" \
    -H "Content-Type: application/json" \
    -d @/tmp/fib-serve-request.json | jq -r '.task_id'
)

curl -s "$BASE_URL/tasks/$TASK_ID?timeout=60" | jq
```

Expected result:

* Top-level `status` should become `completed`.
* `traces[0].evaluation.status` should be `COMPILE_ERROR`.

To get `PASSED` instead, inspect `GET /definitions/{name}` and implement a real solution that matches that definition's inputs and outputs.

## Notes

* The server requires at least one CUDA device.
* Reference results are cached per `(definition, workload)` inside each worker process.
* `GET /health` is intended for operational checks rather than task inspection.
* Submitted solution names are normalized before evaluation.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.