Batch

The Batch API lets you run large numbers of requests asynchronously without real-time responses. It is compatible with the OpenAI Batch API: same JSONL format, Files API for uploads, and batch create/retrieve/cancel/list endpoints. Use https://api.8080.io and your 8080 API key in place of OpenAI’s base URL and key.

Batches complete within the chosen completion window (e.g., 24 hours). Input files can contain up to 50,000 requests and be up to 200 MB.

1. Prepare your JSONL file

   Each line must be a single JSON object with:

   Field	Type	Description
   custom_id	string	Your unique ID for this request (used to match results).
   method	string	HTTP method, e.g., "POST".
   url	string	API path, e.g., "/v1/chat/completions".
   body	object	Request body for that endpoint (same as the underlying API).

   All requests in one file must target the same endpoint. The body must match what the endpoint expects (e.g., model, messages for chat completions).

   Example: chat completions input file

   Save as batch_input.jsonl:

   {"custom_id": "req-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "8080/taalas/llama3.1-8b-instruct", "messages": [{"role": "user", "content": "Say hello in one word."}], "max_tokens": 50}}
   {"custom_id": "req-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "8080/taalas/llama3.1-8b-instruct", "messages": [{"role": "user", "content": "What is 2+2?"}], "max_tokens": 50}}
   {"custom_id": "req-3", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "8080/taalas/llama3.1-8b-instruct", "messages": [{"role": "user", "content": "Name a color."}], "max_tokens": 50}}

   Each line is one request. Use custom_id later to map output lines back to your inputs (output order may not match input order).

2. Upload the file (Files API)

   Upload the JSONL file with purpose batch so it can be used as batch input.

   cURL

   curl https://api.8080.io/v1/files \
     -H "Authorization: Bearer $_8080_API_KEY" \
     -F purpose="batch" \
     -F file="@batch_input.jsonl"

   Python

   import os
   import requests
   
   API_KEY = os.environ.get("_8080_API_KEY")
   BASE = "https://api.8080.io"
   
   with open("batch_input.jsonl", "rb") as f:
       r = requests.post(
           f"{BASE}/v1/files",
           headers={"Authorization": f"Bearer {API_KEY}"},
           files={"file": ("batch_input.jsonl", f, "application/jsonl")},
           data={"purpose": "batch"},
       )
   r.raise_for_status()
   file_id = r.json()["id"]
   print("Uploaded file ID:", file_id)

   The response includes an id (e.g., ae1a17d0-...). Use this as input_file_id when creating the batch.

3. Create the batch

   Request: POST /v1/batches

   Parameter	Type	Required	Description
   input_file_id	string	Yes	File ID from the upload step.
   endpoint	string	Yes	Endpoint for all requests in the file, e.g., "/v1/chat/completions".
   completion_window	string	Yes	Time window to complete the batch; e.g., "24h" if supported.
   metadata	object	No	Key-value pairs (e.g., for labeling).
   output_expires_after	object	No	Expiration for output/error files (e.g., {"anchor": "created_at", "seconds": 86400}).

   cURL

   curl https://api.8080.io/v1/batches \
     -H "Authorization: Bearer $_8080_API_KEY" \
     -H "Content-Type: application/json" \
     -d '{
       "input_file_id": "ae1a17d0-...",
       "endpoint": "/v1/chat/completions",
       "completion_window": "24h"
     }'

   Python

   r = requests.post(
       f"{BASE}/v1/batches",
       headers={
           "Authorization": f"Bearer {API_KEY}",
           "Content-Type": "application/json",
       },
       json={
           "input_file_id": file_id,
           "endpoint": "/v1/chat/completions",
           "completion_window": "24h",
           "metadata": {"job": "nightly-eval"},
       },
   )
   r.raise_for_status()
   batch = r.json()
   batch_id = batch["id"]
   print("Batch ID:", batch_id)

   The response is a batch object with id, status (e.g., validating), input_file_id, output_file_id, error_file_id (often null until the batch progresses), and timestamps.

4. Check batch status

   Request: GET /v1/batches/{batch_id}

   Poll this until status is a terminal state.

   cURL

   curl https://api.8080.io/v1/batches/batch_abc123 \
     -H "Authorization: Bearer $_8080_API_KEY"

   Python

   r = requests.get(
       f"{BASE}/v1/batches/{batch_id}",
       headers={"Authorization": f"Bearer {API_KEY}"},
   )
   r.raise_for_status()
   batch = r.json()
   print("Status:", batch["status"])
   print("Request counts:", batch.get("request_counts"))

   Status values

   Status	Description
   validating	Input file is being validated.
   failed	Validation failed; see errors on the batch.
   in_progress	Batch is running.
   finalizing	Batch finished; results are being prepared.
   completed	Done; use output_file_id (and optionally error_file_id) to download.
   expired	Did not finish within the completion window.
   cancelling	Cancel requested; waiting for in-flight work.
   cancelled	Batch was cancelled.

   When status is completed, output_file_id points to a JSONL file of successful results, and error_file_id (if set) points to a JSONL file of failed requests.

5. Retrieve results and errors

   Request: GET /v1/files/{file_id}/content

   Use the batch’s output_file_id and error_file_id from the completed batch object.

   cURL

   # Output (successful requests)
   curl https://api.8080.io/v1/files/RESULTS_FILE_ID/content \
     -H "Authorization: Bearer $_8080_API_KEY" \
     -o batch_output.jsonl
   
   # Errors (failed requests)
   curl https://api.8080.io/v1/files/ERROR_FILE_ID/content \
     -H "Authorization: Bearer $_8080_API_KEY" \
     -o batch_errors.jsonl

   Python

   # After batch status is "completed"
   output_file_id = batch["output_file_id"]
   error_file_id = batch.get("error_file_id")
   
   r = requests.get(
       f"https://api.8080.io/v1/files/{output_file_id}/content",
       headers={"Authorization": f"Bearer {API_KEY}"},
   )
   r.raise_for_status()
   with open("batch_output.jsonl", "w") as f:
       f.write(r.text)
   
   if error_file_id:
       r = requests.get(
           f"https://api.8080.io/v1/files/{error_file_id}/content",
           headers={"Authorization": f"Bearer {API_KEY}"},
       )
       r.raise_for_status()
       with open("batch_errors.jsonl", "w") as f:
           f.write(r.text)

   Output file format (JSONL)

   One line per successful request. Each line is a JSON object with:

   Field	Description
   id	Batch request ID.
   custom_id	The custom_id from your input line.
   response	Object with status_code, request_id, and body (the API response body).
   error	null for success lines.

   Example output line (chat completions):

   {"id": "batch_req_abc", "custom_id": "req-1", "response": {"status_code": 200, "request_id": "req_xyz", "body": {"id": "chatcmpl-...", "object": "chat.completion", "choices": [{"index": 0, "message": {"role": "assistant", "content": "Hello."}, "finish_reason": "stop"}], "usage": {"prompt_tokens": 12, "completion_tokens": 1, "total_tokens": 13}}}, "error": null}

   Match results to inputs using custom_id; do not rely on line order.

   Error file format (JSONL)

   One line per failed request. Each line has id, custom_id, response: null, and error with code and message, for example:

   {"id": "batch_req_456", "custom_id": "req-2", "response": null, "error": {"code": "invalid_request", "message": "Invalid model."}}

List batches

GET /v1/batches (optional query: limit, after for pagination).

cURL

curl https://api.8080.io/v1/batches?limit=20 \
  -H "Authorization: Bearer $_8080_API_KEY"

Python

import os
import requests

API_KEY = os.environ.get("_8080_API_KEY")
BASE = "https://api.8080.io"

r = requests.get(
    f"{BASE}/v1/batches",
    params={"limit": 20},
    headers={"Authorization": f"Bearer {API_KEY}"},
)
r.raise_for_status()
print(r.json())

Cancel a batch

POST /v1/batches/{batch_id}/cancel

cURL

curl https://api.8080.io/v1/batches/batch_abc123/cancel \
  -H "Authorization: Bearer $_8080_API_KEY" \
  -X POST

Python

import os
import requests

API_KEY = os.environ.get("_8080_API_KEY")
BASE = "https://api.8080.io"
batch_id = "batch_abc123"

r = requests.post(
    f"{BASE}/v1/batches/{batch_id}/cancel",
    headers={"Authorization": f"Bearer {API_KEY}"},
)
r.raise_for_status()
print(r.json())

After cancelling, status moves to cancelling then cancelled (may take a few minutes).