Custom Endpoints

This guide will walk you through the process of getting started with deploying your first project to the 8080 Edge cloud, with a simple example of a chat completions API wrapper.

Prerequisites

First you’ll need to have the 8080 CLI installed and authenticated. In order to successfully deploy your code, you’ll also need to have Docker installed on your system.

Create a project

Next step is to initialize a project:

# first create a new directory and cd into it
mkdir chat && cd chat

# run the 8080 init to create project files
8080 init

Local development

You should now see some files created by the script to help you get started, but let’s direct our attention to the main.py file.

from e80_sdk import Eighty80, eighty80_app
from fastapi import Request

app = eighty80_app()

# Get an OpenAI SDK-compatible object to talk to the 8080 API
api = Eighty80().completion_sdk()

@app.post("/v1/chat/completions")
async def completions(request: Request):
    body = await request.json()
    return api.chat.completions.create(**body)

In this example, you’ll see a very simple web application exposing a single endpoint /v1/chat/completions, essentially serving as a thin wrapper around the OpenAI-compatible API with your own custom code.

For example, try prepending a system prompt that gets added to every completion request:

@app.post("/v1/chat/completions")
async def completions(request: Request):
    body = await request.json()
    body['messages'].insert(0, {
        "role": "developer",
        "content": "Only respond in pig latin no matter what the user prompts."
    })
    response = api.chat.completions.create(**body)
    return response

Running locally

To run this app locally you can do so with the 8080 dev CLI command:

❯ 8080 dev
Fetching secrets for local development for: chat
INFO:     Will watch for changes in these directories: ['~/projects/chat']
INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
INFO:     Started reloader process [4851] using StatReload
INFO:     Started server process [4854]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

If you already have a process running on port 8080, you can specify a different one with the --port option.

Try cURLing your application locally:

curl http://localhost:8080/v1/chat/completions \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{
    "model": "8080/taalas/llama3.1-8b-instruct",
    "messages": [
      {
        "role": "user",
        "content": "Hello, world"
      }
    ]
  }'

Deploy to 8080

Once ready, simply run the following to start a deploy to 8080:

8080 deploy

Watch the progress of your deployment by following the URL returned. Once the deployment successfully completes, your application will now be accessible at {your_project_slug}.hosted.8080.io.

Making requests

This application requires authentication using an API key. Connect to this app by including one of your project’s API keys:

curl https://{{ your_project_slug }}.hosted.8080.io/v1/chat/completions \
  -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $_8080_API_KEY" \
  -d '{
    "model": "8080/taalas/llama3.1-8b-instruct",
    "messages": [
      {
        "role": "user",
        "content": "Hello, world"
      }
    ]
  }'

Alternatively, send a request with the CLI:

8080 apikey set $_8080_API_KEY
8080 call v1/chat/completions '{
  "model": "8080/taalas/llama3.1-8b-instruct",
  "messages": [
    {
      "role": "developer",
      "content": "Hello, world"
    }
  ]
}'

OpenAI compatibility

By wrapping the standard /v1/chat/completions endpoint, this service can be used as the base OpenAI URL in your applications and integrated easily:

export OPENAI_API_KEY=$_8080_API_KEY
export OPENAI_BASE_URL="https://{{ your_project_slug }}.hosted.8080.io/v1/"