Calling the API

The 8080 SDK provides a type-safe way to call the 8080 API, both using asyncio and synchronously. Before you start, make sure the 8080 SDK is installed and set up as outlined in the introduction.

Completion example:

Sync

from e80_sdk import Eighty80

sync_client = Eighty80().api()

completion = sync_client.completion.create(
    model="8080/taalas/llama3.1-8b-instruct",
    messages=[
        {
            "role": "user",
            "content": "Tell me a joke."
        }
    ]
)
print(completion.choices[0].message)

Async

from e80_sdk import Eighty80

async_client = Eighty80().api_async()

async_completion = await async_client.completion.create(
    model="8080/taalas/llama3.1-8b-instruct",
    messages=[
        {
            "role": "user",
            "content": "Tell me a joke."
        }
    ]
)
print(async_completion.choices[0].message)

Streaming example

The 8080 SDK also supports streaming responses.

Sync

from e80_sdk import Eighty80

client = Eighty80().api()
completion = client.completion.create(
    model="8080/taalas/llama3.1-8b-instruct",
    messages=[
        {
            "role": "user",
            "content": "Tell me a joke."
        },
    ],
    stream=True # Required for streaming responses
)

for chunk in completion:
    print(chunk.choices[0].delta)

Async

from e80_sdk import Eighty80

async_client = Eighty80().api_async()

completion = async_client.completion.create(
    model="8080/taalas/llama3.1-8b-instruct",
    messages=[
        {
            "role": "user",
            "content": "Tell me a joke."
        },
    ],
    stream=True # Required for streaming responses
)

# You must use `async for` when iterating for the async API
async for chunk in completion:
    print(chunk.choices[0].delta)

A note on compatibility

The API clients in the SDK are similar to the OpenAI SDK, so switching over to the 8080 SDK should be relatively simple and you should be able to call each API with each SDK.

from openai import OpenAI
from e80_sdk import Eighty80

old_client = OpenAI(...)
drop_me_in = Eighty80().api()

old_client.chat.completions.create(
    model="", messages=[...]
)
drop_me_in.completion.create(
    model="", messages=[...]
)

To see if your use-case is supported by the 8080 SDK, please refer to the API reference.

More info

To see which APIs are available, take a look at the API reference.