Streaming

DigitalOcean Gradient™ AI Agentic Cloud SDK support streaming on every client. We provide support for streaming responses using Server Side Events (SSE).

Examples

Streaming Serverless Inference

For example, you can access streaming serverless inference using the SDK:

Python

import os
from gradient import Gradient

inference_client = Gradient(
    model_access_key=os.environ.get("GRADIENT_MODEL_ACCESS_KEY"), # default
)

stream = inference_client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "What is the capital of Portugal?",
        }
    ],
    stream=True,
    model="llama3.3-70b-instruct",
)

for completion in stream:
    print(completion.choices)

Async Streaming Serverless Inference

The async client uses the exact same interface

Python

from gradient import AsyncGradient

inference_client = AsyncGradient(
    model_access_key=os.environ.get("GRADIENT_MODEL_ACCESS_KEY"), # default
)

stream = await client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "What is the capital of France?",
        }
    ],
    model="ignored",
    stream=True,
)
async for completion in stream:
    print(completion.choices)

Streaming Agent Inference

For example, you can access streaming agent inference using the SDK:

Python

import os
from gradient import Gradient

agent_client = Gradient(
    agent_access_key=os.environ.get("GRADIENT_AGENT_ACCESS_KEY"), # default
    agent_endpoint=os.environ.get("GRADIENT_AGENT_ENDPOINT"), # default
)

stream = agent_client.agents.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "What is the capital of Portugal?",
        }
    ],
    stream=True,
    model="ignored",
)

for completion in stream:
    print(completion.choices)

Async Streaming Agent Inference

The async client uses the exact same interface

Python

from gradient import AsyncGradient

agent_client = AsyncGradient(
    agent_access_key=os.environ.get("GRADIENT_AGENT_ACCESS_KEY"), # default
    agent_endpoint=os.environ.get("GRADIENT_AGENT_ENDPOINT"), # default
)

stream = await agent_client.agents.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "What is the capital of Portugal?",
        }
    ],
    model="ignored",
    stream=True,
)
async for completion in stream:
    print(completion.choices)