Multimodal input

MeshTok exposes 69 models that understand more than text — images, audio, video, and PDFs all flow through the same OpenAI-compatible /v1/chat/completions endpoint. Use the content array to mix text with any modality in a single message.

Modality coverage

The content array pattern

All modalities use the same OpenAI content array. A message is either a plain string (text-only) or an array of typed parts. Mix freely:

from openai import OpenAI
client = OpenAI(base_url="https://meshtok.com/v1", api_key="sk-...")

resp = client.chat.completions.create(
    model="google/gemini-2.5-pro",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Here are three files. Summarize each."},
            {"type": "image_url", "image_url": {"url": "https://.../chart.png"}},
            {"type": "input_audio", "input_audio": {"data": "<base64>", "format": "mp3"}},
        ],
    }],
)

Audio input (transcription & understanding)

13 models accept audio directly in chat — useful for meeting transcription, voice queries, and spoken-language analysis. Use the input_audio content type with base64-encoded audio.

import base64

with open("meeting.mp3", "rb") as f:
    audio_b64 = base64.b64encode(f.read()).decode()

resp = client.chat.completions.create(
    model="google/gemini-2.5-pro",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "Transcribe and extract action items."},
            {"type": "input_audio", "input_audio": {
                "data": audio_b64, "format": "mp3"
            }},
        ],
    }],
)

For pure speech-to-text at scale, audio-capable models are cheaper than full chat models.

Video input

12 models (mostly Gemini family) accept video as a sequence of frames. Upload the video file and the model samples frames automatically. Best for security review, sports analysis, and UI walkthroughs.

Cost notes

• Images: ~765 tokens per 1024×1024 tile. Resizing before upload cuts cost dramatically.
• Audio: billed by audio duration — typically 1 second ≈ 30-50 tokens depending on model.
• Video: billed by frame count × image token rate. Use the lowest frame rate that works.
• Mix modalities freely — there's no separate endpoint or surcharge.