xpark.dataset.VideoCaption#

class xpark.dataset.VideoCaption(*, hint: str | list[str] | None = None, max_words: int = 50, num_frames: int = 16, keyframes_only: bool = True, resize: bool = True, min_token_num: int | None = None, max_token_num: int | None = None, model_context_length: int = 128000, base_url: str | None = None, model: str | None = None, api_key: str = 'NOT_SET', max_new_tokens: int = 256, max_qps: int | None = None, max_concurrency: int | None = None, max_retries: int = 0, fallback_response: str | None = None, **kwargs: Any)#

Video caption processor (remote only, MVBench-style frame sampling).

Generates a natural-language description (caption) for each input video by uniformly sampling num_frames frames along the video timeline (MVBench-style), encoding each frame as an image_url content item, and issuing a single multi-image chat-completion request against a remote OpenAI-compatible endpoint. The VLM compacts the frame sequence into one caption per video. The input column holds video paths or URLs.

Local GPU / CPU inference is not implemented for video yet, so a non-remote configuration raises ValueError.

Parameters:

  • hint – Optional extra instructions or constraints to guide the caption (e.g. focus, tone, output language). Accepts either a single string or a list of strings, where each item is one hint written in plain text. Passing a list is recommended — use one string per hint. Rendered into an <instructions> block of the system prompt.

  • max_words – A non-negative best-effort target for the number of words in the caption. Defaults to 50. If set to 0, there is no word limit. This is a soft, prompt-level constraint rendered into a <response_format> block; it is orthogonal to the hard token cap max_tokens (via max_new_tokens).

  • num_frames – Number of frames to uniformly sample per video. Defaults to 16.

  • keyframes_only – Only decode keyframes (I-frames) when sampling. Default True.

  • resize – If True (default), each sampled frame is passed through Qwen-VL-style smart_resize before base64 encoding. Unlike single-image sizing, the frames of one clip share a total vision-token budget, so the per-frame resolution automatically shrinks as num_frames grows — keeping the multi-image payload and prefill cost bounded. Set to False to send frames at their original resolution.

  • min_token_num – Optional lower bound on the per-frame vision-token budget. Only takes effect when resize is True.

  • max_token_num – Optional upper bound on the per-frame vision-token budget. The effective cap is always the video token budget derived from num_frames; an explicit max_token_num can only further tighten it. Only takes effect when resize is True.

  • model_context_length – Context length (in tokens) of the target VLM. Defaults to 128000, matching Qwen’s default MODEL_SEQ_LEN. The total per-clip vision-token budget shared across all frames follows Qwen (model_context_length * 0.9, reserving ~10% of context for the prompt / generated text); this budget is then split across num_frames to derive the per-frame cap. Larger values allow higher per-frame resolution at high frame counts. Only takes effect when resize is True.

  • base_url – The base URL of the remote OpenAI-compatible endpoint (required).

  • model – The request model name (required).

  • api_key – The request API key.

  • max_new_tokens – Hard upper bound on generated tokens; maps to max_tokens on the remote endpoint. Defaults to 256.

  • max_qps – The maximum query-per-second rate for remote requests.

  • max_concurrency – The maximum number of in-flight remote requests allowed concurrently.

  • max_retries – The maximum number of retries per request in the event of failures. We retry with exponential backoff upto this specific maximum retries.

  • fallback_response – The response value to return when a remote request fails. If set to None, the failed rows fall back to None.

  • **kwargs – Keyword arguments to pass to the openai.AsyncClient.chat.completions.create API.

Examples

import os
from xpark.dataset.expressions import col
from xpark.dataset import VideoCaption, from_items

ds = from_items(["https://example.com/clip.mp4"])
ds = ds.with_column(
    "caption",
    VideoCaption(
        base_url=os.getenv("VLM_ENDPOINT"),
        model="qwen3.5-vl",
        num_frames=16,
        max_words=40,
    )
    .options(num_workers={"IO": 1})
    .with_column(col("item")),
)
print(ds.take(1))

Methods

__call__(videos)

Call self as a function.

options(**kwargs)

with_column(videos)

__call__(videos: pa.ChunkedArray) → pa.Array#

Call self as a function.

options(**kwargs: Unpack[ExprUDFOptions]) → Self#
with_column(videos: pa.ChunkedArray) → pa.Array#