xpark.dataset.ImageCaption#

class xpark.dataset.ImageCaption(_local_model: str | None = 'Qwen/Qwen3.5-0.8B', /, *, hint: str | list[str] | None = None, max_words: int = 50, resize: bool = True, image_quality: Literal['fast', 'balanced', 'accurate'] | None = None, min_token_num: int | None = None, max_token_num: int | None = None, base_url: str | None = None, model: str | None = None, api_key: str = 'NOT_SET', max_new_tokens: int = 1024, max_qps: int | None = None, max_concurrency: int | None = None, max_retries: int = 2, fallback_response: str | None = None, **kwargs: Any)#

Image caption processor for CPU, GPU and remote Http requests.

Generates a natural-language description (caption) for each input image using a Qwen3.5 vision-language model. The backend is chosen automatically per worker: GPU (transformers) when a CUDA device is assigned, remote OpenAI-compatible endpoint on IO workers, otherwise CPU (GGUF + mmproj via xllamacpp).

Parameters:

_local_model – The vision model name for CPU or GPU. Available models: [‘Qwen/Qwen3.5-0.8B’, ‘Qwen/Qwen3.5-4B’, ‘Qwen/Qwen3.5-9B’]. Defaults to Qwen/Qwen3.5-0.8B.
hint – Optional extra instructions or constraints to guide the caption (e.g. focus, tone, output language). Accepts either a single string or a list of strings, where each item is one hint written in plain text. Passing a list is recommended — use one string per hint. Rendered into an <instructions> block of the system prompt.
max_words – A non-negative best-effort target for the number of words in the caption. Defaults to 50. If set to 0, there is no word limit. This is a soft, prompt-level constraint rendered into a <response_format> block; it is orthogonal to the hard token cap max_new_tokens (local) / max_tokens (remote).
resize – If True (default), each image is passed through smart_resize() before being handed to the backend. Applies to all three backends (GPU, CPU, remote HTTP). Set to False to feed the original resolution.
image_quality –
Semantic preset for the per-image vision-token budget. Sets max_token_num under the hood:
- "fast" — max_token_num=256 (~200 K pixels): thumbnails, coarse captioning, low-latency / high-throughput.
- "balanced" — max_token_num=1280 (~1 M pixels): general understanding, recommended default for most downstream tasks.
- "accurate" — max_token_num=16384 (~12.8 M pixels): keeps fine text / small objects intact (OCR-grade).
When None (default) no preset is applied and max_token_num decides. When combined with an explicit max_token_num, the tighter (smaller) of the two wins — the preset states the caller’s intent, and max_token_num can only further tighten it. Only takes effect when resize is True.
min_token_num – Optional lower bound on the per-image vision-token budget, forwarded to smart_resize(). When None (default) the underlying smart_resize() default is used. Independent of image_quality.
max_token_num – Optional upper bound on the per-image vision-token budget, forwarded to smart_resize(). When None (default) the underlying smart_resize() default is used, which preserves detail. Lowering this cap trades image detail for a much smaller vision-token / prefill budget and can substantially speed up local CPU inference and remote payload size. Combined with image_quality via min(...) — see image_quality for details. Only takes effect when resize is True. Applies to all three backends.
base_url – The base URL of the remote OpenAI-compatible endpoint.
model – The request model name (remote).
api_key – The request API key (remote).
max_new_tokens – Hard upper bound on generated tokens; maps to max_tokens on the remote endpoint. Default to 1024.
max_qps – The maximum query-per-second rate for remote requests.
max_concurrency – The maximum number of in-flight remote requests allowed concurrently.
max_retries – The maximum number of retries per request in the event of failures. We retry with exponential backoff upto this specific maximum retries.
fallback_response – The response value to return when a remote request fails. If set to None, the failed rows fall back to None.
**kwargs – Keyword arguments to pass to the openai.AsyncClient.chat.completions.create API.

Examples

from xpark.dataset.expressions import col
from xpark.dataset import ImageCaption, from_items

ds = from_items(["cat.jpg", "dog.jpg"])
ds = ds.with_column(
    "caption",
    ImageCaption(
        # Local vision model.
        "Qwen/Qwen3.5-0.8B",
        # For remote requests.
        base_url="http://127.0.0.1:9997/v1",
        model="qwen3.5-vl",
        hint="Focus on the main subject.",
        max_words=30,
        image_quality="balanced",
    )
    .options(num_workers={"CPU": 4, "IO": 1})
    .with_column(col("item")),
)
print(ds.take(2))

Methods

`__call__`(images)	Call self as a function.
`options`(**kwargs)
`with_column`(images)

__call__(images: pa.ChunkedArray) → pa.Array#: Call self as a function.

options(**kwargs: Unpack[ExprUDFOptions]) → Self#

with_column(images: pa.ChunkedArray) → pa.Array#