xpark.dataset.VideoExtractSubtitle#

class xpark.dataset.VideoExtractSubtitle(*, mode: Literal['ocr', 'asr', 'embedded', 'auto'] = 'auto', subtitle_stream_index: int | None = None, asr_model: str = 'Systran/faster-whisper-large-v3', fps: int = 2, keyframes_only: bool = False, start_time: float = 0.0, end_time: float | None = None, region: tuple[float, float, float, float] = (0.78, 0.99, 0.05, 0.95), similarity_threshold: float = 0.8, ocr_mode: Literal['accurate', 'fast'] = 'accurate', engine: Literal['paddleocr', 'rapidocr'] = 'paddleocr', ocr_batch: int | None = None, frame_window: int = 60, filters: Sequence[Callable[[list[list[TextLayoutBox]]], list[list[TextLayoutBox]]]] | None = None, **ocr_kwargs)#

Extract subtitles from video.

Four extraction modes are available via mode:

"auto" (default): tries "embedded" first, and falls back to "ocr" for videos without a usable embedded subtitle track. Note: the fallback from OCR to ASR can be realized via a pipeline: VideoTextAreaRatio —> VideoExtractSubtitle (asr mode) / VideoExtractSubtitle (auto mode).
"ocr": recognizes subtitle text burned into the video frames.
"asr": transcribes the spoken audio track into subtitles.
"embedded": extracts an existing text-based subtitle track already contained in the video (e.g. srt/ass/ssa/mov_text/webvtt). Image-based subtitle tracks (e.g. PGS) have no extractable text; use "ocr" or "auto" for those.

Input accepts a local path, a remote URI (cos:// / s3:// / http(s)://), or raw video bytes.

Parameters:

mode – Extraction mode, one of "auto" (default), "ocr", "asr", or "embedded".
subtitle_stream_index – [embedded/auto] Which subtitle stream to read when a video has more than one. Default None picks the first subtitle stream.
asr_model – [asr] Speech recognition model to use. Default "Systran/faster-whisper-large-v3".
fps – [ocr/auto] Frame sampling rate in frames per second. A sampled frame identical to the previous one is not OCR’d again (its boxes are reused). Default 2.
keyframes_only – [ocr/auto] Sample only from keyframes. Default False.
start_time – [ocr/auto] Start of the sampled time range in seconds. Default 0.0.
end_time – [ocr/auto] End of the sampled time range in seconds. Default None for the full duration.
region – [ocr/auto] Normalized (top, bottom, left, right) band where subtitle text is expected. Default (0.78, 0.99, 0.05, 0.95).
similarity_threshold – [ocr/auto] Similarity in [0, 1] used to merge consecutive frames showing the same/similar subtitle text. Default 0.8.
ocr_mode – [ocr/auto] OCR accuracy mode, "accurate" (default) or "fast".
engine – [ocr/auto] OCR backend, "paddleocr" (default) or "rapidocr".
ocr_batch – [ocr/auto] Max number of frames fed to the OCR backend in a single call. Defaults to 64 on GPU workers and 4 on CPU workers.
frame_window – [ocr/auto] Decode and OCR in windows of this many seconds, so a long video never holds all its sampled frames in memory at once. Default 60.
filters –
[ocr/auto] Post-process filters applied, in order, to all accumulated boxes before merging. Each filter maps list[list[TextLayoutBox]] -> list[list[TextLayoutBox]] (per-frame box lists, full-frame normalized). Filters run on every accumulated box before the region check. Default None uses DEFAULT_FILTERS. Pass a custom sequence to replace the defaults (an empty sequence disables post-processing); use filter_boxes() to turn a per-box predicate into a filter. Filters must be picklable for distributed runs.

DEFAULT_FILTERS is:
- static_text_filter(): drop static overlay text (logos/watermarks) that persists across most frames.
- small_text_filter(): drop boxes shorter than 3% of the frame height.
- normalize_text_filter(): normalize text with Unicode NFKC (full-width to half-width).
- auto_region_filter(): drop boxes outside the auto-detected subtitle band. Must run last: watermarks and buttons would otherwise pollute the band’s frequency stats.
**ocr_kwargs – [ocr/auto] Extra params forwarded to the OCR backend.

Examples

from xpark.dataset import VideoExtractSubtitle, from_items
from xpark.dataset.expressions import col

ds = from_items([{"video": "example.mp4"}])
# Default: embedded subtitle track if present, on-screen OCR otherwise.
ds = ds.with_column(
    "subtitle",
    VideoExtractSubtitle()
    .options(num_workers={"CPU": 1}, batch_size=1)
    .with_column(col("video")),
)
# Hardcoded (burned-in) subtitles via OCR only.
ds = ds.with_column(
    "subtitle",
    VideoExtractSubtitle(mode="ocr")
    .options(num_workers={"CPU": 1}, batch_size=1)
    .with_column(col("video")),
)
# Spoken subtitles via ASR.
ds = ds.with_column(
    "subtitle",
    VideoExtractSubtitle(mode="asr")
    .options(num_workers={"GPU": 1}, batch_size=1)
    .with_column(col("video")),
)
# Existing subtitle track already muxed into the container.
ds = ds.with_column(
    "subtitle",
    VideoExtractSubtitle(mode="embedded")
    .options(num_workers={"CPU": 1}, batch_size=1)
    .with_column(col("video")),
)

Methods

`__call__`(videos)	Call self as a function.
`options`(**kwargs)
`with_column`(videos)

__call__(videos: pa.ChunkedArray) → pa.Array#: Call self as a function.

options(**kwargs: Unpack[ExprUDFOptions]) → Self#

with_column(videos: pa.ChunkedArray) → pa.Array#