xpark.dataset.PDFToText#

class xpark.dataset.PDFToText(*, mode: Literal['fast'] = 'fast', output_format: Literal['markdown', 'text', 'html', 'json'] = 'markdown', max_pages: int | None = None, fallback_response: str | None = None)#
class xpark.dataset.PDFToText(*, mode: Literal['standard'], output_format: OutputFormat = 'markdown', max_pages: int | None = None, pdf_concurrency: int | None = None, fallback_response: str | None = None, document_timeout: float | None = None, do_ocr: bool = True, force_full_page_ocr: bool = False, do_table_structure: bool = True, do_formula: bool = False, do_code: bool = False)
class xpark.dataset.PDFToText(*, mode: Literal['vlm'], output_format: OutputFormat = 'markdown', max_pages: int | None = None, pdf_concurrency: int | None = None, fallback_response: str | None = None, document_timeout: float | None = None, base_url: str | None = None, model: str | None = None, api_key: str = NOT_SET)

PDF → text operator. Parsing modes pick different quality/speed/hardware trade-offs; each mode exposes only the params that are meaningful for that path.

  • "fast"Sub-second per file via the EdgeParse Rust engine; digital-text

    only (no OCR). Native tables and multi-column reading order.

  • "standard" (default): Layout + (optional) table/text OCR/formula/code. Balanced

    quality, highly customizable.

  • "vlm"VLM. GPU-recommended (local) or via a remote OpenAI-compatible

    endpoint; precise on complex/scanned PDFs.

Parameters tagged [standard only] / [vlm only] apply to a single mode; passing them in another mode is ignored with a warning. Untagged parameters apply to all modes.

Parameters:

  • mode – parsing chain, one of "fast" / "standard" / "vlm". Default "standard".

  • output_format – result format, one of "markdown" (default) / "text" / "html" / "json" / "doctags" ("fast" mode does not support doctags).

  • max_pages – parse only the first N pages (page range [1, N]); None (default) => all pages. Applies to all modes. For vlm, each page is a full VLM forward pass, so large max_pages on long PDFs can dominate GPU memory / latency — prefer slicing the PDF upfront.

  • pdf_concurrency – number of CPU threads a single PDF’s docling pipeline may use (maps to docling AcceleratorOptions.num_threads, i.e. intra-PDF parallelism). Default 1. Raise it for large/few PDFs, keep it at 1 for many small PDFs. Only honored by standard / vlm local (docling); ignored by fast (forced to 1 with a warning) and a no-op in vlm remote (no local model inference).

  • fallback_response – text returned for a PDF whose conversion fails (including document_timeout expiry). When None (default) the underlying exception is raised instead.

  • document_timeout – max seconds spent on a single document before it is treated as failed (see fallback_response above). None (default) means no timeout. Only honored by standard / vlm (docling); ignored by fast.

  • do_ocr – [standard only] enable OCR (needed for scanned PDFs). Default True.

  • force_full_page_ocr – [standard only] force OCR to run on the full page (required for scanned PDFs, since the layout model would otherwise fail; see https://docling-project.github.io/docling/_generated/examples/full_page_ocr/).

  • do_table_structure – [standard only] parse table structure. Default True.

  • do_formula – [standard only] enable formula enrichment (slow). Default False.

  • do_code – [standard only] enable code-block recognition. Default False.

  • base_url – [vlm only] OpenAI-compatible chat-completions endpoint URL. When given, the vlm pipeline runs against this remote service instead of a local model (no GPU needed locally). Choose num_workers={"IO": N} for this remote path — inference happens off-worker, so workers just do I/O and the concurrency budget lives on the IO resource.

  • model –

    [vlm only] VLM model name. Its meaning depends on base_url:

    • Local (base_url unset): local preset. Currently the only supported preset is "granite-docling" — ibm-granite/granite- docling-258M (also the default when model is omitted); fast but lower accuracy. Additional presets (e.g. GLM-OCR) are on the roadmap once their env dependencies are resolved.

    • Remote (base_url set): the served model name passed through to the endpoint (e.g. "hunyuan-ocr").

  • api_key – [vlm only] bearer token for the remote endpoint; when set, sent as Authorization: Bearer <api_key>. Leave unset for endpoints without auth (e.g. self-hosted vLLM without a proxy).

Examples

import os
from xpark.dataset import from_items, PDFToText
from xpark.dataset.expressions import col

ds = from_items([{"path": "/data/pdfs/a.pdf"}, {"path": "/data/pdfs/b.pdf"}])

# Fast path: EdgeParse Rust engine, GFM tables + reading order, digital text only.
ds = ds.with_column(
    "md_fast",
    PDFToText(mode="fast")
    .options(num_workers={"CPU": 4})
    .with_column(col("path")),
)

# Standard pipeline: layout + tables, plus OCR for scanned PDFs.
ds = ds.with_column(
    "md_pipe",
    PDFToText(mode="standard")
    .options(num_workers={"CPU": 2})
    .with_column(col("path")),
)

# VLM, local model (GPU recommended).
ds = ds.with_column(
    "md_vlm",
    PDFToText(mode="vlm")
    .options(num_workers={"GPU": 1})
    .with_column(col("path")),
)

# VLM, remote OpenAI-compatible endpoint (no local GPU).
ds = ds.with_column(
    "md_vlm_api",
    PDFToText(
        mode="vlm",
        base_url=os.getenv("VLM_ENDPOINT"),
        model="hunyuan-ocr",
        api_key=os.getenv("VLM_API_KEY"),
    )
    .options(num_workers={"IO": 4})
    .with_column(col("path")),
)

Methods

__call__(paths)

Call self as a function.

options(**kwargs)

with_column(paths)

__call__(paths: pa.ChunkedArray) → pa.Array#

Call self as a function.

options(**kwargs: Unpack[ExprUDFOptions]) → Self#
with_column(paths: pa.ChunkedArray) → pa.Array#