xpark.dataset.MarkdownChunking#

class xpark.dataset.MarkdownChunking(*, max_tokens: int = 1024)[source]#

Markdown-aware structural text chunking operator.

Preserves document hierarchy by splitting markdown into a virtual file tree. Heading-aware splitting that respects section boundaries and max token limits. Returns a list of structs per input row containing {"id": ..., "content": ...}.

The doc_name column provides a per-row label used to build virtual relative paths. Each input text is paired with its own doc name.

Parameters:: max_tokens – Maximum tokens per chunk. Defaults to 1024.

Examples

from xpark.dataset.expressions import col
from xpark.dataset import MarkdownChunking, from_items

text1 = "# Introduction1\n\nContent here..."
text2 = "# Introduction2\n\nContent here..."

ds = from_items([
    {"text": text1, "doc_name": "document1"},
    {"text": text2, "doc_name": "document2"},
])
ds = ds.with_column(
    "chunks",
    MarkdownChunking(max_tokens=512)
    .options(num_workers={"CPU": 2}, batch_size=32)
    .with_column(col("text"), col("doc_name")),
)
print(ds.take_all())

Methods

`__call__`(texts, doc_names)	Call self as a function.
`options`(**kwargs)
`with_column`(texts, doc_names)

__call__(texts: pa.ChunkedArray, doc_names: pa.ChunkedArray) → pa.Array#: Call self as a function.

options(**kwargs: Unpack[ExprUDFOptions]) → Self#

with_column(texts: pa.ChunkedArray, doc_names: pa.ChunkedArray) → pa.Array#