xpark.dataset.TextPatternCleaner

class xpark.dataset.TextPatternCleaner(patterns: str | list[str], replacement: str = '', flags: int = re.IGNORECASE)

Supports two usage modes:

1. Built-in patterns: Pass a built-in pattern name (string); use a list for multiple. Available names: "uri", "email", "phone", "ipv4".

2. Custom regex: Pass a regex string or list directly. All patterns are merged into a single |-joined regex and applied in one substitution pass.

Both modes can be mixed together.

Parameters:

• patterns – Built-in pattern name(s) or custom regex string(s); accepts a single string or a list.

• replacement – Replacement string. Defaults to "" (delete matched text).

• flags – Flags passed to re.compile(). Defaults to re.IGNORECASE.

Examples

from xpark.dataset.expressions import col
from xpark.dataset import from_items
from xpark.dataset.processors.text_cleaner import TextPatternCleaner

ds = from_items(["Visit https://example.com, email: foo@bar.com"])

ds = ds.with_column(
    "cleaned",
    TextPatternCleaner(patterns=["uri", "email"], replacement="[REDACTED]")
    .options(num_workers={"CPU": 1}, batch_size=10)
    .with_column(col("item")),
)

ds = ds.with_column(
    "cleaned",
    TextPatternCleaner(patterns=r"\d{4}-\d{4}-\d{4}-\d{4}", replacement="[CARD]")
    .options(num_workers={"CPU": 1}, batch_size=10)
    .with_column(col("item")),
)

print(ds.take(1))

Methods

__call__(batch)	Call self as a function.
options(**kwargs)
with_column(batch)

__call__(batch: pa.ChunkedArray) → pa.Array

Call self as a function.

options(**kwargs: Unpack[ExprUDFOptions]) → Self

with_column(batch: pa.ChunkedArray) → pa.Array