Processors

Data Processors

General

xpark.dataset.CommonCompute(*args, **kwargs)
xpark.dataset.HashDedup([hash_algorithm, ...])	Exact-hash-based deduplication for text, binary data, or files.

Text

xpark.dataset.MarkdownChunking(*[, max_tokens])	Markdown-aware structural text chunking operator.
xpark.dataset.TextFuzzyDedup([...])	Text fuzzy deduplication using MinHashLSH.
xpark.dataset.TextExactSubstringDedup([...])	Implements a scalable, distributed exact substring deduplication algorithm using a divide-and-conquer strategy.
xpark.dataset.TextChunking()	Unified text chunking operator with multiple strategies.
xpark.dataset.TextFlaggedWordRatio([asset, ...])	Compute the ratio of flagged words in a text.

Audio

xpark.dataset.AudioCompute(*args, **kwargs)

Image

xpark.dataset.ImageCompute(*args, **kwargs)

Video

xpark.dataset.VideoCompute(*args, **kwargs)
xpark.dataset.VideoShotDetect()	Video shot boundary detection and segmentation.

AI Processors

Text

xpark.dataset.TextEmbedding([_local_model, ...])	Text Embedding processor for CPU, GPU and remote Http requests.
xpark.dataset.TextSummarize(*[, max_words, ...])	TextSummarize processor provides a highly condensed summary of the text.
xpark.dataset.TextMask(labels, /, *, ...[, ...])	TextMask processor replaces sensitive information in the original text with [MASKED] according to the labels.
xpark.dataset.TextGenerate(*, base_url, model)	TextGenerate processor generates content based on the input parameters.
xpark.dataset.TextClassify(labels, /, *, ...)	TextClassify processor extracts the single label that best matches the text content.
xpark.dataset.TextFixGrammar(*, base_url, model)	TextFixGrammar processor corrects grammar mistakes in the input text using LLM model.
xpark.dataset.TextSimilarity(target, /, *[, ...])	TextSimilarity processor calculates similarity between texts using LLM model.
xpark.dataset.TextTranslate([to_lang, ...])	TextTranslate processor responsible for translating the text into the target language.
xpark.dataset.TextSentiment(*[, sentiments, ...])	TextSentiment processor for text sentiment analysis.
xpark.dataset.TextExtract(labels_or_schema, /, *)	TextExtract processor extracts structured information from text based on user-defined
xpark.dataset.TextPredicateEval([predicate, ...])	TextPredicateEval processor evaluates whether input texts satisfy a given predicate condition.
xpark.dataset.TextPerplexity([_local_model, ...])	Computes the perplexity of text using a language model to evaluate fluency and naturalness.
xpark.dataset.TextLanguageDetector([...])	Language detection operator based on a fasttext model.
xpark.dataset.TextLanguageScore([...])	Language score operator based on a fasttext model.
xpark.dataset.LLMReranker(*, base_url, model)	LLM-based reranker; supports pointwise and listwise modes.
xpark.dataset.TextPatternCleaner(patterns[, ...])	Supports two usage modes:

Image

xpark.dataset.ImageNSFWScore([_local_model])	Image NSFW score calculation processor for CPU, GPU.
xpark.dataset.ImageTextSimilarityScore(text)	Image text similarity score calculation processor for CPU, GPU
xpark.dataset.ImageAestheticScore([...])	Image aesthetic score calculation processor for CPU, GPU.
xpark.dataset.ImageExtractTextLayout([mode, ...])	Extract the OCR text layout of an image via PP-OCRv6 (ONNXRuntime).
xpark.dataset.ImageWatermarkDetect([...])	Image watermark detection processor for CPU, GPU.
xpark.dataset.ImageCaption([_local_model, ...])	Image caption processor for CPU, GPU and remote Http requests.

Audio

xpark.dataset.SpeechToText([_local_model, ...])

Speech to text processor for CPU, GPU and remote Http requests.

Video

xpark.dataset.VideoAestheticScore([...])	Video aesthetic score processor based on LAION Aesthetic.
xpark.dataset.VideoNSFWScore([_local_model, ...])	Video NSFW score processor.
xpark.dataset.VideoTextAreaRatio(*[, ...])	Score a video from its on-screen text layout, a float32 in [0, 1].
xpark.dataset.VideoWatermarkDetect([...])	Video watermark detection processor based on LAION watermark classifier.
xpark.dataset.VideoCaption(*[, hint, ...])	Video caption processor (remote only, MVBench-style frame sampling).

Document

xpark.dataset.PDFToText()

PDF → text operator.

Time Series

xpark.dataset.TimeSeriesForecast([...])

Time-series forecasting processor backed by Chronos-2.

Connectors

VectorDB

xpark.dataset.connectors.InsertVectorDB(...)	Operator for batch-inserting records into a vector database.
xpark.dataset.connectors.UpsertVectorDB(...)	Operator for batch-upserting records into a vector database.
xpark.dataset.connectors.DeleteVectorDB(...)	Operator for batch-deleting records from a vector database.
xpark.dataset.connectors.SearchVectorDB(...)	Operator for vector similarity search.

Extensions

Memory

End-to-end LLM-driven memory pipeline. MemoryExtract distills self-contained memory facts from raw conversations; MemoryBuildAction diffs those new facts against the user’s existing memory store and emits per-fact write actions (ADD / UPDATE / DELETE / NONE) for the downstream database connectors. See the operator docstrings for details.

xpark.dataset.extensions.MemoryExtract(*[, ...])	Extract memory facts from conversation messages.
xpark.dataset.extensions.MemoryBuildAction(*)	Decide what action to take ADD / UPDATE / DELETE / NONE for each newly extracted fact given existing memories.