xpark.dataset.connectors.DeleteVectorDB#

class xpark.dataset.connectors.DeleteVectorDB(connection_factory: Callable[[], Any], collection_name: str, *, query_template: dict[str, Any] | str | None = None, **kwargs: Any)#

Operator for batch-deleting records from a vector database.

Supported backends: [‘elasticsearch’, ‘milvus’].

The operator is a pure side-effect: it returns a boolean column and does not produce meaningful output data.

Two operating modes (mutually exclusive):

  • Default mode (no query_template) — delete by id using the backend’s native delete API. Bind exactly one id column via with_column(col('id')). Deleting a non-existent id is a no-op (idempotent delete semantics).

  • Template mode (query_template provided) — render a user-supplied DSL body (dict | str) per row and call the backend’s delete_by_query. Useful when callers want to delete by attribute rather than by surrogate id. Currently supported by the Elasticsearch backend only.

Template placeholders (template mode):

  • $collection — whole-value, str.

  • $N / $N.a.b.c — 1-based positional reference to a column bound via with_column. Unlike the modify/search templates, there is no implicit vector or id column — every bound column maps to a $N. Whole-value form preserves type; interpolated form coerces via str().

  • $ escapes a literal $.

Parameters:

  • connection_factory – A callable that returns a vector database client (e.g. lambda: pymilvus.MilvusClient(uri="...")). The backend is auto-detected from the returned client type.

  • collection_name – Name of the target collection.

  • query_template – Optional dict | str template rendered to a per-row delete-by-query body.

  • **connector_kwargs – Extra keyword arguments. The connector-init reserved keys max_retries, retry_delay, retry_backoff are consumed by the connector itself; all remaining kwargs are forwarded to the underlying delete / delete_by_query call (e.g. timeout).

Examples

Default mode — idempotent delete by id:

import pymilvus
from xpark.dataset import from_items
from xpark.dataset.connectors import DeleteVectorDB
from xpark.dataset.expressions import col

ds = from_items([{"id": "1"}, {"id": "2"}, {"id": "3"}])
ds = ds.with_column(
    "result",
    DeleteVectorDB(
        connection_factory=lambda: pymilvus.MilvusClient(uri="http://localhost:19530"),
        collection_name="my_col",
    )
    .options(num_workers={"IO": 1})
    .with_column(col("id")),
)
print(ds.take_all())

Template mode on Elasticsearch — delete by attribute:

ds = ds.with_column(
    "result",
    DeleteVectorDB(
        connection_factory=es_factory,
        collection_name="memories",
        query_template={
            "query": {
                "bool": {
                    "must": [
                        {"term": {"conv_idx": "$1"}},
                        {"term": {"category": "$2"}},
                    ]
                }
            }
        },
    )
    .options(num_workers={"IO": 1})
    .with_column(col("conv_idx"), col("category")),
)

Methods

__call__(first_col, *extra_cols)

Execute batch delete.

options(**kwargs)

with_column(first_col, *extra_cols)

Execute batch delete.
__call__(first_col: pa.ChunkedArray, *extra_cols: pa.ChunkedArray) → pa.Array#

Execute batch delete.

Parameters:

  • first_col – In default mode, the id column to delete. In template mode, the first positional column ($1); extra columns bind to $2..$N.

  • *extra_cols – Additional positional columns (template mode only; rejected in default mode).

Returns:

A boolean array (one False per row). The operator is used for its side-effects only.

options(**kwargs: Unpack[ExprUDFOptions]) → Self#
with_column(first_col: pa.ChunkedArray, *extra_cols: pa.ChunkedArray) → pa.Array#

Execute batch delete.

Parameters:

  • first_col – In default mode, the id column to delete. In template mode, the first positional column ($1); extra columns bind to $2..$N.

  • *extra_cols – Additional positional columns (template mode only; rejected in default mode).

Returns:

A boolean array (one False per row). The operator is used for its side-effects only.