xpark.dataset.ImageExtractRay#

class xpark.dataset.ImageExtractRay(_local_model: str = 'depth-anything/DA3-BASE', *, use_ray_pose: bool = False, ref_view_strategy: str = 'saddle_balanced', max_resolution: int = 504, resize_method: Literal['upper_bound_resize', 'lower_bound_resize'] = 'upper_bound_resize', precision: Literal['auto', 'bf16', 'fp16', 'fp32'] = 'auto')#

Multi-view reconstruction processor: views -> ray geometry.

Each input row is one scene with N views (N = number of images, 1 or more) and produces one row of ray geometry.

  • cam_condition: per-view camera poses R (N, 3, 3) and t (N, 3), following cam2world in OpenCV convention https://docs.opencv.org/4.13.0/d9/d0c/group__calib3d.html.

  • ray_map: per-pixel ray_directions (N, H, W, 3) (camera frame, z-component 1), depth (N, H, W) and conf (N, H, W) confidence in [0, 1]. H / W are the processing-resolution height / width (max_resolution), not the input size.

Inputs (passed positionally via with_column, in the following order):

  • views (required): one scene per row, as a list of images or a stack of shape (N, H, W, 3) (N = number of views). Rows with an empty view sequence are emitted as empty ray geometry with a warning.

  • cam_condition (optional): a CamConditionArrow struct column of per-view camera pose and lens (R (N, 3, 3), t (N, 3), intrinsics (N, 3, 3), cam2world in OpenCV convention) used to condition the model. When omitted, the model estimates the poses itself. Poses require the intrinsics field alongside them (the model consumes both together; no default calibration is applied).

Parameters:

  • _local_model – The DA3 any-view model name for CPU or GPU, or BYO weights. A name must be one of the registry keys below; an absolute path or an fsspec URI is user weights, always run as any-view (the family is fixed). Only registered models are vetted by xpark. default model is “depth-anything/DA3-BASE” available models: [‘depth-anything/DA3-SMALL’, ‘depth-anything/DA3-BASE’]

  • use_ray_pose – Let the model derive camera poses from its ray head (slower, usually more accurate).

  • ref_view_strategy – Reference view selection for multi-view input.

  • max_resolution – Maximum processing resolution (longest side). Lower it for more throughput, higher for more detail. Defaults to 504.

  • resize_method – Resize strategy, “upper_bound_resize” or “lower_bound_resize”. Defaults to "upper_bound_resize".

  • precision – Inference precision. "auto" = BF16 > FP16 on GPU, FP32 on CPU. Explicit "bf16" / "fp16" / "fp32" force a strategy. Defaults to "auto".

Raises:

ValueError – If _local_model is neither a registered model nor an absolute BYO location, if cam_condition is not a CamConditionArrow struct column, or if camera poses are provided without the intrinsics field.

Note

Needs the ai-git extra (pip install "xpark[ai-git]") — the DA3 package is installed from a git ref, not from PyPI (see pyproject.toml). Missing it raises ImportError when the weights are loaded.

The optional cam_condition column is a CamConditionArrow struct of tensor fields, so it must be built explicitly — from_items cannot infer tensor types:

import numpy as np
import pyarrow as pa
from xpark.dataset.utils import to_arrow_tensor_array

n = len(views)                                # views in this row
K = np.eye(3, dtype=np.float32)               # (3, 3) calibration matrix
cam_condition = pa.chunked_array([pa.StructArray.from_arrays(
    [
        to_arrow_tensor_array([np.tile(np.eye(3, dtype=np.float32), (n, 1, 1))], ndim=3),  # R
        to_arrow_tensor_array([np.zeros((n, 3), dtype=np.float32)], ndim=2),               # t
        to_arrow_tensor_array([np.tile(K, (n, 1, 1))], ndim=3),                            # intrinsics
    ],
    names=["R", "t", "intrinsics"],
)])

Examples

from xpark.dataset import ImageExtractRay, from_items
from xpark.dataset.expressions import col

views = ["bottle_front.jpg", "bottle_left.jpg", "bottle_right.jpg"]
ds = from_items([{"views": views}, {"views": views}])
ds = ds.with_column(
    "ray",
    ImageExtractRay().options(num_workers={"GPU": 1}, batch_size=2).with_column(col("views")),
)
print(ds.take(1))

Methods

__call__(views[, cam_condition])

Call self as a function.

options(**kwargs)

with_column(views[, cam_condition])

__call__(views: pa.ChunkedArray, cam_condition: pa.ChunkedArray | None = None) → pa.Array#

Call self as a function.

options(**kwargs: Unpack[ExprUDFOptions]) → Self#
with_column(views: pa.ChunkedArray, cam_condition: pa.ChunkedArray | None = None) → pa.Array#