biolm.datasets package

Local-first datasets: inventory and addressable artifact bags.

See Managing datasets and Dataset Schema Reference.

Local-first datasets: inventory and addressable artifact bags.

Import from biolm.datasets (not top-level biolm).

exception biolm.datasets.BackendNotAvailableError

Bases: DatasetError

Requested push/pull backend is not installed or not registered.

class biolm.datasets.Dataset(path: str | Path, meta: DatasetMeta | None = None)

Bases: object

A local dataset directory containing dataset.yaml.

add(source: str | Path, *, recursive: bool = False, dest_name: str | None = None) → Path

Copy a file or directory into the dataset content directory.

Returns the destination path.

property attrs: Dict[str, Any]

property created_at: str | None

property data_dir: Path

Preferred content directory: data/ if present, else dataset root.

property description: str | None

files() → List[Path]

Relative paths of files under the dataset (excluding dataset.yaml).

property id: str

property meta: DatasetMeta

open_seqframe()

Open this dataset as a SeqFrame.

Requires biolm-sdk[seqframe]. Resolves attrs.seqframe_path or exactly one *.parquet under the dataset. See biolm.seqframe.dataset_bridge.

property path: Path

push(backend: str = 'mlflow', **opts: Any) → Dict[str, Any]

Push this dataset via a registered backend.

refresh() → Dataset

Re-read dataset.yaml from disk.

property tags: List[str]

to_dict() → Dict[str, Any]

property type: str

class biolm.datasets.DatasetClient(roots: Sequence[str | Path] | None = None, *, primary_root: str | Path | None = None, cwd: str | Path | None = None)

Bases: object

Client for local dataset inventory and push/pull.

create(dataset_id: str, *, type: str = 'files', tags: List[str] | None = None, attrs: Dict[str, Any] | None = None, description: str | None = None, root: str | Path | None = None, force: bool = False) → Dataset

Create //dataset.yaml and an empty data/ directory.

get(id_or_path: str | Path) → Dataset

Resolve a dataset by filesystem path or by id under discovery roots.

init(path: str | Path, *, id: str | None = None, type: str = 'files', tags: List[str] | None = None, attrs: Dict[str, Any] | None = None, description: str | None = None, force: bool = False) → Dataset

Write dataset.yaml into an existing directory without moving files.

list(*, type: str | None = None, tag: str | None = None) → List[Dataset]

List datasets discovered under configured roots.

property primary_root: Path

pull(dataset_id: str, *, backend: str = 'mlflow', path: str | Path | None = None, force: bool = False, **opts: Any) → Dataset

Pull a remote dataset into a local directory and return Dataset.

property roots: List[Path]

exception biolm.datasets.DatasetError

Bases: Exception

Base error for local dataset operations.

exception biolm.datasets.DatasetExistsError

Bases: DatasetError

dataset.yaml already exists and force was not set.

class biolm.datasets.DatasetMeta(id: str, schema_version: int = 1, description: str | None = None, created_at: str | None = None, type: str = 'files', tags: ~typing.List[str] = <factory>, attrs: ~typing.Dict[str, ~typing.Any] = <factory>)

Bases: object

Validated metadata from dataset.yaml.

attrs: Dict[str, Any]

created_at: str | None = None

description: str | None = None

id: str

schema_version: int = 1

tags: List[str]

to_dict() → Dict[str, Any]

type: str = 'files'

exception biolm.datasets.DatasetNotFoundError

Bases: DatasetError

Dataset id or path could not be resolved.

class biolm.datasets.DatasetPushPullBackend(*args, **kwargs)

Bases: Protocol

Adapter that can push/pull a local dataset to a remote store.

name: str

pull(dataset_id: str, dest: Path, **opts: Any) → Dict[str, Any]

Download remote dataset into dest; return status info.

push(dataset: Dataset, **opts: Any) → Dict[str, Any]

Upload local dataset artifacts and metadata.

exception biolm.datasets.DatasetSchemaError

Bases: DatasetError

dataset.yaml is missing required fields or is invalid.

exception biolm.datasets.DuplicateDatasetIdError

Bases: DatasetError

The same dataset id was found under multiple discovery roots.

biolm.datasets.get_backend(name: str) → DatasetPushPullBackend

Return a registered backend, lazy-loading known plugins when needed.

biolm.datasets.register_backend(name: str, backend: DatasetPushPullBackend) → None

Register a push/pull backend under name.