Artifact formats and library integrations¶

Dagster asset observation¶

Install the optional Dagster integration with:

pip install "oclp[dagster]"

@dagster_asset(...) is applied directly inside Dagster's @asset. It opens the OCLP observation context for an existing @run workflow, while that workflow's existing @computation and Artifact declarations continue to produce the only OCLP records. On completion, the adapter adds the OCLP run identity, record-store path, outcome, and an explicit selection of Dagster context fields to the Dagster asset materialization metadata.

import dagster as dg

from oclp.dagster import dagster_asset


@dg.asset
@dagster_asset(
    workflow=train,
    publisher=publisher_for_dagster_context,
    source=source_for_dagster_context,
)
def trained_model(context: dg.AssetExecutionContext) -> None:
    train(...)

This is an integration boundary, not an orchestration DSL. It never derives an OCLP Computation from an asset, creates a parent Execution for a Dagster run, or reflects arbitrary Dagster context. The initial allow-list is run_id, asset_key, partition_key, and retry_number; select only values that are useful to navigate the Dagster materialization.

An OCLP Artifact is format-neutral. The Python SDK's concrete ArtifactType declarations provide the local persistence and loading policy for an Artifact payload. This page is the canonical inventory of integrations shipped by the SDK—not a claim that OCLP itself requires any of these libraries or formats.

The same concrete type is used at both sides of a Computation boundary:

from xgboost import XGBRegressor

from oclp import XGBoostModelArtifact, computation


@computation(
    inputs={"model": XGBoostModelArtifact},
    outputs={"candidate": XGBoostModelArtifact(name="Candidate model")},
)
def promote(model: XGBRegressor) -> XGBRegressor:
    return model

The output declaration tells the SDK how to materialize the normal Python return value. The input declaration validates the Artifact's durable media type; the parameter annotation selects the adapter that loads verified bytes.

Supported representations¶

Media types beginning with application/x- are SDK conventions where a format does not have a registered IANA media type. They make the representation unambiguous within OCLP records without claiming an external standard.

Representation configuration belongs beside the output port¶

The concrete Artifact declaration is the single place to configure how a returned value becomes durable bytes. The SDK validates known options instead of accepting opaque serializer **kwargs that might vary between libraries or silently create an invalid representation.

from oclp import JsonLinesArtifact, YamlArtifact, computation


@computation(
    name="Publish predictions and configuration",
    outputs={
        "predictions": JsonLinesArtifact(
            name="Prediction batch",
            sort_keys=True,
            newline="\n",
        ),
        "config": YamlArtifact(
            name="Training configuration",
            indent=2,
            sort_keys=True,
        ),
    },
)
def publish() -> dict[str, object]:
    return {
        "predictions": [{"id": "a", "score": 0.9}],
        "config": {"learning_rate": 0.1},
    }

Changing a declared representation option changes the stored bytes and hence the Artifact digest. The source-bound Computation identifies the code that made that serialization decision.

JsonLinesArtifact is intentionally separate from JsonArtifact: records with one JSON object per line are a different durable contract, not merely a lines=True rendering option. ArrowIpcArtifact writes the Arrow IPC file format (also known as Feather V2), with Arrow's registered file media type. See Arrow's format documentation.

XML documents¶

XmlArtifact accepts an XML str, persists it as UTF-8 .xml bytes, and validates it with defusedxml. DTDs and entity processing are rejected: XML must be a safe, well-formed document. An explicit XML encoding declaration must say UTF-8; otherwise it must be omitted. A downstream callable can ask for the original str or a safely parsed xml.etree.ElementTree.Element.

The SDK deliberately preserves the returned XML text rather than silently canonicalizing it. Equivalent XML documents can differ physically in attribute order, whitespace, and other permitted syntax; XML canonicalization is an application-level choice, while an OCLP Artifact digest identifies the exact persisted bytes. See Canonical XML and Python's XML security guidance.

Native model formats¶

CatBoostModelArtifact materializes the fitted returned object through CatBoost's native save_model() API as .cbm bytes. The adapter verifies the Artifact digest before loading the payload into the downstream annotated model class.

XGBoostModelArtifact explicitly writes .ubj, XGBoost's UBJSON model format, rather than accepting a version-dependent default. It accepts an XGBoost Booster or sklearn-wrapper model with save_model() and reloads it as the exact annotated XGBoost class. XGBoost also supports JSON, but the SDK uses UBJSON as its shipped native representation to keep artifacts compact without losing model semantics. See XGBoost model I/O.

LightGBMModelArtifact persists the native model returned as a lightgbm.Booster. The LightGBM sklearn wrappers expose a fitted Booster as booster_, but do not provide a stable native wrapper-loader boundary, so the SDK does not reconstruct wrapper internals. Use the Booster for the native artifact contract, or SklearnModelArtifact when the portable Python wrapper object itself is required. LightGBM documents booster_ as the underlying fitted Booster.

SklearnModelArtifact uses the skops .skops format rather than pickle or joblib. It works with fitted scikit-learn BaseEstimator values and returns the same annotated estimator type to the downstream function. The default adapter refuses unknown types in a payload. A project that intentionally uses reviewed custom types must configure an additional SklearnModelAdapter(trusted_types=(...)) in its OclpRun adapter registry; the producer cannot grant that trust through the Artifact itself. This is a Python-environment representation, not a cross-language serving format. See skops secure persistence.

Compatibility and safety¶

The Artifact digest proves that a consumer reads the exact persisted bytes. It does not make a model automatically compatible with every runtime. A model's source framework and version remain part of its execution provenance; applications should pin the environment that loads it.

The optional model libraries retain their own system-runtime requirements. For example, XGBoost and LightGBM installations on some platforms also require an OpenMP runtime; that is a framework requirement, not an OCLP payload concern.

The SDK intentionally does not currently provide a generic pickle/joblib Artifact type. Those formats only belong in trusted, version-compatible Python environments; skops and ONNX have different safety and portability tradeoffs. See scikit-learn's model persistence guidance.