External checkpoints

1. Core requirement: alignment

When you call load_results(), PhenoMe needs to match each row in your HDF5 file to the image files currently on disk. There are two ways.

Option A: ID-based (recommended)

You provide a unique identifier per image.

In HDF5: include /metadata/id (one unique string per row).
In code: call pipeline.find_files(image_dir) first.
Matching: the pipeline matches HDF5 IDs to the IDs generated by find_files. By default find_files uses the filename stem (for example cell_01 for cell_01.png).

Option B: path-based

You provide the literal file path.

In HDF5: store absolute paths in /img_path, or relative paths plus _storage_root root attribute.
In code: call pipeline.set_file_df(df) with a DataFrame that has a file_path column.
Matching: the pipeline matches rows where the path strings are identical.

2. HDF5 file schema

Attributes are metadata attached to the root / of the file.

Root attributes

Every PhenoMe checkpoint must define these four attributes at the root level:

Attribute	Type	Description
`version`	str	Must be `"2.0"`.
`n_committed`	int	Total number of images.
`embedding_dim`	int	Dimension of embedding vectors (set to `0` if none).
`n_committed_props`	int	Number of rows with properties (set to `0` if none).
:::

Attribute	Type	Description
`_storage_root`	str	Base directory for resolving relative paths.
`is_multichannel`	bool	Set to `True` if using `/img_path_channels`.
:::

Datasets and groups

/img_path (Dataset, N strings) - mandatory. The primary image paths.
/embeddings (Dataset, N × D float32) - optional feature vectors.
/metadata/ (Group) - optional.
- id (strings) - required for ID-based alignment.
- [any] (strings / floats) - drug, dose, condition, …
/properties/ (Group) - optional.
- [any] (float32) - cell area, intensity, texture, …

/config/ (Group) - optional but recommended when embeddings were produced with PhenoMe-compatible settings. Attributes only (no full paths):

Attribute / dataset	Type	Description
`channel_mode`	str	`'split'` or `'combined'`.
`resize_size`	str	Square resize size, or `"none"`.
`pad_size`	str	Pad size, or `"none"`.
`force_rgb`	bool	Grayscale expanded to RGB.
`l2_normalize_channels`	bool	Split mode: L2 each channel before concat. Absent → treated as `True`.
`channels`	int dataset `(C,)`	Optional channel subset; omit if all channels.

See HDF5 database protocol for the full /config contract.

3. Full example

import os

import h5py
import numpy as np

DATA_DIR = "/path/to/images"
N, D = 5, 128

img_paths = [f"image_{i}.png" for i in range(N)]
img_ids = [f"image_{i}" for i in range(N)]

embeddings = np.random.randn(N, D).astype("float32")
drug_names = np.array(["DMSO", "DMSO", "DrugA", "DrugB", "Control"], dtype=object)
cell_area = (np.random.rand(N) * 1000).astype("float32")

with h5py.File("external_results.h5", "w") as f:
    f.attrs["version"] = "2.0"
    f.attrs["n_committed"] = N
    f.attrs["embedding_dim"] = D
    f.attrs["n_committed_props"] = N
    f.attrs["_storage_root"] = DATA_DIR
    f.attrs["is_multichannel"] = False

    dt_str = h5py.special_dtype(vlen=str)
    f.create_dataset("img_path", data=np.array(img_paths, dtype=object), dtype=dt_str)
    if D > 0:
        f.create_dataset(
            "embeddings",
            data=embeddings,
            compression="gzip",
            compression_opts=4,
        )

    meta = f.create_group("metadata")
    meta.create_dataset("id", data=np.array(img_ids, dtype=object), dtype=dt_str)
    meta.create_dataset("drug", data=np.array(drug_names, dtype=object), dtype=dt_str)

    props = f.create_group("properties")
    props.create_dataset("cell_area", data=cell_area)

    # Optional: record how embeddings were produced (PhenoMe resume / reproducibility)
    cfg = f.create_group("config")
    cfg.attrs["channel_mode"] = "split"
    cfg.attrs["resize_size"] = "224"
    cfg.attrs["pad_size"] = "none"
    cfg.attrs["force_rgb"] = True
    cfg.attrs["l2_normalize_channels"] = True

from phenome.pipeline import PhenoMe

pipeline = PhenoMe()
pipeline.find_files(DATA_DIR)
pipeline.load_results("external_results.h5")

print(f"Loaded {pipeline.results.n_images} images.")
pipeline.plot_tsne(color_by="drug")

Separate embeddings and properties files

When embeddings and properties live in two checkpoints (common after process_images(..., checkpoint_path="..._embeddings.h5") and compute_properties(..., checkpoint_path="..._properties.h5")), load them in sequence:

pipeline = PhenoMe()
pipeline.find_files(DATA_DIR)
pipeline.load_embeddings("run_embeddings.h5")
pipeline.load_properties("run_properties.h5")

load_properties overlays properties onto the rows established by load_embeddings and keeps the embeddings file open for lazy get_embeddings() access.

Verify the file

Quick sanity check before you open it in PhenoMe:

import h5py

with h5py.File("external_results.h5", "r") as f:
    print("version           :", f.attrs["version"])
    print("n_committed       :", f.attrs["n_committed"])
    print("embedding_dim     :", f.attrs["embedding_dim"])
    print("n_committed_props :", f.attrs["n_committed_props"])
    print("keys              :", list(f.keys()))
    if "config" in f:
        print("config attrs      :", dict(f["config"].attrs))

4. Common variants

Properties only (no embeddings)

Use this pattern if you only want PhenoMe to visualise properties (for example from CellProfiler):

with h5py.File("props_only.h5", "w") as f:
    f.attrs["version"] = "2.0"
    f.attrs["n_committed"] = N
    f.attrs["embedding_dim"] = 0
    f.attrs["n_committed_props"] = N
    # ... create img_path and /metadata/id ...
    # ... create /properties/ datasets ...

Path-only alignment (no IDs)

If you already have absolute paths:

import pandas as pd

file_df = pd.DataFrame(
    {"file_path": ["/abs/path/img1.png", "/abs/path/img2.png"]}
)
pipeline.set_file_df(file_df)
pipeline.load_results("abs_paths.h5")

5. Troubleshooting

Symptom	Probable cause	Solution
0 images loaded	IDs in HDF5 do not match `find_files` IDs.	Check `/metadata/id` - by default, filename without extension.
Images not found	`_storage_root` + relative paths are incorrect.	Print absolute paths in Python and compare to HDF5 `/img_path`.
No PCA / UMAP plots	`embedding_dim` is 0 or dataset missing.	Ensure `embedding_dim > 0` and the `/embeddings` dataset exists.
Runtime / type error	Incorrect dtype.	Paths and string metadata must be `vlen str`; properties must be `float32`.