Discover image and mask files from directories, extract metadata, and cache internally. Recursively scans `image_dir` for image files and optionally aligns them with masks from `mask_dir`. Extracts per-file metadata using `metadata_fn` and stores the result as the internal file DataFrame used by `process_images()`, `compute_properties()`, and `inspect_data()`. **Args:** - **`image_dir`** (`str`): Root directory or glob pattern for image files. Scanned recursively. Example: "path/to/images/" or "path/to/**/*.tif". - **`mask_dir`** (`str or None`): Root directory for corresponding mask files. If provided, adds a 'mask_path' column to the output DataFrame. Default: None (no masks). - **`extensions`** (`list of str or None`): File extensions to include (e.g., [".tif", ".png", ".jpg"]). If None, defaults to common image formats. Default: None. - **`metadata_fn`** (`callable, MetadataBase, or None`): Function or object to extract metadata from file paths. Signature: `fn(file_path: str) -> dict[str, Any]`. Examples: - `DefaultMetadata()`: Simple metadata. - `PathTemplateMetadata(template="...")`: Extract from path pattern. - `lambda p: {'batch': p.split('/')[-2]}`: Custom function. If None, no metadata extracted. Default: None. - **`on_missing_metadata`** (`str`): How to handle files with missing metadata. One of: - 'drop': Remove rows with incomplete metadata. - 'keep': Include rows with NaN metadata. Default: 'drop'. - **`mask_filename_column`** (`str or None`): Metadata column name used to locate mask files. If set, masks are located using this column value. Useful for custom mask naming. Default: None (standard alignment logic). - **`mask_extensions`** (`list of str or None`): Extensions to try when exact mask filename fails (e.g., [".png", ".tif"]). Default: None (exact filename required). - **`channel_names`** (`list of str or None`): Optional labels for image channels, in channel-axis order (same order as list entries in ``file_path`` for multi-file samples). When set, multi-channel property columns use these labels as suffixes (e.g. ``intensity_entropy_DAPI`` instead of ``intensity_entropy_ch0``). Names must be non-empty, unique (case-insensitive), and contain no whitespace. Default: None (integer ``_ch{idx}`` suffixes). **Returns:** pd.DataFrame: File discovery DataFrame with columns: - 'file_path': (str or list of str) Image file path(s). - 'mask_path': (str, if mask_dir set) Corresponding mask path. - Metadata columns: Custom columns from `metadata_fn` if provided. Shape: (N, 2+M) where N is number of images, M is metadata columns. **Raises:** - **`FileNotFoundError`**: If `image_dir` does not exist. - **`ValueError`**: If all files are dropped due to `on_missing_metadata='drop'`, or if `channel_names` is invalid. - **`RuntimeError`**: If metadata extraction fails or masks cannot be resolved. **Example:** ```python Discover images with metadata: >>> pm = PhenoMe() >>> df = pm.find_files( ... "data/images/", ... metadata_fn=PathTemplateMetadata(template="batch_{batch}/sample_{id}"), ... extensions=[".tif", ".png"] ... ) >>> print(df.shape) (150, 4) # 150 images, 4 columns: file_path, batch, id, [others] With masks: >>> df = pm.find_files( ... image_dir="data/images/", ... mask_dir="data/masks/", ... extensions=[".tif"] ... ) >>> print(df.columns) Index(['file_path', 'mask_path'], dtype='object') Named channels for property suffixes: >>> df = pm.find_files( ... "data/images/", ... channel_names=["DAPI", "GFP"], ... ) ``` **See Also:** - `set_file_df`: Manually provide a file DataFrame. - `process_images`: Process discovered images.

Set the internal file DataFrame used for processing. Validates that file_df has a 'file_path' column and optionally 'mask_path'. Stores a normalized copy for use by process_images, compute_properties, inspect_data, and related methods. **Args:** - **`file_df`**: DataFrame with 'file_path' column (str or list of str per row). May include 'mask_path' and metadata columns. - **`channel_names`**: Optional labels for image channels, in channel-axis order (same order as list entries in ``file_path`` for multi-file samples). When set, multi-channel property columns use these labels as suffixes (e.g. ``intensity_entropy_DAPI`` instead of ``intensity_entropy_ch0``). Default: None (integer ``_ch{idx}`` suffixes). **Returns:** The normalized DataFrame that was stored.

Extract embeddings from images using a pretrained or custom model. Reads images from the file DataFrame (set via `find_files()` or `set_file_df()`), processes them through the model in batches, and stores embeddings and metadata in `self.results`. Supports filtering, channel selection, custom preprocessing, and checkpointing for resumable processing. **Embedding storage:** - Eager (default): Embeddings stored in `results.embeddings` (ndarray). - Lazy (with checkpoint): Embeddings stored in HDF5; `results.embeddings` is None and embeddings are loaded on-demand via `get_embeddings()`. - Temporal (incremental): New images added to existing checkpoint via `process_temporal_images()`. **Default processing order** (when ``custom_transformations`` is None): 1. Load image (native dtype) 2. Optional ``preprocessing_fn`` 3. Channel selection / ``channel_mode`` / ``force_rgb`` 4. TypeMaxNorm intensity scaling (identity if max <= 1) 5. Optional pad, then resize (default 224) 6. ImageNet mean/std normalize 7. Vision model -> embedding 8. Split mode: L2-normalize each channel embedding (default True) A concise processing pipeline is logged at INFO. Inspect ``get_processing_pipeline()`` or ``print_processing_pipeline()``. Custom ``custom_transformations`` replace steps 4-6. **Args:** - **`model_wrapper`** (`ModelWrapper`): Model instance (e.g., from `load_dinov2_model()`). Must implement `_get_embeddings(tensor: torch.Tensor) -> torch.Tensor`. For non-PyTorch models (Keras, TensorFlow, etc.), see the [Custom Model Wrapper guide](https://AAitorG.github.io/PhenoMe/guides/custom-model-wrapper/). - **`batch_size`** (`int`): Batch size for processing. Default: 32. Larger batches are faster but use more GPU memory. - **`num_workers`** (`int`): Number of loader processes. In Jupyter, automatically reduced to 0 to avoid multiprocessing issues. Default: 4. - **`filters`** (`dict or None`): Include only rows matching criteria. Format: `{column: [value1, value2, ...]}`. Default: None (no filtering). - **`exclude`** (`dict or None`): Exclude rows matching *any* criterion (OR across fields). Same value format as `filters`. Default: None. - **`channel_mode`** (`str`): How to handle multi-channel images. One of: - 'split': Process each channel separately (default). - 'combined': Process all channels as RGB or grayscale. - **`channels`** (`list of int or None`): Specific channel indices to use. If None, uses all available. Default: None. - **`preprocessing_fn`** (`callable or None`): Custom preprocessing function applied to each image before the model. Signature: `fn(image: ndarray) -> ndarray`. Example: normalization, contrast adjustment, etc. Default: None. - **`custom_transformations`** (`transforms object or None`): PyTorch transforms to apply before the model (e.g., resize, normalize). If None, the default order is TypeMaxNorm, optional pad, optional resize, then ImageNet Normalize. Default: None. - **`preprocessing_description`** (`str or None`): Optional plain-language note for `preprocessing_fn`. Shown in the INFO processing pipeline log and on `get_processing_pipeline()`. Internals are never inspected. Recommended when using custom preprocessing so the log describes what the function does. Default: None. - **`transformations_description`** (`str or None`): Optional plain-language note for `custom_transformations`. Shown in the INFO processing pipeline log and on `get_processing_pipeline()`. Internals are never inspected. Default: None. - **`append`** (`bool`): If True, append to existing embeddings in `results.embeddings`. If False (default), replace. Default: False. - **`resize_size`** (`int or None`): Image resize dimension (square). Default: 224. Set to None to skip resizing. - **`pad_size`** (`int or None`): Padding size (square). Default: None (no padding). - **`checkpoint_path`** (`str or None`): HDF5 file path for lazy storage and resumable processing. If file exists, processing resumes from last checkpoint. If None, embeddings stored in-memory. Default: None. - **`force_rgb`** (`bool`): If True, convert grayscale to RGB before model. Default: True. - **`l2_normalize_channels`** (`bool`): In ``channel_mode='split'``, L2-normalize each channel embedding before concatenation so channels contribute equally. Set False to preserve raw channel magnitudes (one channel may dominate). Ignored in combined mode. Default: True. - **`save_every`** (`int`): Save checkpoint every N batches (when using checkpoint_path). Default: 5. - **`lazy_checkpoint`** (`bool`): If True and checkpoint_path is set, enable lazy loading (embeddings not kept in-memory). If False, load all embeddings after processing. Default: True. - **`force_reprocess`** (`bool`): If True and ``checkpoint_path`` already exists, delete that file and process all requested images from scratch (no resume). Default: False. - **`progress_callback`** (`callable or None`): Optional ``(current, total, desc) -> None`` hook for GUI/notebook progress bars. Invoked after each batch. When set, the default ``tqdm`` bar is disabled. Default: None. **Returns:** None. Modifies `self.results` in-place with embeddings, metadata, and paths. **Raises:** - **`FileNotFoundError`**: If file_df not set (call `find_files()` or `set_file_df()` first). - **`ValueError`**: If filters/exclude reference non-existent columns or produce no data. - **`RuntimeError`**: If model inference fails or checkpoint is corrupted. - **`IOError`**: If checkpoint file cannot be created or read. **Example:** ```python Process images with DINOv2 model: >>> from phenome import PhenoMe, load_dinov2_model >>> pm = PhenoMe(device="cuda") >>> pm.find_files("images/") >>> model = load_dinov2_model("dinov2_vitb14") >>> pm.process_images(model, batch_size=64, resize_size=224) >>> print(pm.results.embeddings.shape) (1000, 768) Optional description for custom preprocessing (appears in the INFO processing pipeline log even though the callable itself is opaque): >>> def enhance_contrast(image): ... p1, p99 = np.percentile(image, (1, 99)) ... return np.clip(image, p1, p99) >>> pm.process_images( ... model, ... preprocessing_fn=enhance_contrast, ... preprocessing_description="clip 1st/99th percentiles", ... ) With filtering and checkpoint: >>> pm.process_images( ... model, ... filters={"batch": ["batch_1", "batch_2"]}, ... checkpoint_path="embeddings.h5", ... lazy_checkpoint=True ... ) With a Jupyter progress bar: >>> from ipywidgets import IntProgress, Label, VBox >>> from IPython.display import display >>> bar, lab = IntProgress(max=1), Label() >>> display(VBox([lab, bar])) >>> def on_progress(current, total, desc): ... bar.max = max(total, 1) ... bar.value = current ... lab.value = f"{desc}: {current}/{total}" >>> pm.process_images(model, progress_callback=on_progress) ``` **See Also:** - `find_files`: Discover and organize image files. - `process_temporal_images`: Incrementally add images to existing checkpoint. - `get_embeddings`: Retrieve embeddings (handles lazy loading). - `compute_properties`: Extract morphological properties from embeddings. - `get_processing_pipeline`: Structured ordered processing pipeline for this run. - `print_processing_pipeline`: Print the same text as the INFO log.

Process new images in-memory (temporary) and append to the current session. Use this to explore additional images (e.g., from a new condition or replicate) without re-running process_images. Temporal images are not persisted to checkpoint; visualize or export before calling clear_temporal_data() or reset(). Requires process_images() to have been run first. Processing parameters (channel_mode, channels, resize_size, etc.) are inherited from the base run when not specified. Temporal rows get metadata['source'] = 'NEW'. Optional ``preprocessing_description`` / ``transformations_description`` are recorded on the session-only temporal pipeline (same optional notes as ``process_images``; they appear in the INFO processing pipeline log). Default processing order matches ``process_images`` (load, optional preprocess, channels/RGB, TypeMaxNorm, pad/resize, ImageNet, model, split L2). A concise processing pipeline is logged at INFO.

Inspect image and mask dimensions, shapes, dtypes, and data ranges. Delegates to FileDiscovery. Uses stored dtypes (not the float32 cast from ``read_image``). Requires file_df from ``set_file_df`` or ``find_files``.

Remove all temporal images (metadata['source'] == 'NEW') from the session. **Returns:** - **`int`**: Number of temporal rows removed.

Return embeddings for the given row indices. When a checkpoint / HDF5 database is active (``self._db`` is not None), only the requested rows are read from disk — the core of the lazy-loading strategy. When temporal embeddings exist (from ``process_temporal_images``), rows with index ``>= self._temporal_start_idx`` are read from ``self._temporal_embeddings``; earlier rows come from the checkpoint. When no database is active, returns ``results.embeddings`` (optionally sliced). If that array holds only checkpoint rows (length ``_temporal_start_idx``) and temporal embeddings are still in memory, they are appended to form a full ``(n_images, D)`` array. If ``results.embeddings`` already has ``n_images`` rows (e.g. materialized on ``checkpoint_context`` exit), temporal data is not appended again. **Args:** - **`indices`**: Optional row index or sequence of row indices (0-based). A single ``int`` returns shape ``(1, D)``. If None, returns all embeddings. **Returns:** np.ndarray shape ``(len(indices), D)`` float32, or None if no embeddings are available.

Return metadata, properties, and optional distance for image idx. **Args:** - **`idx`**: Image index (0 to n_images-1). - **`distance_results`**: Optional DataFrame from compute_reference_distances. **Returns:** - **`dict`**: Keys image_index, image_name, image_path, metadata keys, property keys, distance (if distance_results provided), is_reference (if applicable).

Return sorted metadata keys.

Return sorted list of property keys stored in results.

Return True if embedding data is available (lazy or eager). A properties-only checkpoint has ``n_committed > 0`` but no embedding dataset; we additionally require ``embedding_dim`` to be a positive int.

Return the embedding dimensionality, or 0 if unavailable.

Return the session processing pipeline, or None if none was recorded. The dict has ``version``, ``base``, and optional ``temporal`` and ``properties``. After ``load_results``, embedding and property steps come from ``/processing_pipeline``; temporal exists only in the current session. **See Also:** - `print_processing_pipeline`: Print the concise pipeline text.

Print the ordered processing pipeline for this session. Uses the same text as the INFO log during ``process_images`` and ``compute_properties``. After ``load_results``, prints the steps stored in ``/processing_pipeline`` (embeddings and properties). **See Also:** - `get_processing_pipeline`: Structured dict form of the same record.

Reset all stored data to a clean state. Closes any open HDF5 database handle before clearing results. By default preserves _file_df so compute_properties and inspect_data continue to work after process_images. Set clear_file_df=True for a full reset (e.g. when switching to a completely new dataset).

Reset only the computed properties. **Args:** - **`verbose`**: If True, logs a confirmation message.

Transfer specified metadata columns to properties. **Args:** - **`columns`**: A single metadata column name or a list of names to transfer.

Save results to HDF5 (atomic write or in-place flush). When a checkpoint database is already active (``self._db`` is set), the data is already on disk; this method flushes any uncommitted buffers and logs the existing path. When no database is active, writes ``self.results`` to a new HDF5 file using a temporary-file + atomic rename for crash safety. For full reproducibility, also call [export_experiment_config](/PhenoMe/advanced/api/pipeline/#api-phenome-export_experiment_config) to save seed, use_gpu_for_dr, reference_filters, and model name (not stored in the checkpoint). **Args:** - **`path`**: Explicit output path. - **`compression`**: HDF5 compression algorithm (used only for new files).

Load and use an existing results/checkpoint file. Metadata and properties are loaded immediately. Embeddings are either loaded into RAM (lazy_checkpoint=False) or accessed on-demand from disk (lazy_checkpoint=True). Uses the internally stored file_df (from [find_files](/PhenoMe/advanced/api/pipeline/#api-phenome-find_files) or [set_file_df](/PhenoMe/advanced/api/pipeline/#api-phenome-set_file_df)) to resolve paths so the checkpoint works on this machine. Call find_files or set_file_df first. For separate embeddings and properties checkpoints, prefer [load_embeddings](/PhenoMe/advanced/api/pipeline/#api-phenome-load_embeddings) then [load_properties](/PhenoMe/advanced/api/pipeline/#api-phenome-load_properties). **Args:** - **`path`**: Path to .h5 or .hdf5 file. - **`lazy_checkpoint`**: If True (default), keep the checkpoint file open. If False, load all data into RAM and close the file. **Raises:** - **`FileNotFoundError`**: If file does not exist. - **`ValueError`**: If file format is not recognised as HDF5. - **`RuntimeError`**: If file_df is not available (call find_files or set_file_df first). - **`ConcurrentCheckpointAccessError`**: If the checkpoint is already open in another notebook or process (only one instance can access it at a time).

Context manager that loads a checkpoint and guarantees it is closed on exit. More reliable than relying on __del__ for cleanup. Use when you need to ensure the HDF5 file handle is released (e.g. before moving or deleting the file, or when opening multiple checkpoints in sequence). **Embeddings:** While the context is open, lazy embeddings are read from the HDF5 file via ``get_embeddings()``. On exit, if temporal embeddings were loaded in memory (``process_temporal_images`` while the checkpoint was open), the merged full embedding matrix is copied to ``results.embeddings`` and ``self._temporal_embeddings`` is cleared. Otherwise ``results.embeddings`` may stay ``None``; call ``load_results(..., lazy_checkpoint=True)`` or enter another ``checkpoint_context`` before using ``get_embeddings()`` again. **Args:** - **`path`**: Checkpoint HDF5 path. If None, uses the path from the last ``load_results`` / ``process_images`` session when available. - **`lazy_checkpoint`**: If True, keep the HDF5 file open for on-demand reads inside the context. Default: True. **Example:** ```python pheno.find_files("path/to/images") with pheno.checkpoint_context("results.h5") as p: p.plot_pca(color_by="condition") # Checkpoint closed here ```

Export experiment configuration for reproducibility. Writes a JSON file with init parameters, environment, processing params, and silently recorded method steps. Call after the analysis you want to document. Optional ``model_name`` / ``reference_filters`` override values already captured from ``process_images`` / ``compute_reference_distances``. **Args:** - **`path`**: Output path for config.json. - **`reference_filters`**: Optional dict used for compute_reference_distances (e.g. {'condition': 'Control'}). Include for full traceability. - **`model_name`**: Optional model identifier (e.g. 'dinov2_vitb14_reg'). - **`checkpoint_path`**: Optional checkpoint path used during processing (if not provided and a checkpoint is open, uses self._db.path). **extra: Additional key-value pairs to include in the config.

Write a markdown methods summary from the silent run log. Opt-in companion to [export_experiment_config](/PhenoMe/advanced/api/pipeline/#api-phenome-export_experiment_config) for paper supplements. Does nothing unless this method is called.

Export the dataset as a table (CSV, Parquet, or Excel). **Args:** - **`output_path`**: Path to save the export. If None, only returns the DataFrame. - **`dist_results`**: Optional DataFrame from compute_reference_distances. - **`include_embeddings`**: If True, adds embedding columns; if 'separate', saves embeddings to a companion .npy file. - **`export_format`**: Output format: 'csv', 'parquet', or 'excel'. **Returns:** DataFrame with image_index, image_path, metadata, properties, distances, and embeddings (if requested).

Compute per-image properties using presets and/or custom functions. Each function receives a 2D image slice and/or mask slice and returns a dict[str, float]. When multiple channels exist, properties are suffixed with ``_ch{idx}``, or with labels from ``find_files`` / ``set_file_df`` ``channel_names`` when provided (e.g. ``_DAPI``). Use ``property_preset`` for built-in property sets and ``additional_property_functions`` to add custom functions on top (optional). This method now relies solely on ``file_df`` (typically produced by [PhenoMe.find_files](/PhenoMe/advanced/api/pipeline/#api-phenome-find_files)) for resolving image and mask paths. Legacy ``image_dir`` / ``mask_dir`` parameters are no longer supported. Images used for properties are loaded, laid out as HxWxC, then scaled with ``normalize_by_dtype_max`` (same 0-1 rule as TypeMaxNorm). Masks are loaded and laid out the same way, then binarized (foreground = pixels ``> 0.5``; no intensity scaling). A concise processing pipeline of those applied steps is logged at INFO. **Args:** - **`metadata_config`**: Optional MetadataBase instance (e.g. from `phenome.metadata`). When it has ``mask_dir`` and ``mask_filename_column``, uses explicit mask lookup. - **`property_preset`**: Preset name. Use ``"none"`` or ``None`` for no preset. Valid: ``"none"``, ``"basic"``, ``"shape"``, ``"intensity"``, ``"standard"``, ``"complete"``. - **`additional_property_functions`**: Extra property functions to add on top of preset. Dict of {requirement: fn_or_list} where requirement is ``"image"``, ``"mask"``, ``"both"``, or ``"any"``. Merged with preset when both are used; used alone when ``property_preset`` is ``"none"`` or ``None``. - **`checkpoint_path`**: HDF5 checkpoint for incremental persistence. - **`save_every`**: Commit frequency (images) when checkpointing. - **`n_jobs`**: Number of parallel jobs for property computation. ``1`` (default) = sequential. Use ``-1`` to use all available CPU cores. - **`lazy_checkpoint`**: If True (default), keep the checkpoint file open. If False, load all data into RAM and close the file. - **`force_update`**: If True and ``checkpoint_path`` points to an existing file, clears stored properties in that file, writes current ``_processing_params`` into the checkpoint config, and recomputes properties for every image (no resume). Use this after changing ``property_preset`` or custom functions, or after log warnings about missing checkpoint property keys. If ``checkpoint_path`` is set but the file does not exist yet, only an info log is emitted; computation proceeds as for a new checkpoint. - **`include_metadata`**: If True, add metadata columns to the returned DataFrame. - **`progress_callback`**: Optional ``(current, total, desc) -> None`` hook for GUI/notebook progress bars. Invoked after each image (sequential) or after each parallel batch. When set, the default ``tqdm`` bar is disabled. Default: None. **Returns:** DataFrame with ``image_index``, ``image_path``, ``image_name``, property values, and optional metadata columns. Also populates ``results.properties`` as ``List[dict]`` (one dict per image). **Raises:** - **`ValueError`**: If no images processed or invalid preset/requirement keys are used. - **`TypeError`**: If ``additional_property_functions`` values are not callable. **Example:** ```python Drive a napari/Qt progress bar from a worker thread: >>> def on_progress(current, total, desc): ... # update Qt/napari progress on the main thread ... progress.setValue(int(100 * current / total) if total else 0) ... progress.setLabelText(desc) >>> pm.compute_properties( ... property_preset="basic", ... progress_callback=on_progress, ... ) ```

Group properties, compute per-group statistics, and optionally print a table. Uses `_build_properties_dataframe` from computed ``results.properties`` and metadata (same source as [compute_properties](/PhenoMe/advanced/api/pipeline/#api-phenomeproperties-compute_properties)). **Args:** - **`group_by`**: Metadata columns to group by. If None, auto-selects first 2. - **`properties`**: Property columns to include in aggregation. If None, auto-detects numeric. Matching is case-insensitive; names must be unique ignoring case. - **`print_table`**: If True, log a formatted mean±std table. - **`print_properties`**: Subset of properties to show in the table. If None, shows all. Matching is case-insensitive; names must be unique ignoring case. - **`group_column_width_max`**: Maximum width for each grouping column when printing. - **`content_col_width_max`**: Maximum width for each property statistic column when printing. **Returns:** Aggregated DataFrame with mean/std/min/max per property per group. **Raises:** - **`TypeError`**: If group_by/properties are invalid types. - **`ValueError`**: If group_by keys are not in DataFrame columns.

For each non-reference group, return the top k properties that most differentiate it from the reference. Uses Cohen's d (effect size) by default to rank properties by how different each group is from the reference. Both higher and lower values count as "different" (uses |d|). Cohen's d formula (pooled): d = (mean_group - mean_ref) / s_pooled, where s_pooled = sqrt([(n_ref-1)*std_ref² + (n_other-1)*std_other²] / (n_ref + n_other - 2)). Requires sample std (ddof=1) from property_stats_by_group. **Args:** - **`df`**: Aggregated DataFrame from property_stats_by_group(). - **`reference_group`**: Dict mapping grouping column names to values (e.g. {"drug": "Control", "time": "60_min"}). - **`k`**: Number of top properties per group. - **`properties`**: Property names to consider. If None, uses all in DataFrame. Matching is case-insensitive; names must be unique ignoring case. - **`metric`**: 'cohens_d' (effect size) or 'mean_diff' (absolute mean difference). - **`print_output`**: If True, pretty-print the results. - **`group_column_width_max`**: Maximum width for each grouping column in the printed table. - **`top_property_col_width_max`**: Max width for the property name column in the printed table. - **`top_effect_col_width_min`**: Min width for Cohen's d and mean diff columns when printing. **Returns:** DataFrame with columns: grouping cols, property, effect_size (Cohen's d when ``metric='cohens_d'``), mean_diff, ref_mean, group_mean, rank. One row per (group, property) for top-k only.

Correct plate-to-plate variation using control wells per batch. Call after :meth:`process_images` (and optionally after ``compute_properties`` when ``source='properties'``). For lazy checkpoints, embeddings are read and rewritten in chunks of *chunk_size* rows. **Args:** - **`batch_metadata_key`**: Metadata column for batch / plate id (case-insensitive). - **`method`**: ``"sphering"`` (default) or ``"zscore"``. - **`source`**: ``"embeddings"`` or ``"properties"``. - **`property_keys`**: Property names to correct when ``source='properties'``. If None, uses all keys from :meth:`get_available_property_keys`. - **`control_filters`**: Same semantics as ``filter_indices`` filters (required). - **`inplace`**: If True, updates ``results.embeddings`` or ``results.properties`` in place. - **`checkpoint_path`**: If set, persist batch-correction metadata and/or export results. - If ``None`` (default): Corrections still apply (RAM or an already-open lazy checkpoint), but nothing new is written for persistence—no ``/batch_correction`` stats on disk and no new HDF5 via :meth:`save_results`. - If ``str``: With an open checkpoint, it is cloned to this path first; stats are saved under ``/batch_correction`` when a DB is active; in eager mode (no DB), results are saved to this path after correction. - **`chunk_size`**: Rows per chunk for embedding I/O and property scatter. - **`force`**: If True, allow re-applying correction after a previous run in this session. - **`ridge_multiplier`**: Covariance ridge strength for sphering (see :func:`compute_batch_stats`). - **`min_controls`**: Minimum control wells required per batch. **Returns:** Dict with batch stats summary (``batches``, ``controls_per_batch``, ``method``, …).

Perform clustering and store labels in ``metadata[i]['cluster']`` for each image. **Args:** - **`source`**: ``'embeddings'``, ``'properties'``, or ``'combined'``. - **`n_clusters`**: Number of clusters (k-means, GMM). Ignored for ``clustering_method='dbscan'``. - **`clustering_method`**: ``'kmeans'`` (default), ``'dbscan'``, or ``'gmm'``. - **`property_keys`**: Property subset when *source='properties'*. - **`filters`**: Optional metadata filters. - **`exclude`**: Optional metadata exclusions (same structure as filters). - **`random_state`**: Seed for k-means/GMM. If None, uses pipeline seed when available. - **`normalize`**: Whether to normalize data before clustering (default: True). For embeddings, uses L2 normalization. For properties, uses StandardScaler. - **`reduce_dim`**: If set, apply dimensionality reduction before clustering (default: 100). Only applied when data has more features than *reduce_dim*. Set to None to disable dimensionality reduction. - **`reduce_method`**: Method for dimensionality reduction when *reduce_dim* is set: ``'pca'`` (fast, linear), ``'tsne'`` (non-linear, slower), ``'umap'`` (non-linear, preserves structure). Default: ``'pca'``. - **`return_silhouette`**: If True, compute and return the silhouette score (default: False). Returns ``(df, score)``; score is None if it could not be computed. Silhouette requires at least 2 clusters and 2 samples per cluster. - **`dbscan_eps`**: Maximum distance between two samples for DBSCAN (only when clustering_method='dbscan'). If None, uses 0.5. - **`dbscan_min_samples`**: Minimum samples in a neighborhood for DBSCAN (only when clustering_method='dbscan'). If None, uses max(2, n_samples // 20). - **`include_metadata`**: If True, add metadata columns to the returned DataFrame. **Returns:** If return_silhouette=False: Tier-A DataFrame with image_index, image_path, image_name, cluster, and optional metadata. If return_silhouette=True: Tuple of (df, silhouette_score). Score is None if it could not be computed. Cluster labels are 0..K-1; None for excluded/noise.

Detect outliers based on distance to centroid. When *source* is ``'embeddings'`` and *normalize* is True (default), embeddings are L2-normalized and group centroids use the **arithmetic mean** (not necessarily a unit vector). Euclidean distance to that centroid is not the same as angular distance to the mean direction; consider property-based or combined sources when a different geometry is needed. **Args:** - **`method`**: ``'z-score'`` or ``'iqr'``. - **`threshold`**: Threshold multiplier. - **`group_by`**: Property/metadata key(s) for grouping. Can be a single string or a list of strings. If a list is provided, groups are formed by combining values from all specified columns (e.g., "drug1-10uM"). If *None*, all images are treated as one group. - **`filters`**: Optional metadata filters. - **`exclude`**: Optional metadata exclusions (same structure as filters). - **`source`**: Feature space for outlier detection: - ``'embeddings'``: model embeddings - ``'properties'``: scalar phenotypic properties - ``'combined'``: normalized concatenation of embeddings and properties. - **`property_keys`**: Property subset when *source* is ``'properties'`` or ``'combined'``. - **`normalize`**: Whether to normalize data before outlier detection (default: True). For embeddings, uses L2 normalization. For properties, uses StandardScaler. - **`drop_outliers`**: If True, remove detected outliers from self.results (default: False). - **`plot`**: If True (default), shows one matplotlib figure per group: subplots for detected outliers and a figure title naming the group (``Group: …``). Images are shown without pipeline ``image_transforms`` (same as ``apply_transforms=False`` for display). **Returns:** pd.DataFrame: Tier-A DataFrame containing detected outliers. Columns include: - ``image_index``: Global image index. - ``image_path``: Path to the image file. - ``image_name``: Basename of the image file. - Additional columns for each grouping key (if *group_by* was set). - ``threshold``: The threshold value used for the image's group. - ``distance_to_centroid``: Distance to the group centroid. If no outliers are detected, returns an empty DataFrame. The index is a range index from 0 to N-1.

Find images closest to each group centroid. **Args:** - **`cluster_col`**: Property/metadata key(s) for grouping. Can be a single string or a list of strings. If a list is provided, groups are formed by combining values from all specified columns (e.g., "drug1-10uM"). If *None*, all images are treated as one group. - **`n_prototypes`**: How many prototypes per group. - **`source`**: ``'embeddings'``, ``'properties'``, or ``'combined'``. - **`property_keys`**: Property subset when *source='properties'*. - **`filters`**: Optional metadata filters. - **`exclude`**: Optional metadata exclusions (same structure as filters). - **`metric`**: ``'euclidean'`` or ``'cosine'``. With ``metric='euclidean'``, the group centroid is the arithmetic mean (even when embeddings are L2-normalized). With ``metric='cosine'``, the centroid direction is L2-normalized before ranking by similarity. - **`normalize`**: Whether to normalize data before finding prototypes (default: True). For embeddings, uses L2 normalization. For properties, uses StandardScaler. - **`plot`**: If True (default), shows one matplotlib figure per group: subplots for that group's prototypes and a figure title naming the group (``Group: …``). Images are shown without pipeline ``image_transforms`` (same as ``apply_transforms=False`` for display). **Returns:** pd.DataFrame: DataFrame with one row per prototype. Columns include grouping keys (split into individual columns if multiple), ``image_index``, ``image_path``, ``image_name``, and ``rank_in_group``.

Compute z-score enrichment of properties per group (e.g. cluster labels). For each group, z-scores are computed against a **leave-group-out** population (all samples *except* the current group). When *correct_multiple_testing* is True (default), Welch two-sample t-tests (group vs leave-one-out rest) are run per property and Benjamini-Hochberg FDR correction is applied across all (group, property) pairs; a ``significant`` column is added to the output DataFrame. **Args:** - **`group_by`**: Property/metadata key(s) holding group labels (e.g. ``"cluster"``). Can be a single string or a list of strings for composite grouping. - **`property_keys`**: Properties to analyse (all numeric if *None*). - **`filters`**: Optional metadata filters. - **`exclude`**: Optional metadata exclusions (same structure as filters). - **`plot`**: If True (default), show an interactive Plotly faceted bar chart (one row per group). If False, log a plain-text summary via the package logger instead (unless *return_fig* requests a figure). - **`return_fig`**: If True, return a tuple ``(df, fig)`` where ``fig`` is the Plotly figure. When *return_fig* is True, ``fig.show()`` is not called; use ``plot=True`` with ``return_fig=False`` for the default interactive display. - **`top_k`**: Max properties per group in the figure and in the text summary (``None`` = all). - **`figsize`**: Figure size ``(width, height)`` in inches for the Plotly layout. - **`title`**: Optional figure title. - **`correct_multiple_testing`** (`bool`): If True (default), run Welch t-tests per (group, property), apply Benjamini-Hochberg FDR, and add ``p_value`` / ``significant`` columns (alpha = 0.05). **Returns:** pd.DataFrame or tuple[pd.DataFrame, Any]: - If *return_fig* is False (default): pd.DataFrame with columns: [group_by columns], property, score, mean_group, mean_pop, p_value, abs_score, and optionally significant when *correct_multiple_testing* is True. - If *return_fig* is True: A tuple (enrichment_df, fig).

Correlate dim-reduction components with phenotypic properties. **Args:** - **`method`**: Dimensionality reduction method. - **`n_components`**: Number of components to compute. - **`source`**: ``'embeddings'``, ``'properties'``, or ``'combined'``. - **`property_keys`**: Property subset when *source* is ``'properties'`` or ``'combined'``. - **`filters`**: Optional metadata filters. - **`exclude`**: Optional metadata exclusions (same structure as filters). - **`top_k`**: Number of top correlations to return in summary and per facet in the plot. - **`normalize`**: Whether to normalize data before computing correlations (default: True). - Source data (embeddings/properties): Normalized before dimensionality reduction (L2 normalization for embeddings, StandardScaler for properties) - Properties: Normalized using StandardScaler before correlation computation - **`correlation_method`**: Correlation method to use. Options: - ``'pearson'``: Pearson correlation coefficient (default, linear relationships) - ``'spearman'``: Spearman rank correlation (monotonic relationships, robust to outliers) - ``'distance_correlation'``: Distance correlation (detects non-linear relationships, requires ``dcor`` library: ``pip install dcor``) - ``'mutual_info'``: MI-derived correlation coefficient (detects any dependency, mapped to [0, 1] via the Gaussian bivariate transform, **not** standard NMI) - **`plot`**: If True (default), show an interactive Plotly faceted bar chart (one row per component). If False, log a plain-text summary via the package logger instead (unless *return_fig* requests a figure). - **`return_fig`**: If True, return a tuple of ``(correlation_df, figure)``. When ``return_fig`` is True, ``fig.show()`` is not called. - **`figsize`**: Figure size ``(width, height)`` in inches for the Plotly layout. **Returns:** pd.DataFrame | tuple[pd.DataFrame, Any]: - If ``return_fig`` is False (default): Correlation DataFrame with a ``property`` column and ``component_1``, ``component_2``, … columns. - If ``return_fig`` is True: Returns a tuple of ``(correlation_df, figure)``. **Raises:** - **`ValueError`**: If dimensionality reduction fails, no components are found, no numeric properties are available, or correlation computation fails. **Examples:** ```python Get correlations directly:: corr = pheno.compute_component_correlation(plot=False) Get correlations and save the figure:: corr, fig = pheno.compute_component_correlation(return_fig=True) fig.write_html("component_corr.html") ```

Compute correlations between embedding dimensions and phenotypic scalar properties. This is the time-consuming step that computes raw correlations for each property. **Args:** - **`property_keys`**: Property names (all if *None*). - **`normalize`**: Whether to normalize data before computing correlations (default: True). - Embeddings: L2 normalization (unit norm) - Properties: StandardScaler normalization (zero mean, unit variance) - **`method`**: Correlation method to use. Options: - ``'pearson'``: Pearson correlation coefficient (default, linear relationships) - ``'spearman'``: Spearman rank correlation (monotonic relationships, robust to outliers) - ``'distance_correlation'``: Distance correlation (detects non-linear relationships, requires ``dcor`` library: ``pip install dcor``) - ``'mutual_info'``: MI-derived correlation coefficient (detects any dependency, mapped to [0, 1] via the Gaussian bivariate transform, **not** standard NMI) **Returns:** Dict mapping property names to correlation arrays (n_dims,)

Summarize embedding-property correlations, optionally plot, and/or return a Plotly figure. Computes per-property ``mean_abs``, ``std`` (across dimensions), ``max_abs``, ``min_abs``, and ``mean``, orders rows by ``order_by``, and either shows a horizontal violin plot of the distribution of |r| across dimensions per property or logs a plain-text table when ``plot`` is *False*. **Args:** - **`correlations`**: Output from :meth:`compute_embedding_property_correlations` (dict mapping property names to correlation arrays). - **`order_by`**: Metric used to sort properties (descending): ``mean_abs`` | ``max_abs`` | ``mean`` | ``std``. - **`top_k`**: Number of top properties shown in the plot and listed in ``top_properties``; full sorted table is always in ``summary`` (*None* = all). - **`plot`**: If *True* (default), show a Plotly violin plot of |r| per dimension for the top ``top_k`` properties by ``order_by`` (*None* = all). Highest metric at the **top** of the y-axis. - **`return_fig`**: If *True*, return both the summary DataFrame and the Plotly figure. - **`figsize`**: Figure size in inches, converted to pixels for Plotly layout. - **`title`**: Chart title. - **`embedding_shape`**: Shape of embeddings (n_samples, n_dims) for metadata display. - **`n_properties`**: Number of properties processed for metadata display. - **`correlation_method`**: Method used for correlation computation for metadata display. **Returns:** Sorted DataFrame with ``property``, ``mean_abs``, ``std``, ``max_abs``, ``min_abs``, and ``mean``. If ``return_fig`` is *True*, returns ``(summary, figure)``.

Explain a dimensionality reduction component using LASSO or Random Forest. Calculates which phenotypic properties (features) best explain the variability seen in a deep learning embedding dimension (the target, usually t-SNE 1 or 2). Workflow: dimensionality reduction is run on the full filtered embedding set first; regression then relates classical properties to the resulting axis coordinates. For transductive methods (t-SNE, UMAP), axis coordinates depend on all samples, so the returned ``r2`` (LASSO) describes descriptive on-axis alignment on the same dataset—not independent generalization to new plates or refit embeddings. Prefer ``drivers`` for hypothesis generation; interpret ``r2`` as an effect-size-style summary unless DR and evaluation are nested with proper holdouts. **Args:** - **`method`**: Dimensionality reduction method ('pca', 'tsne', or 'umap'). - **`component`**: Which component to explain (1, 2, ...). - **`model_type`**: The regression model to use ('lasso' or 'random_forest'). 'lasso' uses L1 regularization for linear, sparse explanations. 'random_forest' captures non-linear relationships. - **`property_keys`**: Subset of properties to use as features. - **`filters`**: Optional metadata filters. - **`exclude`**: Optional metadata exclusions. - **`normalize`**: Whether to normalize embeddings before dimensionality reduction (default: True). Property features are scaled fold-wise inside the LASSO pipeline (not globally) to avoid cross-validation leakage. - **`cv`**: Number of cross-validation folds (only for 'lasso'). - **`stratify_by`**: Optional metadata key (e.g. ``"Treatment"``, ``"Batch"``) to stratify LASSO folds by class balance. Requires at least ``cv`` samples per class; otherwise falls back to unstratified KFold. Balances folds for ``lambda`` selection but does not correct global t-SNE/UMAP target leakage. - **`rf_n_estimators`**: Number of trees (only for 'random_forest'). - **`seed`**: Random seed for reproducibility. If None, uses the pipeline's ``seed`` when set. - **`plot`**: If True (default), show an interactive Plotly bar chart of top drivers. If False, log a plain-text summary via the package logger instead. - **`return_fig`**: If True, include the Plotly figure in the return value (see Returns). When ``return_fig`` is True, ``fig.show()`` is not called; use ``plot=True`` with ``return_fig=False`` for interactive display. - **`return_meta`**: If True, include a run-parameters dict in the return value. - **`top_k`**: Number of top drivers to show in the plot or text summary. - **`figsize`**: Figure size ``(width, height)`` passed through to Plotly layout; each value is multiplied by 100 to set width and height in layout pixels. **Returns:** pd.DataFrame or tuple, depending on *return_meta* and *return_fig*: - Default: ``drivers_df`` with columns ``feature``, ``weight``. - ``return_meta=True``: ``(drivers_df, meta)``; *meta* includes r2 (descriptive on-axis fit for globally fit DR targets), n_samples, n_features, method, model_type, target_component. - ``return_fig=True``: ``(drivers_df, fig)`` or ``(drivers_df, meta, fig)`` when *return_meta* is also True.

Generate a comprehensive standalone HTML report from phenotyping results. Uses ReportConfig as the primary source of options. Pass config= for full control, or use **overrides to tweak individual settings (e.g. include_plots=False). **Args:** - **`output_path`**: Path to save the HTML file. - **`title`**: Report title displayed at the top. - **`config`**: Optional ReportConfig for defaults. If None, uses ReportConfig(). **overrides: Any ReportConfig field to override (e.g. include_plots=False, outlier_threshold=2.5, n_clusters=10). **Returns:** Path to the generated HTML file.

Return the silent run log as a JSON-serializable dict. Includes init parameters, environment, recorded method steps, processing_params, and the processing pipeline when available. Does not print or write files.

Rasterize the same view as ``plot_image_by_index`` to PNG bytes. Used by the interactive explorer with ``ipywidgets.Image`` because matplotlib ``display()`` / ``plt.show()`` from Plotly click callbacks does not reliably target a nested ``Output`` widget (output goes to the cell or nowhere). When ``show_extra_info`` is True, also emits the same details as ``plot_image_by_index`` via logging and returns their text for UI display. **Returns:** ``(png_bytes, details_text_or_none)`` — ``details_text_or_none`` is the formatted extra-info block when ``show_extra_info`` is True and details exist.

Load an embeddings checkpoint into the pipeline. Same path/file_df rebase flow as [load_results](/PhenoMe/advanced/api/pipeline/#api-phenome-load_results), but requires the file to contain embeddings. Sets ``self._db`` for lazy ``get_embeddings()`` access when ``lazy_checkpoint=True``. Typical two-file workflow:: pheno.find_files(data_dir) pheno.load_embeddings("run_embeddings.h5") pheno.load_properties("run_properties.h5") **Args:** - **`path`**: Path to .h5 or .hdf5 embeddings checkpoint. - **`lazy_checkpoint`**: If True (default), keep the checkpoint file open. If False, load all data into RAM and close the file. **Raises:** - **`FileNotFoundError`**: If file does not exist. - **`ValueError`**: If file format is not recognised as HDF5, or the checkpoint has no embeddings. - **`RuntimeError`**: If file_df is not available (call find_files or set_file_df first). - **`ConcurrentCheckpointAccessError`**: If the checkpoint is already open in another notebook or process (only one instance can access it at a time).

Load a properties checkpoint into the pipeline. When results already contain images (e.g. after [load_embeddings](/PhenoMe/advanced/api/pipeline/#api-phenome-load_embeddings) or ``process_images``), properties are aligned by metadata key then path and overlaid onto the existing rows without replacing an embeddings ``self._db`` from a different file. When no results are loaded yet, performs a full checkpoint setup like [load_results](/PhenoMe/advanced/api/pipeline/#api-phenome-load_results). For a properties-only file, the HDF5 handle is closed after load (properties are eager); ``self._db`` is kept only if that file also has embeddings. **Args:** - **`path`**: Path to .h5 or .hdf5 properties checkpoint. - **`lazy_checkpoint`**: Used on cold start only. If True (default), keep the file open when it also stores embeddings. Ignored when overlaying onto existing results. **Raises:** - **`FileNotFoundError`**: If file does not exist. - **`ValueError`**: If file format is not recognised as HDF5, or the checkpoint has no property content, or overlay matches no rows. - **`RuntimeError`**: On cold start, if file_df is not available (call find_files or set_file_df first). - **`ConcurrentCheckpointAccessError`**: If the checkpoint is already open in another notebook or process (only one instance can access it at a time).

Plot centroids of groups in reduced embedding space.

Plot a specific image by its index. **Args:** - **`idx`**: Image index in results - **`distance_results`**: Optional distance computation results DataFrame - **`channels`**: specific channels to plot - **`figsize`**: Figure size - **`title_fields`**: Optional list of field names to display in title - **`show_extra_info`**: If True, prints detailed information after plotting - **`apply_transforms`**: If True, apply pipeline image_transforms - **`downsample`**: If not None, approximate desired size (in pixels) for the longest image edge when downsampling. The final image size may differ slightly due to integer stepping. If None, no downsampling. - **`ax`**: Optional matplotlib axes to plot on. If provided, a new figure is not created. - **`return_fig`**: If True, return the displayed image as a NumPy array and do not open a matplotlib figure. Mask overlay, when requested, is composited onto that array. - **`show_mask_overlay`**: If True, draws the segmentation mask as a semi-transparent yellow overlay with a crisp contour. Requires masks to have been discovered via ``mask_dir`` in ``find_files``. Useful for verifying that masks load correctly and spatially align with their images. **Returns:** The image as a NumPy array when ``return_fig`` is True; otherwise ``None``.

Plot PCA of embeddings, properties, or combined features (Plotly, WebGL by default).

Plot t-SNE of embeddings, properties, or combined features (Plotly, WebGL by default).

Plot UMAP of embeddings, properties, or combined features (Plotly, WebGL by default).

Print distance summary statistics without plotting. Safe to use when enable_plots=False. **Args:** - **`distance_results`**: DataFrame containing 'distance' column - **`group_by`**: Metadata key to group by - **`dist_range`**: Tuple of (min, max) distances to include

True if embeddings are stored eagerly (not lazy or absent).

Embedding dimensionality, or 0 if no embeddings are stored.

Reset all fields to empty state.

True if at least one image has computed properties.

Return the basename of the primary path for image *idx*.

Sorted unique metadata keys across all images (case-preserving).

Case-insensitive metadata lookup for image *idx*.

Number of images stored.

Return the primary (or only) file path for image *idx*.

Sorted list of property names from the first non-empty properties dict.

Return a scalar property for image *idx*, or None if absent.

Update all image paths by replacing *old_data_dir* with *data_dir*. This is useful when moving a checkpoint and its dataset to a different machine or directory. **Parameters:** - **`data_dir`** (`str`): The new base directory where the images are located. - **`old_data_dir`** (`str, optional`): The old base directory to be replaced. If not provided, it is automatically detected by finding the common prefix of all stored paths. Special value "relative" indicates that stored paths are already relative and just need to be joined with *data_dir*.

Compute distances from all images to reference group. **Args:** - **`reference_filters`**: Dict mapping metadata keys to values for reference group. Example: {'drug': 'Control', 'time': '60_min'} If a value is a list, matches any value in the list. - **`filters`**: Optional generic filters applied before computing distances. Reference selection still uses reference_filters. - **`exclude`**: Optional metadata exclusions (same structure as filters). - **`source`**: 'embeddings' for model embeddings, 'properties' for scalar properties, or 'combined' for the normalized concatenation of both. - **`mode`**: 'centroid' computes distance to mean embedding of reference, 'all_to_all' computes min distance to any reference image. When *source* is ``'embeddings'`` (L2-normalized by default), the centroid is the **arithmetic mean** of reference vectors (not necessarily unit length). Euclidean distance to that centroid differs from angular distance to the mean direction; use ``distance_type='cosine'`` for angular comparisons on normalized embeddings. - **`distance_type`**: 'euclidean' (default) or 'cosine'. Cosine distance is ``1 - cosine_similarity``, range [0, 2]. Zero-norm vectors produce a distance of 1. - **`property_keys`**: Optional subset of property names when *source* is ``'properties'`` or ``'combined'``. - **`group_by`**: If set, optionally visualize or print grouped statistics after computing distances. Pass ``None`` to skip post-processing. - **`dist_range`**: When using *group_by*, keep distances in ``[min, max]`` for display and summary. - **`figsize`**: Figure size in inches (width, height) when a plot is built. - **`plot`**: If True and *group_by* is set, show or return a violin plot only (no per-group text summary to the logger). If False, log per-group summary statistics where applicable (e.g. text-only mode, or with *return_fig*). - **`return_fig`**: If True and *group_by* is set, include the Plotly figure in the return value (see Returns). When ``return_fig`` is True, ``fig.show()`` is not called. - **`return_meta`**: If True, include a run-parameters dict in the return value. - **`include_metadata`**: If True, add metadata columns to the returned DataFrame. - **`points`**: Violin plot point overlay: ``'all'``, ``'outliers'``, or ``False``; ``None`` auto-selects by data size. **Returns:** pd.DataFrame or tuple, depending on *return_meta* and *return_fig*: - Default: ``distance_df`` (Tier A) with ``image_index``, ``image_path``, ``image_name``, ``distance``, ``is_reference``. - ``return_meta=True``: ``(distance_df, meta)``; *meta* holds reference_filters, filters, mode, source, distance_type. - ``return_fig=True`` (with *group_by*): ``(distance_df, fig)`` or ``(distance_df, meta, fig)`` when *return_meta* is also True. **Example:** ```python >>> dist_df = pipeline.compute_reference_distances( ... reference_filters={'condition': 'Control'}, ... source='embeddings', ... mode='centroid', ... distance_type='euclidean', ... ) >>> distances = dist_df['distance'] ```

Programmatically update the metadata filters and exclusions and recompute. Since filters/exclude change the data subset, this triggers a full recomputation.

Cancel pending debounce timers (call before discarding the explorer).

Display the interactive dashboard. Only one widget tree stays visible. Jupyter clears the cell output before the new dashboard is displayed.

Return the unique ID from metadata. **Raises:** - **`KeyError`**: If the unique ID column is missing from meta.

Inject or return unique ID in meta. Auto-generate if not present. **Parameters:** - **`meta`** (`dict`): Metadata dict (modified in place). - **`paths`** (`str or list of str`): File path(s) for this sample. - **`data_dir`** (`str, optional`): Base directory for ID generation. Uses instance data_dir if None. **Returns:** - **`str`**: The unique ID (existing or newly generated).

Build stable key for checkpoint matching. Uses the unique ID when present and non-path-like; otherwise delegates to metadata_to_stable_key for a JSON-based stable key.

Extract metadata for a path. Guarantees 'id' is present. **Parameters:** - **`path`** (`str`): File path. - **`data_dir`** (`str, optional`): Base directory for ID generation. **Returns:** - **`dict`**: Metadata with file_path, metadata keys, and guaranteed id.

Resolve mask path from metadata when mask_filename_column and mask_dir are set. **Returns:** - **`str or None`**: Full path to mask file, or None if not resolvable.

Root directory for mask files.

Column containing mask filename when using explicit mask lookup.

Column used for multi-channel grouping in find_files.

Return a sample-level capture group, never ``channel_index``.

Extract metadata with data_dir-aware partial path lookup.

Extract per-image embeddings. Delegates to _get_embeddings (heritage class) and validates output shape (B, D). Even for non-PyTorch models, this method ensures the final output is a PyTorch tensor compatible with the rest of the pipeline. **Args:** - **`tensor`**: Input tensor (B, C, H, W) on device. **Returns:** Embedding tensor (B, D).

Move the wrapped model to *device* and set eval mode when supported. Called from ``__init__`` and again by ``EmbeddingExtractor`` before each batch so the model stays on the pipeline device after user changes. **Args:** - **`device`**: Target device. If None, uses the wrapper's current ``self.device``. **Raises:** - **`RuntimeError`**: If ``model.to(device)`` or ``model.eval()`` fails.

Extract per-image embeddings. Delegates to _get_embeddings (heritage class) and validates output shape (B, D). Even for non-PyTorch models, this method ensures the final output is a PyTorch tensor compatible with the rest of the pipeline. **Args:** - **`tensor`**: Input tensor (B, C, H, W) on device. **Returns:** Embedding tensor (B, D).

Move the wrapped model to *device* and set eval mode when supported. Called from ``__init__`` and again by ``EmbeddingExtractor`` before each batch so the model stays on the pipeline device after user changes. **Args:** - **`device`**: Target device. If None, uses the wrapper's current ``self.device``. **Raises:** - **`RuntimeError`**: If ``model.to(device)`` or ``model.eval()`` fails.

Add a batch of embeddings to the in-memory buffer. **Parameters:** - **`embeddings`** (`np.ndarray`): Shape ``(B, D)`` float32. - **`img_paths`** (`list`): Length B. Each element is a ``str`` (single-channel) or a ``List[str]`` (multi-channel). - **`metadata_dicts`** (`list of dict`): Length B.

Buffer per-row internal dicts; must match the next :meth:`buffer_properties` batch size.

Buffer paths and metadata without embeddings (property-only run).

Add property dicts to the in-memory buffer.

Truncate all property datasets to 0 and reset n_committed_props.

Create a full copy of the current checkpoint at *new_path* and return a new manager. If the current manager is lazy, the file is copied on disk. If it is in RAM, the RAM data is written to the new path. **Parameters:** - **`new_path`** (`str`): Path for the new HDF5 file. **Returns:** - **`CheckpointManager`**: A new manager instance pointing to the cloned file.

Flush any remaining buffers and close the HDF5 file.

Flush embedding + path + metadata buffer to HDF5 atomically. **Returns:** - **`int`**: Total n_committed after this commit.

Internal rows are written by :meth:`commit_properties` together with properties. Returns current ``n_committed_props`` (internal uses the same row count).

Flush property buffer to HDF5 atomically. **Returns:** - **`int`**: Total n_committed_props after this commit.

Detect whether *path* is an HDF5 file. Returns ``'hdf5'`` or ``'unknown'``.

Embedding width *D* from file attrs, or ``None`` when not set.

Number of paths/embeddings currently in the write buffer.

Return committed metadata dicts in order.

Return the set of committed primary ``img_path`` values (resolved to absolute when possible).

Return committed primary paths in order (resolved to absolute when storage root available).

Return processing parameters from ``/config`` group, or None.

Return the stored processing-pipeline steps, or None if absent. Stored as ``/processing_pipeline`` (embeddings and properties step lists).

Return True if this file stores real property columns or values. ``n_committed_props > 0`` alone is insufficient: writing a list of empty ``{}`` rows can set that counter and create an empty ``/properties`` group.

Whether the checkpoint stores separate paths per channel.

Load ``/batch_correction`` if present; otherwise ``None``.

Load only the committed properties (avoids loading embeddings/paths). Used when merging partial property results.

Read all committed data and return a [PhenoMeResults](/PhenoMe/advanced/api/pipeline/#class-phenomeresults). Paths are resolved to absolute when _storage_root is available. **Returns:** - **`PhenoMeResults`**: Embeddings are loaded eagerly into the ``embeddings`` field. Returns an empty ``PhenoMeResults`` when no data is committed.

Load only the specified rows of the embedding dataset. Sorts indices before reading for chunk efficiency, then restores original order. **Parameters:** - **`indices`** (`np.ndarray`): 1-D integer array of row indices (0-based, within committed range). **Returns:** - **`np.ndarray`**: Shape ``(len(indices), D)`` float32.

Load and validate an HDF5 results/checkpoint file. Paths are resolved to absolute when _storage_root is stored in the file. **Parameters:** - **`path`** (`str`): Path to the HDF5 file. **Returns:** PhenoMeResults **Raises:** - **`ValueError`**: If the file format is invalid or corrupted.

Load per-row internal checkpoint state (e.g. property-attempt flags). Same length as properties.

Load all committed paths and metadata dicts. Paths are resolved to absolute when _storage_root is available (from file or inference). **Returns:** (paths, metadata_dicts) ``paths`` is a list of ``str`` (single-channel) or ``List[str]`` (multi-channel). ``metadata_dicts`` is a list of dicts.

Load all committed property dicts. **Returns:** list of dict Length n_committed_props. Empty list when no properties stored.

Rows committed for paths + embeddings + metadata.

Rows with internal tracking data; same as ``n_committed_props`` when using /internal.

Rows committed for properties (may lag n_committed).

Number of property dicts in the write buffer.

Names of stored property columns, or empty if none were written.

Set ``n_committed_props`` to 0 when ``/properties`` has no columns. Saving embeddings with a list of empty ``{}`` dicts can stamp ``n_committed_props = n_images`` without writing columns. Later ``commit_properties`` then refuses to append because the counter already equals ``n_committed``.

Persist batch-correction statistics under ``/batch_correction``.

Write processing params to ``/config`` group (creates if needed).

Write ``/processing_pipeline`` as ``step_01``, ``step_02``, … groups. Stores embeddings and property steps when present. A payload that has only one segment keeps the other segment already on disk. Property steps store ``kind=image``, ``kind=mask``, or ``kind=compute``. Removes the group if *payload* is None.

Compare *current_params* against stored params. Returns list of ``(key, stored_value, current_value)`` for mismatches.

In-place write of embedding rows identified by HDF5 row indices. Only rows within the committed range may be written. Flushes the file after updating. For non-lazy checkpoints, updates the in-RAM embedding array instead. **Parameters:** - **`h5_row_indices`** (`np.ndarray`): 1-D int64 indices into ``/embeddings`` (0-based, ``< n_committed``). - **`data`** (`np.ndarray`): ``float32`` array of shape ``(len(h5_row_indices), D)`` aligned with *h5_row_indices*.

Atomically write a full results object to an HDF5 file. Uses a temporary file + ``os.replace`` for crash safety. **Parameters:** - **`results`** (`PhenoMeResults`): Source data (embeddings, img_path, metadata, properties). - **`path`** (`str`): Output file path. compression, compression_level Passed to h5py for the embeddings dataset. - **`processing_params`** (`dict or None`): Written to the ``/config`` group when provided. - **`internal`** (`list of dict, optional`): Per-row internal checkpoint state (e.g. attempted flags), written to ``/internal`` when non-empty. - **`processing_pipeline`** (`dict or None`): Optional processing pipeline written to ``/processing_pipeline`` (ordered title/detail steps for embeddings and properties; property steps store ``kind=image``, ``kind=mask``, or ``kind=compute``).

Inspect image and mask dimensions, shapes, dtypes, and data ranges. Reads each image (and mask when present) in the stored dtype, without the float32 cast used by ``read_image``. Reports shape statistics, image-mask shape matching, and dtype/range. Masks are expected in the ``mask_path`` column of file_df when present. **Args:** - **`file_df`**: pd.DataFrame from find_files(). Must have 'file_path' column. When masks exist, includes 'mask_path' column. **Returns:** pd.DataFrame with columns: file_path, height, width, channels, dtype, min, max, error, shape_match (True/False/None), mask_height, mask_width, mask_dtype, mask_min, mask_max, mask_error.

Build torchvision transform pipeline for image preprocessing. Pipeline steps: ToTensor -> TypeMaxNorm -> [PadToSize] -> [Resize] -> Normalize. When both pad_size and resize_size are given, padding is applied first, then resize (even if resize_size is smaller than pad_size). **Args:** - **`resize_size`**: Target spatial size (H, W) for final resize. If None, no resize. - **`pad_size`**: Minimum size for H and W before resize. If None, no padding. - **`n_channels`**: Number of input channels for Normalize mean/std (default 3). **Returns:** transforms.Compose: Pipeline for (H, W, C) numpy image -> (C, H', W') tensor.

Plot PCA of embeddings, properties, or combined features (Plotly, WebGL by default).

Plot t-SNE of embeddings, properties, or combined features (Plotly, WebGL by default).

Plot UMAP of embeddings, properties, or combined features (Plotly, WebGL by default).

Plot count of images grouped by metadata using Plotly. **Args:** - **`group_by`**: List of metadata keys to group by. - **`return_fig`**: If True, return the Plotly figure instead of displaying it. - **`plot`**: If True (default), create and display a Plotly plot. If False, print results as text.

Plot centroids of groups in reduced embedding space.

Plot a specific image by its index. **Args:** - **`idx`**: Image index in results - **`distance_results`**: Optional distance computation results DataFrame - **`channels`**: specific channels to plot - **`figsize`**: Figure size - **`title_fields`**: Optional list of field names to display in title - **`show_extra_info`**: If True, prints detailed information after plotting - **`apply_transforms`**: If True, apply pipeline image_transforms - **`downsample`**: If not None, approximate desired size (in pixels) for the longest image edge when downsampling. The final image size may differ slightly due to integer stepping. If None, no downsampling. - **`ax`**: Optional matplotlib axes to plot on. If provided, a new figure is not created. - **`return_fig`**: If True, return the displayed image as a NumPy array and do not open a matplotlib figure. Mask overlay, when requested, is composited onto that array. - **`show_mask_overlay`**: If True, draws the segmentation mask as a semi-transparent yellow overlay with a crisp contour. Requires masks to have been discovered via ``mask_dir`` in ``find_files``. Useful for verifying that masks load correctly and spatially align with their images. **Returns:** The image as a NumPy array when ``return_fig`` is True; otherwise ``None``.

Rasterize the same view as ``plot_image_by_index`` to PNG bytes. Used by the interactive explorer with ``ipywidgets.Image`` because matplotlib ``display()`` / ``plt.show()`` from Plotly click callbacks does not reliably target a nested ``Output`` widget (output goes to the cell or nowhere). When ``show_extra_info`` is True, also emits the same details as ``plot_image_by_index`` via logging and returns their text for UI display. **Returns:** ``(png_bytes, details_text_or_none)`` — ``details_text_or_none`` is the formatted extra-info block when ``show_extra_info`` is True and details exist.

Print distance summary statistics without plotting. Safe to use when enable_plots=False. **Args:** - **`distance_results`**: DataFrame containing 'distance' column - **`group_by`**: Metadata key to group by - **`dist_range`**: Tuple of (min, max) distances to include