pymc.backends.zarr.ZarrTrace#

class pymc.backends.zarr.ZarrTrace(store=None, compressors=UNSET, draws_per_chunk=100, include_transformed=False)[source]#

Object that stores and enables access to MCMC draws stored in zarr groups.

This class creates a zarr hierarchy to represent the sampling information which is intended to mimic xarray.DataTree. The hierarchy looks like this:

root
|–> constant_data
|–> observed_data
|–> posterior
|–> unconstrained_posterior
|–> sample_stats
|–> warmup_posterior
|–> warmup_unconstrained_posterior
|–> warmup_sample_stats
|–> _sampling_state

The root group is created when the ZarrTrace object is initialized. The rest of the groups are created once init_trace() is called with a few exceptions: unconstrained_posterior is only created if include_transformed = True, and the groups prefixed with warmup_ are created only after calling split_warmup_groups().

Since ZarrTrace objects are intended to be as close to xarray.DataTree objects as possible, the groups store the dimension and coordinate information following the xarray zarr v3 encoding specification. Arrays store their dimensions in the zarr v3 dimension_names metadata field.

Parameters:
storezarr.abc.store.Store | collections.abc.MutableMapping | None

The store object where the zarr groups and arrays will be stored and read from. Any zarr compatible storage object works. Keep in mind that if None is provided, a zarr.storage.MemoryStore will be used, which means that information won’t be visible to other processes and won’t persist after the ZarrTrace life-cycle ends. If you want to have persistent storage, please use one of the multiple disk backed zarr storage options, e.g. LocalStore or ZipStore. Note that ZipStore must be created with mode="w" to be writable, and that its contents are only persisted once .close() is called on it.

compressorsSequence | None | pymc.util.UNSET

The compressors to use for the underlying zarr arrays. If None, no compressor is used. If UNSET, a default zarr ZstdCodec is used.

draws_per_chunkint

The number of draws that make up a chunk in the variable’s posterior array. Each variable’s array shape is set to (n_chains, n_draws, *rv_shape), but the chunks are set to (1, draws_per_chunk, *rv_shape). This means that each chain will have it’s own chunk to read or write to, allowing for concurrent write operations of different chains not to interfere with each other, and that multiple draws can belong to the same chunk. The variable’s core dimension however, will never be split across different chunks. The default of 100 amortizes the cost of zarr writes; setting it to 1 gives the highest crash resilience, at the cost of a large slowdown.

include_transformedbool

If True, the transformed, unconstrained value variables are included in the storage group.

See also

ZarrChain

Notes

ZarrTrace objects represent the storage information. If the underlying store persists on disk or over the network (e.g. with a zarr.storage.LocalStore pointing to a cloud bucket) multiple processes will be able to concurrently access the same storage and read or write to it.

The intended division of labour is for ZarrTrace to handle the creation and management of the zarr group and storage objects and arrays, and for individual ZarrChain objects to handle recording MCMC samples to the trace. This division was chosen to stay close to the existing pymc.backends.base.MultiTrace and pymc.backends.ndarray.NDArray way of working with the existing samplers.

One extra feature of ZarrTrace is that it enables direct access to any array’s metadata. ZarrTrace takes advantage of this to tag arrays as deterministic or freeRV depending on what kind of variable they were in the defining model.

Methods

ZarrTrace.__init__([store, compressors, ...])

ZarrTrace.create_group(name, data_dict)

ZarrTrace.groups()

ZarrTrace.init_group_with_empty(group, ...)

ZarrTrace.init_sampling_state_group(tune, chains)

ZarrTrace.init_trace(chains, draws, tune, step)

Initialize the trace groups and arrays.

ZarrTrace.split_warmup(group_name[, ...])

Split the arrays of a group into the warmup and regular groups.

ZarrTrace.split_warmup_groups()

Split the warmup and standard groups.

ZarrTrace.to_datatree([save_warmup, eager])

Convert ZarrTrace to DataTree.

ZarrTrace.to_inferencedata([save_warmup, eager])

Convert ZarrTrace to DataTree with arviz attributes.

Attributes

constant_data

observed_data

posterior

sample_stats

sampling_time

tuning_steps

unconstrained_posterior

warnings

Reconstruct the SamplerWarning objects of each chain.

straces