File Formats

MEDYAN.jl uses Zarr v2 zip store for saving snapshots.

NoteOther Languages

To open the snapshots in python use zarr

trajectory output directory

Trajectory outputs and logs are stored in a directory.

The output directory contains a traj sub directory with a header.json file, many $(most significant digits)/$(3 least significant digits).zip files, and finally a footer.json.

The initial state returned by setup is stored in traj/0/000.zip.

Inside the traj/$(i)/$(j).zip there is a snap/medyan group containing the snapshot of the MEDYAN.Context after the step. Other state may be stored in child groups of snap.

The header.json file also has a top level "medyan" key with a value describing the system being simulated.

Other header information may be under other top level keys.

See https://github.com/medyan-dev/MEDYANSimRunner.jl for more details on the output directory structure.

"medyan" Header JSON object.

header.json["medyan"] contains static metadata about the simulation.

Example header.json "medyan" value

using MEDYAN
import JSON3
cinit, s = MEDYAN.example_all_sites_context()
JSON3.pretty(MEDYAN.header(cinit); allow_inf = true)

{
    "version": "0.14.0",
    "medyanInfo": {
        "title": "MEDYAN.jl",
        "version": "0.6.0-dev",
        "sourceCodeUrl": "https://github.com/medyan-dev/MEDYAN.jl"
    },
    "size": {
        "x(nm)": 2000,
        "y(nm)": 500,
        "z(nm)": 500
    },
    "chem_grid_size": {
        "nx": 4,
        "ny": 1,
        "nz": 1,
        "voxel_x(nm)": 500,
        "voxel_y(nm)": 500,
        "voxel_z(nm)": 500
    },
    "diffusing_species": [
        {
            "name": "b"
        },
        {
            "name": "c"
        }
    ],
    "membrane_diffusing_species": [
        {
            "name": "ma"
        },
        {
            "name": "mb"
        },
        {
            "name": "mc"
        }
    ],
    "fixed_species": [
        {
            "name": "d"
        },
        {
            "name": "a"
        }
    ],
    "fila": [
        {
            "name": "a",
            "radius(nm)": 3,
            "monomerstates": [
                "me",
                "a",
                "b",
                "c",
                "pe"
            ],
            "numpercylinder": 40
        },
        {
            "name": "b",
            "radius(nm)": 3,
            "monomerstates": [
                "me",
                "a",
                "b",
                "c",
                "pe"
            ],
            "numpercylinder": 40,
            "twist_per_monomer": -2.8999316802367323
        }
    ],
    "links": [
        {
            "name": "fila_tip_restraint",
            "places": [
                "fila_tip"
            ]
        },
        {
            "name": "fila_mono_restraint",
            "places": [
                "fila_mono"
            ]
        },
        {
            "name": "fila_mono_dummy",
            "places": [
                "fila_mono"
            ]
        },
        {
            "name": "fila_mono_2bonds",
            "places": [
                "fila_mono"
            ]
        },
        {
            "name": "fila_mono_distance_bond",
            "places": [
                "fila_mono",
                "fila_mono"
            ]
        },
        {
            "name": "a",
            "places": [
                "fila_mono",
                "fila_mono"
            ]
        },
        {
            "name": "b",
            "places": [
                "fila_mono",
                "fila_mono"
            ]
        },
        {
            "name": "c",
            "places": [
                "fila_mono",
                "fila_mono"
            ]
        },
        {
            "name": "memb_vert_restraint",
            "places": [
                "memb_vert"
            ]
        },
        {
            "name": "fila_mono_anchor_distance",
            "places": [
                "fila_mono",
                "anchor"
            ]
        },
        {
            "name": "fila_mono_ball_distance",
            "places": [
                "fila_mono",
                "ball"
            ]
        }
    ]
}

Snapshot group

A snapshot describes the state of a Context at a single point in time.

A snapshot can be used to set the state of a Context if it was constructed with the same system and parameters as the Context used to create the snapshot. Trying to load a snapshot into a Context constructed with a different system or parameters may lead to unexpected results.

By default units are in nm, pN, and s. Coordinates are relative to the center of the grid.

The snapshot doesn’t hold the exact full state of a Context, simulations restarted from a snapshot may not have the exact same results because:

1. The snapshot doesn’t store the state of the random number generator.
2. Coordinates may be rounded to save disk space.
3. Dictionaries and other internal data structures may get reordered and or resorted when the snapshot is loaded.
4. Multithreading may be non deterministic.

But if rounding isn’t too extreme, the reloaded Context should have the same statistics.

Snapshot Versioning

The snapshot format is versioned by the “version” attribute. The current snapshot version is:

using MEDYAN
MEDYAN.SNAPSHOT_VERSION

v"0.14.0"

Currently before snapshot version 1.0.0 anything goes.

After snapshot version 1.0.0 is released.

Snapshots written with a previous snapshot version above v"1" should be readable. Snapshots written with a newer snapshot version are generally not readable.

If new agent types are added to MEDYAN.jl, usually only the minor version needs to be updated, as nothing special needs to be done to read older snapshot versions without that added agent type.

If new data is added to an existing agent type, also usually only the minor version needs to be increment, though the case of the new data not existing must be handled with some default.

If the way an existing agent type is stored significantly changes, such that external code analyzing the snapshot would need to be modified, the major version must be incremented. Also if possible there should be a function to update a snapshot from the old version to the new version.

“#experimental” and “#comment” prefixes

Any group, dataset, or attribute name prefixed with with “#” can change format or be removed without changing the snapshot version.

“#experimental” is used for saving new types of agents or other data that doesn’t have a stable format yet.

“#comment” is used for saving human readable comments that could change in format or wording.

Example snapshot

using MEDYAN
using SmallZarrGroups
cinit, s = MEDYAN.example_all_sites_context()
group = MEDYAN.snapshot(cinit)

NoteSnapshot 📂

Note🏷️

time (s)

attrs(group)["time (s)"]

Default: No change
See MEDYAN.set_time! MEDYAN.get_time

version

attrs(group)["version"]

Default: No change
See Snapshot Versioning

uuid

attrs(group)["uuid"]

Must be set to exactly “37eee81f-88ae-4d11-b6b3-d38e1ccf0a08”
to be considered a valid MEDYAN snapshot.

Note📂

Notechem

Note🏷️

version

attrs(group["chem"])["version"]

chem data major version, version 1 is described here.

🔢 dc

collect(group["chem/dc"])

Default: Empty

A 4D Array of Int32 indexed by [species id, cartesian index] to give count.

See MEDYAN.add_diffusing_count!

🔢 fc

collect(group["chem/fc"])

Default: Empty

A 4D array of Int64 indexed by [species id, cartesian index] to give count. The count is a fixed point number Q31f32 reinterpreted as an integer. To get the count, divide the integers stored by 2^32.

See MEDYAN.add_fixed_count!

🔢 a

collect(group["chem/a"])

Default: Full areas

A 4D array indexed by (side, cartesian index) to get the area of a minus side of a chem voxel. side 1 is x, 2 is y, and 3 is z. These can be recalculated based on the chem boundaries and membranes by MEDYAN.apply_chem_boundaries!

🔢 v

collect(group["chem/v"])

Default: Full volumes

A 3D array indexed by a cartesian index to get the volume of a chem voxel. voxels can have zero volume if they were outside the boundaries when MEDYAN.apply_chem_boundaries! was called. These can be recalculated based on the chem boundaries and membranes by MEDYAN.apply_chem_boundaries!

There is a dataset for each type of boundary, the total boundary is an intersection of the following.

See MEDYAN.Boundary

🔢 ca

collect(group["chem/ca"])

Default: Empty

Each column of the dataset is a capsule.

🔢 pl

collect(group["chem/pl"])

Default: Empty

Each column of the dataset is a plane.

Noteball

Note🏷️

version

attrs(group["ball"])["version"]

ball data major version, version 1 is described here.

position_scale

attrs(group["ball"])["position_scale"]

Default: No effect

Ball positions were rounded to the nearest 2^-position_scale nm when saved.

🔢 p

collect(group["ball/p"])

Each row is a ball position in nm.

🔢 r

collect(group["ball/r"])

Ball radius in nm.

🔢 k

collect(group["ball/k"])

Ball stiffness in pN/nm.

🔢 s

collect(group["ball/s"])

Each row is a ball state integers.

🔢 im

collect(group["ball/im"])

Boolean vector indicating whether each ball is marked as minimized. true means the ball’s position has been updated by minimize_energy! since it was last created or modified. May be absent in older snapshots (defaults to false for all balls when loading).

Notefila

Note🏷️

version

attrs(group["fila"])["version"]

fila data major version, version 1 is described here.

position_scale

attrs(group["fila"])["position_scale"]

Default: No effect

Filament positions were rounded to the nearest 2^-position_scale nm when saved.

There is a subgroup for each filament type with at least one filament. The subgroups are named by their filament typeid. For example:

Note📂 1

🔢 load

collect(group["fila/1/load"])

Filament end load forces (pN).

Each row is the load force on the minus and plus ends of a filament.

🔢 newm

collect(group["fila/1/newm"])

Number of newly added monomers to the filament minus ends since last minimization.

🔢 newp

collect(group["fila/1/newp"])

Number of newly added monomers to the filament plus ends since last minimization.

🔢 clen

collect(group["fila/1/clen"])

Number of cylinders per filament.

🔢 mlen

collect(group["fila/1/mlen"])

Number of monomers per filament.

🔢 nm

collect(group["fila/1/nm"])

The monomer ids at the minus ends of the cylinders.

                                   |
                        -----+-----|-----+-----
    minus end <----       M  |  M  | (M) |  M        ----> plus end
                        -----+-----|-----+-----
                                   |
                                   ^ A nodeposition is indicated by the line.

The monomer id with parenthesis (M) will in node_mids

🔢 np

collect(group["fila/1/np"])

Each row is a node position in nm.

🔢 ms

collect(group["fila/1/ms"])

Monomer states.

🔢 mdir

collect(group["fila/2/mdir"])

This array only exists if the filament type is twistable. Each row is the material direction associated with the center of each chem cylinder.

🔢 tsm

collect(group["fila/1/tsm"])

Snapshotted minus-end tip positions (nm).

Each row is the minus-end tip position of a filament, set at the end of minimize_energy! / baoab!. NaN before the first mechanics cycle. Used by fila_tip_cutoff reactions.

May be absent in older snapshots (defaults to NaN when loading).

See fila_tip_snapshot.

🔢 tsp

collect(group["fila/1/tsp"])

Snapshotted plus-end tip positions (nm).

Same semantics as tsm but for plus ends.

See fila_tip_snapshot.

🔢 tom

collect(group["fila/1/tom"])

Minus-end tip extensions in monomer spacings.

Each entry is the tip extension for a filament’s minus end. 0=monomer center (default).

May be absent in older snapshots (defaults to 0.0 when loading).

See fila_tip_extension.

🔢 top

collect(group["fila/1/top"])

Plus-end tip extensions in monomer spacings.

Same semantics as tom but for plus ends.

See fila_tip_extension.

Notelinks

Note🏷️

version

attrs(group["links"])["version"]

link data major version, version 1 is described here.

There is a subgroup for each link type with at least one current or past link. The subgroups are named by their link typeid. For example:

Note📂 8

Note🏷️

num_links

attrs(group["links/8"])["num_links"]

Number of links of this type.

next_lid

attrs(group["links/8"])["next_lid"]

The default next link id for new links.

This should be greater than all the existing link ids of this type.

🔢 be

collect(group["links/8/be"])

The bonds that are enabled.

Each row of this matrix represents a link, each column represents a bond.

🔢 ids

collect(group["links/8/ids"])

The link ids.

🔢 im

collect(group["links/8/im"])

Is the link minimized. A link is marked as not minimized when created, and then marked as minimized after running mechanics.

🔢 re

collect(group["links/8/re"])

The reactions that are enabled.

Each row of this matrix represents a link, each column represents a reaction.

🔢 tags

collect(group["links/8/tags"])

The tags of the places the links are attached to.

Each row of this matrix represents a link.

Each place has two adjacent columns. First a index into the tags, and next a tag generation. The place type can be found in the header file. If the place is not attached, the generation and index will be zero.

📂 bs

Default: default bond states

See MEDYAN.update_link!.

The state of link bonds, organized in a nested struct of vector like form. Any static arrays will be unwrapped into a tuple of vectors, in column major order. The “name” attribute of each subgroup and dataset is the corresponding property name in the StructArray Any property in the default state that isn’t in the snapshot will stay at its default value.

group["links/8/bs"]

📂
└─ 📂 1 🏷️ name => "1",
   ├─ 🔢 1: 4 Float64  🏷️ name => "k",
   └─ 🔢 2: 4 Float64  🏷️ name => "L0",

📂 s

Default: default states

See MEDYAN.update_link!.

The state of link, organized in a nested struct of vector like form. Any static arrays will be unwrapped into a tuple of vectors, in column major order. The “name” attribute of each subgroup and dataset is the corresponding property name in the StructArray Any property in the default state that isn’t in the snapshot will stay at its default value.

group["links/8/s"]

📂
├─ 🔢 1: 4 Int64  🏷️ name => "a",
└─ 🔢 2: 4 Float64  🏷️ name => "b",

Notemechboundary

There is a dataset for each type of boundary, the total boundary is an intersection of the following.

See MEDYAN.Boundary

🔢 capsules

collect(group["mechboundary/capsules"])

Default: Empty

Each column of the dataset is a capsule.

🔢 planes

collect(group["mechboundary/planes"])

Default: Empty

Each column of the dataset is a plane.

Notememb

Note🏷️

version

attrs(group["memb"])["version"]

memb data major version, version 1 is described here.

num_membranes

attrs(group["memb"])["num_membranes"]

Total number of membranes.

position_scale

attrs(group["memb"])["position_scale"]

Default: No effect

Positions were rounded to the nearest 2^-position_scale nm when saved.

There is a subgroup for each membrane. The subgroups are named by the membrane index. For example:

Note📂 1

Note🏷️

typeid

attrs(group["memb/1"])["typeid"]

This membrane’s type id.

🔢 trilist

collect(group["memb/1/trilist"])

Each column is the 3 vertex indexes of a triangle. Indexes are one based, and follow the right hand rule. Looking at the triangle from the outside in, they have counterclockwise winding.

🔢 vertlist

collect(group["memb/1/vertlist"])

Each column is a vertex coordinate in nm.

optional 🔢 copynumbers

collect(group["memb/1/copynumbers"])

Array of vertex membrane species copynumbers. If the membrane has no defined species this dataset will not exist.

NoteTags

There is a subgroup for each tag type with at least one current or past tag. Each tag type stores tag generations as a vector of UInt32 in g and packed places in p_n where n is the packed format version. If the generation is odd the tag is valid. otherwise the tag is an empty slot à la https://docs.rs/slotmap/latest/slotmap/.

Note📂 fila_mono

🔢 p_1

collect(group["tags/fila_mono/p_1"])

The filament monomer places for each tag stored as a vector of UInt64. Not all monomers will have tags. For tag i of n total tags, p_1[i]>>>32 is the filament type id, (p_1[i]%UInt32)>>1 is the filament index, and p_1[i+n] is the zigzag encoded monomer id.

Note📂 fila_tip

🔢 p_1

collect(group["tags/fila_tip/p_1"])

The filament tip places for each tag stored as a vector of UInt64. All filament tips will have tags. For tag i of n total tags, p_1[i]>>>32 is the filament type id, (p_1[i]%UInt32)>>1 is the filament index, and p_1[i] is odd for the minus end of the filament and even for the plus end.