File Formats

MEDYAN.jl uses Zarr v2 zip store for saving snapshots.

Other Languages

To open the snapshots in python use zarr

trajectory output directory

Trajectory outputs and logs are stored in a directory.

The output directory contains a traj sub directory with a header.json file, many $(most significant digits)/$(3 least significant digits).zip files, and finally a footer.json.

The initial state returned by setup is stored in traj/0/000.zip.

Inside the traj/$(i)/$(j).zip there is a snap/medyan group containing the snapshot of the MEDYAN.Context after the step. Other state may be stored in child groups of snap.

The header.json file also has a top level "medyan" key with a value describing the system being simulated.

Other header information may be under other top level keys.

See https://github.com/medyan-dev/MEDYANSimRunner.jl for more details on the output directory structure.

"medyan" Header JSON object.

header.json["medyan"] contains static metadata about the simulation.

Example header.json "medyan" value

using MEDYAN
import JSON3
cinit, s = MEDYAN.example_all_sites_context()
JSON3.pretty(MEDYAN.header(cinit); allow_inf = true)

{
    "version": "0.13.0",
    "medyanInfo": {
        "title": "MEDYAN.jl",
        "version": "0.2.1",
        "sourceCodeUrl": "https://github.com/medyan-dev/MEDYAN.jl"
    },
    "size": {
        "x(nm)": 2000,
        "y(nm)": 500,
        "z(nm)": 500
    },
    "chem_grid_size": {
        "nx": 4,
        "ny": 1,
        "nz": 1,
        "voxel_x(nm)": 500,
        "voxel_y(nm)": 500,
        "voxel_z(nm)": 500
    },
    "diffusing_species": [
        {
            "name": "b"
        },
        {
            "name": "c"
        }
    ],
    "membrane_diffusing_species": [
        {
            "name": "ma"
        },
        {
            "name": "mb"
        },
        {
            "name": "mc"
        }
    ],
    "fixed_species": [
        {
            "name": "d"
        },
        {
            "name": "a"
        }
    ],
    "fila": [
        {
            "name": "a",
            "radius(nm)": 3,
            "monomerstates": [
                "me",
                "a",
                "b",
                "c",
                "pe"
            ]
        },
        {
            "name": "b",
            "radius(nm)": 3,
            "monomerstates": [
                "me",
                "a",
                "b",
                "c",
                "pe"
            ]
        }
    ],
    "links": [
        {
            "name": "fila_tip_restraint",
            "places": [
                "fila_tip"
            ]
        },
        {
            "name": "fila_mono_restraint",
            "places": [
                "fila_mono"
            ]
        },
        {
            "name": "fila_mono_dummy",
            "places": [
                "fila_mono"
            ]
        },
        {
            "name": "fila_mono_2bonds",
            "places": [
                "fila_mono"
            ]
        },
        {
            "name": "fila_mono_distance_bond",
            "places": [
                "fila_mono",
                "fila_mono"
            ]
        },
        {
            "name": "a",
            "places": [
                "fila_mono",
                "fila_mono"
            ]
        },
        {
            "name": "b",
            "places": [
                "fila_mono",
                "fila_mono"
            ]
        },
        {
            "name": "c",
            "places": [
                "fila_mono",
                "fila_mono"
            ]
        },
        {
            "name": "memb_vert_restraint",
            "places": [
                "memb_vert"
            ]
        }
    ]
}

Snapshot group

A snapshot describes the state of a Context at a single point in time.

A snapshot can be used to set the state of a Context if it was constructed with the same system and parameters as the Context used to create the snapshot. Trying to load a snapshot into a Context constructed with a different system or parameters may lead to unexpected results.

By default units are in nm, pN, and s. Coordinates are relative to the center of the grid.

The snapshot doesn’t hold the exact full state of a Context, simulations restarted from a snapshot may not have the exact same results because:

1. The snapshot doesn’t store the state of the random number generator.
2. Coordinates may be rounded to save disk space.
3. Dictionaries and other internal data structures may get reordered and or resorted when the snapshot is loaded.
4. Multithreading may be non deterministic.

But if rounding isn’t too extreme, the reloaded Context should have the same statistics.

Snapshot Versioning

The snapshot format is versioned by the “version” attribute. The current snapshot version is:

using MEDYAN
MEDYAN.SNAPSHOT_VERSION

v"0.13.0"

Currently before snapshot version 1.0.0 anything goes.

After snapshot version 1.0.0 is released.

Snapshots written with a previous snapshot version above v"1" should be readable. Snapshots written with a newer snapshot version are generally not readable.

If new agent types are added to MEDYAN.jl, usually only the minor version needs to be updated, as nothing special needs to be done to read older snapshot versions without that added agent type.

If new data is added to an existing agent type, also usually only the minor version needs to be increment, though the case of the new data not existing must be handled with some default.

If the way an existing agent type is stored significantly changes, such that external code analyzing the snapshot would need to be modified, the major version must be incremented. Also if possible there should be a function to update a snapshot from the old version to the new version.

“#experimental” and “#comment” prefixes

Any group, dataset, or attribute name prefixed with with “#” can change format or be removed without changing the snapshot version.

“#experimental” is used for saving new types of agents or other data that doesn’t have a stable format yet.

“#comment” is used for saving human readable comments that could change in format or wording.

Example snapshot

using MEDYAN
using SmallZarrGroups
cinit, s = MEDYAN.example_all_sites_context()
group = MEDYAN.snapshot(cinit)

Snapshot 📂

🏷️

time (s)

attrs(group)["time (s)"]

Default: No change
See MEDYAN.set_time!

version

attrs(group)["version"]

Default: No change
See Snapshot Versioning

uuid

attrs(group)["uuid"]

Must be set to exactly “37eee81f-88ae-4d11-b6b3-d38e1ccf0a08”
to be considered a valid MEDYAN snapshot.

🔢

diffusingcounts

collect(group["diffusingcounts"])

Default: Empty

Indexed by [species id, compartment id] to give count.

See MEDYAN.chem_adddiffusingcount!

fixedcounts

collect(group["fixedcounts"])

Default: Empty

Indexed by [species id, compartment id] to give count.

See MEDYAN.chem_addfixedcount!

📂

chemboundary

There is a dataset for each type of boundary, the total boundary is an intersection of the following.

See MEDYAN.Boundary

🔢 capsules

collect(group["chemboundary/capsules"])

Default: Empty

Each column of the dataset is a capsule.

🔢 planes

collect(group["chemboundary/planes"])

Default: Empty

Each column of the dataset is a plane.

fila

🏷️

version

attrs(group["fila"])["version"]

fila data major version, version 1 is described here.

position_scale

attrs(group["fila"])["position_scale"]

Default: No effect

Filament positions were rounded to the nearest 2^-position_scale nm when saved.

There is a subgroup for each filament type with at least one filament. The subgroups are named by their filament typeid. For example:

📂 1

🔢 load

collect(group["fila/1/load"])

Filament end load forces (pN).

Each row is the load force on the minus and plus ends of a filament.

🔢 newm

collect(group["fila/1/newm"])

Number of newly added monomers to the filament minus ends since last minimization.

🔢 newp

collect(group["fila/1/newp"])

Number of newly added monomers to the filament plus ends since last minimization.

🔢 clen

collect(group["fila/1/clen"])

Number of cylinders per filament.

🔢 mlen

collect(group["fila/1/mlen"])

Number of monomers per filament.

🔢 nm

collect(group["fila/1/nm"])

The monomer ids at the minus ends of the cylinders.

                                   |
                        -----+-----|-----+-----
    minus end <----       M  |  M  | (M) |  M        ----> plus end
                        -----+-----|-----+-----
                                   |
                                   ^ A nodeposition is indicated by the line.

The monomer id with parenthesis (M) will in node_mids

🔢 np

collect(group["fila/1/np"])

Each row is a node position in nm.

🔢 ms

collect(group["fila/1/ms"])

Monomer states.

links

🏷️

version

attrs(group["links"])["version"]

link data major version, version 1 is described here.

There is a subgroup for each link type with at least one current or past link. The subgroups are named by their link typeid. For example:

📂 8

🏷️

num_links

attrs(group["links/8"])["num_links"]

Number of links of this type.

next_lid

attrs(group["links/8"])["next_lid"]

The default next link id for new links.

This should be greater than all the existing link ids of this type.

🔢 be

collect(group["links/8/be"])

The bonds that are enabled.

Each row of this matrix represents a link, each column represents a bond.

🔢 ids

collect(group["links/8/ids"])

The link ids.

🔢 im

collect(group["links/8/im"])

Is the link minimized. A link is marked as not minimized when created, and then marked as minimized after running mechanics.

🔢 re

collect(group["links/8/re"])

The reactions that are enabled.

Each row of this matrix represents a link, each column represents a reaction.

🔢 tags

collect(group["links/8/tags"])

The tags of the places the links are attached to.

Each row of this matrix represents a link.

Each place has two adjacent columns. First a index into the tags, and next a tag generation. The place type can be found in the header file. If the place is not attached, the generation and index will be zero.

📂 bs

Default: default bond states

See MEDYAN.update_link!.

The state of link bonds, organized in a nested struct of vector like form. Any static arrays will be unwrapped into a tuple of vectors, in column major order. The “name” attribute of each subgroup and dataset is the corresponding property name in the StructArray Any property in the default state that isn’t in the snapshot will stay at its default value.

group["links/8/bs"]

📂
└─ 📂 1 🏷️ name => "1",
   ├─ 🔢 1: 4 Float64  🏷️ name => "k",
   └─ 🔢 2: 4 Float64  🏷️ name => "L0",

📂 s

Default: default states

See MEDYAN.update_link!.

The state of link, organized in a nested struct of vector like form. Any static arrays will be unwrapped into a tuple of vectors, in column major order. The “name” attribute of each subgroup and dataset is the corresponding property name in the StructArray Any property in the default state that isn’t in the snapshot will stay at its default value.

group["links/8/s"]

📂
├─ 🔢 1: 4 Int64  🏷️ name => "a",
└─ 🔢 2: 4 Float64  🏷️ name => "b",

mechboundary

There is a dataset for each type of boundary, the total boundary is an intersection of the following.

See MEDYAN.Boundary

🔢 capsules

collect(group["mechboundary/capsules"])

Default: Empty

Each column of the dataset is a capsule.

🔢 planes

collect(group["mechboundary/planes"])

Default: Empty

Each column of the dataset is a plane.

memb

🏷️

version

attrs(group["memb"])["version"]

memb data major version, version 1 is described here.

num_membranes

attrs(group["memb"])["num_membranes"]

Total number of membranes.

position_scale

attrs(group["memb"])["position_scale"]

Default: No effect

Positions were rounded to the nearest 2^-position_scale nm when saved.

There is a subgroup for each membrane. The subgroups are named by the membrane index. For example:

📂 1

🏷️

typeid

attrs(group["memb/1"])["typeid"]

This membrane’s type id.

🔢 trilist

collect(group["memb/1/trilist"])

Each column is the 3 vertex indexes of a triangle. Indexes are one based, and follow the right hand rule. Looking at the triangle from the outside in, they have counterclockwise winding.

🔢 vertlist

collect(group["memb/1/vertlist"])

Each column is a vertex coordinate in nm.

optional 🔢 copynumbers

collect(group["memb/1/copynumbers"])

Array of vertex membrane species copynumbers. If the membrane has no defined species this dataset will not exist.

Tags

There is a subgroup for each tag type with at least one current or past tag. Each tag type stores tag generations as a vector of UInt32 in g and packed places in p_n where n is the packed format version. If the generation is odd the tag is valid. otherwise the tag is an empty slot à la https://docs.rs/slotmap/latest/slotmap/.

📂 fila_mono

🔢 p_1

collect(group["tags/fila_mono/p_1"])

The filament monomer places for each tag stored as a vector of UInt64. Not all monomers will have tags. For tag i of n total tags, p_1[i]>>>32 is the filament type id, (p_1[i]%UInt32)>>1 is the filament index, and p_1[i+n] is the zigzag encoded monomer id.

📂 fila_tip

🔢 p_1

collect(group["tags/fila_tip/p_1"])

The filament tip places for each tag stored as a vector of UInt64. All filament tips will have tags. For tag i of n total tags, p_1[i]>>>32 is the filament type id, (p_1[i]%UInt32)>>1 is the filament index, and p_1[i] is odd for the minus end of the filament and even for the plus end.