run_fargate_calculations#

Workflow for running containerized calculations using Fargate.

This workflow is used to run registered calculation flows across different simulation conditions and random seeds in parallel. The configurations for the selected calculation are passed into the corresponding flows.

Some calculations can be chunked (in which the calculation is only run on a subset of the cells for the given condition, seed, and tick) in order to further parallelize the flow. For chunked calculations, re-running the flow will merge completed chunks into a single file.

The flow will aim to avoid re-running any existing calculations. Calculations are skipped if the calculation output file already exists, or if the specific chunk already exists. Calculations for additional ticks are appended into the existing calculation output file.

If the submit tasks option is turned off, the flow will print the full pipeline command instead, which can then be run locally. Running commands locally can be useful for conditions that require more CPUs/memory than are available.

Note that this workflow works only if working location is an S3 bucket.

Flows

run_flow

Main run fargate calculations flow.

run_flow(context: ContextConfig, series: SeriesConfig, parameters: ParametersConfig) → None[source]#: Main run fargate calculations flow.

Configs

`Calculation`	Registered calculation types.
`ContextConfig`	Context configuration for run fargate calculations flow.
`ParametersConfig`	Parameter configuration for run fargate calculations flow.
`SeriesConfig`	Series configuration for run fargate calculations flow.

class Calculation[source]#

Registered calculation types.

COEFFICIENTS = ('calculate_coefficients', 'COEFFICIENTS', True)#

NEIGHBORS = ('calculate_neighbors', 'NEIGHBORS', False)#

POSITIONS = ('calculate_positions', 'POSITIONS', False)#

PROPERTIES = ('calculate_properties', 'PROPERTIES', True)#

IMAGE_PROPERTIES = ('calculate_image_properties', 'PROPERTIES', False)#

class ContextConfig[source]#

Context configuration for run fargate calculations flow.

working_location: str#: Location for input and output files (local path or S3 bucket).

account: str#: AWS account number.

region: str#: AWS region.

user: str#: User name prefix.

vcpus: int#: Requested number of vcpus for AWS Fargate task.

memory: int#: Requested memory for AWS Fargate task.

cluster: str#: AWS Fargate cluster name.

security_groups: str#: AWS Fargate security groups, separated by colon.

subnets: str#: AWS Fargate subnets groups, separated by colon.

class ParametersConfig[source]#

Parameter configuration for run fargate calculations flow.

image: str#: Name of pipeline image.

ticks: list[int]#: List of ticks to run flow on.

calculate: Calculation | None = None#: Calculation type.

chunk: int | None = None#: Chunk size, if possible for the given calculation type.

submit_tasks: bool = True#: True to submit calculation tasks, False otherwise.

overrides: dict#: Overrides for the specific calculation type.

class SeriesConfig[source]#

Series configuration for run fargate calculations flow.

name: str#: Name of the simulation series.

seeds: list[int]#: List of series random seeds.

conditions: list[dict]#: List of series condition dictionaries (must include unique condition “key”).