group_resource_usage#

Workflow for grouping resource usage.

Working location structure:

(name)
├── data
│   └── data.(category)
│       └── (name)_(key)_(seed).(category).tar.xz
├── groups
│   └── groups.RESOURCE_USAGE
│       ├── (name).object_storage.csv
│       └── (name).wall_clock.csv
└── logs
    └── (job_id).log

Different groups use inputs from data and logs. Grouped data are saved to groups.RESOURCE_USAGE.

Flows

`run_flow`	Main group resource usage flow.
`run_flow_group_object_storage`	Group resource usage subflow for object storage size.
`run_flow_group_wall_clock`	Group resource usage subflow for wall clock time.

run_flow(context: ContextConfig, series: SeriesConfig, parameters: ParametersConfig) → None[source]#

Main group resource usage flow.

Calls the following subflows, if the group is specified:

run_flow_group_object_storage()
run_flow_group_wall_clock()

run_flow_group_object_storage(context: ContextConfig, series: SeriesConfig, parameters: ParametersConfigObjectStorage) → None[source]#: Group resource usage subflow for object storage size.

run_flow_group_wall_clock(context: ContextConfig, series: SeriesConfig, parameters: ParametersConfigWallClock) → None[source]#: Group resource usage subflow for wall clock time.

Configs

`ContextConfig`	Context configuration for group resource usage flow.
`ParametersConfig`	Parameter configuration for group resource usage flow.
`ParametersConfigObjectStorage`	Parameter configuration for group resource usage subflow - object storage.
`ParametersConfigWallClock`	Parameter configuration for group resource usage subflow - wall clock.
`SeriesConfig`	Series configuration for group resource usage flow.

class ContextConfig[source]#

Context configuration for group resource usage flow.

working_location: str#: Location for input and output files (local path or S3 bucket).

class ParametersConfig[source]#

Parameter configuration for group resource usage flow.

groups: list[str]#: List of resource usages groups.

object_storage: ParametersConfigObjectStorage = ParametersConfigObjectStorage(search_locations=[], categories=['CELLS', 'LOCATIONS'], pattern='[_]*([A-z0-9\\s\\_]*)_([0-9]{4})\\.')#: Parameters for group object storage subflow.

wall_clock: ParametersConfigWallClock = ParametersConfigWallClock(search_locations=[], pattern='simulation \\[ ([A-z0-9\\s\\_]+) \\| ([0-9]{4}) \\] finished in ([0-9\\.]+) minutes', exceptions=[])#: Parameters for group wall clock subflow.

class ParametersConfigObjectStorage[source]#

Parameter configuration for group resource usage subflow - object storage.

search_locations: list[str]#: List of locations (local paths or S3 buckets) to search for files.

categories: list[str]#: List of object storage categories.

pattern: str = '[_]*([A-z0-9\\s\\_]*)_([0-9]{4})\\.'#: Pattern to match for object key and seed.

class ParametersConfigWallClock[source]#

Parameter configuration for group resource usage subflow - wall clock.

search_locations: list[str]#: List of locations (local paths or S3 buckets) to search for files.

pattern: str = 'simulation \\[ ([A-z0-9\\s\\_]+) \\| ([0-9]{4}) \\] finished in ([0-9\\.]+) minutes'#: Pattern to match for object key, seed, and time.

exceptions: list[str]#: List of exception strings used to filter log files.

class SeriesConfig[source]#

Series configuration for group resource usage flow.

name: str#: Name of the simulation series.