cesm_hawc.dispatch¶

source

Run independent jobs serially or across a pool of worker processes.

run_jobs¶

def run_jobs(
    fn: Callable,
    jobs: list[tuple],
    n_workers: int,
    max_tasks_per_child: int | None = None,
    on_result: Callable[[object], None] | None = None,
) -> list

source

Run fn(*job) for every job, serially or in parallel.

Parameters¶

• fn (callable) — Job function. Must be defined at module level so it can be pickled for worker processes.

• jobs (list of tuple) — Positional arguments for each call of fn.

• n_workers (int) — Number of worker processes. 1 or less runs the jobs serially in this process, which is easiest to debug; otherwise up to min(n_workers, len(jobs)) processes are used.

• max_tasks_per_child (int, optional) — Replace each worker process after this many jobs, which limits memory growth from state not released between jobs. Use 1 for jobs known to leak memory. Default: workers are never replaced.

• on_result (callable, optional) — Called with each result as it completes, e.g. for progress logging.

Returns¶

• list — The result of every job. In parallel runs they are in completion order, not job order.

Raises¶

• concurrent.futures.process.BrokenProcessPool — If a worker process is killed abruptly (e.g. out of memory). The whole pool is lost, including jobs still in progress; output already written by finished jobs is unaffected, so resumable job functions can continue when the same job list is re-run.

Notes¶

cesm-hawc’s job functions return a status string starting with "OK" or "FAIL", but this function doesn’t depend on the result type.