Launching Local & Cluster Tasks

Manages local and cluster deployment. Ideal for development, testing, executing on local resources, or deployment on a computing cluster environment.

usage: clowdr local [-h] [--verbose] [--dev] [--workdir WORKDIR]
                    [--volumes VOLUMES] [--groupby GROUPBY] [--sweep SWEEP]
                    [--setup] [--cluster {slurm}] [--clusterargs CLUSTERARGS]
                    [--jobname JOBNAME] [--simg SIMG] [--user]
                    [--rerun {all,select,failed,incomplete}] [--run_id RUN_ID]
                    [--task_ids TASK_IDS [TASK_IDS ...]] [--s3 S3] [--bids]
                    descriptor invocation provdir

Positional Arguments

descriptor	Local path to Boutiques descriptor for the tool you wish to run. To learn about descriptors and Boutiques, go to: https://boutiques.github.io.
invocation	Local path to Boutiques invocation (or directory containing multiple invocations) for the analysis you wish to run. To learn about invocations and Boutiques, go to: https://boutiques.github.io.
provdir	Local directory for Clowdr provenance records and other captured metadata to be stored. This directory needs to exist prior to running Clowdr.

Named Arguments

--verbose, -V	Toggles verbose output statements. Default: False
--dev, -d	Launches only the first created task. This is intended for development purposes. Default: False
--workdir, -w	Specifies the working directory to be used by the tasks created.
--volumes, -v	Specifies any volumes to be mounted to the container. This is usually related to the path of any data files as specified in your invocation(s).
--groupby, -g	If you wish to run tasks in batches, specify the number of tasks to group here. For imperfect multiples, the last group will be the remainder.
--sweep	If you wish to perform a parameter sweep with Clowdr, you can use this flag and provide Boutiques parameter ID as the argument here. This requires: 1) the parameter exists in the provided invocation, and 2) that field contains a list of the parameter values to be used (if it is ordinarily a list, this means it must be a list of lists here). This option does not work with directories of invocations, but only single files.
--setup	If you wish to generate metadata but not launch tasks then you can use this mode. Default: False
--cluster, -c	Possible choices: slurm If you wish to submit your local tasks to a scheduler, you must specify it here. Currently this only supports SLURM clusters.
--clusterargs, -a
	This allows users to supply arguments to the cluster, such as specifying RAM or requesting a certain amount of time on CPU. These are provided in the form of key:value pairs, and separated by commas. For example: –clusterargs time:4:00,mem:2048,account:ABC
--jobname, -n	If running on a cluster, and you wish to specify a unique identifier to appear in thesubmitted tasks, you can specify it with this flag.
--simg, -s	If the Boutiques descriptor summarizes a tool wrapped in Singularity, and the image has already been downloaded, this option allows you to specify that image file.
--user, -u	If the Boutiques descriptor summarizes a tool wrapped in Docker, toggles propagating the current user within the container. Default: False
--rerun, -R	Possible choices: all, select, failed, incomplete Allows user to re-run jobs in a previous execution that either failed or didn’t finish, etc. This requires the –run_id argument to also be supplied. Four choices are: ‘all’ to re-run all tasks, ‘select’ to re-run specific tasks, ‘failed’ to re-run tasks which finished with a non-zero exit-code, ‘incomplete’ to re-run tasks which have not yet indicated job completion. While the descriptor and invocations will be adopted from the previous executions, other options such as clusterargs or volume can be set to different values, if they were the source of errors. Pairing the incomplete mode with the –dev flag allows you to walk through your dataset one group at a time.
--run_id	Pairs with –rerun. This ID is the directory within the supplied provdir which contains execution you wish to relaunch. These IDs/directories are in the form: year-month-day_hour-minute-second-8digitID.
--task_ids	Pairs with –rerun. This list of task IDs are the task numbers within the directory supplied with –run_id and provdir. These IDs are integers greater than or equal to 0.
--s3	Amazon S3 bucket and path for remote data. Accepted in the format: s3://{bucket}/{path}
--bids, -b	Indicates that the tool being launched is a BIDS app. BIDS is a data organization format in neuroimaging. For more information about this, go to https://bids.neuroimaging.io. Default: False