clowdr package


clowdr.driver module, invocation, provdir, s3, cloud, credentials, **kwargs)[source]

Launches a pipeline locally at scale through Clowdr.

descriptor : str
Path to a boutiques descriptor for the tool to be run
invocation : str
Path to a boutiques invocation for the tool and parameters to be run
provdir : str
Path on S3 for storing Clowdr intermediate files and outputs
s3 : str
Path on S3 for accessing input data
cloud : str
Which endpoint to use for deployment
credentials : str
Credentials for Amazon with access to dataloc, clowdrloc, and Batch
**kwargs : dict
Arbitrary keyword arguments (i.e. {‘verbose’: True})
The exit-code returned by the task being executed
clowdr.driver.local(descriptor, invocation, provdir, backoff_time=36000, sweep=[], verbose=False, workdir=None, simg=None, rerun=None, run_id=None, task_ids=[], volumes=[], s3=None, cluster=None, jobname=None, clusterargs=None, dev=False, groupby=1, user=False, setup=False, bids=False, **kwargs)[source]

cluster Launches a pipeline locally through the Clowdr wrappers.

descriptor : str
Path to a boutiques descriptor for the tool to be run.
invocation : str
Path to a boutiques invocation for the tool and parameters to be run.
provdir : str
Path for storing Clowdr intermediate files and output logs.
backoff_time : int (default = 36000)
Maximum delay time before attempting resubmission of jobs that failed to be submitted to a scheduler, in seconds.
sweep : list (default = [])
List of parameters to sweep over in the provided invocations.
verbose : bool (default = False)
Flag toggling verbose output printing
workdir : str (default = None)
Working directory to be used in execution, if different from provdir.
simg : str (default = None)
Path to local copy of Singularity image to be used during execution.
rerun : str (default = None)
One of “all”, “select”, “failed”, and “incomplete,” which enables re-launching tasks from a previous execution either individually or in commonly-desired groups.
run_id : str (default = None)

Required when using rerun, above, this specifies the experiment ID to be re-run. This is the directory created for metadata, of the form:

task_ids : list (default = [])
If re-running with the “select” mode, a list of task IDs within the directory specified by run_id which are to be re-run.
volumes : list (default = [])
List of volume mount-path strings, specified using the standard:
s3 : str (default = None)
Path for accessing input data on an S3 bucket. Must include s3://.
cluster : str (default = None)
Scheduler on the cluster being used. Currently only slurm is supported.
jobname : str (default = None)
Base-name for the jobs as they will appear in the scheduler.
clusterargs : str (default = None)
Comma-separated list of arguments to be provided to the cluster on job submission. Such as: time:4:00,mem:2048,account:ABC
dev : bool (default = False)
Flag to toggle dev mode which only runs the first execution in the set.
groupby : int (default = 1)
Value which dictates the grouping of tasks. Particularly useful when tasks are short or a cluster restricts the number of unique jobs.
user : bool (default = False)
When running with Docker, toggles whether or not the host-user’s UID is used within the container.
setup : bool (default = False)
Flag which prevents execution of tasks after the metadata task and invocation files are generated.
bids : bool (default = False)
Flag toggling BIDS-aware metadata preparation.
**kwargs : dict
Arbitrary additional keyword arguments which may be passed.
The path to the created directory containing Clowdr experiment metadata.

Command-line API wrapper for Clowdr as a CLI, not Python API. For information about the command-line wrapper and arguments it accepts, please try running “clowdr –help”.

args: list
List of all command-line arguments being passed.
The exit-code returned by the driver.
clowdr.driver.runtask(tasklist, **kwargs)[source]
clowdr.driver.share(provdir, prepare=False, host='', port=8050, verbose=False, debug=False, **kwargs)[source]

Launches a simple web server which showcases all runs at the clowdrloc.

provdir : str
Path with Clowdr metdata files (returned from “local” and “deploy”)
**kwargs : dict
Arbitrary keyword arguments (i.e. {‘verbose’: True})


clowdr.server module

clowdr.task module

class clowdr.task.TaskHandler(taskfile, **kwargs)[source]

Bases: object

manageTask(taskfile, provdir=None, verbose=False, **kwargs)[source]
monitor(target, **kwargs)[source]
provLaunch(options, **kwargs)[source]

clowdr.utils module

clowdr.utils.backoff(function, posargs, optargs, backoff_time=36000, verbose=False, **kwargs)[source]
clowdr.utils.get(remote, local, verbose=False, **kwargs)[source]
clowdr.utils.getContainer(savedir, container, simg=None, verbose=False, **kwargs)[source], remote, verbose=False, **kwargs)[source]