clowdr package

Subpackages

• clowdr.controller package
  • Submodules
  • clowdr.controller.launcher module
  • clowdr.controller.metadata module
  • Module contents
• clowdr.endpoint package
  • Submodules
  • clowdr.endpoint.AWS module
  • clowdr.endpoint.remote module
  • Module contents

Submodules

clowdr.driver module

clowdr.driver.cloud(descriptor, invocation, provdir, s3, cloud, credentials, **kwargs)

Launches a pipeline locally at scale through Clowdr.

descriptor : str

Path to a boutiques descriptor for the tool to be run

invocation : str

Path to a boutiques invocation for the tool and parameters to be run

provdir : str

Path on S3 for storing Clowdr intermediate files and outputs

s3 : str

Path on S3 for accessing input data

cloud : str

Which endpoint to use for deployment

credentials : str

Credentials for Amazon with access to dataloc, clowdrloc, and Batch

**kwargs : dict

Arbitrary keyword arguments (i.e. {‘verbose’: True})

int

The exit-code returned by the task being executed

clowdr.driver.local(descriptor, invocation, provdir, backoff_time=36000, sweep=[], verbose=False, workdir=None, simg=None, rerun=None, run_id=None, task_ids=[], volumes=[], s3=None, cluster=None, jobname=None, clusterargs=None, dev=False, groupby=1, user=False, setup=False, bids=False, **kwargs)

cluster Launches a pipeline locally through the Clowdr wrappers.

descriptor : str

Path to a boutiques descriptor for the tool to be run.

invocation : str

Path to a boutiques invocation for the tool and parameters to be run.

provdir : str

Path for storing Clowdr intermediate files and output logs.

backoff_time : int (default = 36000)

Maximum delay time before attempting resubmission of jobs that failed to be submitted to a scheduler, in seconds.

sweep : list (default = [])

List of parameters to sweep over in the provided invocations.

verbose : bool (default = False)

Flag toggling verbose output printing

workdir : str (default = None)

Working directory to be used in execution, if different from provdir.

simg : str (default = None)

Path to local copy of Singularity image to be used during execution.

rerun : str (default = None)

One of “all”, “select”, “failed”, and “incomplete,” which enables re-launching tasks from a previous execution either individually or in commonly-desired groups.

run_id : str (default = None)

Required when using rerun, above, this specifies the experiment ID to be re-run. This is the directory created for metadata, of the form:

year-month-day_hour-minute-second-8digitID.

task_ids : list (default = [])

If re-running with the “select” mode, a list of task IDs within the directory specified by run_id which are to be re-run.

volumes : list (default = [])

List of volume mount-path strings, specified using the standard:

/path/on/host/:/path/in/container/

s3 : str (default = None)

Path for accessing input data on an S3 bucket. Must include s3://.

cluster : str (default = None)

Scheduler on the cluster being used. Currently only slurm is supported.

jobname : str (default = None)

Base-name for the jobs as they will appear in the scheduler.

clusterargs : str (default = None)

Comma-separated list of arguments to be provided to the cluster on job submission. Such as: time:4:00,mem:2048,account:ABC

dev : bool (default = False)

Flag to toggle dev mode which only runs the first execution in the set.

groupby : int (default = 1)

Value which dictates the grouping of tasks. Particularly useful when tasks are short or a cluster restricts the number of unique jobs.

user : bool (default = False)

When running with Docker, toggles whether or not the host-user’s UID is used within the container.

setup : bool (default = False)

Flag which prevents execution of tasks after the metadata task and invocation files are generated.

bids : bool (default = False)

Flag toggling BIDS-aware metadata preparation.

**kwargs : dict

Arbitrary additional keyword arguments which may be passed.

str

The path to the created directory containing Clowdr experiment metadata.

clowdr.driver.main(args=None)

clowdr.driver.makeparser()

Command-line API wrapper for Clowdr as a CLI, not Python API. For information about the command-line wrapper and arguments it accepts, please try running “clowdr –help”.

args: list

List of all command-line arguments being passed.

int

The exit-code returned by the driver.

clowdr.driver.runtask(tasklist, **kwargs)

clowdr.driver.share(provdir, prepare=False, host='0.0.0.0', port=8050, verbose=False, debug=False, **kwargs)

Launches a simple web server which showcases all runs at the clowdrloc.

provdir : str

Path with Clowdr metdata files (returned from “local” and “deploy”)

**kwargs : dict

Arbitrary keyword arguments (i.e. {‘verbose’: True})

None

clowdr.server module

clowdr.task module

class clowdr.task.TaskHandler(taskfile, **kwargs)

Bases: object

execWrapper(sender)

manageTask(taskfile, provdir=None, verbose=False, **kwargs)

monitor(target, **kwargs)

provLaunch(options, **kwargs)

clowdr.utils module

clowdr.utils.backoff(function, posargs, optargs, backoff_time=36000, verbose=False, **kwargs)

clowdr.utils.get(remote, local, verbose=False, **kwargs)

clowdr.utils.getContainer(savedir, container, simg=None, verbose=False, **kwargs)

clowdr.utils.post(local, remote, verbose=False, **kwargs)

clowdr.utils.randstring(k)

clowdr.utils.remove(local)

clowdr.utils.splitS3Path(path)

clowdr.utils.truepath(path)