clowdr package

Subpackages

• clowdr.controller package
  • Submodules
  • clowdr.controller.launcher module
  • clowdr.controller.metadata module
  • Module contents
• clowdr.endpoint package
  • Submodules
  • clowdr.endpoint.AWS module
  • clowdr.endpoint.remote module
  • Module contents

Submodules

clowdr.driver module

clowdr.driver.cloud(descriptor, invocation, provdir, s3, cloud, credentials, **kwargs)

Launches a pipeline locally at scale through Clowdr.

descriptor : str

Path to a boutiques descriptor for the tool to be run

invocation : str

Path to a boutiques invocation for the tool and parameters to be run

provdir : str

Path on S3 for storing Clowdr intermediate files and outputs

s3 : str

Path on S3 for accessing input data

cloud : str

Which endpoint to use for deployment

credentials : str

Credentials for Amazon with access to dataloc, clowdrloc, and Batch

**kwargs : dict

Arbitrary keyword arguments (i.e. {‘verbose’: True})

int

The exit-code returned by the task being executed

clowdr.driver.local(descriptor, invocation, provdir, backoff_time=36000, sweep=[], verbose=False, workdir=None, simg=None, rerun=None, run_id=None, volumes=None, s3=None, cluster=None, jobname=None, clusterargs=None, dev=False, groupby=None, user=False, setup=False, **kwargs)

cluster Launches a pipeline locally through the Clowdr wrappers.

tool : str

Path to a boutiques descriptor for the tool to be run

invocation : str

Path to a boutiques invocation for the tool and parameters to be run

clowdrloc : str

Path for storing Clowdr intermediate files and outputs

dataloc : str

Path for accessing input data. If local, provide the hostname and optionally a path. If on S3, provide an S3 path.

cluster : str

Scheduler on the cluster being used. Currently, the only supported mode is slurm.

**kwargs : dict

Arbitrary keyword arguments. Currently supported arguments: - account : str

Account for the cluster scheduler

• jobname : str

  Base-name for the jobs as they will appear in the scheduler

• backoff_time: int

  Time limit for wait times when resubmitting jobs to a scheduler

• verbose : bool

  Toggle verbose output printing

• dev : bool

  Toggle dev mode (only runs first execution in the specified set)

Additionally, transfers all keyword arguments accepted by both of “controller.metadata.consolidateTask” and “task.TaskHandler”

int

The exit-code returned by the task being executed

clowdr.driver.main(args=None)

clowdr.driver.makeparser()

Command-line API wrapper for Clowdr as a CLI, not Python API. For information about the command-line wrapper and arguments it accepts, please try running “clowdr –help”.

args: list

List of all command-line arguments being passed.

int

The exit-code returned by the driver.

clowdr.driver.runtask(tasklist, **kwargs)

clowdr.driver.share(provdir, **kwargs)

Launches a simple web server which showcases all runs at the clowdrloc.

provdir : str

Path with Clowdr metdata files (returned from “local” and “deploy”)

**kwargs : dict

Arbitrary keyword arguments (i.e. {‘verbose’: True})

None

clowdr.server module

clowdr.task module

class clowdr.task.TaskHandler(taskfile, **kwargs)

Bases: object

execWrapper(sender)

manageTask(taskfile, provdir=None, verbose=False, **kwargs)

monitor(target, **kwargs)

provLaunch(options, **kwargs)

clowdr.utils module

clowdr.utils.backoff(function, posargs, optargs, backoff_time=36000, **kwargs)

clowdr.utils.get(remote, local, **kwargs)

clowdr.utils.getContainer(savedir, container, **kwargs)

clowdr.utils.post(local, remote, **kwargs)

clowdr.utils.randstring(k)

clowdr.utils.remove(local)

clowdr.utils.splitS3Path(path)

clowdr.utils.truepath(path)