Last updated: 2023-09-21

Checks: 2 0

Knit directory: snakemake_tutorial/

This reproducible R Markdown analysis was created with workflowr (version 1.7.0). The Checks tab describes the reproducibility checks that were applied when the results were created. The Past versions tab lists the development history.

R Markdown file: up-to-date

Great! Since the R Markdown file has been committed to the Git repository, you know the exact version of the code that produced these results.

Repository version: 8972212

Great! You are using Git for version control. Tracking code development and connecting the code version to the results is critical for reproducibility.

The results in this page were generated with repository version 8972212. See the Past versions tab to see a history of the changes made to the R Markdown and HTML files.

Note that you need to be careful to ensure that all relevant files for the analysis have been committed to Git prior to generating the results (you can use wflow_publish or wflow_git_commit). workflowr only checks the R Markdown file, but you know if there are other scripts or data files that it depends on. Below is the status of the Git repository when the results were generated: 


Ignored files:
    Ignored:    .Rproj.user/

Unstaged changes:
    Deleted:    data/chr20.vcf.gz

Note that any generated files, e.g. HTML, png, CSS, etc., are not included in this status report because it is ok for generated content to have uncommitted changes.

These are the previous versions of the repository in which changes were made to the R Markdown (analysis/cluster.Rmd) and HTML (docs/cluster.html) files. If you’ve configured a remote Git repository (see ?wflow_git_remote), click on the hyperlinks in the table below to view the files as they were in that past version.

File Version Author Date Message
Rmd 8972212 Jean Morrison 2023-09-21 updates

Introduction

In this section, we will learn how to use Snakemake to submit jobs to the cluster.

In this section we will:

  1. Learn about cluster related command-line options
  2. Create a file containing options for submitted jobs
  3. Learn about local rules

Submitting Jobs to the Cluster with --slurm

A big advantage of using Snakemake is that it can take care of submitting lots of jobs to the cluster, leaving you free time to do more interesting tasks. In order for Snakemake to do this, we need to tell it how to submit jobs.

All three of the Biostatistics, CSG, and GreatLakes clusters use SLURM. As of recent versions of Snakemake, we can simply add --slurm to the command line execution.

On the CSG and Biostatistics clusters, the following line will work

snakemake --slurm --jobs 10 -p

The --jobs flag indicates the maximum number of jobs to submit at a time. You may want to increase this if you have a larger job underway.

On GreatLakes you will (probably) also need to specify the account you are using.

snakemake --slurm --jobs 10 --default-resources slurm_account=<your SLURM account>

Resources

Every job that gets submitted to the cluster is allocated resources such as memory, time, and number of cores. These can be specified either as defaults used for every job using --default-resources as shown above for specifying the slurm account or they can be specified differently for each rule. To specify resources for a specific rule, add a line resources: to that rule with the desired specifications. For example, using

rule combine_data:
    input: expand("data/chr{c}.vcf.gz", c = range(20, 23))
    output: "data/all.vcf.gz"
    resources: mem_mb = 1000
    shell: "bcftools concat -o {output} {input}"

specifies 1Gb of memory to be allocated for the combine_data rule. A full listing of the available cluster resources can be found in the Snakemake documentation here.