Last updated: 2019-12-06
Checks: 7 0
Knit directory: PSYMETAB/
This reproducible R Markdown analysis was created with workflowr (version 1.5.0). The Checks tab describes the reproducibility checks that were applied when the results were created. The Past versions tab lists the development history.
Great! Since the R Markdown file has been committed to the Git repository, you know the exact version of the code that produced these results.
Great job! The global environment was empty. Objects defined in the global environment can affect the analysis in your R Markdown file in unknown ways. For reproduciblity it’s best to always run the code in an empty environment.
The command set.seed(20191126) was run prior to running the code in the R Markdown file. Setting a seed ensures that any results that rely on randomness, e.g. subsampling or permutations, are reproducible.
Great job! Recording the operating system, R version, and package versions is critical for reproducibility.
Nice! There were no cached chunks for this analysis, so you can be confident that you successfully produced the results during this run.
Great job! Using relative paths to the files within your workflowr project makes it easier to run your code on other machines.
Great! You are using Git for version control. Tracking code development and connecting the code version to the results is critical for reproducibility. The version displayed above was the version of the Git repository at the time these results were generated.
Note that you need to be careful to ensure that all relevant files for the analysis have been committed to Git prior to generating the results (you can use wflow_publish or wflow_git_commit). workflowr only checks the R Markdown file, but you know if there are other scripts or data files that it depends on. Below is the status of the Git repository when the results were generated:
Ignored files:
Ignored: .drake/
Ignored: analysis/QC/
Ignored: data/processed/
Ignored: data/raw/
Untracked files:
Untracked: post_impute_qc.out
Unstaged changes:
Deleted: post_imputation_qc.log
Deleted: pre_imputation_qc.log
Modified: pre_impute_qc.out
Deleted: qc_part2.out
Note that any generated files, e.g. HTML, png, CSS, etc., are not included in this status report because it is ok for generated content to have uncommitted changes.
These are the previous versions of the R Markdown and HTML files. If you’ve configured a remote Git repository (see ?wflow_git_remote), click on the hyperlinks in the table below to view them.
| File | Version | Author | Date | Message |
|---|---|---|---|---|
| Rmd | 487b5f5 | Sjaarda Jennifer Lynn | 2019-12-06 | update website, add qc description |
| html | 9f1ba5e | Jenny Sjaarda | 2019-12-06 | Build site. |
| Rmd | 5e454c3 | Sjaarda Jennifer Lynn | 2019-12-06 | add more details to website |
| html | 9220e22 | Jenny | 2019-11-26 | Build site. |
| Rmd | f809d9a | Jenny | 2019-11-26 | Add info |
Last updated: 2019-12-06
Code version: 487b5f56d9bdfd9bc979405b72e23980daf9dd69
All processing scripts were run from the root sgg directory. Project was initialized using workflowr rpackage, see here.
On sgg server:
project_name <- "PSYMETAB"
library("workflowr")
wflow_start(project_name) # creates directory called project_name
options("workflowr.view" = FALSE) # if using cluster
wflow_build() # create directories
wflow_publish(c("analysis/index.Rmd", "analysis/about.Rmd", "analysis/license.Rmd"),
"Publish the initial files for myproject")
wflow_use_github("jennysjaarda")
# select option 2. Create the remote repository yourself by going to https://github.com/new and entering the Repository name that matches the name of the directory of your workflowr project.
options(workflowr.sysgit = "")
wflow_git_push()After successfully creating a github repository, on personal computer:
cd ~/Dropbox/UNIL/projects/
git clone https://github.com/jennysjaarda/PSYMETAB.git PSYMETABadd the following to the git .Rprofile options(workflowr.sysgit = "") and options(workflowr.sysgit = "")
Create internal folders:
project_dir=/data/sgg2/jenny/projects/PSYMETAB_GWAS
mkdir $project_dir/data/raw/reference_files
mkdir $project_dir/data/raw/phenotype_data
mkdir $project_dir/data/raw/extraction
mkdir $project_dir/data/raw/imputation
mkdir $project_dir/data/processed/imputationMost steps are saved in shell scripts. This code should be able to run in any Unix-like environment (of course after installing the necessary software). To process the data efficiently on our HPC, we wrote a wrapper script, submit-array.sh, to combine the many parallel jobs into one array job submitted to the Sun Grid Engine (SGE) scheduler. Not only does this make it easier for the scheduler (and you) to manage thousands of tasks, we also made it so that it won’t re-run jobs if the output file already exists. This is convenient when a small fraction of jobs fail and need to be re-run. However, if your HPC uses a scheduler other than SGE, you will have to decide how best to submit the jobs.
create-genome.sh downloads the fasta files for human genome hg19 (chromosomes 1-22, X, Y, M) and the ERRC spike-ins. It indexes the genome with subread-buildindex. The output is saved in the directory genome.
submit-array.sh create-genome.sh 8g genomerun-fastqc.sh runs FastQC on each raw fastq file. It also counts the number of reads. The output is saved in the directory fastqc.
submit-array.sh run-fastqc.sh 2g fastq/*fastq.gzls fastqc/*zip | wc -l
ls fastqc/*count.txt | wc -l
grep -w success ~/log/run-fastqc.sh/* | wc -l
grep -w failure ~/log/run-fastqc.sh/* | wc -ltrim.sh removes the 5 bp UMI at the 5’ end of the read using the program umitools trim. We used umitools v2.1.1. Output fastq files are written to the directory trim. Reads without a valid UMI are written to the directory invalid.
submit-array.sh trim.sh 2g fastq/*fastq.gzTo confirm that the jobs ran successfully:
ls trim/*fastq.gz | wc -l
ls invalid/*fastq.gz | wc -l
grep -w success ~/log/trim.sh/* | wc -l
grep -w failure ~/log/trim.sh/* | wc -l
sessionInfo()R version 3.5.3 (2019-03-11)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS Linux 7 (Core)
Matrix products: default
BLAS: /data/sgg2/jenny/bin/R-3.5.3/lib64/R/lib/libRblas.so
LAPACK: /data/sgg2/jenny/bin/R-3.5.3/lib64/R/lib/libRlapack.so
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
loaded via a namespace (and not attached):
[1] workflowr_1.5.0 Rcpp_1.0.3 rprojroot_1.3-2 digest_0.6.23
[5] later_1.0.0 R6_2.4.1 backports_1.1.5 git2r_0.26.1
[9] magrittr_1.5 evaluate_0.14 stringi_1.4.3 rlang_0.4.1
[13] fs_1.3.1 promises_1.1.0 whisker_0.4 rmarkdown_1.18
[17] tools_3.5.3 stringr_1.4.0 glue_1.3.1 httpuv_1.5.2
[21] xfun_0.11 yaml_2.2.0 compiler_3.5.3 htmltools_0.4.0
[25] knitr_1.26