Process data for Buenrostro 2018 scATAC-seq dataset

Last updated: 2020-11-18

Checks: 7 0

Knit directory: scATACseq-topics/

This reproducible R Markdown analysis was created with workflowr (version 1.6.2). The Checks tab describes the reproducibility checks that were applied when the results were created. The Past versions tab lists the development history.

R Markdown file: up-to-date

Great! Since the R Markdown file has been committed to the Git repository, you know the exact version of the code that produced these results.

Environment: empty

Great job! The global environment was empty. Objects defined in the global environment can affect the analysis in your R Markdown file in unknown ways. For reproduciblity it’s best to always run the code in an empty environment.

Seed: set.seed(20200729)

The command set.seed(20200729) was run prior to running the code in the R Markdown file. Setting a seed ensures that any results that rely on randomness, e.g. subsampling or permutations, are reproducible.

Session information: recorded

Great job! Recording the operating system, R version, and package versions is critical for reproducibility.

Cache: none

Nice! There were no cached chunks for this analysis, so you can be confident that you successfully produced the results during this run.

File paths: relative

Great job! Using relative paths to the files within your workflowr project makes it easier to run your code on other machines.

Repository version: ee608fd

Great! You are using Git for version control. Tracking code development and connecting the code version to the results is critical for reproducibility.

The results in this page were generated with repository version ee608fd. See the Past versions tab to see a history of the changes made to the R Markdown and HTML files.

Note that you need to be careful to ensure that all relevant files for the analysis have been committed to Git prior to generating the results (you can use wflow_publish or wflow_git_commit). workflowr only checks the R Markdown file, but you know if there are other scripts or data files that it depends on. Below is the status of the Git repository when the results were generated:


Ignored files:
    Ignored:    .Rhistory
    Ignored:    .Rproj.user/

Untracked files:
    Untracked:  analysis/clusters_pca_structure_Cusanovich2018.Rmd
    Untracked:  analysis/process_data_Buenrostro2018_Chen2019.Rmd
    Untracked:  buenrostro2018.RData

Unstaged changes:
    Modified:   analysis/plots_Lareau2019_bonemarrow.Rmd
    Modified:   scripts/fit_all_models_Buenrostro_2018.sbatch

Note that any generated files, e.g. HTML, png, CSS, etc., are not included in this status report because it is ok for generated content to have uncommitted changes.

These are the previous versions of the repository in which changes were made to the R Markdown (analysis/process_data_Buenrostro2018.Rmd) and HTML (docs/process_data_Buenrostro2018.html) files. If you’ve configured a remote Git repository (see ?wflow_git_remote), click on the hyperlinks in the table below to view the files as they were in that past version.

File	Version	Author	Date	Message
Rmd	ee608fd	kevinlkx	2020-11-18	process counts for aggregated single-cell peaks using chromVAR
html	594fd86	kevinlkx	2020-11-10	Build site.
Rmd	614814b	kevinlkx	2020-11-10	process Buenrostro2018 data using Downloaded scATAC-seq processed data from GEO
Rmd	c113844	kevinlkx	2020-11-10	wflow_rename(“analysis/process_data_Buenrostro2018.Rmd”, “analysis/process_data_Buenrostro2018_Chen2019pipeline.Rmd”)
html	c113844	kevinlkx	2020-11-10	wflow_rename(“analysis/process_data_Buenrostro2018.Rmd”, “analysis/process_data_Buenrostro2018_Chen2019pipeline.Rmd”)
html	1905c58	kevinlkx	2020-11-05	Build site.
Rmd	1da32f8	kevinlkx	2020-11-05	set binarized counts as sparse matrix
html	89b45be	kevinlkx	2020-11-04	Build site.
Rmd	857e1e8	kevinlkx	2020-11-04	process data from Buenrostro 2018 paper
html	907fa65	kevinlkx	2020-11-04	Build site.
Rmd	12bf4b3	kevinlkx	2020-11-04	process data from Buenrostro 2018 paper

About Buenrostro 2018 dataset

Reference: Buenrostro, J. D. et al. Integrated Single-Cell Analysis Maps the Continuous Regulatory Landscape of Human Hematopoietic Differentiation. Cell 173, 1535–1548.e16 (2018).

Data were downloaded from GEO: GSE96772

RCC directory: /project2/mstephens/kevinluo/scATACseq-topics/data/Buenrostro_2018/

Prepare data for topic modeling

Using peaks called from bulk ATAC-seq samples

The Buenrostro et al paper called ATAC-seq peaks from the bulk ATAC-seq samples. Downloaded scATAC-seq processed data from GEO: GSE96769

Peaks called from bulk ATAC-seq samples GSE96769_PeakFile_20160207.bed.gz
scATAC-seq sparse count matrix GSE96769_scATACseq_counts.txt.gz

library(Matrix)
library(tools)
library(readr)
library(data.table)

Load the fragment counts as a 2,953 x 491,437 sparse matrix.

# The first row has the sample names
file_counts <- "/project2/mstephens/kevinluo/scATACseq-topics/data/Buenrostro_2018/GEO_data/GSE96769_scATACseq_counts.txt.gz"
sample_names <- readLines(file_counts,n = 1)
sample_names <- unlist(strsplit(sample_names,"\t",fixed = TRUE))
sample_names <- unlist(strsplit(sample_names,";",fixed = TRUE))
sample_names <- sample_names[-1]

# Load the fragment counts as sparse matrix.
dat <- fread(file_counts,sep = "\t",skip = 1)
class(dat) <- "data.frame"
names(dat) <- c("i","j","x")
counts <- sparseMatrix(i = dat$i,j = dat$j,x = dat$x)
counts <- t(counts)
rownames(counts) <- sample_names
dim(counts)

[1]   2953 491437

peaks

peaks <- fread("/project2/mstephens/kevinluo/scATACseq-topics/data/Buenrostro_2018/GEO_data/GSE96769_PeakFile_20160207.bed.gz")
peak_names <- paste(peaks$V1,peaks$V2,peaks$V3,sep = "_")
rownames(peaks) <- peak_names
cat(sprintf("Number of peaks: %d\n",nrow(peaks)))

Number of peaks: 491437

colnames(counts) <- peak_names

Plot the distribution of fragment counts.

y <- table(summary(counts)$x)
x <- names(y)
y <- as.numeric(y)
plot(x,y,pch = 20,log = "y",cex = 0.65,
     xlab = "number of fragments mapping to peak",
     ylab = "number of peaks")
lines(x,y)

Version	Author	Date
594fd86	kevinlkx	2020-11-10

The supplementary Table S1 provides more details about these samples. https://ars.els-cdn.com/content/image/1-s2.0-S009286741830446X-mmc1.xlsx

Use the metadata.tsv from Chen et al. Genome Biology 2019 paper https://github.com/pinellolab/scATAC-benchmarking/tree/master/Real_Data/Buenrostro_2018/input/metadata.tsv)

samples_filtered <- read.table('/project2/mstephens/kevinluo/scATACseq-topics/data/Buenrostro_2018/data/input_Chen_2019/metadata.tsv', header = TRUE, stringsAsFactors=FALSE, quote="",row.names=1)
samples_filtered$name <- rownames(samples_filtered)

idx_samples_filtered <- match(samples_filtered$name, sample_names)
sample_names_filtered <- sample_names[idx_samples_filtered]
counts <- counts[idx_samples_filtered,]
samples <- samples_filtered[,c("name", "label")]
dim(counts)

[1]   2034 491437

Remove peaks not exist in any of the cells

j <- which(colSums(counts > 0) >= 1)
peaks <- peaks[j,]
counts <- counts[,j]
cat(length(j), "peaks after filtering.  \n")

465536 peaks after filtering.

Plot the distribution of filtered fragment counts.

y <- table(summary(counts)$x)
x <- names(y)
y <- as.numeric(y)
plot(x,y,pch = 20,log = "y",cex = 0.65,
     xlab = "number of fragments mapping to peak",
     ylab = "number of peaks")
lines(x,y)

Version	Author	Date
594fd86	kevinlkx	2020-11-10

Binarize counts

binarized_counts <- as.matrix((counts > 0) + 0)
binarized_counts <- Matrix(binarized_counts, sparse = TRUE) 

dim(binarized_counts)

[1]   2034 465536

data.dir <- "/project2/mstephens/kevinluo/scATACseq-topics/data/Buenrostro_2018/processed_data/"
dir.create(data.dir, showWarnings = FALSE, recursive = TRUE)

saveRDS(counts, file.path(data.dir, "counts_Buenrostro_2018.rds"))
saveRDS(binarized_counts, file.path(data.dir, "binarized_counts_Buenrostro_2018.rds"))

save(list = c("samples", "peaks", "counts"), 
     file = file.path(data.dir, "Buenrostro_2018.RData"))

counts <- binarized_counts
save(list = c("samples", "peaks", "counts"), 
     file = file.path(data.dir, "Buenrostro_2018_binarized.RData"))

data.dir <- "/project2/mstephens/kevinluo/scATACseq-topics/data/Buenrostro_2018/processed_data/"
load(file.path(data.dir, "Buenrostro_2018.RData"))
cat(sprintf("Loaded %d x %d counts matrix.\n",nrow(counts),ncol(counts)))

Loaded 2034 x 465536 counts matrix.

cat(sprintf("Number of samples (cells): %d\n",nrow(counts)))

Number of samples (cells): 2034

cat(sprintf("Number of peaks: %d\n",ncol(counts)))

Number of peaks: 465536

cat(sprintf("Proportion of counts that are non-zero: %0.1f%%.\n",
            100*mean(counts > 0)))

Proportion of counts that are non-zero: 1.5%.

Using peaks called from aggregated single-cell ATAC-seq data processed by Chen et al. Genome Biology 2019

Downloaded processed peaks and BAM files from the scATAC-benchmarking website (Chen et al. Genome Biology 2019) (https://github.com/pinellolab/scATAC-benchmarking/)
Count paired-end scATAC-seq reads in peaks using chromVAR, following the pipeline at https://nbviewer.jupyter.org/github/pinellolab/scATAC-benchmarking/blob/master/Real_Data/Buenrostro_2018/run_methods/chromVAR/chromVAR_buenrostro2018_motifs.ipynb?flush_cache=true

Load chromVAR and related packages

# if (!requireNamespace("BiocManager", quietly = TRUE))
#     install.packages("BiocManager")
# BiocManager::install("chromVAR")
# BiocManager::install("motifmatchr")
# BiocManager::install("BSgenome.Hsapiens.UCSC.hg19")
# BiocManager::install("JASPAR2016")

library(chromVAR)

library(motifmatchr)
library(Matrix)
library(SummarizedExperiment)

Loading required package: GenomicRanges

Loading required package: stats4

Loading required package: BiocGenerics

Loading required package: parallel


Attaching package: 'BiocGenerics'

The following objects are masked from 'package:parallel':

    clusterApply, clusterApplyLB, clusterCall, clusterEvalQ,
    clusterExport, clusterMap, parApply, parCapply, parLapply,
    parLapplyLB, parRapply, parSapply, parSapplyLB

The following object is masked from 'package:Matrix':

    which

The following objects are masked from 'package:stats':

    IQR, mad, sd, var, xtabs

The following objects are masked from 'package:base':

    anyDuplicated, append, as.data.frame, basename, cbind, colnames,
    dirname, do.call, duplicated, eval, evalq, Filter, Find, get, grep,
    grepl, intersect, is.unsorted, lapply, Map, mapply, match, mget,
    order, paste, pmax, pmax.int, pmin, pmin.int, Position, rank,
    rbind, Reduce, rownames, sapply, setdiff, sort, table, tapply,
    union, unique, unsplit, which, which.max, which.min

Loading required package: S4Vectors


Attaching package: 'S4Vectors'

The following objects are masked from 'package:data.table':

    first, second

The following object is masked from 'package:Matrix':

    expand

The following object is masked from 'package:base':

    expand.grid

Loading required package: IRanges


Attaching package: 'IRanges'

The following object is masked from 'package:data.table':

    shift

Loading required package: GenomeInfoDb

Loading required package: Biobase

Welcome to Bioconductor

    Vignettes contain introductory material; view with
    'browseVignettes()'. To cite Bioconductor, see
    'citation("Biobase")', and for packages 'citation("pkgname")'.

Loading required package: DelayedArray

Loading required package: matrixStats


Attaching package: 'matrixStats'

The following objects are masked from 'package:Biobase':

    anyMissing, rowMedians

Loading required package: BiocParallel


Attaching package: 'DelayedArray'

The following objects are masked from 'package:matrixStats':

    colMaxs, colMins, colRanges, rowMaxs, rowMins, rowRanges

The following objects are masked from 'package:base':

    aperm, apply, rowsum

library(BiocParallel)
library('JASPAR2016')
library(BSgenome.Hsapiens.UCSC.hg19)

Loading required package: BSgenome

Loading required package: Biostrings

Loading required package: XVector


Attaching package: 'Biostrings'

The following object is masked from 'package:DelayedArray':

    type

The following object is masked from 'package:base':

    strsplit

Loading required package: rtracklayer

register(MulticoreParam(10))

data.dir <- "/project2/mstephens/kevinluo/scATACseq-topics/data/Buenrostro_2018/processed_data_Chen2019pipeline/chromVAR/"
dir.create(data.dir, showWarnings = FALSE, recursive = TRUE)

samples <- read.table('/project2/mstephens/kevinluo/scATACseq-topics/data/Buenrostro_2018/data/input_Chen_2019/metadata.tsv', header = TRUE, stringsAsFactors=FALSE, quote="",row.names=1)

peakfile <- "/project2/mstephens/kevinluo/scATACseq-topics/data/Buenrostro_2018/data/input_Chen_2019/combined.sorted.merged.bed"
peaks <- getPeaks(peakfile, sort_peaks = TRUE)

Warning in getPeaks(peakfile, sort_peaks = TRUE): Peaks are not equal width!Use
resize(peaks, width = x, fix = "center") to make peaks equal in size, where x is
the desired size of the peaks)

Peaks sorted

peaks <- resize(peaks, width = 500, fix = "center")

cat(length(peaks), "peaks \n")

237450 peaks

dir_bamfiles <- "/project2/mstephens/kevinluo/scATACseq-topics/data/Buenrostro_2018/data/input_Chen_2019/sc-bams_nodup/"
bamfiles <- list.files(path = dir_bamfiles, pattern = "\\.bam$")
cellnames <- sapply(strsplit(bamfiles,'.',fixed = TRUE), "[[", 1)
sum(cellnames == rownames(samples))

[1] 2034

count scATAC-seq paired-end fragments in the peaks

fragment_counts <- getCounts(file.path(dir_bamfiles,bamfile), 
                             peaks, 
                             paired = TRUE, 
                             by_rg = TRUE, 
                             format = "bam", 
                             colData = data.frame(celltype = cellnames))

saveRDS(fragment_counts, file.path(data.dir, "fragment_counts_scPeaks_chromVAR_Buenrostro_2018.rds"))

fragment_counts <- readRDS(file.path(data.dir, "fragment_counts_scPeaks_chromVAR_Buenrostro_2018.rds"))
fragment_counts

class: RangedSummarizedExperiment 
dim: 237450 2034 
metadata(0):
assays(1): counts
rownames: NULL
rowData names(0):
colnames(2034): BM1077-CLP-Frozen-160106-13 BM1077-CLP-Frozen-160106-14
  ... singles-PB1022-mono-160128-95 singles-PB1022-mono-160128-96
colData names(2): celltype depth

counts <- assay(fragment_counts)
counts <- t(counts)

peaks <- as.data.frame(peaks)[,1:3]
colnames(peaks) <- c("chr", "start", "end")
peak_names <- paste(peaks$chr, peaks$start, peaks$end, sep = "_")
colnames(counts) <- peak_names

Filter peaks using filterPeaks in chromVAR

idx_peaks_filtered <- filterPeaks(fragment_counts, min_fragments_per_peak = 1, non_overlapping = TRUE, ix_return = TRUE)
peaks <- peaks[idx_peaks_filtered,]
counts <- counts[,idx_peaks_filtered]
cat(length(idx_peaks_filtered), "peaks after filtering.  \n")

228965 peaks after filtering.